AzureFixes Logo
Published on

ExpressRoute Design Changes: Scaling for Segmentation & HA

Authors

Goal

This document outlines a recent shift in ExpressRoute architecture to improve scalability, segmentation, and high availability (HA). The previous design had several technical limitations that became bottlenecks as the cloud footprint expanded.

Previous Topology

The legacy ExpressRoute configuration was built using:

  • Standard SKU ExpressRoute circuit
  • A single Virtual Network Gateway (VNG) in the Hub VNet
  • Spoke VNets peered to the hub for ER access
  • No Global Reach or cross-region BCDR

Limitations Observed:

  • Prefix limits exceeded during routing updates (max 10,000 for Standard SKU)
  • Lack of regional segmentation – one blast radius for all apps
  • No Global Reach – cross-geo traffic routed inefficiently via VPN/Internet
  • Throughput constraints due to shared VNG and circuit

New Design Highlights

The revised architecture leverages:

  • Premium SKU ExpressRoute Circuit
  • Dedicated VNG per region (or per functional Hub)
  • Enabled ExpressRoute Global Reach for east-west BCDR
  • Segmented VNets by workload/region via multiple peering domains

Architecture Diagrams

Previous Design

graph TD
    A[On-Prem Router] -->|BGP/Private Peering| B[ER Circuit (Standard)]
    B --> C[ER Gateway (Hub VNet)]
    C --> D1[Spoke VNet 1]
    C --> D2[Spoke VNet 2]
    C --> D3[Spoke VNet 3]

New Design

graph TD
    A1[On-Prem DC - South Central] -->|BGP| B1[ER Circuit - South Central (Premium)]
    A2[On-Prem DC - East] -->|BGP| B2[ER Circuit - East (Premium)]

    B1 --> C1[ER GW - Hub South Central US]
    B2 --> C2[ER GW - Hub East US]

    C1 --> D1[Spoke VNet 1 - South Central]
    C1 --> D2[Spoke VNet 2 - South Central]
    C2 --> E1[Spoke VNet 1 - East]
    C2 --> E2[Spoke VNet 2 - East]

    B1 <--> B2[Global Reach]

Technical Changes Implemented

ChangeDescription
Circuit UpgradeUpgraded ER from Standard to Premium to lift prefix limits and allow Global Reach
VNG RedesignCreated multiple ER Gateways in separate Hubs (regional or functional)
Peering ModelImplemented dual-peering strategy per environment
Route FilteringEnabled to control learned prefixes
Traffic SegmentationUsed UDRs and NVAs to route regionally and isolate workloads

Sample Change Log Entry

Change ID: CHG-AZR-ER-2025-07
Date: 2025-07-15
Summary: ER Circuit upgrade + Hub redesign for segmentation and HA
Action:
- Upgraded ER circuit to Premium
- Deployed new ER GW in Central US
- Connected 3 spoke VNets using regional hub
- Enabled Global Reach with East US circuit
Impact: Improved BCDR and removed prefix limit error observed during routing updates

Risk Mitigation Steps

  • Staged Deployment: Implemented regional hub design incrementally
  • Backout Plan: Preserved previous ER circuit until routing validation completed
  • Prefix Review: Analyzed BGP route tables before rollout to avoid blackholing
  • Monitoring: Enabled NPM, Log Analytics for route and latency tracking

PowerShell Validation Script

# Get ExpressRoute Circuit Details
$circuit = Get-AzExpressRouteCircuit -ResourceGroupName "rg-network" -Name "er-circuit-premium"
$circuit | Format-List Name,ServiceProviderProperties,BandwidthInMbps,PeeringLocation,Sku

# List all Peerings
Get-AzExpressRouteCircuitPeering -ExpressRouteCircuit $circuit

# Validate Peering Connection Status
$circuit.Peerings | ForEach-Object {
    Write-Host "Peering: $($_.PeeringType) | State: $($_.State) | AzureASN: $($_.AzureASN)"
}

# Get associated VNet Gateway and Connections
$vng = Get-AzVirtualNetworkGateway -ResourceGroupName "rg-network" -Name "er-gw-central"
Get-AzVirtualNetworkGatewayConnection -VirtualNetworkGateway1 $vng | Format-Table Name,ConnectionType,ConnectionStatus

Terraform Validation Snippet

data "azurerm_express_route_circuit" "er_circuit" {
  name                = "er-circuit-premium"
  resource_group_name = "rg-network"
}

output "er_circuit_details" {
  value = {
    service_provider_name = data.azurerm_express_route_circuit.er_circuit.service_provider_name
    peering_location      = data.azurerm_express_route_circuit.er_circuit.peering_location
    bandwidth             = data.azurerm_express_route_circuit.er_circuit.bandwidth_in_mbps
    sku                   = data.azurerm_express_route_circuit.er_circuit.sku[0].name
  }
}

data "azurerm_virtual_network_gateway" "er_gateway" {
  name                = "er-gw-central"
  resource_group_name = "rg-network"
}

output "gateway_bgp_settings" {
  value = data.azurerm_virtual_network_gateway.er_gateway.bgp_settings
}