AWS Transit Gateway

"The center of the universe..."

Allan A. B. Thomsen - March 2021

When you are a global enterprise customer with branches and offices worldwide and your AWS presence is considerable, connectivity becomes a critical part of your landscape. With requirements to low global latency, high security and ease of maintenance, you want a network infrastructure that satisfies all your requirements, while at the same time being ready for change as the company grows.

AWS introduced the AWS Transit Gateway (TGW) in November 2018 and is the easy way to connect VPCs with other clouds or on-premise networks though, for example, a Software Defined - Wide Area Network (SD-WAN). Before that, linking VPCs and routing traffic in bigger infrastructures was not easy.

TGW has many features and can enable scanning and/or firewalling of all traffic. In the sample we have created below we have dissected some of the parts, and included the new Connect Attachment setup for SD-WAN integration.

After setting up the TGW in IaC (Infrastructure as Code) using CloudFormation (CFN), I have a few takeaways which hopefully will give you an advantage, if you try the same.

This is a technical post, and if you are not familiar with network principles and concepts, then this might be too technical.

"The Target Architecture..."

RAM

From the top left: RAM (AWS Resource Access Manager) is required to support an enterprise organisation setup, separating projects, and central functionality in different accounts, but still enabling the central TGW to be shared and thus only needed once per region.

This is simply what it takes to allow all accounts in the organization access to create TGW Attachments, and in this case enabling the integration to other VPCs in AWS, all SD-WAN connected clouds/datacenters and internet access.

# Share the TGW to make it available to all accounts in the org

TransitGatewayShare:

Type: "AWS::RAM::ResourceShare"

Properties:

Name: "TGWShare"

AllowExternalPrincipals: false

ResourceArns:

- !Sub "arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:transit-gateway/${TransitGateway}"

Principals:

- !Ref OrganizationArn

Tags:

- Key: Name

Value: TGW Share

- Key: VpcName

Value: !Sub "${AWS::StackName}"

- Key: Role

Value: Sharing the TGW with the entire organization

The project VPC depicted at the top in the middle, represents any VPC in the AWS organisation, and if that causes you challenges, then the rest of this blog post is not for you :)

In the middle to the left is the Internet Egress VPC, which only purpose is to route traffic to the Internet. Here it would be possible to add specialised exfiltration appliances, or dedicated WAF/NWFW configurations. This is not the subject of this post, so will skip that gracefully.

Transit Gateway

In the center of the drawing the TGW is placed as the center of the universe, enabling VPC peering in a star model.

The TGW is also simple to add, and in our sample we do not want default association, nor propagation, as having the FW VPC for traffic scanning, isn't lending itself to the default settings.

TransitGateway:

Type: "AWS::EC2::TransitGateway"

Properties:

AmazonSideAsn: !Ref ASN

AutoAcceptSharedAttachments: enable

DefaultRouteTableAssociation: disable

DefaultRouteTablePropagation: disable

Description: Transit Gateway

DnsSupport: enable

VpnEcmpSupport: enable

Tags:

- Key: Name

Value: Transit Gateway

If you are part of a larger SD-WAN setup, you probably have ASN planning too, and if you (like us) need multiple regions, they should be different if propagated.

After creating the TGW, all subnets needing access to it, will need a TGW attachment created

Here we attach Internet Egress (IE) subnets to the TGW:

# Create the attachment to the IE subnets

TransitGatewayAttachmentIE:

Type: "AWS::EC2::TransitGatewayAttachment"

Properties:

SubnetIds:

- !Ref SubnetIETGW1a

- !If [CreateAZ-IE-1b, !Ref SubnetIETGW1b, !Ref AWS::NoValue]

- !If [CreateAZ-IE-1c, !Ref SubnetIETGW1c, !Ref AWS::NoValue]

Tags:

- Key: Name

Value: Transit Gateway for Internet Egress VPC

TransitGatewayId: !Ref TransitGateway

VpcId: !Ref VPCIE

The TGW has two routing tables in this setup, one for all VPCs to be attached to, to ensure all traffic goes to the Firewall VPC.

This is the table I am referring to:

The default route is the only one needed for VPC traffic to be routed correctly, but the latter route is used in the propagation to the SD-WAN. This CIDR block is the one covering the network segment handle by this TGW. More can be added or larger/smaller CIDR blocks be used. If you have TGWs in different regions, then each region needs their own block(s).

The SD-WAN configuration will need to have an ignore rule for the default route, as that is not applicable for the rest of the SD-WAN network.

This default route table, where no more than the configured routes shown above, are needed, is created like this:

TransitGatewayDEFRouteTable:

Type: AWS::EC2::TransitGatewayRouteTable

Properties:

Tags:

- Key: Name

Value: Default Transit Gateway FW Route Table used by all other VPCs

TransitGatewayId: !Ref TransitGateway

The above routes are added like this:

# Add the static routes for all attachments

TransitGatewayFWRoute:

Type: AWS::EC2::TransitGatewayRoute

Properties:

TransitGatewayAttachmentId: !Ref TransitGatewayAttachmentFWTGW

DestinationCidrBlock: 0.0.0.0/0

TransitGatewayRouteTableId: !Ref TransitGatewayDEFRouteTable

# Add a special route propagated to the SD-WAN to ensure all AWS traffic is propagated from SD-WAN to the AWS TGW

# We point it to the FW as it is not really needed by us

# Additionally the SD-WAN has to filter the default route in here

TransitGatewayFWAWSRoute:

Type: AWS::EC2::TransitGatewayRoute

Properties:

TransitGatewayAttachmentId: !Ref TransitGatewayAttachmentFWTGW

DestinationCidrBlock: !Ref AWSCIDR

TransitGatewayRouteTableId: !Ref TransitGatewayDEFRouteTable

All the VPC's attached (except the FW VPC) are attached to the default route table like this:

# Associate the Default route table with IE subnets

TransitGatewayIEDEFRouteTableAssociation:

Type: AWS::EC2::TransitGatewayRouteTableAssociation

Properties:

TransitGatewayAttachmentId: !Ref TransitGatewayAttachmentIE

TransitGatewayRouteTableId: !Ref TransitGatewayDEFRouteTable

The other routing table is the one associated with the firewall VPC, to route to the relevant VPCs, known by the TGW.

Connecting different networks requires IPAM (IP Address Management), to avoid NATting. If you use the new connection feature Connect Attachment (which we do in this sample), then the TGW requires a CIDR block to connect the GRE (Generic Routing Encapsulation) tunnels to. Luckily this CIDR block does not need to be a company reserved IP range, but can be a reusable (between TGWs) routable CIDR block, as it is only used between the TGW and the Connect Attachment device.This CIDR block requires a CLI call after the TGW has been deployed and before the Connect Attachment Peers are added.

aws ec2 modify-transit-gateway --transit-gateway-id ${tgwid} --options AddTransitGatewayCidrBlocks=100.64.0.0/24

The CIDR block chosen here is from RFC 6598, which specifies an unrouted shared address space.

The second route table looks like this:

The first route (which will be multiplied by the number of connected AWS VPCs) is the route to the Cloud Projects VPC. The second route is for Internet Egress traffic, which is not captured by rule 1, or 3.

The last rule should cover everything which is in all internal clouds or on-prem. In this case we do not use any other reserved ranges than this class A CIDR block: 10.0.0.0/8. As all known VPCs are more specific, they will get their traffic, before considering this last "catch all internal" rule is considered. This rule can either be added in CFN or propagated from SD-WAN.

If you would like it propagated, then you just tell the TGW:

# Propagate the SDW routes from the ConnectAttachment to the FW Route table

# Traffic from the FW will then know what needs to be sent to the SD-WAN

# We are not currently doing this as the # of prefixes from the SD-WAN if too high for the TGW to handle

# It still works as the catch all internal route has been added to the CH route table, pointing to the SD-WAN

TransitGatewayVersaCHRouteTablePropagation:

Type: AWS::EC2::TransitGatewayRouteTablePropagation

Properties:

TransitGatewayAttachmentId: !Ref TransitGatewayConnectSDW

TransitGatewayRouteTableId: !Ref TransitGatewayFWRouteTable

The route table, the routes, and the attachment are created in the same way as for the default shown previously. A picture and naming standard is essential to stay sane while creating the CFN, so take the time to create both - it saves a lot of time and rework.

Network Firewall

The ability to scan all traffic is in this case enabled through a Network Firewall, which gives you stateless and stateful firewall rules. These combined with IPAM can be very fine grained.

To ensure the traffic for the stateful firewall rules stay within the same AZ in both directions, the VPC attachment needs to have appliance mode enabled. This requires another CLI call, as this is also not possible using CFN.

aws ec2 modify-transit-gateway-vpc-attachment --transit-gateway-attachment-id ${tgwattid} \
--options '{"ApplianceModeSupport": "enable"}'

This is how the NW Firewall is attached to the FW subnets

NWFirewall:

Type: AWS::NetworkFirewall::Firewall

Properties:

FirewallName: NetworkFirewall

FirewallPolicyArn: !Ref NWFirewallPolicy

VpcId: !Ref VPCFW

SubnetMappings:

- SubnetId: !Ref SubnetFWFW1a

- !If [CreateAZ-FW-1b, SubnetId: !Ref SubnetFWFW1b, !Ref AWS::NoValue]

- !If [CreateAZ-FW-1c, SubnetId: !Ref SubnetFWFW1c, !Ref AWS::NoValue]

Description: The NW Firewall for all TGW traffic

Tags:

- Key: Name

Value: NetworkFirewall

- Key: Role

Value: Filtering all TGW traffic

Here is a FW policy sample:

NWFirewallPolicy:

Type: "AWS::NetworkFirewall::FirewallPolicy"

Properties:

FirewallPolicyName: DefaultNWFWPolicy

FirewallPolicy:

StatelessDefaultActions:

- "aws:forward_to_sfe"

StatelessFragmentDefaultActions:

- "aws:forward_to_sfe"

StatefulRuleGroupReferences:

- ResourceArn: !Ref NWFWStatefulRulegroup

Description: Default NW Firewall

Tags:

- Key: Name

Value: NetworkFirewallPolicy

- Key: Role

Value: Default NW FW Policy

and this is a RuleGroup sample

NWFWStatefulRulegroup:

Type: "AWS::NetworkFirewall::RuleGroup"

Properties:

RuleGroupName: awsfw-policy-euwe-001

Type: STATEFUL

RuleGroup:

RulesSource:

StatefulRules:

- Action: DROP

Header:

Destination: ANY

DestinationPort: ANY

Direction: ANY

Protocol: IP

Source: ANY

SourcePort: ANY

RuleOptions:

- Keyword: "sid"

Settings:

- "1000"

- Keyword: "rev"

Settings:

- "1"

- Action: PASS

Header:

Destination: ANY

DestinationPort: ANY

Direction: ANY

Protocol: ICMP

Source: ANY

SourcePort: ANY

RuleOptions:

- Keyword: "sid"

Settings:

- "1500"

- Keyword: "rev"

Settings:

- "1"

- Action: PASS

Header:

Destination: ANY

DestinationPort: "443"

Direction: FORWARD

Protocol: TCP

Source: !Ref AWSCIDR

SourcePort: ANY

RuleOptions:

- Keyword: "sid"

Settings:

- "2000"

- Keyword: "rev"

Settings:

- "1"

Capacity: 500

Description: This group prevents all but ICMP and HTTPS to the internet (enables SSM too)

Tags:

- Key: Name

Value: NetworkFirewallStatefulRuleGroup

- Key: Role

Value: Blocking all but ICMP and HTTPS to the internet

If you are looking for documentation for the NW Firewall RuleOptions in CFN, you can find them here: Suricata rule-options

Internet Egress

The Egress VPC gives you an option to do exfiltration scanning as part of an IDP/IDS setup. A simple version is using the NW firewall as it uses Suricata compatible rules.

SD-WAN

The SD-WAN appliances we have chosen, supports the AWS Connect Attachment option.

Configuring the Connect Attachment is a multi step process. First the attachment to the TGW endpoints is configured:

# Create the attachment to the SDW boxes

TransitGatewayConnectSDW:

Type: "AWS::EC2::TransitGatewayConnect"

Properties:

Options:

Protocol: "gre"

Tags:

- Key: Name

Value: Transit Gateway GRE to SDW

TransportTransitGatewayAttachmentId: !Ref TransitGatewayAttachmentSDWTGW

Then you attach that to the default TGW route table too:

# Associate the SDW ConnectAttachment with Default route table

TransitGatewaySDWANDEFRouteTableAssociation:

Type: AWS::EC2::TransitGatewayRouteTableAssociation

Properties:

TransitGatewayAttachmentId: !Ref TransitGatewayConnectSDW

TransitGatewayRouteTableId: !Ref TransitGatewayDEFRouteTable

Setting up the last part of the Connect Attachments, connecting the peers, requires the last CLI command

aws ec2 create-transit-gateway-connect-peer --transit-gateway-attachment-id $tgwattid \
--peer-address $sdwan-lanip \
--bgp-options PeerAsn=64xxx --inside-cidr-blocks 169.254.xx.0/29

Each peer creates two GRE tunnels, so in our case it is 6 in total. Currently you will not need more than two AZs, as it is the BGP4 configuration which determines where the traffic is routed (which AZ it goes to), so you cannot avoid cross AZ traffic costs.

We had a challenge here with the number of prefixes propagated, and chose to limit it to a static "catch all internal". This issue can only be spotted if you have the Network Manager configured and have enabled the sharing of events with cloudwatch (a manual console task for now).

Network Manager

The network manager is a service which provides a graphical representation of your TGW setup. In this same I have simply provided it to get the benefit of debugging needed in the previous section.

The NW manager is set up like this:

NMGlobalNetwork:

Type: AWS::NetworkManager::GlobalNetwork

Properties:

Description: Global AWS network

Tags:

- Key: Name

Value: GlobalAWSNetwork

- Key: Role

Value: Visibility of TGW connections between sites

A site is needed, in order to be able to add the TGW:

NMSite:

Type: AWS::NetworkManager::Site

Properties:

Description: "Transit GW Hub"

GlobalNetworkId: !Ref NMGlobalNetwork

Location:

Address: "AWS eu-west-1 region"

Tags:

- Key: Network

Value: Central TGW Site

and finally the TGW registration - easy as pie

NMTGWRegistration:

Type: AWS::NetworkManager::TransitGatewayRegistration

Properties:

GlobalNetworkId: !Ref NMGlobalNetwork

TransitGatewayArn: !Sub "arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:transit-gateway/${TransitGateway}"

Wrapping up

This became a bit longer than intended, but I hope it gave you an overview or the missing piece for your TGW setup. I haven't touched upon the DNS setup, but I might pick that up in a later post.

As always, if you need further assistance, we are here (email address below)