AWS VPN to NSX Edge Appliance

Working with the AWS default hardware VPN solution often leaves a lot to be desired, especially when trying to establish a tunnel to a policy-based VPN like the NSX Edge Appliance. In the past, we often turned to third-party software VPNs to work around the limitations and compatibility issues imposed by AWS’s VPN solution.

Recently, though, we came upon a request to establish a site–to-site VPN between an AWS VPC and NSX Edge. Upon initial configuration, the tunnel appeared to work as expected, but we soon discovered a strange issue. The tunnel would bounce sporadically and only allow connectivity between one of the three networks we had defined. When the tunnel came back, connectivity would rotate to another network.

As it turns out, buried in the AWS support documentation was one blob of text that explained the issue perfectly.

“You are limited to 1 unique Security Association (SA) pair per tunnel (1 inbound and 1 outbound), and therefore 2 unique SA pairs in total for 2 tunnels (4 SAs). Some devices use a policy-based VPN and create as many SAs as ACL entries. Therefore, you may need to consolidate your rules and then filter so you don’t permit unwanted traffic.”

Source: https://docs.aws.amazon.com/vpc/latest/adminguide/Introduction.html#DevicesTested

In simple terms, you can only have one network defined in the encryption domain. Luckily, in our case, the three networks local to the NSX Edge were able to be combined into one supernet without overlapping the network used in the AWS VPC. Once we made this change, we were able to pass traffic between all three local networks and the AWS VPC simultaneously. Also, the sporadic bouncing of the tunnel ceased and has remained stable.