EnRoute Gateway With AWS

EnRoute HowTos

EnRoute Gateway works in any cloud environment that runs a Kubernetes Cluster. EnRoute gateway works with public cloud provider Kubernetes or any other distribution.

This guide provides details on running EnRoute gateway on AWS, most common deployment configuration and different options available in AWS environment.

In AWS environment, EnRoute service annotations can program the AWS load balancer. AWS has several different loadbalancers that differ in the functionality provided. We explore different configurations to work with AWS LoadBalancers, the details of configuration and how good architectural practices.

We also discuss how less reliance on cloud provided load balancer can help achieve a better cloud-agnostic architecture which can be ported to other cloud environments.

AWS LoadBalancer #

AWS LoadBalancer options are

• Classic Load Balancer
• Application LoadBalancer (ALB)
• Network Load Balancer (NLB)

When a Kubernetes service of type LoadBalancer is created, it provisions a Load Balancer on AWS. By default, a classic load balancer is provisioned by the Kubernetes Controller on AWS. Provisioning of an Application Load Balancer has to be manual. Annotations can be used on the LoadBalancer service to provision a network load balancer.

Load Balancer annotations used on the LoadBalancer service can be used to provide configuration options to the Classic Load Balancer or the Network Load Balancer

The Classic Load Balancer was the first one created by AWS. Subsequently, a Network Load Balancer (NLB) and Application Load Balancer (ALB) were created as evolved versions of the load balancer. While Kubernetes LoadBalancer service can be run with either NLB or ALB, the preferred mechanism is using the NLB -

• Using a NLB load balancer keeps most of the L7 functionality inside Kubernetes and hence portable across clouds
• A NLB load balancer is cheaper to use compared to the ALB
• A NLB does not inspect traffic at L7. This also makes it the most efficient Load Balancer on AWS capable of handling millions of connections
• Annotations on the LoadBalancer service can be use to provide configuration options to the NLB

Network LoadBalancer (NLB) #

The NLB runs at TCP/SSL layer to load balance TCP/IP traffic. It does not look at any HTTP or HTTPs state to make load balancing decisions. The NLB can however terminate SSL connections.

NLB is the recommended mechanism to work in Kubernetes environments on AWS to keep costs minimal and delegate all the functionality to the L7 Proxy inside the cluster. See the section on Architectural Choices below to understand why delegating SSL termination and load balancing to a service inside Kubernetes is more desirable.

NLB Configuration #

Kubernetes controller on AWS provisions a cloud Load Balancer for LoadBalancer type of service. Configuration to the load balancer can be provided by specifying annotations on service definition.

EnRoute helm chart includes support for provisioning a NLB on AWS along with switches to control annotations. We discuss the most common options that are recommended to run EnRoute with AWS NLB load balancer

Annotations #

EnRoute helm chart has a switch to enable AWS annotations to achieve this configuration when setting up the enroute service

    helm upgrade enroute-demo saaras/enroute \
      --set enrouteService.rbac.create=true  \
      --set awsSettings.enable=true

Example Working configuration for EnRoute service #

Here is an example of how a working sample configuration looks like for an EnRoute deployment using NLB with original client IP preserved

 apiVersion: v1
 kind: Service
 metadata:
   annotations:
     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
     service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
     service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
     service.beta.kubernetes.io/aws-load-balancer-type: nlb
   name: enroute
   namespace: enroutedemo
 spec:
   externalTrafficPolicy: Local
   ports:
   - name: http
     port: 80
     protocol: TCP
     targetPort: 8080
   - name: https
     port: 443
     protocol: TCP
     targetPort: 8443
   selector:
     app: enroute
   sessionAffinity: None
   type: LoadBalancer

Architectural Choices #

We discuss here some of the architectural choices that can be made for an application. Appropriate choices help application make more cloud agnostic and provide better control and visibility over the application.

TLS Termination #

For example, terminating TLS inside Kubernetes using EnRoute make TLS termination and cipher selection cloud agnostic. EnRoute also integrates with external certificate providers using Jetstack and helps fetch and install certificates using one step. This greatly improves developer experience.

The same mechanism to terminate SSL traffic using EnRoute can be used on Google Cloud where certificates can be fetched an installed in one-step using the ACME protocol.

EnRoute support for ACME protocol using Jetstack Cert manager also provides the support for Let’s Encrypt Certificate Authority using one-step. When using Let’s Encrypt, a helm switch can be used to switch between staging and production Let’s Encrypt servers

Retain real client-IP address #

EnRoute is built on unmodified Envoy proxy and can leverage the PROXY protocol to retain the original client-IP. Using NLB with annotations described above help with retaining the original source-IP.

HTTP to HTTPS redirection #

EnRoute supports HTTP -> HTTPs redirection automatically when a certificate is installed for an application. No additional steps are required from the user.

L7 Logging #

L7 logging is an integral aspect for both troubleshooting and compliance. EnRoute integrates with external logging systems like fluentd and datadog to pipe log traffic directly to them. Logs can then be received at a choice of destination. For AWS environments, logs can be directed to CloudWatch destination to integrate with lest of the logging.

Predictable Pricing #

Since EnRoute can be horizontally scaled inside Kubernetes, it is very cost effective. EnRoute pricing is predictable and simple. It is simple consumption based pricing for billions of requests that matches the cloud consumption and pricing models. Typical EnRoute pricing results in over 90% savings over alternative solutions

Cloud Agnostic Architecture #

Using Envoy / EnRoute inside Kubernetes for L7 function makes the architecture, logging and visibility, debugging and L7 security cloud-agnostic. The same architecture can be recreated for edge use-cases inside a Kubernetes cluster inside any other cloud. EnRoute’s flexibility and integration with independent logging providers also helps with this approach.

Envoy’s deep visibility #

Envoy proxy provides deep visibility that provides deep insights into the working of the application. This is critical for operational efficiency and reducing the mean time to resolving problems when they arise.