End-to-end encryption using EnRoute and Istio

Istio Integration

Introduction

This document describes how easy it is to integrate Istio with EnRoute. Istio is a service mesh based on Envoy Proxy that encrypts traffic between micro-services inside a Kubernetes Cluster

Enabling EnRoute integration with Istio can be done in one step by setting a flag and running a container along with EnRoute (to serve secrets) to participate in Istio trust framework.

An End-to-end encryption of traffic using EnRoute and istio includes -

  • Encryption from client to EnRoute
  • Encryption from EnRoute to mTLS inside the mesh

We install EnRoute and Istio, enable cluster-wide mTLS, configure EnRoute for Istio environment and make the secrets available to EnRoute. We will go through each of these steps by -

  • Setting up a cluster, installing Istio, setting up example workload (bookinfo app)
  • Next we install EnRoute, a container to serve Istio secrets to EnRoute to make it a part of the mesh
  • Enable mesh-wide mTLS enforce strict zero-trust environment and make EnRoute a part of Istio mesh
  • We install a certificate on the GatewayHost to achieve end-to-end encryption

We trace through each of the above steps while monitoring the cluster to verify end-to-end encrypted traffic.

We also verify some of the steps above using the open source Kiali project for observing a Kubernetes Istio deployment

Deployment of EnRoute showing meshed with HTTP Metrics

Pre-requisites

Check Kubernetes Cluster 

kubectl get nodes

NAME                                                STATUS   ROLES    AGE   VERSION
k3s-feature-test-03c4-fa116a-node-pool-f572-w4zm6   Ready    <none>   54d   v1.22.11+k3s1
k3s-feature-test-03c4-fa116a-node-pool-f572-nnm4h   Ready    <none>   54d   v1.22.11+k3s1

Install and Check Istio Installation

The installation steps for Istio are covered on the Istio website, they aren’t covered here. Once Istio is installed, we can verify the installation.

We installed Istio using helm, and the installation can be verified using helm

helm status istiod -n istio-system

NAME: istiod
LAST DEPLOYED: Wed Nov 23 01:10:47 2022
NAMESPACE: istio-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
"istiod" successfully installed!

To learn more about the release, try:
  $ helm status istiod
  $ helm get all istiod

Next steps:
  * Deploy a Gateway: https://istio.io/latest/docs/setup/additional-setup/gateway/
  * Try out our tasks to get started on common configurations:
    * https://istio.io/latest/docs/tasks/traffic-management
    * https://istio.io/latest/docs/tasks/security/
    * https://istio.io/latest/docs/tasks/policy-enforcement/
    * https://istio.io/latest/docs/tasks/policy-enforcement/
  * Review the list of actively supported releases, CVE publications and our hardening guide:
    * https://istio.io/latest/docs/releases/supported-releases/
    * https://istio.io/latest/news/security/
    * https://istio.io/latest/docs/ops/best-practices/security/

For further documentation see https://istio.io website

Tell us how your install/upgrade experience went at https://forms.gle/99uiMML96AmsXY5d6

We will be running the bookinfo sample application packaged with istio. Ensure that the bookinfo sample app is installed (if not already installed)

kubectl get pods

NAME                              READY   STATUS    RESTARTS   AGE
details-v1-7d88846999-d7q4w       2/2     Running   0          4d
reviews-v1-55b668fc65-mb28q       2/2     Running   0          4d
reviews-v3-7886dd86b9-7k5tp       2/2     Running   0          4d
ratings-v1-754f9c4975-s7wv2       2/2     Running   0          4d
productpage-v1-7795568889-cl9sw   2/2     Running   0          4d
reviews-v2-858f99c99-tpjbt        2/2     Running   0          4d

Install and Check Kiali Installation

Installing kiali is optional, but it makes it easier to ensure mesh-wide TLS and other aspects of visualizing istio. We will use kiali to check traffic flow graph, mTLS and security aspects of our installation.

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.16/samples/addons/kiali.yaml

serviceaccount/kiali created
configmap/kiali created
clusterrole.rbac.authorization.k8s.io/kiali-viewer created
clusterrole.rbac.authorization.k8s.io/kiali created
clusterrolebinding.rbac.authorization.k8s.io/kiali created
role.rbac.authorization.k8s.io/kiali-controlplane created
rolebinding.rbac.authorization.k8s.io/kiali-controlplane created
service/kiali created
deployment.apps/kiali create

Ensure automatic injection of Istio sidecar is enabled

Istio installs a side-car in every container (just like linkerd-proxy) to create a mesh. These proxies are a part of the Istio trust system that can fetch and use Istio secrets.

For EnRoute to be a part of this Istio trust system, we have an Istio proxy injected in the EnRoute pod. This helps EnRoute work with the Istio trust framework, to enforce a zero-trust environment.

To make the EnRoute pod a part of the Istio mesh, we enable automatic sidecar injection for EnRoute namespace

Create Namespace

kubectl create namespace enroute-system

namespace/enroute-system created

Label Namespace for automatic SideCar injection

kubectl label namespace enroute-system istio-injection=enabled

namespace/enroute-system labeled

Install EnRoute in Istio environment

EnRoute helm chart has support for installation in Istio environment. The helm chart sets up EnRoute in an Istio environment creating all the necessary environment variables while also installing a cert-server pod for serving secrets to EnRoute

Note the istio.enable flag during EnRoute installation. This flag ensures that all the necessary artifacts are setup for EnRoute to be a part of Istio trust framework.

helm install --set istio.enable=true -n enroute-system \
  enroute-istio saaras/enroute

NAME: enroute-istio
LAST DEPLOYED: Sun Nov 27 20:10:38 2022
NAMESPACE: enroute-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
π™΄πš—πšπš˜πšžπšπšŽ Ingress API Gateway Community Edition Installed!
-------------------------------------------------------
Request a free evaluation license for enterprise version
by sending an email to contact@saaras.io

Slack Channel         - https://slack.saaras.io
Getting Started Guide - https://getenroute.io/docs/getting-started-enroute-ingress-controller/
EnRoute Features      - https://getenroute.io/features/

Ensure that Istio proxy got injected in EnRoute pod

kubectl get pods -n enroute-system

NAME                      READY   STATUS    RESTARTS   AGE
enroute-b94698958-dg84n   5/5     Running   0          8m50s

Get the list of containers in the pod

kubectl get pods -n enroute-system enroute-b94698958-dg84n -o jsonpath='{.spec.containers[*].name}'

enroute cert-server redis envoy istio-proxy

Note that istio-proxy and cert-server got setup along with enroute envoy and redis containers

EnRoute is now installed in an Istio environment, and setup to use secrets generated by Istio (through cert-server) for traffic inside the cluster.

Next, we lock down the cluster by enforcing strict traffic policies inside the cluster. We enable a cluster wide mTLS inside the cluster where the identity of all cluster traffic is verified.

Enable EnRoute to use Istio Secrets for upstream traffic

The above steps setup a mechanism to make Istio secrets accessible to EnRoute. We now need to instruct EnRoute to use these secrets when communicating with service (bookinfo) inside the cluster.

We use the GlobalConfig type of resource for this -

apiVersion: enroute.saaras.io/v1
kind: GlobalConfig
metadata:
  labels:
    app: web
  name: enable-istio
  namespace: default
spec:
  config: |
    {
      "tlsContext" : { "alpnProtos" : ["istio", "http/1.1", "h2"] },
      "istio_enabled": true
    }
  name: global-config
  type: globalconfig_globals

The above configuration programs EnRoute use mTLS to communicate with service inside the cluster.

Enable strict mTLS for Istio

Next we force all traffic inside the cluster to be mTLS. We do this by creating istio configuration to only allow mTLS type of traffic globally

kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: global
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
EOF
EnRoute with Istio

One the above policy is configured, we can verify it by checking the Kiali console. Note the lock sign on the top right corner of the kiali console. It indicates that mesh-wide mTLS is enabled.

Istio Settings

Controlling Istio Certificate Validity Interval

Environment variables can be used to control the validity duration of certificates loaded by EnRoute. One such environment variable is SECRET_TTL which controls how long the secrets should be valid. They can be set in EnRoute annotations. Starting istio 1.6, such annotations will be honored by istio -

proxy.istio.io/config: |
  proxyMetadata:
    SECRET_TTL: 60m

Controlling Certificate output directory

When we need to access the key and certificate for a workload, we can provide a path to save them. This is applicable in case of workloads that are not natively a part of Istio system. An example of such a workload would be the one running on a Virtual Machine.

The path specified in OUTPUT_CERTS is used to persist the latest key and certificate of the workload.

proxy.istio.io/config: |
  proxyMetadata:
    OUTPUT_CERTS: /etc/istio-certs

Send traffic to test application

The productpage front page for the bookinfo application is setup to receive traffic on port 9080

kubectl get svc productpage

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
productpage   ClusterIP   10.43.116.119   <none>        9080/TCP   4d2h

We setup a GatewayHost to send traffic to the the productpage app.

apiVersion: enroute.saaras.io/v1
kind: GatewayHost
metadata:
  labels:
    app: productpage
    app.kubernetes.io/managed-by: Helm
  name: productpage-9080-gatewayhost
  namespace: default
spec:
  routes:
  - conditions:
    - prefix: /
    services:
    - name: productpage
      port: 9080
  virtualhost:
    fqdn: '*'

This can also be done easily using the helm chart for service-host-route

helm install product-host saaras/service-host-route \
  --namespace=default                               \
  --set service.prefix=/                            \
  --set service.name=productpage                    \
  --set service.port=9080

Note that this GatewayHost is setup without SSL termination. We’ll add a certificate in the next-step, to ensure end-to-end encryption.

Get the public IP of EnRoute

kubectl get svc -n enroute-system

NAME      TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
enroute   LoadBalancer   10.43.177.204   212.2.241.93   80:31411/TCP,443:30422/TCP   39m

Send test traffic

curl -sS 212.2.241.93/productpage | grep -o "<title>.*</title>"

<title>Simple Bookstore App</title>

Ensure mTLS on kiali console

While we can access the productpage above, we verify mTLS on the kiali console. There are couple of ways to check that -

  • We note that only mTLS traffic allowed in the cluster with the lock on the upper right
  • There is a lock sign on top of the edge from EnRoute to productpage
  • We note that when the edge to productpage is selected, we can verify the namespace enroute-system principals
EnRoute with Istio

Encrypt traffic from client to EnRoute

To encrypt traffic from client to EnRoute, we create a kubernetes TLS secret and install it on the GatewayHost

Create a Kubernetes secret using Key and Certificate

kubectl create secret tls bookinfo-saaraslabs \
  --cert=bookinfo-saaraslabs-cert.pem \
  --key=bookinfo-saaraslabs-key.pem

secret/bookinfo-saaraslabs created

Associate the secret with GatewayHost

apiVersion: enroute.saaras.io/v1
kind: GatewayHost
metadata:
  labels:
    app: productpage
  name: productpage-9080-gatewayhost
  namespace: default
spec:
  routes:
  - conditions:
    - prefix: /
    services:
    - name: productpage
      port: 9080
  virtualhost:
    fqdn: bookinfo.saaraslabs.com
    tls:
      secretName: bookinfo-saaraslabs

Send some traffic over HTTPs

curl -sS https://bookinfo.saaraslabs.com/productpage | grep -o "<title>.*</title>"

<title>Simple Bookstore App</title>

Conclusion

EnRoute provides an extremely easy integration with Istio

EnRoute integrates with Istio and Linkerd. The integration with Linkerd is described in another article.

← Why Every API Needs a Clock?
End-to-end encryption using EnRoute and Linkerd →