Enabling Auto-TLS on EnRoute with Jetstack Cert Manager
EnRoute HowTos
Introduction - Jetstack Cert Manager
Jetstack Cert Manager is a Kubernetes package that helps working with Certificate Authorities like Let’s Encrypt to automatically sign and renew certificates. Helm is a popular package manager choice for Kubernetes. Installation of software, managing versions, upgrading versions and finding charts from registry are key benefits of Helm.
Installing TLS Certificates for EnRoute Ingress using Jetstack Cert Manager
EnRoute helm chart installs the EnRoute Ingress Controller to define L7 policy for a service running inside Kubernetes. The helm chart provides a fine-grained control to define L7 policies with its ability to enable/disable plugins for a service using configuration options that can be specified when helm is invoked.
EnRoute also supports plugins/filters to extend functionality and enforce policies. The features page lists the available plugins for the Gateway. More details about each of the plugins can also be found on plugin pages.
The helm chart integrates with Jetstack Cert manager to install a TLS certificate for a service. Once Jetstack Cert manager is installed, for example the following command sets up Jetstack Cert manager and issues a certificate -
helm upgrade httpbin-service-policy local-saaras/service-policy \
--set service.namespace=demo-service \
--set service.name=httpbin \
--set service.prefix=/get \
--set service.port=80 \
--set service.enableTLS=true \
--set autoTLS.certificateCN=httpbin.enroutedemo.com \
--set autoTLS.enableProd=true
In the next few sections we describe what artifacts for certificate issue are created and how to debug it.
Artifacts created to support TLS Certificate Generation
Jetstack Cert Manager needs the creation of a ClusterIssuer that represent a certificate authority for certificates. EnRoute helm chart sets up a ClusterIssuer to work with Let’s encrypt staging, and another one to work with Let’s Encrypt production server -
ClusterIssuer
kubectl describe --all-namespaces clusterissuers.cert-manager.io
Name: letsencrypt-prod
Namespace:
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: httpbin-service-policy
meta.helm.sh/release-namespace: default
API Version: cert-manager.io/v1
Kind: ClusterIssuer
Metadata:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
Self Link: /apis/cert-manager.io/v1/clusterissuers/letsencrypt-prod
UID: 370db8e7-fb61-4e41-a740-6e0f3bcabc1c
Spec:
Acme:
Email: contact@example.com
Preferred Chain:
Private Key Secret Ref:
Name: letsencrypt-prod
Server: https://acme-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: enroute
Selector:
Status:
Acme:
Last Registered Email: contact@example.com
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/118404211
Conditions:
Last Transition Time: 2021-04-16T23:33:02Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Name: letsencrypt-staging
Namespace:
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: httpbin-service-policy
meta.helm.sh/release-namespace: default
API Version: cert-manager.io/v1
Kind: ClusterIssuer
Metadata:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
Self Link: /apis/cert-manager.io/v1/clusterissuers/letsencrypt-staging
UID: 34db7e69-f9a4-4009-bc68-7f1651f23d93
Spec:
Acme:
Email: contact@example.com
Preferred Chain:
Private Key Secret Ref:
Name: letsencrypt-staging
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: enroute
Selector:
Status:
Acme:
Last Registered Email: contact@example.com
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/18972900
Conditions:
Last Transition Time: 2021-04-16T23:33:02Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Certificates
You get a certificate to install on the service by creating a Certificate object. This initiates the certificate signing process. Depending on the Issuer used, the staging or production server is used to get a certificate.
kubectl describe --all-namespaces certificates.cert-manager.io
Name: httpbin.enroutedemo.com
Namespace: demo-service
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: httpbin-service-policy
meta.helm.sh/release-namespace: default
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2021-04-17T00:00:26Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
API Version: cert-manager.io/v1
Time: 2021-04-17T00:02:36Z
Resource Version: 47879688
Self Link: /apis/cert-manager.io/v1/namespaces/demo-service/certificates/httpbin.enroutedemo.com
UID: e41092bf-6773-49d3-9f5d-4338798c97ab
Spec:
Common Name: httpbin.enroutedemo.com
Dns Names:
httpbin.enroutedemo.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: httpbin.enroutedemo.com
Status:
Conditions:
Last Transition Time: 2021-04-17T00:02:36Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2021-07-15T23:02:34Z
Not Before: 2021-04-16T23:02:34Z
Renewal Time: 2021-06-15T23:02:34Z
Revision: 1
Events: <none>
Tracing Certificate creation logs
Jetstack Cert Manager installation creates services in the cert manager space.
kubectl get all -n cert-manager
NAME READY STATUS RESTARTS AGE
pod/cert-manager-56f5c44b5d-wshwn 1/1 Running 1 3d23h
pod/cert-manager-cainjector-5f494679f9-q8jhq 1/1 Running 0 3d23h
pod/cert-manager-webhook-5f59d4c7bd-xhqgj 1/1 Running 0 3d23h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.96.82.166 <none> 9402/TCP 3d23h
service/cert-manager-webhook ClusterIP 10.96.239.164 <none> 443/TCP 3d23h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cert-manager 1/1 1 1 3d23h
deployment.apps/cert-manager-cainjector 1/1 1 1 3d23h
deployment.apps/cert-manager-webhook 1/1 1 1 3d23h
NAME DESIRED CURRENT READY AGE
replicaset.apps/cert-manager-56f5c44b5d 1 1 1 3d23h
replicaset.apps/cert-manager-cainjector-5f494679f9 1 1 1 3d23h
replicaset.apps/cert-manager-webhook-5f59d4c7bd 1 1 1 3d23h
Tailing logs for the cert-manager service provides logs to debug certificate issue
kubectl logs -n cert-manager cert-manager-56f5c44b5d-wshwn --follow --since=5m
I0415 19:03:14.188036 1 conditions.go:173] Setting lastTransitionTime for Certificate "httpbin.enroutedemo.com" condition "Issuing" to 2021-04-15 19:03:14.188026423 +0000 UTC m=+3382.297097405
I0415 19:03:14.191331 1 conditions.go:173] Setting lastTransitionTime for Certificate "httpbin.enroutedemo.com" condition "Ready" to 2021-04-15 19:03:14.191322087 +0000 UTC m=+3382.300393029
...
I0415 19:03:16.904672 1 pod.go:70] cert-manager/controller/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" "dnsName"="httpbin.enroutedemo.com" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
...
I0415 19:03:17.018426 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.018685 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.127299 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.127613 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
E0415 19:03:17.139881 1 controller.go:156] ingress 'enroutedemo/cm-acme-http-solver-kd82k' in work queue no longer exists
E0415 19:03:17.150861 1 controller.go:156] ingress 'enroutedemo/cm-acme-http-solver-drhzw' in work queue no longer exists
I0415 19:03:17.269384 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.269793 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.536732 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.537821 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.538186 1 ingress.go:92] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-vglq7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
...
I0415 19:03:27.071540 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:27.071676 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:27.071761 1 ingress.go:92] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-vglq7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
...
I0415 19:03:38.290830 1 pod.go:118] cert-manager/controller/challenges/cleanupPods "msg"="deleting pod resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:38.309099 1 pod.go:126] cert-manager/controller/challenges/cleanupPods "msg"="successfully deleted pod resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
E0415 19:03:38.415267 1 controller.go:156] ingress 'enroutedemo/cm-acme-http-solver-vglq7' in work queue no longer exists
I0415 19:03:38.832551 1 acme.go:198] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="httpbin.enroutedemo.com-f4xqt" "resource_namespace"="enroutedemo" "resource_version"="v1"
I0415 19:03:38.833597 1 conditions.go:222] Found status change for CertificateRequest "httpbin.enroutedemo.com-f4xqt" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-04-15 19:03:38.833587237 +0000 UTC m=+3406.942658189
...