Kubernetes Tutorial: Managing TLS Certificates with Ambassador

TL;DR: In this article, you will learn how to leverage the Ambassador API Gateway to secure the apps running in your Kubernetes clusters with TLS certificates. The approach that the article describes will enable you to use Let's Encrypt to issue certificates for free. With that, you will be able to enhance the security of your clusters (and its apps) for free, even if you are hosting multiple domains on them. You can use this GitHub repository as a reference if you need to double check the Kubernetes resources presented here.

"Learn how Ambassador, cert-manager, and Let's Encrypt can work together to issue TLS certificates to your Kubernetes cluster's domains for free!"
Tweet This

Prerequisites

To follow along with this article, you will need some previous experience with Kubernetes. If you are new to this platform, check out the "Step by Step Introduction to Basic Kubernetes Concepts" tutorial. There, you will learn everything you need to follow the instructions here.

Besides that, you will need kubectl, a Command-Line Interface (CLI) tool that will enable you to control your cluster from a terminal. If you don't have this tool, check the instructions on the "Installing Kube Control (kubectl)" section available on the article mentioned above.

Another thing you will need is a couple of domains to test the steps described here. If you already have some domains hanging around, you can use them (or you can use subdomains under one that you own, for example: kubernetes-tutorial.yourdomain.com). If you don't have any domains you can use, you can get some for free on Freenom.

Lastly, you will also need a Kubernetes cluster. If you already have a cluster running, you can skip the next section and jump into action. If you don't, keep reading as you will learn how to use DigitalOcean to spin up a cluster for free.

Spinning up a DO Kubernetes cluster

For starters, you can use this DO referral link to get some credit to avoid paying, while you read this article. If you don't use a referral link, you will end up paying for your cluster from the very beginning. This might not be a big problem because DigitalOcean prices are quite fair and you will need to keep the cluster running for less than an hour. That is, it won't cost more than a few cents but, well, not paying is better than paying, right?

After using this link to create your account on DigitalOcean, you will get an email confirmation. Use the link sent to you to confirm your email address. Confirming your address will make DigitalOcean ask you for a credit card. Don't worry about this. If you don't spend more than $100 USD, they won't charge a dime.

After inputting a valid credit card, you can use the next screen to create a project, or you can use this link to skip this unnecessary step and to head to DO's Managed Kubernetes dashboard.

From the dashboard, you can hit the Create a Kubernetes cluster button (you might have to click on Enable Limited Access first). Then, DigitalOcean will show you a new page with a form that you can fill in as follows:

Select a Kubernetes version: The instructions on this article were tested with the 1.13.5-do.1 version. If you feel like testing other versions, feel free to do so. Just let us know how it went.
Choose a datacenter region: Feel free to choose whatever region you prefer.
Choose cluster capacity: Make sure you have just one node pool, that you choose the $10/Month per node option, and that you have at least three nodes.
Add Tags: Don't worry about tagging anything.
Choose a name: You can name your cluster whatever you want (e.g., "kubernetes-tutorial"). Just make sure DigitalOcean accepts the name (e.g., names can't contain spaces).

After filling in this form, you can click on the Create Cluster button. It will take a few minutes before DigitalOcean finishes creating your cluster for you. However, you can already download the cluster's config file now.

This file contains the credentials needed for you to act as the admin of the cluster, and you can find it on the cluster's dashboard. After you clicked on the Create Cluster button, DigitalOcean redirected you to your cluster's dashboard. From there, if you scroll to the bottom, you will see a button called Download Config File. Click on this button to download the file.

When you finish downloading this file, open a terminal and move the file to the .kube directory in your home dir (you might have to create this directory first):

# make sure .kube exists
mkdir ~/.kube

# move the config file to it
mv ~/Downloads/kubernetes-tutorial-kubeconfig.yaml ~/.kube

If needed, adjust the last command with the correct path of the downloaded file.

The ~/.kube directory is a good place to keep your Kubernetes credentials. By default, kubectl will use a file named config (if it finds one inside the .kube dir) to communicate with clusters. To use a different file, you have three alternatives:

First, you can specify another file by using the --kubeconfig flag in your kubectl commands, but this is too cumbersome.
Second, you can define the KUBECONFIG environment variable to avoid having to type --kubeconfig all the time.
Third, you can merge contexts in the same config file and then you can switch contexts.

The second option (setting the KUBECONFIG environment variable) is the easiest one, but feel free to choose another approach if you prefer. To set this environment, you can issue the following command:

export KUBECONFIG=~/.kube/kubernetes-tutorial-kubeconfig.yaml

Note: Your file path might be different. Make sure the command above contains the right path.

Keep in mind that this command will set this environment only on this terminal's session. If you open a new terminal, you will have to execute this command again.

What is Ambassador? The API Gateway

If you don't know Ambassador, the Kubernetes-native API gateway that you will use to manage TLS certificates, don't worry. Ambassador is built on a declarative configuration model that exposes a bunch of really neat features in Envoy Proxy. This tool is an open-source project that helps you "secure, scale, and ship your microservices" and that is really easy to use. You don't really need to learn about it before diving into this article. But after you finish reading the instructions here, make sure you check out their documentation so you can see how it works.

Note: Before moving to the next section, make sure your Kubernetes cluster is up and running. To do so, execute kubectl get nodes and check if all nodes show Ready on the STATUS column.

Deploying Ambassador

After these introductory steps, it's time to see some action. For starters, you will need to install Ambassador in your Kubernetes cluster. To do so, execute the following code:

# install Ambassador
kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-rbac.yaml

Note: If you are not using DigitalOcean, you might need to issue a few different commands. In this case, please, check the instructions on the Deploying Ambassador section of this resource.

After installing Ambassador, you will need to create a Kubernetes service to expose a LoadBalancer that is integrated with the gateway. To do so, you will need to create a YAML file. However, before doing this, as you will need to create a few different YAML files throughout this article, you will be better off creating and using a directory to group these files. So, in your preferred location, create a directory called ambassador-tls; then move into it:

# create a directory to add YAML files
mkdir ambassador-tls

# move into it
cd ambassador-tls

After that, create a file called ambassador-service.yaml inside this directory and add the following code to it:

---
apiVersion: v1
kind: Service
metadata:
  name: ambassador
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
  selector:
    service: ambassador

As mentioned, this file creates a service (kind: Service) that exposes your cluster through a LoadBalancer that is attached to Ambassador (service: ambassador). Now, execute the following command to deploy this service to your cluster:

kubectl apply -f ambassador-service.yaml

This command will finish executing promptly. However, the load balancer that this service creates is an external element controlled by the Kubernetes vendor (e.g., DigitalOcean) that needs a couple of minutes to be up.

To check if your load balancer is already available, issue the following command:

kubectl get svc ambassador

If the EXTERNAL-IP column that this command presents is showing <pending>, then your load balancer is not ready yet. Nevertheless, you can continue with the instructions here until further notice.

As you will need an application up and running to see the whole configuration in action, you will deploy a sample application. To do so, create a file called deployment.yaml and add the following code to it:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: auth0blog/kubernetes-tutorial
        ports:
          - containerPort: 3000

When applied, this file will deploy a simple application that emulates a much simpler Twitter application where users can share their thoughts. However, note that this app doesn't even handle user authentication (by the way, if you read the Kubernetes introduction, you will notice that this is the same application).

After creating this file, execute the following command to apply the deployment:

kubectl apply -f deployment.yaml

Then, you will need to create another service. This time the goal is to expose the deployment inside your cluster. So, create a file called service.yaml and add the following code to it:

---
apiVersion: v1
kind: Service
metadata:
  name: kubernetes-tutorial-service
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: kubernetes-tutorial-mapping
      prefix: /
      service: kubernetes-tutorial-service
      host_rewrite: kubernetes-tutorial-service
spec:
  ports:
    - port: 80
      targetPort: 3000
      name: http
      protocol: TCP
  selector:
    app: kubernetes-tutorial-deployment

This file is doing two important things: first, it is creating a Kubernetes service to represent your deployment and its pods; second, it is leveraging Ambassador to create a mapping (kind: Mapping) to define that this service must handle requests to the root path (prefix: /).

To apply this service, execute the following command:

kubectl apply -f service.yaml

Then, execute kubectl get svc ambassador once more and copy the external IP address of your load balancer. Now, if you use this IP address in a browser, you will be able to see the sample application running.

Issuing a TLS Certificate with Ambassador

After deploying the sample application on Kubernetes and confirming that Ambassador is working, it is time to install and configure your first TLS certificate. This task will involve three steps:

You will configure one or more domains to point to your cluster.
You will install a tool called cert-manager to help Ambassador manage certificates.
You will use cert-manager and Ambassador to issue a TLS certificate.

Pointing domains to Kubernetes' load balancer

Since Let's Encrypt will issue certificates to specific domains, you will have to configure the DNS settings of one or more domains to point to your new load balancer.

For example, if you are using Freenom, you can go to the My Domains section of your dashboard and click on the Manage Domain button. After that, move to the Manage Freenom DNS tab and add an A record that points to the IP address of the load balancer; then, hit Save Changes to persist the new setting.

Note: After adding a new record to your domain, the DNS servers that support the internet will need a few minutes to sync. That is, you won't be able to open http://yourdomain.com in a web browser instantaneously. However, after these minutes, you will see that if you request this domain in your browser, the sample application you deployed earlier will load.

Installing cert-manager

Having configured your domain to point to your cluster, the next thing you will do is to install cert-manager. This tool is a Kubernetes add-on that facilitates management and issuance of TLS certificates from various issuing sources. Combining cert-manager with Ambassador will make TLS certificate management an easy task.

To install it, you will need to create a Kubernetes namespace where you will deploy the resources that belong to cert-manager:

# create a namespace for cert-manager's resources
kubectl create namespace cert-manager

After that, you will need to disable resource validation on the namespace so the installation doesn't end up on a deadlock:

# disable resource validation on the cert-manager namespace
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

Then, you can install cert-manager:

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/cert-manager.yaml

Issue certificates with Ambassador and cert-manager

Now that your domain is pointing to your cluster and that you installed cert-manager, you can issue a TLS certificate to your domain. To do so, you will need to create and deploy two Kubernetes resources: a ClusterIssuer and a Certificate.

So, create a file called certificate-issuer.yaml and add the following resources to it:

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: kubernetes-tutorial-tk-issuer
spec:
  acme:
    email: bruno.krebs@auth0.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: kubernetes-tutorial-tk-issuer
    http01: {}
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kubernetes-tutorial-tk-certificate
  namespace: default
spec:
  secretName: kubernetes-tutorial-tk-cert-secret
  issuerRef:
    name: kubernetes-tutorial-tk-issuer
    kind: ClusterIssuer
  dnsNames:
  - kubernetes-tutorial.tk
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - kubernetes-tutorial.tk

You will have to replace a few properties on this file. For example, you will need to replace bruno.krebs@auth0.com with your email address. Besides that, you will have to replace the two places that use kubernetes-tutorial.tk with the domain you are using. Lastly, you can replace the kubernetes-tutorial-tk- prefix that this file uses to something more meaningful to you (e.g., your-domain-com-). This last step is optional but will help you identify the resources later.

Then, you can issue the following command to apply these resources to your cluster:

# create cluster issuer and certificate
kubectl apply -f certificate-issuer.yaml

Now, if you check the Certificate status:

# check the status of the certificate
kubectl describe certificates kubernetes-tutorial-tk-certificate

You will notice that the last entry on Events is OrderCreated. No matter how long you wait, cert-manager will never be able to fulfill this order. The problem is that cert-manager uses an Ingress resource to issue the challenge to /.well-known/acme-challenge. However, since Ambassador is not an Ingress, you will need to create a Mapping so cert-manager can reach the temporary pod.

To do so, first, you will need to find the cert-manager pod:

# find cert-manager name
kubectl get pods -n cert-manager

This command will fetch all pods on the cert-manager namespace. More specifically, this command will return three pods: one that starts with cert-manager-webhook-; one that starts with cert-manager-cainjector-; and one that starts with cert-manager- only. You are interested in the latter. Copy its full name (it will be something like cert-manager-5c54bbcc4d-2dpzg), and use it on the following command:

# get cert-manager logs
kubectl logs -n cert-manager cert-manager-5c54bbcc4d-2dpzg | grep acme-http-domain

This command will fetch the logs of the pod; then it will use grep to find lines that contain acme-http-domain. The lines that this command returns will show two properties separated by a comma. The properties will be similar to this:

certmanager.k8s.io/acme-http-domain=1745815783;
and certmanager.k8s.io/acme-http-token=744820138

You will need to use the values after the equals sign (i.e., after acme-http-domain= and after acme-http-token=) in the next file.

Now, create a file called challenge-service.yaml, and add the following code to it:

---
apiVersion: v1
kind: Service
metadata:
  name: acme-challenge-service
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: acme-challenge-mapping
      prefix: /.well-known/acme-challenge
      rewrite: ""
      service: acme-challenge-service 
spec:
  ports:
  - port: 80
    targetPort: 8089
  selector:
    certmanager.k8s.io/acme-http-domain: "1745815783"
    certmanager.k8s.io/acme-http-token: "744820138"

This file is using an Ambassador mapping to expose the /.well-known/acme-challenge route (under your domain) to enable Let's Encrypt to confirm that your cluster is indeed "attached" to the domain (or, in other words, that you own the domain you are using to issue a certificate). Before proceeding, make sure you replace the values passed to certmanager.k8s.io/acme-http-domain and certmanager.k8s.io/acme-http-token with the ones you fetched with grep; then apply the service to your cluster:

kubectl apply -f challenge-service.yaml

After a couple of minutes, Let's Encrypt and cert-manager will have accomplished their goal and will have issued a valid certificate to your domain. To confirm that, issue the following command again:

# check the status of the certificate
kubectl describe certificates kubernetes-tutorial-tk-certificate

Now, you will see that the last event is CertIssued and that the message says "Certificate issued successfully". To wrap this section, open the ambassador-service.yaml and replace its code with this:

---
apiVersion: v1
kind: Service
metadata:
  name: ambassador
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Module
      name: tls
      config:
        server:
          enabled: True
          redirect_cleartext_from: 8080
          secret: kubernetes-tutorial-tk-cert-secret
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443
  selector:
    service: ambassador

Note: If you changed the name of the secret while creating the certificate-issuer.yaml file, you will need to fix the code above to reference the correct one (i.e., you will have to replace kubernetes-tutorial-tk-cert-secret to whatever you chose).

The new version of this file is going to make three changes to Ambassador. First, it will configure Ambassador's service to use the TLS Module (note: name: tls is case sensitive, don't change it). Second, it will make Ambassador listen to port 443, the default one used on secure connections (HTTPS). Third, it will redirect requests from port 80 (unsecured) to port 443 (redirect_cleartext_from: 8080).

After changing this file, use the following command to apply the changes:

kubectl apply -f ambassador-service.yaml

If everything works as expected, you will be able to load the sample app by using the https: protocol plus your domain (e.g., https://kubernetes-tutorial.tk) in a web browser. Also, you will notice that, no matter how hard you try, even if you ask for an insecure version of your site first (i.e., http:// instead of https://), your browser will load the secured version (you can thank redirect_cleartext_from for that).

"I just used Ambassador, cert-manager, and Let's Encrypt to issue TLS certificates to my Kubernetes cluster's domain for free!"
Tweet This

Managing Multiple Domains with Ambassador

Another great feature that ships with Ambassador is the support to Server Name Indication (SNI). This feature lets you supply separate TLS certificates for different domains instead of using a single TLS certificate for all domains.

To see this in action, you will need another domain, and you will need to point this domain's DNS to your load balancer's IP address. If you don't have another domain to use, you can get one for free at Freenom.

Then, you will have to go to the dashboard where you manage your domain's DNS and add an A record that points to the IP address of the load balancer. As a reminder, you can get the external IP address of your Kubernetes cluster by issuing kubectl get svc ambassador.

After configuring the DNS, the first thing you will have to do is to deactivate the tls module on your Ambassador installation. If you don't do this, Let's Encrypt won't be able to issue a certificate to your second domain. This happens because, for the moment, your cluster is configured to use the certificate issued for the first domain. If Let's Encrypt (or anyone for that matter) tries to issue a request to the second domain, they will get an error saying that the certificate is not valid for that domain.

So, replace the contents of the ambassador-service.yaml file with this:

---
apiVersion: v1
kind: Service
metadata:
  name: ambassador
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
  selector:
    service: ambassador

Then, apply this to your cluster by issuing kubectl apply -f ambassador-service.yaml. This will reconfigure the Ambassador service to allow unsecured (plain HTTP) requests again.

After that, you will have to define the same resources you defined for your first domain: a ClusterIssuer and a Certificate. So, create a file called certificate-issuer-second-domain.yaml and add the following to it:

---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: second-domain-tk-issuer
spec:
  acme:
    email: bruno.krebs@auth0.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: second-domain-tk-issuer
    http01: {}
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: second-domain-tk-certificate
  namespace: default
spec:
  secretName: second-domain-tk-cert-secret
  issuerRef:
    name: second-domain-tk-issuer
    kind: ClusterIssuer
  dnsNames:
  - second-domain.tk
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - second-domain.tk

Just like before, you will have to replace a few properties on this file. For example, you will need to replace bruno.krebs@auth0.com with your email address. Besides that, you will have to replace the two places that use second-domain.tk with the domain you own. Lastly, you can replace the second-domain-tk- prefix that this file uses to something more meaningful to you (e.g., your-domain-com-). This last step is optional but will help you identify the resources later.

Then, you can issue the following command to apply these resources to your cluster:

# create cluster issuer and certificate
kubectl apply -f certificate-issuer-second-domain.yaml

If you recall, the first time you were issuing a TLS certificate, you had to create a service to make .well-known/acme-challenge a valid path under your domain. This is necessary so Let's Encrypt can confirm the ownership of the domain. So, repeating the steps you executed before, issue the following command to get the name of the cert-manager pod:

# find cert-manager name
kubectl get pods -n cert-manager

Then, use it to get two properties:

# get cert-manager logs
kubectl logs -n cert-manager cert-manager-5c54bbcc4d-2dpzg | grep acme-http-domain

Note: You will have to replace the pod name in this last command.

After running this command, you will see an output that shows the following properties:

certmanager.k8s.io/acme-http-domain=1745815783;
and certmanager.k8s.io/acme-http-token=744820138.

You will need to use the values after the equals sign (i.e., after acme-http-domain= and after acme-http-token=) in the challenge service. So, create a file called second-domain-challenge-service.yaml, and add the following code to it:

---
apiVersion: v1
kind: Service
metadata:
  name: second-domain-acme-challenge-service
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: second-domain-acme-challenge-mapping
      prefix: /.well-known/acme-challenge
      rewrite: ""
      service: second-domain-acme-challenge-service 
spec:
  ports:
  - port: 80
    targetPort: 8089
  selector:
    certmanager.k8s.io/acme-http-domain: "894240303"
    certmanager.k8s.io/acme-http-token: "1107562110"

Note: Replace the values passed to acme-http-domain and acme-http-token with the values you copied from the cert-manager pod. Also, feel free to replace the second-domain- prefix with something more meaningful.

Now, apply this service to your cluster:

kubectl apply -f second-domain-challenge-service.yaml

After a couple of minutes, Let's Encrypt and cert-manager will have issued the certificate for your second domain. To confirm that, issue the following command:

# check the status of the certificate
kubectl describe certificates second-domain-tk-certificate

If the last event is CertIssued and the message says "Certificate issued successfully", then you are good to go. If the last event still is OrderCreated, you will have to wait a couple more minutes before proceeding.

Lastly, you will have to reconfigure your Ambassador service to use the two certificates. So, replace the contents of the ambassador-service.yaml file with this:

---
apiVersion: v1
kind: Service
metadata:
  name: ambassador
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Module
      name: tls
      config:
        server:
          enabled: True
          redirect_cleartext_from: 8080
      ---
      apiVersion: ambassador/v1
      kind: TLSContext
      name: kubernetes-tutorial-tls-context
      hosts:
      - kubernetes-tutorial.tk
      secret: kubernetes-tutorial-tk-cert-secret
      ---
      apiVersion: ambassador/v1
      kind: TLSContext
      name: second-domain-tls-context
      hosts:
      - second-domain.tk
      secret: second-domain-tk-cert-secret
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443
  selector:
    service: ambassador

Then, if you apply this service (kubectl apply -f ambassador-service.yaml), you will be reconfiguring your Ambassador service to use both domains with separate certificates. Just note that you will have to fix the contents of this file to reflect your domains (i.e., you will have to replace kubernetes-tutorial.tk and second-domain.tk).

Now, if you open both domains on a browser, you will see that both will load securely. Also, if you click on the

"Using Ambassador to handle multiple domains on a Kubernetes cluster is easy. And each domain can use its own TLS certificate!"
Tweet This

Recap

In this article, you learned how to take advantage of Ambassador, cert-manager, and Let's Encrypt to manage TLS certificates for free. With this, you will enhance the security of the apps you run in your cluster and will have more confidence that your data will not be tampered in motion (for example, by a MITM attack).

What do you think about this approach? Did you know that Ambassador also supports SNI (Server Name Indication) to be able to manage different TLS certificates to different domains? If so, would you be interested in learning how to use this feature? Let us know in the comment box.

Blog

Kubernetes Tutorial: Managing TLS Certificates with Ambassador

Tweet This

Prerequisites

Spinning up a DO Kubernetes cluster

What is Ambassador? The API Gateway

Deploying Ambassador

Issuing a TLS Certificate with Ambassador

Pointing domains to Kubernetes' load balancer

Installing cert-manager

Issue certificates with Ambassador and cert-manager

Tweet This

Managing Multiple Domains with Ambassador

Tweet This

Recap