Cert-Manager failed to create ClusterIssuer for letsencrypt due to TLS handshake error

First off, I’m new here so apologies if Applications isn’t the best category here. I was split between this, Network, and Virtualization.

I deployed a new basic install of MicroOS, installed k3s using their script on version 1.26.7 of kubernetes. From here I’ve installed Rancher using helm, and then added the jetstack repo to get the latest version of cert-manager, v1.12.4 at the time of this post.

One of the CRD’s provided by cert-manager is for a ClusterIssuer. This gets crated correctly, but when I try to create a ClusterIssuer pointing to letsencrypt I get an error that states TLS handshake error. I’ve ensured that their site is using a cipher that is enabled by default, and the exact same steps to get this deployed work on both Fedora and an Ubuntu server.

Does anyone else here have experience with Kubernetes and had issues adding letsencrypt (or others) as a ClusterIssuer and getting a similar error?

Happy to pull any logs or more information as needed. I’m not certain if this is an issue with k3s, MicroOS, cert-manager, etc. Figured I would start here, though.

Thanks!

@TechnoMario I wonder if letsencrypt is trying to write to the read-only filesystem. I’ve only ever used cert-manager and rancher, no letsencrypt.

I’m assuming the letsencrypt stuff is done in a transactional-update shell?

My normal deployment is jetstack first, then rancher…

I have tried deploying jetstack/cert-manager before rancher as well, and it didn’t make any difference. Since cert-manager is running as a helm app in the cluster I would not expect it to be writing anything to the RO system, but I may be incorrect there.

In case it wasn’t clear, I am only running cert-manager and rancher in the cluster, but I’m trying to set it up so that letsencrypt is one of the issuers instead of letting cert-manager only generate self-signed certs.

I’ll have to look at my other cluster that’s working correctly to see if the letsencrypt ClusterIssuer is actually trying to write out a CA or something on the system somewhere… that just doesn’t seem right since it’s running as a container though.

I’m not certain that’s it actually, as their documentation states that CA’s and other certificates are all saved as secrets within the appropriate namespace… so they should be contained within the K8s system.

@TechnoMario Might be worth a read as well? https://ranchermanager.docs.rancher.com/v2.7/getting-started/installation-and-upgrade/resources/upgrade-cert-manager

What version of Rancher?

I’ve seen that page before, too. I just rolled back to a base install, k3s is installed as well as helm. From there I added the jetstack helm repo, and following the commands below. This is without Rancher even, just trying to get cert-manager working on it’s own. I was trying this with Rancher v2.6.7 as well, though.

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.26.7+k3s1 sh -

transactional-update pkg in helm

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.12.4 --set installCRDs=true

Then I created the file on the sever below, and ran kubectl apply -f cert-setup.yml

# cert-setup.yml
---
# cloudflare token secret
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: [redacted]

---
# cluster issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [redacted]
    privateKeySecretRef:
      name: letsencrypt
    solvers:
      - dns01:
          cloudflare:
            email: [redacted]
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token

That’s are far as it gets. From there the secret and ClusterIssuer are generated in the cert-manager namespace, but the ClusterIssuer never reaches a ready state due to “Failed to register ACME account : Get “https://acme-v02.api.letsencrypt.org/directory”: remote error: tls: handshake failure.”

This exact same process, and file, works fine on other distributions. Since it’s all in pods inside of K8s, I wouldn’t expect it to be trying to write something to the RO OS, but that is an interesting idea. I just need to try to identify what may be trying to do that.

@TechnoMario Rancher 2.6.7 only supports k3s 1.24… https://www.suse.com/suse-rancher/support-matrix/all-supported-versions/rancher-v2-6-7/

Oops, definitely typed that backwards. I was on the latest version of Rancher, v2.7.6. I am on track with the version matrix.

Still in my last post that’s without Rancher installed at all. Just k3s 1.26.7 and cert-manager 1.12.4.

@TechnoMario I did wonder :wink:

I’m not sure, do you use Slack at all. there is a Rancher-Users channel which I can send an invite for?

I can blow the dust off my Slack account. I haven’t use it in ages. Feel free to PM me the invite, or however else you prefer to do it.

I primarily use Discord if there are equivalent groups there which are you are active in as well? I’m in a “Kubernetes @Home” channel on Discord as well, which I’ll probably end up posting this same question to.

1 Like