SSL client certificate secured repo - sometimes tlsv13 alert certificate required

Hello together!

I have a small private (non-public) repo in our company to deploy some proprietary and very special software here as RPMs. The repo system is build upon Leap 15.5 + OpenBuildServer + Apache2 (but I don’t think it is an OBS issue). This repo must be secured in terms of licenses. I have decided to use the SSL client certificate method for this.

Since a bunch of months I get very often the following message when a package has to be installed manually (with zypper/cli):

Download (curl) error for 'https://repo.example.com/example-main/15.5/noarch/rasst-8.2.1-lp155.1.1.noarch.rpm?ssl_clientcert=/var/lib/zypp/internal-client-d.crt&ssl_clientkey=/var/lib/zypp/internal-client-d.key':
Error code: Curl error 56
Error message: OpenSSL SSL_read: OpenSSL/1.1.1l-fips: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0

Abort, retry, ignore? [a/r/i/...? shows all options] (a): a

The curious part is, at the same time:

  • Check with curl -v --cert ... --key ... <url> is working
  • Invoking ‘retry’ (2nd try) is always successful
  • A own background update script started as systemd-timer is always successful too (at least I’ve found nothing about the “Error: 56” in zypper.log)

/etc/pki/trust/anchors/xzy.ca.pem (update-ca-certificates) is given. Issue is independent of Leap 15.5, 15.6 or Tumbleweed clients.

The important apache2 configuration lines look like the following example:

<VirtualHost *:443>
...
SSLCACertificateFile "/etc/apache2/ssl.crt/ca.crt"
SSLCARevocationCheck chain
SSLCARevocationFile "/etc/apache2/ssl.crl/ca.crl"
SSLVerifyClient none
...
    <Directory "/srv/www/repos/example-main">
        <If "%{Request_URI} !~ m#.*/repodata.*#">
            SSLVerifyClient require
            SSLVerifyDepth 2
            SSLRequire (( %{SSL_CLIENT_V_REMAIN} > 0 ) and (( %{SSL_CLIENT_S_DN_CN} in {"internal-client-d"}) ))
            ErrorDocument 403 "You need a valid client certificate on this site."
        </If>
    </Directory>
...

But no issues with curl or firefox. The apache2 settings should be fine in my humble opinion.

Thankful for every idea! :slight_smile:

Is the certificate linked to a valid CA for the system?

What type of certificate are you using?

You can test what is being returned by using the openssl s_client -connect <hostname:port> command .

Have a look at the output and check for the Protocol. It looks like your certificate isn’t a TLSv1.3 certificate, based on the error message received.

Here’s what my host returns (I use a Let’s Encrypt certificate):

$ openssl s_client -connect <myhost>:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.<myhost>
verify return:1
---
Certificate chain
 0 s:CN = *.<myhost>
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 21 17:24:15 2024 GMT; NotAfter: Jul 20 17:24:14 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
[public certificate]
-----END CERTIFICATE-----
subject=CN = *.<myhost>
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2701 bytes and written 384 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: EDC9B000F002991BA50BF384C8DB6C7CBD84C8C19C0CA22F5F3A19C5CC7FF021
    Session-ID-ctx: 
    Resumption PSK: 4813534C797365190569ED8E64621AFE683AF1AB5E89BBED6F6EB3BACB216A44
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 7b 22 88 dd b5 49 1e 6d-4e ee 4a ce f2 e0 df 47   {"...I.mN.J....G
    0010 - 95 da b9 fc 89 4d 79 75-f4 60 09 bc 19 91 18 84   .....Myu.`......

    Start Time: 1718392663
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed
$ 

You can see that the Protocol line at the end shows “TLSv1.3”, and you can also see that the certificate has a valid trust chain.

Thank you for your participation. :slightly_smiling_face:
Yes, it’s pretty much the same here. The https URL to the repo has a valid Let’s Encrypt certificate.

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R10
verify return:1
depth=0 CN = xxx
verify return:1
---
Certificate chain
 0 s:CN = xxx
   i:C = US, O = Let's Encrypt, CN = R10
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 13 01:17:38 2024 GMT; NotAfter: Sep 11 01:17:37 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R10
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
...

Nothing special and it is working fine.
Completely independent of this Let’s Encrypt certificate, is the client side certificate, which uses zypper to authenticate against the server (rather than username/passwords). This certificate was generated by my own internal CA.
The web server and the client knows about the own certificate chain - and it is working as long I use firefox or curl directly like:

curl -v --cert /var/lib/zypp/example-client-d.crt --key /var/lib/zypp/example-client-d.key https://repo.example.org/example-main/


* Host repo.example.org:443 was resolved.
* IPv6: (none)
* IPv4: xxx
*   Trying xxx:443...
* Connected to repo.example.org (xxx) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=repo.example.org
*  start date: Jun 13 01:17:38 2024 GMT
*  expire date: Sep 11 01:17:37 2024 GMT
*  subjectAltName: host "repo.example.org" matched cert's "repo.example.org"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /example-main/ HTTP/1.1
> Host: repo.example.org
> User-Agent: curl/8.8.0
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< Date: Fri, 14 Jun 2024 20:43:02 GMT
< Server: Apache
< Content-Length: 829
< Content-Type: text/html;charset=ISO-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /example-main</title>
 </head>
 <body>
<h1>Index of /example-main</h1>
<pre><img src="/icons/blank.gif" alt="Icon "> <a href="?C=N;O=D">Name</a>                    <a href="?C=M;O=A">Last modified</a>      <a href="?C=S;O=A">Size</a>  <a href="?C=D;O=A">Description</a><hr><img src="/icons/back.gif" alt="[PARENTDIR]"> <a href="/">Parent Directory</a>                             -   
<img src="/icons/folder.gif" alt="[DIR]"> <a href="15.5/">15.5/</a>                   2024-06-11 19:21    -   
<img src="/icons/folder.gif" alt="[DIR]"> <a href="15.6/">15.6/</a>                   2024-06-11 19:20    -   
<hr></pre>
</body></html>
* Connection #0 to host repo.example.org left intact

But when it comes to zypper (which is using curl too as far as I know) the first try to authenticate against the sever to download a rpm will fail (not always but very often), the second one is always successfully. That drives me crazy and is a bit annoying in terms of scripting (zypper -n in ... which will fail).
zypper ref has also no issues.

I would probably be inclined to capture the traffic with wireshark (if you know how to do that with certificates so you can see what’s in the request/replies, that would be best).

If you don’t have experience with that, something like httptoolkit might be useful for diagnosing further - you can use that to launch a terminal window that will let httptoolkit sniff the encrypted data to see if there’s anything weird going on beyond the certificate.

May be a good point… I will try it out.
(need coffee and a bit of time)

But it is a bit strange…

  • it works with curl
  • but not always with zypper at 1st try
  • it works always on the 2nd (re)try
  • it works always in background (non interactive update script)
  • it is happening on ~100 computer
  • and suse flavor independent

:crazy_face:

It really calls for a bug report. Both curl and zypper are using libcurl, but apparently zypper is using it differently. It may be zypper bug or it may trigger libcurl bug.

I would agree that it’s probably a bug. Getting inside the request/reply might provide some additional insight, but yeah, it sounds like a bug to me as well.

Thank you both for your feedback.
It looks like a bug to me as well but on the other side, how is it working with SLES and non public resources? I have no information but I can not believe that a password authentication is used on their own solution. Isn’t it?

Anyway, I tested some different apache configurations and SSL options this weekend (unfortunately without success). I think I have to be a bit more transparent in terms of urls/setup and logfiles. After I collected more information and make it reproducible I will open a bug report and add a link to it here.

May be it has to do with the exceptional clause ( <If "%{Request_URI} !~ m#.*/repodata.*#"> ...) in case the URL has ‘repodata’ in it. I thought this is needed because without that exception zypper ref won’t be able to checkout the repository for any news.
:thinking:
(FilesMatch constructions instead of if make no difference - just as a side node.)

I did an additional test and gave nginx a chance with a very minimal temporary configuration… and it worked without any issues. This threw me back to the beginnings of my repository server as there was only one repo existent.
It turns out the error begins with directory specific SSLVerifyClient require statements in the apache2 config. I changed my apache2 setup to follows:

<VirtualHost *:443>
...
SSLCACertificateFile "/etc/apache2/ssl.crt/ca.crt"
SSLCARevocationCheck chain
SSLCARevocationFile "/etc/apache2/ssl.crl/ca.crl"
SSLVerifyClient require
SSLVerifyDepth 2
ErrorDocument 403 "You need a valid client certificate on this site."
...

    # In dir - only check for kind of cert
    <Directory "/srv/www/repos/example-main">
        SSLRequire ( %{SSL_CLIENT_V_REMAIN} > 0 and %{SSL_CLIENT_S_DN_CN} in {"internal-client-d"} )
    </Directory>

    <Directory "/srv/www/repos/example-testing">
        SSLRequire ( %{SSL_CLIENT_V_REMAIN} > 0 and %{SSL_CLIENT_S_DN_CN} in {"test-client-b"} )
    </Directory>

...

All issues are solved and even the repodata directory can be secured now. So I am happy.

Unfortunately I am still confused why the former setup wasn’t working with zypper - which should have been… Is it a zypper issue? Is it a performance topic with apache2… :thinking:
For the moment I have no explanation to this.

It doesn’t seems wrong what I did before:

Anyway, thanks a lot for your support. :+1:

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.