apache 2.4.29-1ubuntu4.12 authentication with client certificate broken

Bug #1865900 reported by Riho Kalbus
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Release Notes for Ubuntu
Confirmed
Undecided
Unassigned
apache2 (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
New
Undecided
Marc Deslauriers
Jammy
Fix Released
Undecided
Unassigned
python-urllib3 (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
New
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned
requests (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
New
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned

Bug Description

Ubuntu 18.04.4 LTS, after update from apache 2.4.29-1ubuntu4.11 to apache 2.4.29-1ubuntu4.12 authentication with client certificate stopped working. No certificate is requested from client browser and apahce log has error:

[Tue Mar 03 16:03:34.964389 2020] [ssl:debug] [pid 12384:tid 139853354215168] ssl_engine_kernel.c(2217): AH02041: Protocol: TLSv1.3, Cipher: TLS_AES_256_GCM_SHA384 (256/256 bits)
[Tue Mar 03 16:03:36.499614 2020] [ssl:debug] [pid 12383:tid 139853481088768] ssl_engine_io.c(1106): AH02001: Connection closed to child 1 with standard shutdown
[Tue Mar 03 16:03:37.714744 2020] [ssl:debug] [pid 12384:tid 139853481088768] ssl_engine_kernel.c(383): AH02034: Initial (No.1) HTTPS request received for child 65 (server devel.liisi.ee:443), referer: https://devel.liisi.ee:8950/accounts/login/
[Tue Mar 03 16:03:37.714941 2020] [ssl:error] [pid 12384:tid 139853481088768] AH: verify client post handshake, referer: https://devel.liisi.ee:8950/accounts/login/

A temporary workaround is to disable the whole TLSv1.3 protocol in the vhost configuration.
---
ProblemType: Bug
Apache2ConfdDirListing: False
Apache2Modules:
 AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.20.4.138. Set the 'ServerName' directive globally to suppress this message
 httpd (pid 13567) already running
ApportVersion: 2.20.9-0ubuntu7.11
Architecture: amd64
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2010-05-21 (3576 days ago)
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
Package: apache2 2.4.29-1ubuntu4.12
PackageArchitecture: amd64
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.15.0-88.88-generic 4.15.18
Tags: bionic
Uname: Linux 4.15.0-88-generic x86_64
UpgradeStatus: Upgraded to bionic on 2018-10-16 (505 days ago)
UserGroups:

_MarkForUpload: True
error.log:
 [Thu Mar 05 06:25:05.942445 2020] [ssl:warn] [pid 13567:tid 140475868056512] AH01909: klient.liisi.ee:443:0 server certificate does NOT include an ID which matches the server name
 [Thu Mar 05 06:25:05.945212 2020] [mpm_worker:notice] [pid 13567:tid 140475868056512] AH00292: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
 [Thu Mar 05 06:25:05.945234 2020] [core:notice] [pid 13567:tid 140475868056512] AH00094: Command line: '/usr/sbin/apache2'
modified.conffile..etc.apache2.mods-available.reqtimeout.conf: [modified]
modified.conffile..etc.apache2.ports.conf: [modified]
modified.conffile..etc.apache2.sites-available.000-default.conf: [modified]
mtime.conffile..etc.apache2.mods-available.reqtimeout.conf: 2020-03-03T16:33:43.294515
mtime.conffile..etc.apache2.ports.conf: 2014-10-22T16:31:31.217125
mtime.conffile..etc.apache2.sites-available.000-default.conf: 2019-10-16T13:29:08.811073

Revision history for this message
Bryce Harrington (bryce) wrote :

Hi Riho,

Thank you for taking the time to report this bug. I've mentioned this on bug LP: #1845263 as a possible regression related to the
2.4.29-1ubuntu4.12 update that backported the TLSv1.3 support to bionic. That update indicated some expectation that certain environments might be adversely affected when the new protocol is added, so it would be helpful to understand in more detail about your particular setup. That may help identify what went wrong precisely in this case.

Please execute the following command, as it will automatically gather debugging information, in a terminal:

  apport-collect 1865900

Alternatively, if you want to manually attach things (e.g. so you can remove any sensitive information), the files this collects includes:

/etc/apache2/apache2.conf
/etc/apache2/sites-enabled/*
/etc/apache2/conf.d
/var/log/apache2/error.log
`/usr/sbin/apachectl -D DUMP_MODULES`

Obviously the piece that'll need more examination is the client certificate configuration, so if there are other config files or logs of relevance to that you're aware of, those details could be useful as well.

Changed in apache2 (Ubuntu):
status: New → Incomplete
tags: added: regression-update
Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

This is likely a dupe of bug 1834671...

Revision history for this message
Ken Dreyer (Red Hat) (kdreyer-redhat) wrote :

I can confirm this as well. I have a CI job that uses python-requests to contact Apache with SSL x590 client authentication. This job passed with apache 2.4.29-1ubuntu4.11 and it fails with apache 2.4.29-1ubuntu4.12.

Passing: https://travis-ci.org/ktdreyer/koji-ansible/builds/655568368
Failing: https://travis-ci.org/ktdreyer/koji-ansible/builds/657818117

Revision history for this message
Riho Kalbus (rihokalbus) wrote : Dependencies.txt

apport information

tags: added: apport-collected bionic
description: updated
Revision history for this message
Riho Kalbus (rihokalbus) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Riho Kalbus (rihokalbus) wrote : linkweb.conf.txt

apport information

Revision history for this message
Riho Kalbus (rihokalbus) wrote : tangoweb.conf.txt

apport information

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Most clients don't support post handshake authentication, hence can't use client side certificates with TLSv1.3.

In environments where client side certificates are used, TLSv1.3 has to be disabled in the Apache configuration until browsers and other clients support post handshake authentication.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Bug #1834671 also has this possible workaround:
"""
Another workaround is to move the SSLVerifyClient config to the vhost level. It it applied to the whole vhost, and there are no exceptions in specific blocks, then a re-negotiation isn't triggered and the problem doesn't happen.
"""

i.e., it's the change in ssl configuration inside a vhost that triggers the PHA, from my understanding.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

> I can confirm this as well. I have a CI job that uses python-requests to contact
> Apache with SSL x590 client authentication. This job passed with
> apache 2.4.29-1ubuntu4.11 and it fails with apache 2.4.29-1ubuntu4.12.

Is this a case where python or python-requests could be updated to handle PHA? Just like firefox was updated in bionic to handle PHA.

Robie Basak (racb)
tags: added: bionic-openssl-1.1
Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Firefox in bionic added an option to handle PHA, but it's disabled by default because it conflicts with http2.

I'm not aware if there's an equivalent "fix" for python-requests.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :
Revision history for this message
Ken Dreyer (Red Hat) (kdreyer-redhat) wrote :

From https://bugzilla.redhat.com/show_bug.cgi?id=1761403:
"The fix is available in urllib3 1.25.4. The fix requires Python 3.7.4 or newer with fix https://bugs.python.org/issue37428 ."

I upgraded urllib3 and requests to the Disco versions:

Unpacking python3-urllib3 (1.24.1-1ubuntu0.1) over (1.22-1ubuntu0.18.04.1) ...
Unpacking python3-requests (2.21.0-1) over (2.18.4-2ubuntu0.1) ...

I still get "HTTPError: 403 Client Error: Forbidden for url: https://localhost/kojihub/ssllogin" in my Bionic VM when I try those versions.

Revision history for this message
Ken Dreyer (Red Hat) (kdreyer-redhat) wrote :

With Bionic's apache 2.4.29-1ubuntu4.12:

"SSLProtocol TLSv1.3 TLSv1.2" - works
"SSLProtocol TLSv1.3 +TLSv1.2" - does not work
"SSLProtocol all -SSLv3" - does not work

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

There also seems to be https://bugs.python.org/issue37440

In any case, I think this bug is not about apache, other than it's a change introduced there that made tls v1.3 available for clients to use. The clients need to be updated now.

Revision history for this message
Ken Dreyer (Red Hat) (kdreyer-redhat) wrote :

"SSLProtocol all -SSLv3" is in the default /etc/apache2/mods-enabled/ssl.conf. Why does the behavior change when I set "SSLProtocol TLSv1.3 TLSv1.2"?

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I guess depends where you change it. If you do it on a specific location or directory, it's my understanding that this is what triggers PHA.

Revision history for this message
Riho Kalbus (rihokalbus) wrote :

> With Bionic's apache 2.4.29-1ubuntu4.12:
>
>"SSLProtocol TLSv1.3 TLSv1.2" - works

Tried with Firefox 73.0.1 - works, but connection is established using TLS1.2 protocol
when "SSLProtocol TLSv1.3 TLSv1.2 TLSv1.1" is specified, then TLS1.1 is used.

Robie Basak (racb)
tags: added: server-next
Revision history for this message
Robie Basak (racb) wrote :

I think we have enough information on this report now; all that remains is some difficult decision making on what, if anything, we can do about it. Depending on the answer, we might need to assign this bug to a different package, etc.

Changed in apache2 (Ubuntu):
status: Incomplete → New
Revision history for this message
Christian Ehrhardt (paelzer) wrote :

FYI there is a similar bug 1867223 which has a patch suggested at least for some cases of this.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

I have uploaded an apache2 package to the security team PPA for testing here: