Use system CA Certificates in Python3

[ken@…]

When Python is installed in LFS, we install pip3 as part of that, and various other modules are installed by pip3 in site-packages/pip/_vendor/. One of those modules in Certifi. For current 3.11.0 that is Certifi-2022-09-24 with the mozilla certs which were current when it was packaged, and those will not be updated unless, or until, pip or python3 are updated.

It is good practice for the System Administrator to control which certificates are used (in the same way that in perl modules, when building individually from source, we prefer the system certificates rather than a (probably very old) copy of Mozilla::CA).

Unfortunately, in LFS itself we cannot do that. But once make-ca and p11-kit have been installed, and make-ca has been configured, we can modify the relevant file.

I have a patch, Python-3.11.0-use_system_certs-1.patch, which I will be uploading. This is based on fedora from earlier this month.

It might probably need to be recreated at some point when the vendored copy of Certifi changes, so here I'm recording how it looks:

"""
certifi.py
~~~~~~~~~~

This module returns the installation location of cacert.pem or its contents.
"""

# Always use the system certificates
def where() -> str:
    return '/etc/pki/tls/certs/ca-bundle.crt'

def contents() -> str:
    with open(where(), encoding='utf=8') as data:
        return data.read()

[ken@…]

First cut for new page 'System Certificates for Python-3.11' in Security chapter is at ​https://www.linuxfromscratch.org/~ken/python-system-certs/

also a diff of the current status (git add -N to avoid committing, I did not bother making a branch for this).

The patch is in patches/

[ken@…]

I have been pointed to 'requests' which is also vendored by pip but includes a comment about overriding the certs with system certs. Oddly, requests says that the preferred default CA bundle comes fro mCertifi, but Certifi's web page says the (curated mozilla) certs have been extracted from the Requests project.

There is also a pip-system-certs which claims to fix pip to use system certs (I have not, on a quick look, worked out where it might look), and ​https://pip.pypa.io/en/stable/topics/https-certificates/ mentions using environment variables.

Stalled, pending further time to review this on a 3.11 system.

[ken@…]

Confirmed that changing vendored certifi to point to a non-existent file allows virtual env to still install collected packages, so that first attempt does nothing useful

[ken@…]

Interesting. The fedora patch for requests-2.28.1 does not look as if their original file matches what is in our 2.28.1 _vendored file, and also creates a file with a .new suffix.

The Arch patch for requests-2.28.1 looks like the effective part of the fedora patch (not dropping dependency on certifi) but applying to the original file.

I can confirm that the Arch patch applies to separate requests-2.28.1 with an offset (it started from an older version) but does NOT apply to what pip installed in its vendored version of 2.28.1:

root@deluxe /usr/lib/python3.11/site-packages/pip/_vendor 

#cat /sources/requests-2.28.1-arch-patch | patch -p1 --dry-run

checking file requests/certs.py
Hunk #1 FAILED at 12.
1 out of 1 hunk FAILED

This is because what pip has installed in its vendored version tests for the _PIP_STANDALONE_CERT envvar. If I point that to a non-existent certs file (note the added x on the end) with

export _PIP_STANDALONE_CERT=/etc/pki/tls/certs/ca-bundle.crtx 

the venv of charset normalizer looks for that and fails:

Successfully built charset-normalizer
ken@deluxe /tmp/charset-normalizer-2.1.1 $python3 -m venv --system-site-packages testenv
ken@deluxe /tmp/charset-normalizer-2.1.1 $source testenv/bin/activate
(testenv) ken@deluxe /tmp/charset-normalizer-2.1.1 $pip3 install pytest-cov
ERROR: Could not install packages due to an OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/pki/tls/certs/ca-bundle.crtx

So for handling pip the correct (and easy) fix is to use that environment variable. Changing the export to point to the real system certs, the tests for charset_normalizer completed with 127 passes and 1 deprecation warning.

I also noticed that the vendored version of the requests file claims to use

#!/usr/bin/env python

which does not exist on my system (no python2 on this system) but would, if present, perhaps be amusing on a system with both python2 and python3.

Weird, looking at certifi that too uses plain env python throughout.

Checking what I have installed, /usr/bin/env python is present in:

cython requests (some of pygments) recommonmark youtube_dl (not in the book)

I've run the sphinx tests which needed cython, and created llvm docs and manpages in this run, so this envocation of just python appears to work.

A 'quick' test of youtube-dl (a very short video downloaded very slowly) also seems to work. But how ?

For anything using system certifi and/or requests, patches before building them should do the job.

Memo to self: when doing initial tests for certs, point to non-existent certs to ensure something is looking where I think it will be looking.

[pierre]

Replying to ken@…:

I also noticed that the vendored version of the requests file claims to use

#!/usr/bin/env python

which does not exist on my system (no python2 on this system) but would, if present, perhaps be amusing on a system with both python2 and python3.

Guess it is obvious to everybody, but this #! is only interpreted by the shell. It means nothing in python's world. For example if you type python3 requests, the requests script will be executed by python3.

[ken@…]

Replying to pierre:

Replying to ken@…: Guess it is obvious to everybody, but this #! is only interpreted by the shell. It means nothing in python's world. For example if you type python3 requests, the requests script will be executed by python3.

After struggling with python, my brain usually switches off. For skimming it to try to get a view of what it might do, I can usually get close. But for anything deeper I'd prefer a regular language rather than counting the whitespace in indentations.

Anyway, I've proved the envvar works. Still continuing on (will do more later tests which I had failed to run, and try to fix Requests to use the same envvar and to drop Certifi). I might have to run that 'python3 requests' style of invocation to check kapidox if nothing else exercises Requests. But that will be for your branch (assuming it works).

Looks as if a short explanation at the bottom of Configuring make-ca' with a small file in /etc/profile.d can do the job.

[ken@…]

Fixed in 012af225d4d14d6c4e896ab13b55ef9138e481c7