Install Python 3.11 to /opt/python3.11 for Mozilla and/or QtWebEngine

[Bruce Dubbs]

Mozilla is just too slow to adapt for any breaking changes in dependencies (they ignore any announcements or even warnings for future breaking changes until the change is really released and everyone start to complain). And QtWebEngine 5 is not maintained. We are working around the issue with patches or seds but doing so is nasty for users.

The build time for Python 3.11 is about 40 seconds at -j8 on xry111's laptop, with the configuration:

./configure --prefix=/opt/python3.11 --with-system-expat --disable-shared --without-ensurepip

I used Python 3.11.0 and it works for Firefox. But we'd better use 3.11.1 or above because 3.11.0 will fail to build once we update to GCC 14 (per Fedora developers).

We'll not add Python 3.11 into currency, nor update it (unless building Mozilla and/or QtWebEngine is broken) even if a security issue is reported because the Python 3.11 build will be only intended for building Mozilla and/or QtWebEngine, and we consider the Mozilla & QtWebEngine tarballs "trusted". We should make our intention clearly in the upcoming Python 3.11 page.

Checklist

• Add Python 3.11
• QtWebEngine:
• Firefox:
• SpiderMonkey: I tend to keep the current work around for SpiderMonkey because it's simple.
• Thunderbird:
• Seamonkey:

[Douglas R. Reno]

I'm already going to be doing something similar for Thunderbird and Seamonkey, both of which have large amounts of seds that make them very difficult to follow and read. We should do the same to Firefox as well

[ken@…]

Sounds as if this will cause problems for anybody updating old systems (e.g. those with python-3.11 in particular).

The current large number of fixes show how much is broken nowadays with what are supposed to be stable releases.

[Douglas R. Reno]

Replying to ken@…:

Sounds as if this will cause problems for anybody updating old systems (e.g. those with python-3.11 in particular).

The current large number of fixes show how much is broken nowadays with what are supposed to be stable releases.

I'll be adding some text to state that you should not apply the patch if you aren't on Python 3.12/ICU 74 so that we can maintain information for security advisories (and testing against Python 3.11 and 3.12)

I'm still of the opinion that Python's minor releases are really Major ones with plenty of compatibility changes. Since Python 3.10 I've felt that 3.11 should've been 4, and now 12 should have been 4. This cycle has been particularly nasty with libxml2, python 3.12, ICU 74, etc. and the amount of fixes required has been hectic (we're almost at 1,000 commits since 12.0!)

[Douglas R. Reno]

Added a checklist to the description of the ticket.

[Douglas R. Reno]

Seamonkey done at 4b3d2071e1a3ae32431948861c8c719cfd61f552

[Xi Ruoyao]

Frankly I don't see an advantage of patch instead of sed.

[Xi Ruoyao]

Currently bdubbs and I think installing Python 3.11 into /opt/python3.11 would be easier.

[Rahul Chandra]

I created a branch to do this in (python311), 1 test failed for me test_curses, but this might have something to do with me doing this over SSH and using a non-standard terminal emulator (kitty). NCurses dialogs still run fine though. I added Python 3.11 and hopefully my explanations are up to par. If someone could do a test build and read through it that would be great.

Firefox and Thunderbird both build great if you use python3.11 before mach, but I only modified the firefox xml as it's 02:00 right now

If all is well I will merge the branch and add it to the changelog whenever it is checked over.

(sorry forgot to add the rahul/ before the branch)

[Xi Ruoyao]

Replying to Rahul Chandra:

I created a branch to do this in (python311), 1 test failed for me test_curses, but this might have something to do with me doing this over SSH and using a non-standard terminal emulator (kitty).

Try LC_ALL=en_US.UTF-8 as what we are doing for 3.12. It happens for me when all locale environment variables are not set.

But anyway we should just not mention Python 3.11 test suite in the book at all, IMO. We should make it clear that "if Python 3.11 builds Mozilla and QtWebEngine fine, then it's fine, other uses are completely unsupported".

[Xi Ruoyao]

And we should not create a symlink in /usr/bin because other packages shipping a Python binding may try to find all Python installations in $PATH and build the Python binding multiple times (once for each installation), and we don't like it to happen (it will waste CPU time and potentially cause a build failure).

[Douglas R. Reno]

Yeah I don't think we need the symlink to /usr either, that might just cause additional problems.

Reading through this page a bit, other than some typos that we should clean up (extra parenthesis at the end of the description and in the list of installed files), everything looks good in terms of the page's text to me.

We might want to remove the description for --enable-optimizations from this page, and as Xi suggested we might want to disable the test suite too.

For the dependencies list, I think we might just want to keep the sqlite section, and drop the rest of it.

Once you're done with this I'm good with you merging it, I can take care of the rest (reverting my patch to Seamonkey, and removing stuff from other pages) afterwards since I'm going to need to update two of those packages anyway. Thank you for doing it for Firefox :)

[Rahul Chandra]

I was contemplating the test suite but since most of the tests passed anyways I left it in. I do agree though that the best course of action is probably to remote it entirely.

Removing --enable-optimizations should be fine, no one should need it anyways if it's just for Mozilla apps and qtwebengine.

Thanks for the insight into the symlink Xi, I was mostly trying to keep versioned commands as I've seen a couple times before on some Windows and older linux hosts but I can see how this could cause a problem, I'll comment it out.

I'll fix the typos, thanks for finding them Doug.

[Rahul Chandra]

Seems like Bruce just did that, thanks.

[Rahul Chandra]

If another person could test the build then I think we'll be good to merge.

[Rahul Chandra]

Merged Python-3.11 and Trunk @ 6dd695f77fd75e59f749904b068b4b107dacd19e

Changelog entries added @ ddf9c814791a91a6502c1de4762095bac78831a9

Related Commits

105d06c49aac026ef8d3895304fb57f0585ddbd0 - Original Addition
5bd4e666b233b4ab86019e08a8a6e1750762ae40 - Simplify python3.11 page and firefox page.
01c1ac6affda439597c4f7e40380dece1183dd0f - Typos and small corrections.
6dd695f77fd75e59f749904b068b4b107dacd19e - Typo

Thanks for doing the fixes, Bruce.

[Douglas R. Reno]

Seamonkey is good to go at 6848f3f59b098edf40203af2864bec0b504d159e

[Douglas R. Reno]

Thunderbird is good to go at b0f39cc3f12a55a7c28efe47e0c12a2d3ab7b2bf

I believe that just leaves QtWebEngine now

[Douglas R. Reno]

Update on QtWebEngine:

I have created an updated version of the build_fixes patch. It retains what is already there and I've added hunks to the end of it with descriptions, just adding onto what Ken has in there.

Here's what I've added into it:

• The fix for the shipped ffmpeg build failure
• The fixes for Python 3.11's changes
• The changes for ICU-74, but with a catch - updated it to work with systems with previous versions of ICU as well, bringing in a patch from Arch (​https://gitlab.archlinux.org/archlinux/packaging/packages/qt5-webengine/-/blob/main/qt5-webengine-icu-74.patch?ref_type=heads)
• The NINJAJOBS change
• Chunks from a patch from ArchLinux 32 to prevent instantaneous crashes on 32-bit systems. Some of the patch was applied upstream, but other parts were not. That patch originally came from: ​https://git.archlinux32.org/packages/tree/extra/qt5-webengine/qtwebengine-everywhere-src-5.15.2-glibc-2.33.patch

I will remove the libxml2 change since we don't build libxml2 with ICU support by default (now you have to pass --with-icu to it), and this has caused confusion with other editors and users recently.

This will leave us with the pulseaudio sed, the two patches, the 'mkdir', the sed for qmake.conf to make QtWebEngine match the Qt version, and the find command.

This will allow us to maintain compatibility with previous versions of BLFS (especially due to the ICU change) while also cleaning up the instructions on the page significantly (dropping at least six instruction blocks).

When the page is updated, I will also have it use the copy of Python 3.11 from /opt. To do this, we just need to do "PATH=/opt/python3.11/bin:$PATH qmake .. [...]", it'll hardcode that PATH into the Makefiles that it generates.

I'm currently waiting on testing for this to complete on a different machine, this may take me another day or so to get in.

[Joe Locash]

Replying to Douglas R. Reno:

Seamonkey is good to go at 6848f3f59b098edf40203af2864bec0b504d159e

I'm seeing 2 issues with the change for seamonkey:

$ export PYTHON311=/opt/python3.11/bin/python3.11 && $PYTHON311 AUTOCONF=true ./mach build
/opt/python3.11/bin/python3.11: can't open file '/tmp/build/seamonkey-2.53.18/AUTOCONF=true': [Errno 2] No such file or directory

Moved settting AUTOCONF to true before $PYTHON311:

$ export PYTHON311=/opt/python3.11/bin/python3.11 && AUTOCONF=true $PYTHON311 ./mach build
 <snip>
 0:00.46 Traceback (most recent call last):
 0:00.46   File "/tmp/build/seamonkey-2.53.18/configure.py", line 26, in <module>
 0:00.46     from mozbuild.configure import (
 0:00.46   File "/tmp/build/seamonkey-2.53.18/python/mozbuild/mozbuild/configure/__init__.py", line 13, in <module>
 0:00.46     from six.moves import builtins as __builtin__
 0:00.46 ModuleNotFoundError: No module named 'six.moves'
 0:00.46 *** Fix above errors and then restart with\
 0:00.46                "./mach build"
 0:00.46 make: *** [client.mk:114: configure] Error 1

The issue is that with seamonkey mach is a shell sript and not a python script and has a function to find a suitable python. Setting PATH to /opt/python3.11/bin:$PATH and running AUTOCONF=true ./mach build works.

[Xi Ruoyao]

I'll take a look at Seamonkey.

[Xi Ruoyao]

Replying to Xi Ruoyao:

I'll take a look at Seamonkey.

Done at r12.0-1102-gf94402b44d.

[Douglas R. Reno]

QtWebEngine fixed at c0f71f24ae11dbc959d599105b2b25deb3a6078b