Fixes for packages affected by Python-3.12

[Bruce Dubbs]

Transferred from #18664

• QtWebEngine (done at c51bc3f4d9)
• Node.js (done at 73ac81bdd426bee0adf57df76967e1578ddbbb5a)
• Spidermonkey (done at e3b21095897b39c898446a794e7f5c99aa451b3e)
• itstool (thousands of "invalid escape sequence" warnings on every invocation) - done at 87463485b21e78e544e6dbc4ef55a0868b6768e3
• asciidoc (generates thousands of errors on every invocation)
• Firefox/Thunderbird (Done at r12.0-532-ge86ded4104/r12.0-535-g1dcb11c64e)
• Seamonkey (r12.0-590-gf56feddcab)
• Samba (f-strings changes) (not affected)
• gobject-introspection (fixed at 3d349f41148cbdc8e41563d26b8488ae3f2e9f8b)
• Qemu (ignored, only 8 warning messages "harmless" for now)
• Wireshark (fixed at f17571081b)

[Joe Locash]

Upstream fix for qemu: ​https://gitlab.com/qemu-project/qemu/-/commit/e6d8e5e6e366ab4c9ed7d8ed1572f98c6ad6a38e

[Joe Locash]

For itstool and asciidoc see https://wiki.linuxfromscratch.org/blfs/ticket/18664#comment:14

[Bruce Dubbs]

Replying to Joe Locash:

Upstream fix for qemu: ​https://gitlab.com/qemu-project/qemu/-/commit/e6d8e5e6e366ab4c9ed7d8ed1572f98c6ad6a38e

Thanks Joe. Not sure yet, but it looks like a sed to me.

[Xi Ruoyao]

I've not seen the issue with samba. Is there a link to a bug report or patch etc.?

[Xi Ruoyao]

Firefox done at r12.0-532-ge86ded4104.

[Xi Ruoyao]

Thunderbird done at r12.0-535-g1dcb11c64e.

[Xi Ruoyao]

There are only 8 warning messages building QEMU so IMO we'll just ignore them.

[Xi Ruoyao]

asciidoc fixed at r12.0-559-g8ef3f050af.

[pierre]

Got this error with qtwebengine:

[103/23995] ...
FAILED: gen/chrome/browser/resource_coordinator/lifecycle_unit_state.mojom-modul
e ...
ModuleNotFoundError: No module named 'imp'

This module has been removed in python3.12, and python devs tell to use "importlib" instead. Problem is this is not a one-to-one replacement. And upstream fixes seem either absent or partially done (see ​https://groups.google.com/a/chromium.org/g/chromium-reviews/c/sVH9Y0VgNpw)

[Xi Ruoyao]

qtwebengine:

• Comment out import imp in src/3rdparty/chromium/mojo/public/tools/mojom/mojom/fileutil.py and src/3rdparty/chromium/mojo/public/tools/mojom/mojom/parse/lexer.py
• In src/3rdparty/chromium/components/resources/protobufs/binary_proto_generator.py:
  • Replace import imp with import importlib.util
  • Rewrite return imp.load_source(fullname, filepath) as:

        filepath = self._fullname_to_filepath(fullname)
        spec = importlib.util.spec_from_file_location(fullname, filepath)
        mod = importlib.util.module_from_spec(spec);
        spec.loader.exec_module(mod)
        return mod
    

• Remove src/3rdparty/chromium/tools/grit/third_party/six
• Comment out from six.moves import range in src/3rdparty/chromium/third_party/protobuf/python/google/protobuf/internal/python_message.py

[pierre]

Seamonkey is broken, and the changes for firefox are not enough. Actually, third_party/python/setuptools does not exist, so that it is unclear what change is needed. FIrst, there are a lot of warnings SyntaxWarning: invalid escape sequence in various files, then the first error is:

Command /sources/seamonkey/s...init_py3/bin/python3 -m pip config list had error code 1

And indeed obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/lib/python3.12/ does not contain a pip module.

Then the build system does

Installing setuptools, pip, wheel...

but this result in

File "/sources/seamonkey/seamonkey-2.53.17.1/third_party/python/virtualenv/virtualenv_support/pip-19.3.1-py2.py3-none-any.whl/pip/_internal/cli/cmdoptions.py", line 19, in <module>

ModuleNotFoundError: No module named 'distutils'

There are two alternatives here: have the virtualenv copy system pip, or patch the vendored pip-19.3.1-py2.py3-none-any.whl. Problem is: this is a .zip file.

[Xi Ruoyao]

Replying to pierre:

Seamonkey is broken, and the changes for firefox are not enough. Actually, third_party/python/setuptools does not exist, so that it is unclear what change is needed. FIrst, there are a lot of warnings SyntaxWarning: invalid escape sequence in various files, then the first error is:

Command /sources/seamonkey/s...init_py3/bin/python3 -m pip config list had error code 1

And indeed obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/lib/python3.12/ does not contain a pip module.

Then the build system does

Installing setuptools, pip, wheel...

but this result in

File "/sources/seamonkey/seamonkey-2.53.17.1/third_party/python/virtualenv/virtualenv_support/pip-19.3.1-py2.py3-none-any.whl/pip/_internal/cli/cmdoptions.py", line 19, in <module>

ModuleNotFoundError: No module named 'distutils'

There are two alternatives here: have the virtualenv copy system pip, or patch the vendored pip-19.3.1-py2.py3-none-any.whl. Problem is: this is a .zip file.

Is it a part of the ensurepip module? If true we can try to overwrite the ensurepip module with /usr/lib/python3.12/ensurepip.

[pierre]

Replying to Xi Ruoyao:

Replying to pierre:

File "/sources/seamonkey/seamonkey-2.53.17.1/third_party/python/virtualenv/virtualenv_support/pip-19.3.1-py2.py3-none-any.whl/pip/_internal/cli/cmdoptions.py", line 19, in <module>

ModuleNotFoundError: No module named 'distutils'

There are two alternatives here: have the virtualenv copy system pip, or patch the vendored pip-19.3.1-py2.py3-none-any.whl. Problem is: this is a .zip file.

Is it a part of the ensurepip module? If true we can try to overwrite the ensurepip module with /usr/lib/python3.12/ensurepip.

There is no ensurepip module either. I think that if pip is not found in the venv, then they use the (old) pip-19.3.1-py2.py3-none-any.whl. The best would be to have the venv creator copy pip from host, but I have not been able to find where in "mach", the venv is created (something like "manager.build" in python, but I have not found the cource (yet))

[Xi Ruoyao]

SeaMonkey:

(for i in $(find -name six.py); do
   ln -sfv /usr/lib/python3.12/site-packages/six.py $i
   [ $? = 0 ] || exit $?
 done) &&

sed '/ConfigParser/s/Safe//'                          \
    -i testing/mozbase/mozprofile/mozprofile/prefs.py &&

sed 's/distutils/setuptools._&/'           \
    -i python/mozbuild/mozbuild/nodeutil.py &&

sed -e '/^import/s/imp$/importlib.util/'             \
    -e 's/imp.new_module/__import__/'                \
    -e "s/imp.load_source\(.*\)/spec =               \
        importlib.util.spec_from_file_location\1;    \
        mod = importlib.util.module_from_spec(spec); \
        spec.loader.exec_module(mod)/" -i python/mach/mach/main.py &&

sed 's/readfp/read_file/' -i ipc/ipdl/ipdl.py &&

ln -sv /usr/lib/python3.12/ensurepip/_bundled/pip-23.2.1-py3-none-any.whl \
       third_party/python/virtualenv/virtualenv_support &&

# additional download
tar xf ../setuptools-68.2.2.tar.gz &&
pip3 wheel --no-build-isolation --no-deps --no-cache-dir \
           $PWD/setuptools-68.2.2 \
           -w third_party/python/virtualenv/virtualenv_support

Then rewrite python/mozbuild/mozbuild/action/file_generate.py and netwerk/dns/prepare_tlds.py to use importlib.util instead of imp.load_module.

For example, in file_generate.py:

        spec = importlib.util.spec_from_file_location('script', script)
        module = importlib.util.module_from_spec(spec)
        spec.loader.exec_module(module)
        #module = imp.load_module('script', fh, script,
        #                         ('.py', 'r', imp.PY_SOURCE))

[Douglas R. Reno]

Replying to Xi Ruoyao:

I've not seen the issue with samba. Is there a link to a bug report or patch etc.?

Samba appears to be fine, the reports were for a previous version (4.18.x).

[Douglas R. Reno]

The itstool issue can be fixed by:

sed -i 's/re.compile(/re.compile(r/' itstool.in
sed -i 's/re.sub(/re.sub(r/' itstool.in

I'll put that into the book later today.

[Douglas R. Reno]

itstool has been adapted at 87463485b21e78e544e6dbc4ef55a0868b6768e3

[pierre]

Wireshark cmake step exits with:

CMake Error at cmake/modules/LocatePythonModule.cmake:47 (message):
  Could NOT find python module asn2wrs
Call Stack (most recent call first):
  cmake/modules/UseAsn2Wrs.cmake:11 (locate_python_module)
  epan/dissectors/asn1/lix2/CMakeLists.txt:34 (ASN2WRS)

The problem is that the LocatePythonModule.cmake file uses:

execute_process(COMMAND "${PYTHON_EXECUTABLE}" "-c"
                        "import imp; print(imp.find_module('${module}')[1])"
                        RESULT_VARIABLE _${module}_status
                        OUTPUT_VARIABLE _${module}_location
                        ERROR_QUIET OUTPUT_STRIP_TRAILING_WHITESPACE)

So this should be changed to use importlib, but actually upstream has a much more drastic fix (removing completely the LocatePythonModule.cmake file and references to it): ​https://gitlab.com/wireshark/wireshark/-/merge_requests/12808/diffs. Will try that one.

[Bruce Dubbs]

All packages appear to be fixed. Closing.