python3-3.12.0 (Wait for LFS)

[Bruce Dubbs]

New minor version.

[Xi Ruoyao]

When I configure it, I get:

configure: WARNING: unrecognized options: --with-system-ffi

[Xi Ruoyao]

This option is removed in ​https://github.com/python/cpython/pull/100544. We can drop it from LFS and BLFS.

[Xi Ruoyao]

Summary -- Release highlights

Python 3.12 is the latest stable release of the Python programming language, with a mix of changes to the language and the standard library. The library changes focus on cleaning up deprecated APIs, usability, and correctness. Of note, the distutils package has been removed from the standard library. Filesystem support in os and pathlib has seen a number of improvements, and several modules have better performance.

The language changes focus on usability, as f-strings have had many limitations removed and 'Did you mean ...' suggestions continue to improve. The new type parameter syntax and type statement improve ergonomics for using generic types and type aliases with static type checkers.

This article doesn't attempt to provide a complete specification of all new features, but instead gives a convenient overview. For full details, you should refer to the documentation, such as the Library Reference and Language Reference. If you want to understand the complete implementation and design rationale for a change, refer to the PEP for a particular new feature; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented.

The complete release note is at ​https://github.com/python/cpython/blob/main/Doc/whatsnew/3.12.rst.

[Xi Ruoyao]

We can drop the test_distutils test failure as distutils is gone.

[Xi Ruoyao]

It seems ensurepip does not install setuptools anymore (pip is still installed). We need to install setuptools as a separate package.

[Xi Ruoyao]

Some gobject-introspection tests fail due to lack of distutils. g-i can still be installed but I'm not sure if there will be some runtime issue.

[Xi Ruoyao]

I've rebuilt the following Python modules (some are not in BLFS) with Python-3.12 (pip3 list output):

Package            Version
------------------ -------
asciidoc           10.2.0
Brotli             1.1.0
dbus-python        1.3.2
docutils           0.20.1
editables          0.3
flit_core          3.9.0
fonttools          4.43.0
gi-docgen          2023.1
gitdb              4.0.10
GitPython          3.1.34
gpg                1.22.0
hatch-vcs          0.3.0
hatchling          1.12.2
iniconfig          2.0.0
Jinja2             3.1.2
Mako               1.2.4
Markdown           3.4.1
MarkupSafe         2.1.3
meson              1.2.2
meson-python       0.12.0
packaging          23.2
pathspec           0.10.3
pip                23.2.1
pluggy             1.0.0
polib              1.2.0
pwquality          1.4.5
py                 1.11.0
pycairo            1.25.0
Pygments           2.15.1
PyGObject          3.46.0
pyproject-metadata 0.7.0
pytest             7.4.2
python-dbusmock    0.29.1
setuptools         68.2.2
setuptools-scm     7.1.0
six                1.16.0
smartypants        2.0.1
smmap              5.0.0
typing_extensions  4.4.0
typogrify          2.0.7
wheel              0.41.2

[Xi Ruoyao]

Note that distutils is not removed: it's a part of separated setuptools package now. We are adding setuptools into LFS (meson needs it) anyway.

The g-i test failure seems unrelated to distutils. There seems a namespace conflict in the testing environment due to the usage of PYTHONPATH, but I'm 99% sure it won't be a problem when the package is installed.

[Xi Ruoyao]

Replying to Xi Ruoyao:

Note that distutils is not removed: it's a part of separated setuptools package now. We are adding setuptools into LFS (meson needs it) anyway.

The g-i test failure seems unrelated to distutils. There seems a namespace conflict in the testing environment due to the usage of PYTHONPATH, but I'm 99% sure it won't be a problem when the package is installed.

It can be worked around with sed "/PYTHONPATH/a'/usr/lib/python3.12'," -i tests/warn/meson.build.

[Xi Ruoyao]

Mozilla packages (Firefox, Thunderbird, SpiderMonkey, and maybe SeaMonkey) needs adjustment: ​https://bugzilla.mozilla.org/show_bug.cgi?id=1857492

[Xi Ruoyao]

Replying to Xi Ruoyao:

Mozilla packages (Firefox, Thunderbird, SpiderMonkey, and maybe SeaMonkey) needs adjustment: ​https://bugzilla.mozilla.org/show_bug.cgi?id=1857492

For SpiderMonkey, for i in $(find -name six.py); do cp -v /usr/lib/python3.12/site-packages/six.py $i; done works. I'll test Firefox and Thunderbird too.

[Xi Ruoyao]

The work around for Firefox:

for i in $(find -name six.py); do
  cp -v /usr/lib/python3.12/site-packages/six.py $i;
done

cp /usr/lib/python3.12/site-packages/{setuptools,distutils-precedence.pth,pkg_resources,_distutils_hack} -R third_party/python/setuptools

sed -e '/^import/s/imp$/importlib.util/' \
    -e 's/imp.new_module/__import__/' \
    -e "s/imp.load_source\(.*\)/spec = \
        importlib.util.spec_from_file_location\1; \
        mod = importlib.util.module_from_spec(spec); \
        spec.loader.exec_module(mod)/" -i python/mach/mach/main.py

sed '/ConfigParser/s/Safe//' -i testing/mozbase/mozprofile/mozprofile/prefs.py
sed 's/distutils/setuptools._&/' -i python/mozbuild/mozbuild/nodeutil.py
sed '/import imp/d' -i netwerk/dns/prepare_tlds.py

[Xi Ruoyao]

Replying to Xi Ruoyao:

The work around for Firefox:

for i in $(find -name six.py); do
  cp -v /usr/lib/python3.12/site-packages/six.py $i;
done

cp /usr/lib/python3.12/site-packages/{setuptools,distutils-precedence.pth,pkg_resources,_distutils_hack} -R third_party/python/setuptools

sed -e '/^import/s/imp$/importlib.util/' \
    -e 's/imp.new_module/__import__/' \
    -e "s/imp.load_source\(.*\)/spec = \
        importlib.util.spec_from_file_location\1; \
        mod = importlib.util.module_from_spec(spec); \
        spec.loader.exec_module(mod)/" -i python/mach/mach/main.py

sed '/ConfigParser/s/Safe//' -i testing/mozbase/mozprofile/mozprofile/prefs.py
sed 's/distutils/setuptools._&/' -i python/mozbuild/mozbuild/nodeutil.py
sed '/import imp/d' -i netwerk/dns/prepare_tlds.py

Same for Thunderbird.

[Joe Locash]

Node.js needs a fix to configure to support python-3.12, the same was done when python-3.11 first came out. BTW, the node LTS release is now 18.18.0 and was released last month.

Another thing I noticed is a LOT of "SyntaxWarning: invalid escape sequence" python messages coming from the build and installed packages. In my build I had over 25,000 of these warnings with most coming from itstool.

An upstream merge request to fix this in itstool can be found @ ​https://github.com/itstool/itstool/pull/51

asciidoc can also generate a large amount of errors and there's a patch that fedora has for it @ ​https://src.fedoraproject.org/rpms/asciidoc/blob/rawhide/f/asciidoc-table-separator.patch

[Xi Ruoyao]

Replying to Joe Locash:

Node.js needs a fix to configure to support python-3.12, the same was done when python-3.11 first came out. BTW, the node LTS release is now 18.18.0 and was released last month.

Another thing I noticed is a LOT of "SyntaxWarning: invalid escape sequence" python messages coming from the build and installed packages. In my build I had over 25,000 of these warnings with most coming from itstool.

An upstream merge request to fix this in itstool can be found @ ​https://github.com/itstool/itstool/pull/51

asciidoc can also generate a large amount of errors and there's a patch that fedora has for it @ ​https://src.fedoraproject.org/rpms/asciidoc/blob/rawhide/f/asciidoc-table-separator.patch

Also available from upstream: ​https://github.com/asciidoc-py/asciidoc-py/commit/cf72db316e3281249c6f69c5591a0092b231b53d

Is this an "error" or simply "warning"? We generally don't fix warnings.

[Joe Locash]

Replying to Xi Ruoyao:

Is this an "error" or simply "warning"? We generally don't fix warnings.

I think it's just a warning. I don't know enough about the internals of python to know for sure. Just wanted to bring attention to it because of the number of them that I saw and it can cause failures in tests.

[Douglas R. Reno]

Replying to Xi Ruoyao:

Replying to Joe Locash:

Node.js needs a fix to configure to support python-3.12, the same was done when python-3.11 first came out. BTW, the node LTS release is now 18.18.0 and was released last month.

Another thing I noticed is a LOT of "SyntaxWarning: invalid escape sequence" python messages coming from the build and installed packages. In my build I had over 25,000 of these warnings with most coming from itstool.

An upstream merge request to fix this in itstool can be found @ ​https://github.com/itstool/itstool/pull/51

asciidoc can also generate a large amount of errors and there's a patch that fedora has for it @ ​https://src.fedoraproject.org/rpms/asciidoc/blob/rawhide/f/asciidoc-table-separator.patch

Also available from upstream: ​https://github.com/asciidoc-py/asciidoc-py/commit/cf72db316e3281249c6f69c5591a0092b231b53d

Is this an "error" or simply "warning"? We generally don't fix warnings.

This is definitely a case where we need to fix it. 25,000+ warnings would drown out any meaningful output in the event that we need to troubleshoot another problem.

[Joe Locash]

Replying to Douglas R. Reno:

Replying to Xi Ruoyao:

Replying to Joe Locash:

Node.js needs a fix to configure to support python-3.12, the same was done when python-3.11 first came out. BTW, the node LTS release is now 18.18.0 and was released last month.

Another thing I noticed is a LOT of "SyntaxWarning: invalid escape sequence" python messages coming from the build and installed packages. In my build I had over 25,000 of these warnings with most coming from itstool.

An upstream merge request to fix this in itstool can be found @ ​https://github.com/itstool/itstool/pull/51

asciidoc can also generate a large amount of errors and there's a patch that fedora has for it @ ​https://src.fedoraproject.org/rpms/asciidoc/blob/rawhide/f/asciidoc-table-separator.patch

Also available from upstream: ​https://github.com/asciidoc-py/asciidoc-py/commit/cf72db316e3281249c6f69c5591a0092b231b53d

Is this an "error" or simply "warning"? We generally don't fix warnings.

This is definitely a case where we need to fix it. 25,000+ warnings would drown out any meaningful output in the event that we need to troubleshoot another problem.

In my last build with the itstool merge request and asciidoc patch applied it knocked it down to 183 warnings.

[martyj19]

The deletion of the "imp" package with the intended replacement "importlib" will need to be addressed in qtwebengine. Research indicates that there is not necessarily a one to one mapping of calls from one to the other.

I am finding that the beautifulsoup4 (bs4) package that is bundled in the qtwebengine tarball fails with a diagnostic that it cannot find html5lib, even though it is present. I worked around this by deleting the requirement for html5lib in those calls. Thus far I have not been able to determine the cause for this failure.

(The above findings apply to Qt6 qtwebengine. I am assuming they also would apply to Qt5 because they are in the Chromium portion.)

[Bruce Dubbs]

Fixed at commits

7420ff72f1 Update to Python-3.12.0.
1459c3b271 Update to shadow-4.14.2.

[Xi Ruoyao]

Reopening because various downstream packages need fix or workaround.

[Bruce Dubbs]

Replying to Xi Ruoyao:

Reopening because various downstream packages need fix or workaround.

Which ones?

[Douglas R. Reno]

Replying to Bruce Dubbs:

Replying to Xi Ruoyao:

Reopening because various downstream packages need fix or workaround.

Which ones?

From the comments here I'm aware of:

• QtWebEngine
• Node.js (done at 73ac81bdd426bee0adf57df76967e1578ddbbb5a)
• Spidermonkey (done at e3b21095897b39c898446a794e7f5c99aa451b3e)
• itstool (thousands of "invalid escape sequence" warnings on every invocation)
• asciidoc (generates thousands of errors on every invocation)
• Firefox/Thunderbird (see comment #12)
• Seamonkey is very likely broken as well
• Samba (f-strings changes)
• gobject-introspection (fixed at 3d349f41148cbdc8e41563d26b8488ae3f2e9f8b)

... and likely many more to come. Most of the issues that have been seen happen at runtime, rather than at build time (which is relatively common for Python)

There are a few big changes here that will cause these problems to happen. One of them is f-strings. The removal of the 'imp' and 'distutils' modules are two more, and the 'tokenize' function being rewritten with a known behavior change.

[Douglas R. Reno]

Replying to Xi Ruoyao:

The work around for Firefox:

for i in $(find -name six.py); do
  cp -v /usr/lib/python3.12/site-packages/six.py $i;
done

cp /usr/lib/python3.12/site-packages/{setuptools,distutils-precedence.pth,pkg_resources,_distutils_hack} -R third_party/python/setuptools

sed -e '/^import/s/imp$/importlib.util/' \
    -e 's/imp.new_module/__import__/' \
    -e "s/imp.load_source\(.*\)/spec = \
        importlib.util.spec_from_file_location\1; \
        mod = importlib.util.module_from_spec(spec); \
        spec.loader.exec_module(mod)/" -i python/mach/mach/main.py

sed '/ConfigParser/s/Safe//' -i testing/mozbase/mozprofile/mozprofile/prefs.py
sed 's/distutils/setuptools._&/' -i python/mozbuild/mozbuild/nodeutil.py
sed '/import imp/d' -i netwerk/dns/prepare_tlds.py

The changes necessary for Firefox/Thunderbird

[Douglas R. Reno]

Note that we are not aware of any fix for QtWebEngine yet, for either the Qt6 version or the Qt5 version that we have in the book.

[Bruce Dubbs]

Replying to Xi Ruoyao:

Reopening because various downstream packages need fix or workaround.

Closing at transferring package issues to #18798