python3-3.12.1 (Wait for LFS)

[Bruce Dubbs]

New point version.

[Xi Ruoyao]

Python News

What's New in Python 3.12.1 final?

*Release date: 2023-12-07*

Core and Builtins

• gh-issue-112125: Fix None.ne(None) returning NotImplemented instead of False

• gh-issue-112625: Fixes a bug where a bytearray object could be cleared while iterating over an argument in the bytearray.join() method that could result in reading memory after it was freed.

• gh-issue-105967: Workaround a bug in Apple's macOS platform zlib library where :func:zlib.crc32 and :func:binascii.crc32 could produce incorrect results on multi-gigabyte inputs. Including when using :mod:zipfile on zips containing large data.

• gh-issue-112356: Stopped erroneously deleting a LOAD_NULL bytecode instruction when optimized twice.

• gh-issue-111058: Change coro.cr_frame/gen.gi_frame to return None after the coroutine/generator has been closed. This fixes a bug where :func:~inspect.getcoroutinestate and :func:~inspect.getgeneratorstate return the wrong state for a closed coroutine/generator.

• gh-issue-112388: Fix an error that was causing the parser to try to overwrite tokenizer errors. Patch by pablo Galindo

• gh-issue-112387: Fix error positions for decoded strings with backwards tokenize errors. Patch by Pablo Galindo

• gh-issue-112367: Avoid undefined behaviour when using the perf trampolines by not freeing the code arenas until shutdown. Patch by Pablo Galindo

• gh-issue-112243: Don't include comments in f-string debug expressions. Patch by Pablo Galindo

• gh-issue-112266: Change docstrings of :attr:~object.__dict__ and :attr:~object.__weakref__.

• gh-issue-111654: Fix runtime crash when some error happens in opcode LOAD_FROM_DICT_OR_DEREF.

• gh-issue-109181: Speed up :obj:Traceback object creation by lazily compute the line number. Patch by Pablo Galindo

• gh-issue-102388: Fix a bug where iso2022_jp_3 and iso2022_jp_2004 codecs read out of bounds

• gh-issue-111366: Fix an issue in the :mod:codeop that was causing :exc:SyntaxError exceptions raised in the presence of invalid syntax to not contain precise error messages. Patch by Pablo Galindo

• gh-issue-111380: Fix a bug that was causing :exc:SyntaxWarning to appear twice when parsing if invalid syntax is encountered later. Patch by Pablo galindo

• gh-issue-94438: Fix a regression that prevented jumping across is None and is not None when debugging. Patch by Savannah Ostrowski.

• gh-issue-110938: Fix error messages for indented blocks with functions and classes with generic type parameters. Patch by Pablo Galindo

• gh-issue-109894: Fixed crash due to improperly initialized static :exc:MemoryError in subinterpreter.

• gh-issue-110782: Fix crash when :class:typing.TypeVar is constructed with a keyword argument. Patch by Jelle Zijlstra.

• gh-issue-110696: Fix incorrect error message for invalid argument unpacking. Patch by Pablo Galindo

• gh-issue-110543: Fix regression in Python 3.12 where :meth:types.CodeType.replace would produce a broken code object if called on a module or class code object that contains a comprehension. Patch by Jelle Zijlstra.

• gh-issue-110514: Add PY_THROW to :func:sys.setprofile events

• gh-issue-110455: Guard assert(tstate->thread_id > 0) with #ifndef HAVE_PTHREAD_STUBS. This allows for for pydebug builds to work under WASI which (currently) lacks thread support.

• gh-issue-110259: Correctly identify the format spec in f-strings (with single or triple quotes) that have multiple lines in the expression part and include a formatting spec. Patch by Pablo Galindo

• gh-issue-110237: Fix missing error checks for calls to PyList_Append in _PyEval_MatchClass.

• gh-issue-109889: Fix the compiler's redundant NOP detection algorithm to skip over NOPs with no line number when looking for the next instruction's lineno.

• gh-issue-109853: sys.path[0] is now set correctly for subinterpreters.

• gh-issue-105716: Subinterpreters now correctly handle the case where they have threads running in the background. Before, such threads would interfere with cleaning up and destroying them, as well as prevent running another script.

• gh-issue-109793: The main thread no longer exits prematurely when a subinterpreter is cleaned up during runtime finalization. The bug was a problem particularly because, when triggered, the Python process would always return with a 0 exitcode, even if it failed.

• gh-issue-109596: Fix some tokens in the grammar that were incorrectly marked as soft keywords. Also fix some repeated rule names and ensure that repeated rules are not allowed. Patch by Pablo Galindo

• gh-issue-109351: Fix crash when compiling an invalid AST involving a named (walrus) expression.

• gh-issue-109216: Fix possible memory leak in :opcode:BUILD_MAP.

• gh-issue-109207: Fix a SystemError in repr of symtable entry object.

• gh-issue-109179: Fix bug where the C traceback display drops notes from :exc:SyntaxError.

• gh-issue-109052: Use the base opcode when comparing code objects to avoid interference from instrumentation

• gh-issue-88943: Improve syntax error for non-ASCII character that follows a numerical literal. It now points on the invalid non-ASCII character, not on the valid numerical literal.

• gh-issue-106931: Statically allocated string objects are now interned globally instead of per-interpreter. This fixes a situation where such a string would only be interned in a single interpreter. Normal string objects are unaffected.

Library

• gh-issue-79325: Fix an infinite recursion error in :func:tempfile.TemporaryDirectory cleanup on Windows.

• gh-issue-112645: Remove deprecation error on passing onerror to :func:shutil.rmtree.

• gh-issue-112618: Fix a caching bug relating to :data:typing.Annotated. Annotated[str, True] is no longer identical to Annotated[str, 1].

• gh-issue-112334: Fixed a performance regression in 3.12's :mod:subprocess on Linux where it would no longer use the fast-path vfork() system call when it should have due to a logic bug, instead always falling back to the safe but slower fork().

Also fixed a related 3.12 security regression: If a value of extra_groups=[] was passed to :mod:subprocess.Popen or related APIs, the underlying setgroups(0, NULL) system call to clear the groups list would not be made in the child process prior to exec(). This has been assigned CVE-2023-6507.

This was identified via code inspection in the process of fixing the first bug.

• gh-issue-110190: Fix ctypes structs with array on Arm platform by setting MAX_STRUCT_SIZE to 32 in stgdict. Patch by Diego Russo.

• gh-issue-112578: Fix a spurious :exc:RuntimeWarning when executing the :mod:zipfile module.

• gh-issue-112509: Fix edge cases that could cause a key to be present in both the required_keys and optional_keys attributes of a :class:typing.TypedDict. Patch by Jelle Zijlstra.

• gh-issue-112414: Fix regression in Python 3.12 where calling :func:repr on a module that had been imported using a custom :term:loader could fail with :exc:AttributeError. Patch by Alex Waygood.

• gh-issue-112358: Revert change to :class:struct.Struct initialization that broke some cases of subclassing.

• gh-issue-94722: Fix bug where comparison between instances of :class:~doctest.DocTest fails if one of them has None as its lineno.

• gh-issue-112105: Make :func:readline.set_completer_delims work with libedit

• gh-issue-111942: Fix SystemError in the TextIOWrapper constructor with non-encodable "errors" argument in non-debug mode.

• gh-issue-109538: Issue warning message instead of having :class:RuntimeError be displayed when event loop has already been closed at :meth:StreamWriter.__del__.

• gh-issue-111942: Fix crashes in :meth:io.TextIOWrapper.reconfigure when pass invalid arguments, e.g. non-string encoding.

• gh-issue-111460: :mod:curses: restore wide character support (including :func:curses.unget_wch and :meth:~curses.window.get_wch) on macOS, which was unavailable due to a regression in Python 3.12.

• gh-issue-103791: :class:contextlib.suppress now supports suppressing exceptions raised as part of a :exc:BaseExceptionGroup, in addition to the recent support for :exc:ExceptionGroup.

• gh-issue-111804: Remove posix.fallocate() under WASI as the underlying posix_fallocate() is not available in WASI preview2.

• gh-issue-111841: Fix truncating arguments on an embedded null character in :meth:os.putenv and :meth:os.unsetenv on Windows.

• gh-issue-111541: Fix :mod:doctest for :exc:SyntaxError not-builtin subclasses.

• gh-issue-110894: Call loop exception handler for exceptions in client_connected_cb of :func:asyncio.start_server so that applications can handle it. Patch by Kumar Aditya.

• gh-issue-111531: Fix reference leaks in bind_class() and bind_all() methods of :mod:tkinter widgets.

• gh-issue-111356: Added :func:io.text_encoding(), :data:io.DEFAULT_BUFFER_SIZE, and :class:io.IncrementalNewlineDecoder to io.all.

• gh-issue-111342: Fixed typo in :func:math.sumprod.

• gh-issue-68166: Remove mention of not supported "vsapi" element type in :meth:tkinter.ttk.Style.element_create. Add tests for element_create() and other ttk.Style methods. Add examples for element_create() in the documentation.

• gh-issue-75666: Fix the behavior of :mod:tkinter widget's unbind() method with two arguments. Previously, widget.unbind(sequence, funcid) destroyed the current binding for *sequence*, leaving *sequence* unbound, and deleted the *funcid* command. Now it removes only *funcid* from the binding for *sequence*, keeping other commands, and deletes the *funcid* command. It leaves *sequence* unbound only if *funcid* was the last bound command.

• gh-issue-79033: Another attempt at fixing :func:asyncio.Server.wait_closed(). It now blocks until both conditions are true: the server is closed, *and* there are no more active connections. (This means that in some cases where in 3.12.0 this function would *incorrectly* have returned immediately, it will now block; in particular, when there are no active connections but the server hasn't been closed yet.)

• gh-issue-111295: Fix :mod:time not checking for errors when initializing.

• gh-issue-111253: Add error checking during :mod:!_socket module init.

• gh-issue-111251: Fix :mod:_blake2 not checking for errors when initializing.

• gh-issue-111174: Fix crash in :meth:io.BytesIO.getbuffer called repeatedly for empty BytesIO.

• gh-issue-111187: Postpone removal version for locale.getdefaultlocale() to Python 3.15.

• gh-issue-111159: Fix :mod:doctest output comparison for exceptions with notes.

• gh-issue-110910: Fix invalid state handling in :class:asyncio.TaskGroup and :class:asyncio.Timeout. They now raise proper RuntimeError if they are improperly used and are left in consistent state after this.

• gh-issue-111092: Make turtledemo run without default root enabled.

• gh-issue-110488: Fix a couple of issues in :meth:pathlib.PurePath.with_name: a single dot was incorrectly considered a valid name, and in :class:PureWindowsPath, a name with an NTFS alternate data stream, like a:b, was incorrectly considered invalid.

• gh-issue-110392: Fix :func:tty.setraw and :func:tty.setcbreak: previously they returned partially modified list of the original tty attributes. :func:tty.cfmakeraw and :func:tty.cfmakecbreak now make a copy of the list of special characters before modifying it.

• gh-issue-110590: Fix a bug in :meth:!_sre.compile where :exc:TypeError would be overwritten by :exc:OverflowError when the *code* argument was a list of non-ints.

• gh-issue-65052: Prevent :mod:pdb from crashing when trying to display undisplayable objects

• gh-issue-110519: Deprecation warning about non-integer number in :mod:gettext now alwais refers to the line in the user code where gettext function or method is used. Previously it could refer to a line in gettext code.

• gh-issue-110395: Ensure that :func:select.kqueue objects correctly appear as closed in forked children, to prevent operations on an invalid file descriptor.

• gh-issue-110378: :func:~contextlib.contextmanager and :func:~contextlib.asynccontextmanager context managers now close an invalid underlying generator object that yields more then one value.

• gh-issue-110365: Fix :func:termios.tcsetattr bug that was overwritting existing errors during parsing integers from term list.

• gh-issue-109653: Fix a Python 3.12 regression in the import time of :mod:random. Patch by Alex Waygood.

• gh-issue-110196: Add reduce method to :class:IPv6Address in order to keep scope_id

• gh-issue-110036: On Windows, multiprocessing Popen.terminate() now catchs :exc:PermissionError and get the process exit code. If the process is still running, raise again the :exc:PermissionError. Otherwise, the process terminated as expected: store its exit code. Patch by Victor Stinner.

• gh-issue-110038: Fixed an issue that caused :meth:KqueueSelector.select to not return all the ready events in some cases when a file descriptor is registered for both read and write.

• gh-issue-109631: :mod:re functions such as :func:re.findall, :func:re.split, :func:re.search and :func:re.sub which perform short repeated matches can now be interrupted by user.

• gh-issue-109747: Improve errors for unsupported look-behind patterns. Now re.error is raised instead of OverflowError or RuntimeError for too large width of look-behind pattern.

• gh-issue-109818: Fix :func:reprlib.recursive_repr not copying type_params from decorated function.

• gh-issue-109047: :mod:concurrent.futures: The *executor manager thread* now catches exceptions when adding an item to the *call queue*. During Python finalization, creating a new thread can now raise :exc:RuntimeError. Catch the exception and call terminate_broken() in this case. Patch by Victor Stinner.

• gh-issue-109782: Ensure the signature of :func:os.path.isdir is identical on all platforms. Patch by Amin Alaee.

• gh-issue-109590: :func:shutil.which will prefer files with an extension in PATHEXT if the given mode includes os.X_OK on win32. If no PATHEXT match is found, a file without an extension in PATHEXT can be returned. This change will have :func:shutil.which act more similarly to previous behavior in Python 3.11.

• gh-issue-109786: Fix possible reference leaks and crash when re-enter the next() method of :class:itertools.pairwise.

• gh-issue-109593: Avoid deadlocking on a reentrant call to the multiprocessing resource tracker. Such a reentrant call, though unlikely, can happen if a GC pass invokes the finalizer for a multiprocessing object such as SemLock.

• gh-issue-109613: Fix :func:os.stat and :meth:os.DirEntry.stat: check for exceptions. Previously, on Python built in debug mode, these functions could trigger a fatal Python error (and abort the process) when a function succeeded with an exception set. Patch by Victor Stinner.

• gh-issue-109375: The :mod:pdb alias command now prevents registering aliases without arguments.

• gh-issue-107219: Fix a race condition in concurrent.futures. When a process in the process pool was terminated abruptly (while the future was running or pending), close the connection write end. If the call queue is blocked on sending bytes to a worker process, closing the connection write end interrupts the send, so the queue can be closed. Patch by Victor Stinner.

• gh-issue-50644: Attempts to pickle or create a shallow or deep copy of :mod:codecs streams now raise a TypeError. Previously, copying failed with a RecursionError, while pickling produced wrong results that eventually caused unpickling to fail with a RecursionError.

• gh-issue-108987: Fix :func:_thread.start_new_thread race condition. If a thread is created during Python finalization, the newly spawned thread now exits immediately instead of trying to access freed memory and lead to a crash. Patch by Victor Stinner.

• gh-issue-108791: Improved error handling in :mod:pdb command line interface, making it produce more concise error messages.

• gh-issue-105829: Fix concurrent.futures.ProcessPoolExecutor deadlock

• gh-issue-106584: Fix exit code for unittest if all tests are skipped. Patch by Egor Eliseev.

• gh-issue-102956: Fix returning of empty byte strings after seek in zipfile module

• gh-issue-84867: :class:unittest.TestLoader no longer loads test cases from exact :class:unittest.TestCase and :class:unittest.FunctionTestCase classes.

• gh-issue-91133: Fix a bug in :class:tempfile.TemporaryDirectory cleanup, which now no longer dereferences symlinks when working around file system permission errors.

• gh-issue-73561: Omit the interface scope from an IPv6 address when used as Host header by :mod:http.client.

• gh-issue-86826: :mod:zipinfo now supports the full range of values in the TZ string determined by RFC 8536 and detects all invalid formats. Both Python and C implementations now raise exceptions of the same type on invalid data.

• bpo-43153: On Windows, tempfile.TemporaryDirectory previously masked a PermissionError with NotADirectoryError during directory cleanup. It now correctly raises PermissionError if errors are not ignored. Patch by Andrei Kulakov and Ken Jin.

• bpo-35332: The :func:shutil.rmtree function now ignores errors when calling :func:os.close when *ignore_errors* is True, and :func:os.close no longer retried after error.

• bpo-41422: Fixed memory leaks of :class:pickle.Pickler and :class:pickle.Unpickler involving cyclic references via the internal memo mapping.

• bpo-40262: The :meth:ssl.SSLSocket.recv_into method no longer requires the *buffer* argument to implement len and supports buffers with arbitrary item size.

Documentation

• gh-issue-111699: Relocate smtpd deprecation notice to its own section rather than under locale in What's New in Python 3.12 document

• gh-issue-108826: :mod:dis module command-line interface is now mentioned in documentation.

Tests

---

• gh-issue-112769: The tests now correctly compare zlib version when :const:zlib.ZLIB_RUNTIME_VERSION contains non-integer suffixes. For example zlib-ng defines the version as 1.3.0.zlib-ng.

• gh-issue-110367: Make regrtest --verbose3 option compatible with --huntrleaks -jN options. The ./python -m test -j1 -R 3:3 --verbose3 command now works as expected. Patch by Victor Stinner.

• gh-issue-111165: Remove no longer used functions run_unittest() and run_doctest() from the :mod:test.support module.

• gh-issue-110932: Fix regrtest if the SOURCE_DATE_EPOCH environment variable is defined: use the variable value as the random seed. Patch by Victor Stinner.

• gh-issue-110995: test_gdb: Fix detection of gdb built without Python scripting support. Patch by Victor Stinner.

• gh-issue-110918: Test case matching patterns specified by options --match, --ignore, --matchfile and --ignorefile are now tested in the order of specification, and the last match determines whether the test case be run or ignored.

• gh-issue-110647: Fix test_stress_modifying_handlers() of test_signal. Patch by Victor Stinner.

• gh-issue-103053: Fix test_tools.test_freeze on FreeBSD: run "make distclean" instead of "make clean" in the copied source directory to remove also the "python" program. Patch by Victor Stinner.

• gh-issue-110167: Fix a deadlock in test_socket when server fails with a timeout but the client is still running in its thread. Don't hold a lock to call cleanup functions in doCleanups(). One of the cleanup function waits until the client completes, whereas the client could deadlock if it called addCleanup() in such situation. Patch by Victor Stinner.

• gh-issue-110388: Add tests for :mod:tty.

• gh-issue-81002: Add tests for :mod:termios.

• gh-issue-110267: Add tests for pickling and copying PyStructSequence objects. Patched by Xuehai Pan.

• gh-issue-110031: Skip test_threading tests using thread+fork if Python is built with Address Sanitizer (ASAN). Patch by Victor Stinner.

• gh-issue-110088: Fix test_asyncio timeouts: don't measure the maximum duration, a test should not measure a CI performance. Only measure the minimum duration when a task has a timeout or delay. Add CLOCK_RES to test_asyncio.utils. Patch by Victor Stinner.

• gh-issue-109974: Fix race conditions in test_threading lock tests. Wait until a condition is met rather than using :func:time.sleep with a hardcoded number of seconds. Patch by Victor Stinner.

• gh-issue-110033: Fix test_interprocess_signal() of test_signal. Make sure that the subprocess.Popen object is deleted before the test raising an exception in a signal handler. Otherwise, Popen.del() can get the exception which is logged as Exception ignored in: ... and the test fails. Patch by Victor Stinner.

• gh-issue-109594: Fix test_timeout() of test_concurrent_futures.test_wait. Remove the future which may or may not complete depending if it takes longer than the timeout ot not. Keep the second future which does not complete before wait() timeout. Patch by Victor Stinner.

• gh-issue-109972: Split test_gdb.py file into a test_gdb package made of multiple tests, so tests can now be run in parallel. Patch by Victor Stinner.

• gh-issue-103053: Skip test_freeze_simple_script() of test_tools.test_freeze if Python is built with ./configure --enable-optimizations, which means with Profile Guided Optimization (PGO): it just makes the test too slow. The freeze tool is tested by many other CIs with other (faster) compiler flags. Patch by Victor Stinner.

• gh-issue-109580: Skip test_perf_profiler if Python is built with ASAN, MSAN or UBSAN sanitizer. Python does crash randomly in this test on such build. Patch by Victor Stinner.

• gh-issue-104736: Fix test_gdb on Python built with LLVM clang 16 on Linux ppc64le (ex: Fedora 38). Search patterns in gdb "bt" command output to detect when gdb fails to retrieve the traceback. For example, skip a test if Backtrace stopped: frame did not save the PC is found. Patch by Victor Stinner.

• gh-issue-108927: Fixed order dependence in running tests in the same process when a test that has submodules (e.g. test_importlib) follows a test that imports its submodule (e.g. test_importlib.util) and precedes a test (e.g. test_unittest or test_compileall) that uses that submodule.

Build

• gh-issue-112088: Add Tools/build/regen-configure.sh script to regenerate the configure with an Ubuntu container image. The quay.io/tiran/cpython_autoconf:271 container image (tiran/cpython_autoconf <https://github.com/tiran/cpython_autoconf>_) is no longer used. Patch by Victor Stinner.

• gh-issue-111046: For wasi-threads, memory is now exported to fix compatibility issues with some wasm runtimes.

• gh-issue-103053: "make check-clean-src" now also checks if the "python" program is found in the source directory: fail with an error if it does exist. Patch by Victor Stinner.

• gh-issue-109191: Fix compile error when building with recent versions of libedit.

Windows

• gh-issue-111856: Fixes :func:~os.fstat on file systems that do not support file ID requests. This includes FAT32 and exFAT.

• gh-issue-111293: Fix :data:os.DirEntry.inode dropping higher 64 bits of a file id on some filesystems on Windows.

• gh-issue-110913: WindowsConsoleIO now correctly chunks large buffers without splitting up UTF-8 sequences.

• gh-issue-110437: Allows overriding the source of VC redistributables so that releases can be guaranteed to never downgrade between updates.

• gh-issue-109286: Update Windows installer to use SQLite 3.43.1.

macOS

• gh-issue-109981: Use /dev/fd on macOS to determine the number of open files in test.support.os_helper.fd_count to avoid a crash with "guarded" file descriptors when probing for open files.

• gh-issue-110950: Update macOS installer to include an upstream Tcl/Tk fix for the Secure coding is not enabled for restorable state`` warning encountered in Tkinter on macOS 14 Sonoma.

• gh-issue-111015: Ensure that IDLE.app and Python Launcher.app are installed with appropriate permissions on macOS builds.

• gh-issue-109286: Update macOS installer to use SQLite 3.43.1.

• gh-issue-71383: Update macOS installer to include an upstream Tcl/Tk fix for the ttk::ThemeChanged error encountered in Tkinter.

• gh-issue-92603: Update macOS installer to include a fix accepted by upstream Tcl/Tk for a crash encountered after the first :meth:tkinter.Tk instance is destroyed.

IDLE

• bpo-35668: Add docstrings to the IDLE debugger module. Fix two bugs: initialize Idb.botframe (should be in Bdb); in Idb.in_rpc_code, check whether prev_frame is None before trying to use it. Greatly expand test_debugger.

C API

• gh-issue-106560: Fix redundant declarations in the public C API. Declare PyBool_Type and PyLong_Type only once. Patch by Victor Stinner.

• gh-issue-112438: Fix support of format units "es", "et", "es#", and "et#" in nested tuples in :c:func:`PyArg_ParseTuple`-like functions.

• gh-issue-109521: :c:func:`PyImport_GetImporter` now sets RuntimeError if it fails to get :data:sys.path_hooks or :data:sys.path_importer_cache or they are not list and dict correspondingly. Previously it could return NULL without setting error in obscure cases, crash or raise SystemError if these attributes have wrong type.

[Xi Ruoyao]

Elevate for CVE-2023-6507.

[Bruce Dubbs]

Fixed at commits

2c758f05b7 Update to Python-3.12.1.
92202f73f5 Update to mesa-23.3.1.
7450bb6c4d Update to librsvg-2.57.1.