python3-3.9.1

[Bruce Dubbs]

New point version.

[Bruce Dubbs]

What's New in Python 3.9.1 final?

*Release date: 2020-12-07*

Core and Builtins

• bpo-42576: types.GenericAlias will now raise a TypeError when attempting to initialize with a keyword argument. Previously, this would cause the interpreter to crash if the interpreter was compiled with debug symbols. This does not affect interpreters compiled for release. Patch by Ken Jin.

Library

• bpo-5054: CGIHTTPRequestHandler.run_cgi() HTTP_ACCEPT improperly parsed. Replace the special purpose getallmatchingheaders with generic get_all method and add relevant tests.

• bpo-17735: :func:inspect.findsource now raises :exc:OSError instead of :exc:IndexError when :attr:co_lineno of a code object is greater than the file length. This can happen, for example, when a file is edited after it was imported.

• bpo-42116: Fix handling of trailing comments by :func:inspect.getsource.

• bpo-42487: ChainMap.iter no longer calls getitem on underlying maps

• bpo-42482: :class:~traceback.TracebackException no longer holds a reference to the exception's traceback object. Consequently, instances of TracebackException for equivalent but non-equal exceptions now compare as equal.

• bpo-42406: We fixed an issue in pickle.whichmodule in which importing multiprocessing could change the how pickle identifies which module an object belongs to, potentially breaking the unpickling of those objects.

• bpo-34215: Clarify the error message for :exc:asyncio.IncompleteReadError when expected is None.

• bpo-12800: Extracting a symlink from a tarball should succeed and overwrite the symlink if it already exists. The fix is to remove the existing file or symlink before extraction.

Tests

• bpo-41473: Reenable test_gdb on gdb 9.2 and newer: ​https://bugzilla.redhat.com/show_bug.cgi?id=1866884 bug is fixed in gdb 10.1.

• bpo-42553: Fix test_asyncio.test_call_later() race condition: don't measure asyncio performance in the call_later() unit test. The test failed randomly on the CI.

macOS

• bpo-41116: If no explicit macOS SDK was specified, setup.py should check for Tcl and TK frameworks in /Library/Frameworks; the previous commit inadvertently broke that test.

• bpo-42504: Fix build on macOS Big Sur when MACOSX_DEPLOYMENT_TARGET=11

IDLE

• bpo-42508: Keep IDLE running on macOS. Remove obsolete workaround that prevented running files with shortcuts when using new universal2 installers built on macOS 11.

What's New in Python 3.9.1 release candidate 1?

*Release date: 2020-11-24*

Security

• bpo-42103: Prevented potential DoS attack via CPU and RAM exhaustion when processing malformed Apple Property List files in binary format.

• bpo-42051: The :mod:plistlib module no longer accepts entity declarations in XML plist files to avoid XML vulnerabilities. This should not affect users as entity declarations are not used in regular plist files.

• bpo-40791: Add volatile to the accumulator variable in hmac.compare_digest, making constant-time-defeating optimizations less likely.

Core and Builtins

• bpo-41686: On Windows, the SIGINT event, _PyOS_SigintEvent(), is now created even if Python is configured to not install signal handlers (if :c:member:`PyConfig.install_signal_handlers` equals to 0, or Py_InitializeEx(0)).

• bpo-42381: Allow assignment expressions in set literals and set comprehensions as per PEP 572.

• bpo-42374: Fix a regression introduced by the new parser, where an unparenthesized walrus operator was not allowed within generator expressions.

• bpo-42296: On Windows, fix a regression in signal handling which prevented to interrupt a program using CTRL+C. The signal handler can be run in a thread different than the Python thread, in which case the test deciding if the thread can handle signals is wrong.

• bpo-42332: :class:types.GenericAlias objects can now be the targets of weakrefs.

• bpo-42218: Fixed a bug in the PEG parser that was causing crashes in debug mode. Now errors are checked in left-recursive rules to avoid cases where such errors do not get handled in time and appear as long-distance crashes in other places.

• bpo-42214: Fixed a possible crash in the PEG parser when checking for the '!=' token in the barry_as_flufl rule. Patch by Pablo Galindo.

• bpo-42143: Fix handling of errors during creation of PyFunctionObject, which resulted in operations on uninitialized memory. Patch by Yonatan Goldschmidt.

• bpo-41659: Fix a bug in the parser, where a curly brace following a primary didn't fail immediately. This led to invalid expressions like `a {b} to throw a :exc:SyntaxError` with a wrong offset, or invalid expressions ending with a curly brace like a { to not fail immediately in the REPL.

• bpo-42150: Fix possible buffer overflow in the new parser when checking for continuation lines.

• bpo-42123: Run the parser two times. On the first run, disable all the rules that only generate better error messages to gain performance. If there's a parse failure, run the parser a second time with those enabled.

• bpo-41910: Document the default implementation of object.__eq__.

• bpo-42057: Fix peephole optimizer misoptimize conditional jump + JUMP_IF_NOT_EXC_MATCH pair.

• bpo-41984: The garbage collector now tracks all user-defined classes.

• bpo-41993: Fixed potential issues with removing not completely initialized module from sys.modules when import fails.

• bpo-41979: Star-unpacking is now allowed for with item's targets in the PEG parser.

• bpo-41909: Fixed stack overflow in :func:issubclass and :func:isinstance when getting the bases attribute leads to infinite recursion.

• bpo-41894: When loading a native module and a load failure occurs, prevent a possible UnicodeDecodeError when not running in a UTF-8 locale by decoding the load error message using the current locale's encoding.

• bpo-39934: Correctly count control blocks in 'except' in compiler. Ensures that a syntax error, rather a fatal error, occurs for deeply nested, named exception handlers.

Library

• bpo-42328: Fixed :meth:tkinter.ttk.Style.map. The function accepts now the representation of the default state as empty sequence (as returned by Style.map()). The structure of the result is now the same on all platform and does not depend on the value of wantobjects.

• bpo-42345: Fix various issues with typing.Literal parameter handling (flatten, deduplicate, use type to cache key).

• bpo-42350: Fix the :class:threading.Thread class at fork: do nothing if the thread is already stopped (ex: fork called at Python exit). Previously, an error was logged in the child process.

• bpo-42014: The onerror callback from shutil.rmtree now receives correct function when os.open fails.

• bpo-42237: Fix os.sendfile() on illumos.

• bpo-42249: Fixed writing binary Plist files larger than 4 GiB.

• bpo-35455: On Solaris, :func:~time.thread_time is now implemented with gethrvtime() because clock_gettime(CLOCK_THREAD_CPUTIME_ID) is not always available. Patch by Jakub Kulik.

• bpo-42233: The :func:repr of :mod:typing types containing :ref:Generic Alias Types <types-genericalias> previously did not show the parameterized types in the GenericAlias. They have now been changed to do so.

• bpo-41754: webbrowser: Ignore *NotADirectoryError* when calling xdg-settings.

• bpo-29566: binhex.binhex() consisently writes macOS 9 line endings.

• bpo-42183: Fix a stack overflow error for asyncio Task or Future repr().

The overflow occurs under some circumstances when a Task or Future recursively returns itself.

• bpo-42146: Fix memory leak in :func:subprocess.Popen in case an uid (gid) specified in user (group, extra_groups) overflows uid_t (gid_t).

• bpo-42140: Improve asyncio.wait function to create the futures set just one time.

• bpo-42103: :exc:~plistlib.InvalidFileException and :exc:RecursionError are now the only errors caused by loading malformed binary Plist file (previously ValueError and TypeError could be raised in some specific cases).

• bpo-41052: Pickling heap types implemented in C with protocols 0 and 1 raises now an error instead of producing incorrect data.

• bpo-41491: plistlib: fix parsing XML plists with hexadecimal integer values

• bpo-42065: Fix an incorrectly formatted error from :meth:_codecs.charmap_decode when called with a mapped value outside the range of valid Unicode code points.

• bpo-41966: Fix pickling pure Python :class:datetime.time subclasses.

• bpo-41976: Fixed a bug that was causing :func:ctypes.util.find_library to return None when triying to locate a library in an environment when gcc>=9 is available and ldconfig is not.

• bpo-41900: C14N 2.0 serialisation in xml.etree.ElementTree failed for unprefixed attributes when a default namespace was defined.

• bpo-41840: Fix a bug in the :mod:symtable module that was causing module-scope global variables to not be reported as both local and global.

• bpo-41831: str() for the type attribute of the tkinter.Event object always returns now the numeric code returned by Tk instead of the name of the event type.

• bpo-41817: fix tkinter.EventType Enum so all members are strings, and none are tuples

• bpo-41815: Fix SQLite3 segfault when backing up closed database. Patch contributed by Peter David McCormick.

• bpo-41316: Fix the :mod:tarfile module to write only basename of TAR file to GZIP compression header.

• bpo-16936: Allow ctypes.wintypes to be imported on non-Windows systems.

• bpo-40592: :func:shutil.which now ignores empty entries in :envvar:PATHEXT instead of treating them as a match.

• bpo-40550: Fix time-of-check/time-of-action issue in subprocess.Popen.send_signal.

• bpo-40492: Fix --outfile for :mod:cProfile / :mod:profile not writing the output file in the original directory when the program being profiled changes the working directory.

• bpo-40105: ZipFile truncates files to avoid corruption when a shorter comment is provided in append ("a") mode.

• bpo-27321: Fixed KeyError exception when flattening an email to a string attempts to replace a non-existent Content-Transfer-Encoding header.

Documentation

• bpo-42153: Fix the URL for the IMAP protocol documents.

• bpo-42061: Document format functionality for IP addresses.

• bpo-42010: Clarify that subscription expressions are also valid for certain :term:classes <class> and :term:types <type> in the standard library, and for user-defined classes and types if the classmethod :meth:__class_getitem__ is provided.

• bpo-41805: Documented :ref:generic alias type <types-genericalias> and :data:types.GenericAlias. Also added an entry in glossary for :term:generic types <generic type>.

• bpo-41774: In Programming FAQ "Sequences (Tuples/Lists)" section, add "How do you remove multiple items from a list".

• bpo-35293: Fix RemovedInSphinx40Warning when building the documentation.

• bpo-41726: Update the refcounts info of PyType_FromModuleAndSpec.

• bpo-39693: Fix tarfile's extractfile documentation

• bpo-39416: Document some restrictions on the default string representations of numeric classes.

Tests

• bpo-40754: Include _testinternalcapi module in Windows installer for test suite

• bpo-41739: Fix test_logging.test_race_between_set_target_and_flush(): the test now waits until all threads complete to avoid leaking running threads.

• bpo-41970: Avoid a test failure in test_lib2to3 if the module has already imported at the time the test executes. Patch by Pablo Galindo.

• bpo-41944: Tests for CJK codecs no longer call eval() on content received via HTTP.

• bpo-41939: Fix test_site.test_license_exists_at_url(): call urllib.request.urlcleanup() to reset the global urllib.request._opener. Patch by Victor Stinner.

• bpo-41561: test_ssl: skip test_min_max_version_mismatch when TLS 1.0 is not available

• bpo-41602: Add tests for SIGINT handling in the runpy module.

• bpo-41306: Fixed a failure in test_tk.test_widgets.ScaleTest happening when executing the test with Tk 8.6.10.

Build

• bpo-42398: Fix a race condition in "make regen-all" when make -jN option is used to run jobs in parallel. The clinic.py script now only use atomic write to write files. Moveover, generated files are now left unchanged if the content does not change, to not change the file modification time.

• bpo-41617: Fix building pycore_bitutils.h internal header on old clang version without builtin_bswap16() (ex: Xcode 4.6.3 on Mac OS X 10.7). Patch by Joshua Root and Victor Stinner.

• bpo-38249: Update :c:macro:`Py_UNREACHABLE` to use builtin_unreachable() if only the compiler is able to use it. Patch by Dong-hee Na.

• bpo-40998: Addressed three compiler warnings found by undefined behavior sanitizer (ubsan).

Windows

• bpo-42120: Remove macro definition of copysign (to _copysign) in headers.

• bpo-38439: Updates the icons for IDLE in the Windows Store package.

• bpo-41744: Fixes automatic import of props file when using the Nuget package.

• bpo-41557: Update Windows installer to use SQLite 3.33.0.

• bpo-38324: Avoid Unicode errors when accessing certain locale data on Windows.

macOS

• bpo-41116: Ensure distutils.unixxcompiler.find_library_file can find system provided libraries on macOS 11.

• bpo-41100: Add support for macOS 11 and Apple Silicon systems.

It is now possible to build "Universal 2" binaries using "--enable-universalsdk --with-universal-archs=universal2".

Binaries build on later macOS versions can be deployed back to older versions (tested up to macOS 10.9), when using the correct deployment target. This is tested using Xcode 11 and later.

• bpo-38443: The --enable-universalsdk and --with-universal-archs options for the configure script now check that the specified architectures can be used.

• bpo-41471: Ignore invalid prefix lengths in system proxy excludes.

• bpo-41557: Update macOS installer to use SQLite 3.33.0.

IDLE

• bpo-42426: Fix reporting offset of the RE error in searchengine.

• bpo-42415: Get docstrings for IDLE calltips more often by using inspect.getdoc.

• bpo-33987: Mostly finish using ttk widgets, mainly for editor, settings, and searches.

• bpo-41775: Use 'IDLE Shell' as shell title

• bpo-35764: Rewrite the Calltips doc section.

• bpo-40181: In calltips, stop reminding that '/' marks the end of positional-only arguments.

• bpo-40511: Typing opening and closing parentheses inside the parentheses of a function call will no longer cause unnecessary "flashing" off and on of an existing open call-tip, e.g. when typed in a string literal.

• bpo-38439: Add a 256×256 pixel IDLE icon to the Windows .ico file. Created by Andrew Clover. Remove the low-color gif variations from the .ico file.

C API

• bpo-42015: Fix potential crash in deallocating method objects when dynamically allocated PyMethodDef's lifetime is managed through the self argument of a PyCFunction.

• bpo-41986: :c:data:`Py_FileSystemDefaultEncodeErrors` and :c:data:`Py_UTF8Mode` are available again in limited API.

[Douglas R. Reno]

---

Security

bpo-42103: Prevented potential DoS attack via CPU and RAM exhaustion when processing malformed Apple Property List files in binary format.

bpo-42051: The :mod:plistlib module no longer accepts entity declarations in XML plist files to avoid XML vulnerabilities. This should not affect users as entity declarations are not used in regular plist files.

bpo-40791: Add volatile to the accumulator variable in hmac.compare_digest, making constant-time-defeating optimizations less likely.

---

Marking high due to security fixes.

[Bruce Dubbs]

Fixed at revision 12069.