Merge upstream 3.12.4 release

Summary:
This merges the upstream changes from 3.12.3 -> 3.12.4 into our fork.

Merge-point in upstream is commit `8e8a4baf652f6e1cee7acde9d78c4b6154539748` (also pushed in cinder github as a `meta/3.12.4-base` tag).

Merging required a few manual merge conflict resolutions, called out as diff comments, and summarized here:
1. `patchlevel.h` -- update the meta version patch to "3.12.4+meta"
1. `python3.12.abi` -- fake news, probably due to EOL-style shenanigans
1. `site.py` -- reapplied patch from lazy imports in a block that had minor upstream changes
1. `generated_cases.c.h` -- needed to regenerate to resolve conflicts with the lazy imports patch (`python3 Tools/cases_generator/generate_cases.py`)

Reviewed upstream changes to configure and Makefile to look for needed buckification changes:

1. new C file, `_testcapi/run.c`, added upstream -- needed to update the buckified `_testcapi` module to also add this C file (tests failed without it).
1. `Modules/expat/xmltok_impl.c` and `Modules/expat/xmltok_ns.c` added to `LIBEXPAT_HEADERS` -- looks like an upstream bug that was fixed - these files were *already* in the buckified headers list since the build didn't work without these.
1. additional dirs in the `TESTSUBDIRS` list -- buckification unaffected
1. changes in the values of `PGO_PROF_USE_FLAG` and `LLVM_PROF_FILE` -- buckification unaffected, we don't use these directly.

Reviewed added/removed files in case they need to be reflected in the buckification (C file mentioned above, others don't affect our build):

```
$ git diff --name-status HEAD^ HEAD | sort | less
A       Doc/howto/mro.rst
A       Lib/test/test_capi/test_run.py
A       Lib/test/test_import/data/package3/__init__.py
A       Lib/test/test_import/data/package3/submodule.py
A       Lib/test/test_import/data/package4/__init__.py
A       Lib/test/test_import/data/package4/submodule.py
A       Lib/test/test_winapi.py
A       Lib/test/typinganndata/ann_module695.py
A       Misc/NEWS.d/3.12.4.rst
A       Modules/_testcapi/run.c
D       Mac/BuildScript/backport_gh92603_fix.patch
...
```

Reviewed changes in public header files. the additions in `pycore_global_strings` (generated init and fini) might be concerning for native python binaries until new RPMs rollout.

Reviewed changes (or lack-thereof) in a few places where we have internal-only patches to make sure they're not lost:
- zipimport (zip64 support)
- multiprocessing track param backport (D58048029)
- ceval.c
- import.c and other lazy imports related places

Reviewed By: zsol

Differential Revision: D58316054

fbshipit-source-id: 66f415a98ccaf26158f8ed8abfcc3d4f2674493a

.github/workflows/build.yml

@@ -228,6 +228,9 @@ jobs:
     uses: ./.github/workflows/reusable-macos.yml
     with:
       config_hash: ${{ needs.check_source.outputs.config_hash }}
+      # Cirrus and macos-14 are M1, macos-13 is default GHA Intel.
+      # Cirrus used for upstream, macos-14 for forks.
+      os-matrix: '["ghcr.io/cirruslabs/macos-runner:sonoma", "macos-14", "macos-13"]'
 
   build_ubuntu:
     name: 'Ubuntu'
@@ -244,7 +247,7 @@ jobs:
 
   build_ubuntu_ssltests:
     name: 'Ubuntu SSL tests with OpenSSL'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     timeout-minutes: 60
     needs: check_source
     if: needs.check_source.outputs.run_tests == 'true'
@@ -302,7 +305,7 @@ jobs:
 
   test_hypothesis:
     name: "Hypothesis tests on Ubuntu"
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     timeout-minutes: 60
     needs: check_source
     if: needs.check_source.outputs.run_tests == 'true' && needs.check_source.outputs.run_hypothesis == 'true'
@@ -415,7 +418,7 @@ jobs:
 
   build_asan:
     name: 'Address sanitizer'
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     timeout-minutes: 60
     needs: check_source
     if: needs.check_source.outputs.run_tests == 'true'

.github/workflows/reusable-docs.yml

@@ -62,7 +62,8 @@ jobs:
         python Doc/tools/check-warnings.py \
           --annotate-diff '${{ env.branch_base }}' '${{ env.branch_pr }}' \
           --fail-if-regression \
-          --fail-if-improved
+          --fail-if-improved \
+          --fail-if-new-news-nit
 
   # This build doesn't use problem matchers or check annotations
   build_doc_oldest_supported_sphinx:
@@ -74,7 +75,7 @@ jobs:
     - name: 'Set up Python'
       uses: actions/setup-python@v5
       with:
-        python-version: '3.11'  # known to work with Sphinx 4.2
+        python-version: '3.12'  # known to work with Sphinx 6.2.1
         cache: 'pip'
         cache-dependency-path: 'Doc/requirements-oldest-sphinx.txt'
     - name: 'Install build dependencies'

.github/workflows/reusable-macos.yml

@@ -8,10 +8,13 @@ on:
         required: false
         type: boolean
         default: false
+      os-matrix:
+        required: false
+        type: string
 
 jobs:
   build_macos:
-    name: 'build and test'
+    name: build and test (${{ matrix.os }})
     timeout-minutes: 60
     env:
       HOMEBREW_NO_ANALYTICS: 1
@@ -22,10 +25,14 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [
-          "macos-14",  # M1
-          "macos-13",  # Intel
-        ]
+        os: ${{fromJson(inputs.os-matrix)}}
+        is-fork:
+          - ${{ github.repository_owner != 'python' }}
+        exclude:
+          - os: "ghcr.io/cirruslabs/macos-runner:sonoma"
+            is-fork: true
+          - os: "macos-14"
+            is-fork: false
     runs-on: ${{ matrix.os }}
     steps:
     - uses: actions/checkout@v4

.github/workflows/reusable-ubuntu.yml

@@ -12,7 +12,7 @@ jobs:
   build_ubuntu_reusable:
     name: 'build and test'
     timeout-minutes: 60
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     env:
       OPENSSL_VER: 3.0.13
       PYTHONSTRICTEXTENSIONBUILD: 1

Doc/Makefile

@@ -32,6 +32,7 @@ help:
 	@echo "  clean      to remove build files"
 	@echo "  venv       to create a venv with necessary tools"
 	@echo "  html       to make standalone HTML files"
+	@echo "  gettext    to generate POT files"
 	@echo "  htmlview   to open the index page built by the html target in your browser"
 	@echo "  htmllive   to rebuild and reload HTML files in your browser"
 	@echo "  htmlhelp   to make HTML files and a HTML help project"
@@ -140,14 +141,23 @@ pydoc-topics: build
 	@echo "Building finished; now run this:" \
 	      "cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py"
 
+.PHONY: gettext
+gettext: BUILDER = gettext
+gettext: SPHINXOPTS += '-d build/doctrees-gettext'
+gettext: build
+
 .PHONY: htmlview
 htmlview: html
 	$(PYTHON) -c "import os, webbrowser; webbrowser.open('file://' + os.path.realpath('build/html/index.html'))"
 
+.PHONY: ensure-sphinx-autobuild
+ensure-sphinx-autobuild: venv
+	$(VENVDIR)/bin/sphinx-autobuild --version > /dev/null || $(VENVDIR)/bin/python3 -m pip install sphinx-autobuild
+
 .PHONY: htmllive
 htmllive: SPHINXBUILD = $(VENVDIR)/bin/sphinx-autobuild
 htmllive: SPHINXOPTS = --re-ignore="/venv/" --open-browser --delay 0
-htmllive: html
+htmllive: ensure-sphinx-autobuild html
 
 .PHONY: clean
 clean: clean-venv

Doc/bugs.rst

@@ -22,6 +22,10 @@ have a suggestion on how to fix it, include that as well.
 You can also open a discussion item on our
 `Documentation Discourse forum <https://discuss.python.org/c/documentation/26>`_.
 
+If you find a bug in the theme (HTML / CSS / JavaScript) of the
+documentation, please submit a bug report on the `python-doc-theme bug
+tracker <https://github.com/python/python-docs-theme>`_.
+
 If you're short on time, you can also email documentation bug reports to
 [email protected] (behavioral bugs can be sent to [email protected]).
 'docs@' is a mailing list run by volunteers; your request will be noticed,

Doc/c-api/buffer.rst

@@ -147,9 +147,9 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
       or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard
       :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``.
 
-   .. c:member:: const char *format
+   .. c:member:: char *format
 
-      A *NUL* terminated string in :mod:`struct` module style syntax describing
+      A *NULL* terminated string in :mod:`struct` module style syntax describing
       the contents of a single item. If this is ``NULL``, ``"B"`` (unsigned bytes)
       is assumed.
 

Doc/c-api/exceptions.rst

@@ -208,13 +208,14 @@ For convenience, some of these functions will always return a
 
 .. c:function:: PyObject* PyErr_SetFromWindowsErr(int ierr)
 
-   This is a convenience function to raise :exc:`WindowsError`. If called with
+   This is a convenience function to raise :exc:`OSError`. If called with
    *ierr* of ``0``, the error code returned by a call to :c:func:`!GetLastError`
    is used instead.  It calls the Win32 function :c:func:`!FormatMessage` to retrieve
    the Windows description of error code given by *ierr* or :c:func:`!GetLastError`,
-   then it constructs a tuple object whose first item is the *ierr* value and whose
-   second item is the corresponding error message (gotten from
-   :c:func:`!FormatMessage`), and then calls ``PyErr_SetObject(PyExc_WindowsError,
+   then it constructs a :exc:`OSError` object with the :attr:`~OSError.winerror`
+   attribute set to the error code, the :attr:`~OSError.strerror` attribute
+   set to the corresponding error message (gotten from
+   :c:func:`!FormatMessage`), and then calls ``PyErr_SetObject(PyExc_OSError,
    object)``. This function always returns ``NULL``.
 
    .. availability:: Windows.

Doc/c-api/init.rst

@@ -760,7 +760,7 @@ Process-wide parameters
       It is recommended that applications embedding the Python interpreter
       for purposes other than executing a single script pass ``0`` as *updatepath*,
       and update :data:`sys.path` themselves if desired.
-      See `CVE-2008-5983 <https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5983>`_.
+      See :cve:`2008-5983`.
 
       On versions before 3.1.3, you can achieve the same effect by manually
       popping the first :data:`sys.path` element after having called

Doc/c-api/object.rst

@@ -16,7 +16,7 @@ Object Protocol
 
    Properly handle returning :c:data:`Py_NotImplemented` from within a C
    function (that is, create a new :term:`strong reference`
-   to NotImplemented and return it).
+   to :const:`NotImplemented` and return it).
 
 
 .. c:macro:: Py_PRINT_RAW

Doc/c-api/tuple.rst

@@ -59,6 +59,12 @@ Tuple Objects
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
    negative or out of bounds, return ``NULL`` and set an :exc:`IndexError` exception.
 
+   The returned reference is borrowed from the tuple *p*
+   (that is: it is only valid as long as you hold a reference to *p*).
+   To get a :term:`strong reference`, use
+   :c:func:`Py_NewRef(PyTuple_GetItem(...)) <Py_NewRef>`
+   or :c:func:`PySequence_GetItem`.
+
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
 

Doc/c-api/typeobj.rst

@@ -1052,7 +1052,8 @@ and :c:data:`PyType_Type` effectively act as defaults.)
       the type, and the type object is INCREF'ed when a new instance is created, and
       DECREF'ed when an instance is destroyed (this does not apply to instances of
       subtypes; only the type referenced by the instance's ob_type gets INCREF'ed or
-      DECREF'ed).
+      DECREF'ed). Heap types should also :ref:`support garbage collection <supporting-cycle-detection>`
+      as they can form a reference cycle with their own module object.
 
       **Inheritance:**
 

Doc/c-api/weakref.rst

@@ -35,7 +35,7 @@ as much as it can.
    callable object that receives notification when *ob* is garbage collected; it
    should accept a single parameter, which will be the weak reference object
    itself. *callback* may also be ``None`` or ``NULL``.  If *ob* is not a
-   weakly referencable object, or if *callback* is not callable, ``None``, or
+   weakly referenceable object, or if *callback* is not callable, ``None``, or
    ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 
@@ -47,7 +47,7 @@ as much as it can.
    be a callable object that receives notification when *ob* is garbage
    collected; it should accept a single parameter, which will be the weak
    reference object itself. *callback* may also be ``None`` or ``NULL``.  If *ob*
-   is not a weakly referencable object, or if *callback* is not callable,
+   is not a weakly referenceable object, or if *callback* is not callable,
    ``None``, or ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 

Doc/conf.py

@@ -12,6 +12,8 @@
 sys.path.append(os.path.abspath('tools/extensions'))
 sys.path.append(os.path.abspath('includes'))
 
+from pyspecific import SOURCE_URI
+
 # General configuration
 # ---------------------
 
@@ -24,6 +26,7 @@
     'pyspecific',
     'sphinx.ext.coverage',
     'sphinx.ext.doctest',
+    'sphinx.ext.extlinks',
 ]
 
 # Skip if downstream redistributors haven't installed it
@@ -280,8 +283,8 @@
     'languages': ['ja', 'fr', 'zh_TW', 'zh_CN'], 'builders': ['man', 'text'],
 }
 
-# Avoid a warning with Sphinx >= 2.0
-master_doc = 'contents'
+# Avoid a warning with Sphinx >= 4.0
+root_doc = 'contents'
 
 # Allow translation of index directives
 gettext_additional_targets = [
@@ -355,6 +358,8 @@
 # Split the index
 html_split_index = True
 
+# Split pot files one per reST file
+gettext_compact = False
 
 # Options for LaTeX output
 # ------------------------
@@ -416,6 +421,10 @@
 epub_author = 'Python Documentation Authors'
 epub_publisher = 'Python Software Foundation'
 
+# index pages are not valid xhtml
+# https://github.com/sphinx-doc/sphinx/issues/12359
+epub_use_index = False
+
 # Options for the coverage checker
 # --------------------------------
 
@@ -499,6 +508,19 @@
     r'https://unix.org/version2/whatsnew/lp64_wp.html',
 ]
 
+# Options for sphinx.ext.extlinks
+# -------------------------------
+
+# This config is a dictionary of external sites,
+# mapping unique short aliases to a base URL and a prefix.
+# https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html
+extlinks = {
+    "cve": ("https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-%s", "CVE-%s"),
+    "cwe": ("https://cwe.mitre.org/data/definitions/%s.html", "CWE-%s"),
+    "pypi": ("https://pypi.org/project/%s/", "%s"),
+    "source": (SOURCE_URI, "%s"),
+}
+extlinks_detect_hardcoded_links = True
 
 # Options for extensions
 # ----------------------