gh-107305: Update the C-API Docs for PEP 684 (gh-107324)

Author: ericsnowcurrently

Doc/c-api/init.rst

@@ -1226,7 +1226,95 @@ You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap`
 function. You can create and destroy them using the following functions:
 
 
-.. c:function:: PyThreadState* Py_NewInterpreter()
+.. c:type:: PyInterpreterConfig
+
+   Structure containing most parameters to configure a sub-interpreter.
+   Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and
+   never modified by the runtime.
+
+   .. versionadded:: 3.12
+
+   Structure fields:
+
+   .. c:member:: int use_main_obmalloc
+
+      If this is ``0`` then the sub-interpreter will use its own
+      "object" allocator state.
+      Otherwise it will use (share) the main interpreter's.
+
+      If this is ``0`` then
+      :c:member:`~PyInterpreterConfig.check_multi_interp_extensions`
+      must be ``1`` (non-zero).
+      If this is ``1`` then :c:member:`~PyInterpreterConfig.gil`
+      must not be :c:macro:`PyInterpreterConfig_OWN_GIL`.
+
+   .. c:member:: int allow_fork
+
+      If this is ``0`` then the runtime will not support forking the
+      process in any thread where the sub-interpreter is currently active.
+      Otherwise fork is unrestricted.
+
+      Note that the :mod:`subprocess` module still works
+      when fork is disallowed.
+
+   .. c:member:: int allow_exec
+
+      If this is ``0`` then the runtime will not support replacing the
+      current process via exec (e.g. :func:`os.execv`) in any thread
+      where the sub-interpreter is currently active.
+      Otherwise exec is unrestricted.
+
+      Note that the :mod:`subprocess` module still works
+      when exec is disallowed.
+
+   .. c:member:: int allow_threads
+
+      If this is ``0`` then the sub-interpreter's :mod:`threading` module
+      won't create threads.
+      Otherwise threads are allowed.
+
+   .. c:member:: int allow_daemon_threads
+
+      If this is ``0`` then the sub-interpreter's :mod:`threading` module
+      won't create daemon threads.
+      Otherwise daemon threads are allowed (as long as
+      :c:member:`~PyInterpreterConfig.allow_threads` is non-zero).
+
+   .. c:member:: int check_multi_interp_extensions
+
+      If this is ``0`` then all extension modules may be imported,
+      including legacy (single-phase init) modules,
+      in any thread where the sub-interpreter is currently active.
+      Otherwise only multi-phase init extension modules
+      (see :pep:`489`) may be imported.
+
+      This must be ``1`` (non-zero) if
+      :c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``.
+
+   .. c:member:: int gil
+
+      This determines the operation of the GIL for the sub-interpreter.
+      It may be one of the following:
+
+      .. c:namespace:: NULL
+
+      .. c:macro:: PyInterpreterConfig_DEFAULT_GIL
+
+         Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`).
+
+      .. c:macro:: PyInterpreterConfig_SHARED_GIL
+
+         Use (share) the main interpreter's GIL.
+
+      .. c:macro:: PyInterpreterConfig_OWN_GIL
+
+         Use the sub-interpreter's own GIL.
+
+      If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then
+      :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``.
+
+
+.. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config)
 
    .. index::
       pair: module; builtins
@@ -1246,16 +1334,47 @@ function. You can create and destroy them using the following functions:
    ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying
    file descriptors).
 
-   The return value points to the first thread state created in the new
+   The given *config* controls the options with which the interpreter
+   is initialized.
+
+   Upon success, *tstate_p* will be set to the first thread state
+   created in the new
    sub-interpreter.  This thread state is made in the current thread state.
    Note that no actual thread is created; see the discussion of thread states
-   below.  If creation of the new interpreter is unsuccessful, ``NULL`` is
-   returned; no exception is set since the exception state is stored in the
-   current thread state and there may not be a current thread state.  (Like all
-   other Python/C API functions, the global interpreter lock must be held before
-   calling this function and is still held when it returns; however, unlike most
-   other Python/C API functions, there needn't be a current thread state on
-   entry.)
+   below.  If creation of the new interpreter is unsuccessful,
+   *tstate_p* is set to ``NULL``;
+   no exception is set since the exception state is stored in the
+   current thread state and there may not be a current thread state.
+
+   Like all other Python/C API functions, the global interpreter lock
+   must be held before calling this function and is still held when it
+   returns.  Likewise a current thread state must be set on entry.  On
+   success, the returned thread state will be set as current.  If the
+   sub-interpreter is created with its own GIL then the GIL of the
+   calling interpreter will be released.  When the function returns,
+   the new interpreter's GIL will be held by the current thread and
+   the previously interpreter's GIL will remain released here.
+
+   .. versionadded:: 3.12
+
+   Sub-interpreters are most effective when isolated from each other,
+   with certain functionality restricted::
+
+      PyInterpreterConfig config = {
+          .use_main_obmalloc = 0,
+          .allow_fork = 0,
+          .allow_exec = 0,
+          .allow_threads = 1,
+          .allow_daemon_threads = 0,
+          .check_multi_interp_extensions = 1,
+          .gil = PyInterpreterConfig_OWN_GIL,
+      };
+      PyThreadState *tstate = Py_NewInterpreterFromConfig(&config);
+
+   Note that the config is used only briefly and does not get modified.
+   During initialization the config's values are converted into various
+   :c:type:`PyInterpreterState` values.  A read-only copy of the config
+   may be stored internally on the :c:type:`PyInterpreterState`.
 
    .. index::
       single: Py_FinalizeEx()
@@ -1290,19 +1409,79 @@ function. You can create and destroy them using the following functions:
    .. index:: single: close() (in module os)
 
 
+.. c:function:: PyThreadState* Py_NewInterpreter(void)
+
+   .. index::
+      pair: module; builtins
+      pair: module; __main__
+      pair: module; sys
+      single: stdout (in module sys)
+      single: stderr (in module sys)
+      single: stdin (in module sys)
+
+   Create a new sub-interpreter.  This is essentially just a wrapper
+   around :c:func:`Py_NewInterpreterFromConfig` with a config that
+   preserves the existing behavior.  The result is an unisolated
+   sub-interpreter that shares the main interpreter's GIL, allows
+   fork/exec, allows daemon threads, and allows single-phase init
+   modules.
+
+
 .. c:function:: void Py_EndInterpreter(PyThreadState *tstate)
 
    .. index:: single: Py_FinalizeEx()
 
-   Destroy the (sub-)interpreter represented by the given thread state. The given
-   thread state must be the current thread state.  See the discussion of thread
-   states below.  When the call returns, the current thread state is ``NULL``.  All
-   thread states associated with this interpreter are destroyed.  (The global
-   interpreter lock must be held before calling this function and is still held
-   when it returns.)  :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
+   Destroy the (sub-)interpreter represented by the given thread state.
+   The given thread state must be the current thread state.  See the
+   discussion of thread states below.  When the call returns,
+   the current thread state is ``NULL``.  All thread states associated
+   with this interpreter are destroyed.  The global interpreter lock
+   used by the target interpreter must be held before calling this
+   function.  No GIL is held when it returns.
+
+   :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
    haven't been explicitly destroyed at that point.
 
 
+A Per-Interpreter GIL
+---------------------
+
+Using :c:func:`Py_NewInterpreterFromConfig` you can create
+a sub-interpreter that is completely isolated from other interpreters,
+including having its own GIL.  The most important benefit of this
+isolation is that such an interpreter can execute Python code without
+being blocked by other interpreters or blocking any others.  Thus a
+single Python process can truly take advantage of multiple CPU cores
+when running Python code.  The isolation also encourages a different
+approach to concurrency than that of just using threads.
+(See :pep:`554`.)
+
+Using an isolated interpreter requires vigilance in preserving that
+isolation.  That especially means not sharing any objects or mutable
+state without guarantees about thread-safety.  Even objects that are
+otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared
+because of the refcount.  One simple but less-efficient approach around
+this is to use a global lock around all use of some state (or object).
+Alternately, effectively immutable objects (like integers or strings)
+can be made safe in spite of their refcounts by making them "immortal".
+In fact, this has been done for the builtin singletons, small integers,
+and a number of other builtin objects.
+
+If you preserve isolation then you will have access to proper multi-core
+computing without the complications that come with free-threading.
+Failure to preserve isolation will expose you to the full consequences
+of free-threading, including races and hard-to-debug crashes.
+
+Aside from that, one of the main challenges of using multiple isolated
+interpreters is how to communicate between them safely (not break
+isolation) and efficiently.  The runtime and stdlib do not provide
+any standard approach to this yet.  A future stdlib module would help
+mitigate the effort of preserving isolation and expose effective tools
+for communicating (and sharing) data between interpreters.
+
+.. versionadded:: 3.12
+
+
 Bugs and caveats
 ----------------
 

Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst

@@ -0,0 +1,3 @@
+Add documentation for :c:type:`PyInterpreterConfig` and
+:c:func:`Py_NewInterpreterFromConfig`.  Also clarify some of the nearby docs
+relative to per-interpreter GIL.