Combined PR to show ContextVar changes together

Doc/c-api/contextvars.rst

@@ -157,6 +157,14 @@ Context variable functions:
    a default value for the context variable, or ``NULL`` for no default.
    If an error has occurred, this function returns ``NULL``.
 
+.. c:function:: PyObject *PyContextVar_NewThreadInheritable(const char *name, PyObject *def)
+
+   Create a new ``ContextVar`` object whose bindings are inherited by new
+   :class:`threading.Thread` instances.  The parameters and return value are
+   the same as for :c:func:`PyContextVar_New`.
+
+   .. versionadded:: 3.16
+
 .. c:function:: int PyContextVar_Get(PyObject *var, PyObject *default_value, PyObject **value)
 
    Get the value of a context variable.  Returns ``-1`` if an error has

Doc/howto/free-threading-python.rst

@@ -152,8 +152,9 @@ is set to true by default which causes threads created with
 :class:`threading.Thread` to start with a copy of the
 :class:`~contextvars.Context()` of the caller of
 :meth:`~threading.Thread.start`.  In the default GIL-enabled build, the flag
-defaults to false so threads start with an
-empty :class:`~contextvars.Context()`.
+defaults to false, so threads start with a context containing only the caller's
+current bindings for context variables created by
+:meth:`~contextvars.ContextVar.thread_inheritable`.
 
 
 Warning filters

Doc/library/contextvars.rst

@@ -45,6 +45,72 @@ Context Variables
    :class:`!ContextVar`\s are :ref:`generic <generics>` over the type of
    their contained value.
 
+   .. classmethod:: ContextVar.thread_inheritable(name, [*, default])
+
+      Return a new context variable whose binding is inherited by new
+      :class:`threading.Thread` instances.  When
+      :meth:`threading.Thread.start` would otherwise start the thread with
+      an empty context (that is, when
+      :data:`sys.flags.thread_inherit_context` is false and no explicit
+      :class:`Context` was supplied), the new thread's context is initialized
+      with the caller's current bindings for all thread-inheritable variables.
+      This allows individual context variables to opt in to thread inheritance
+      on a per-variable basis.
+
+      These are ordinary bindings in the new thread's context: they are
+      visible to :func:`copy_context`, may be changed independently with
+      :meth:`ContextVar.set`, and are in turn inherited by threads started
+      from the new thread.  Threads started with an explicitly supplied
+      context are unaffected, as are threads started by means other than
+      :meth:`threading.Thread.start`, such as
+      :func:`_thread.start_new_thread` or the C API.
+
+      The *name* and *default* parameters have the same meaning as for the
+      :class:`ContextVar` constructor.  When
+      :data:`sys.flags.thread_inherit_context` is true, a variable created
+      with this method behaves identically to one created with
+      :class:`ContextVar`, so it is safe to use unconditionally.
+
+      Libraries that also support Python versions without this method must
+      choose how to degrade on those versions.  For state with a safe
+      default in a new thread (the behavior that :class:`threading.local`
+      based state has always had), fall back to an ordinary context
+      variable::
+
+          _new_var = getattr(ContextVar, "thread_inheritable", ContextVar)
+          var = _new_var("var")
+
+      With this fallback, on older versions new threads see the variable's
+      default rather than the starting thread's binding, unless the
+      application enables :option:`-X thread_inherit_context <-X>`.
+
+      For state that was previously a module-level global, reverting to the
+      default in new threads may break existing threaded code, since such
+      code has always seen the current global value.  Those libraries can
+      instead pick the best semantics each interpreter supports::
+
+          if hasattr(ContextVar, "thread_inheritable"):
+              # Context-local and inherited by new threads.
+              _new_var = ContextVar.thread_inheritable
+          elif getattr(sys.flags, "thread_inherit_context", False):
+              # Threads inherit the whole context, so an ordinary
+              # context variable is inherited too.
+              _new_var = ContextVar
+          else:
+              # Keep the library's historical global semantics.
+              _new_var = _GlobalVar
+
+      Here ``_GlobalVar`` is a class provided by the library that stores a
+      single process-wide value.  To be substitutable for
+      :class:`ContextVar` it must accept the same constructor arguments
+      (*name* positional, keyword-only *default*), distinguish a missing
+      *default* from ``default=None``, and provide ``get()``, ``set()``
+      returning a token, and ``reset()``.  This case gives up task-local
+      isolation: concurrent tasks and threads share one value, exactly as
+      they did before the library adopted context variables.
+
+      .. versionadded:: 3.16
+
    .. attribute:: ContextVar.name
 
       The name of the variable.  This is a read-only property.

Doc/library/threading.rst

@@ -539,12 +539,24 @@ since it is impossible to detect the termination of alien threads.
    the thread.  The default value is ``None`` which indicates that the
    :data:`sys.flags.thread_inherit_context` flag controls the behaviour.  If
    the flag is true, threads will start with a copy of the context of the
-   caller of :meth:`~Thread.start`.  If false, they will start with an empty
-   context.  To explicitly start with an empty context, pass a new instance of
+   caller of :meth:`~Thread.start`.  If false, they will start with a
+   context containing only the caller's current bindings for context variables
+   created by :meth:`~contextvars.ContextVar.thread_inheritable`.  To explicitly
+   start with an empty context, pass a new instance of
    :class:`~contextvars.Context()`.  To explicitly start with a copy of the
    current context, pass the value from :func:`~contextvars.copy_context`. The
    flag defaults true on free-threaded builds and false otherwise.
 
+   On GIL-enabled builds, when the flag has its default value of false and
+   *context* is ``None``, looking up a context variable that was set in the
+   caller of :meth:`~Thread.start` but is absent from the new thread's context
+   emits a :exc:`DeprecationWarning`.  In a future release, the flag will
+   default to true on all builds, so threads will inherit context by default.
+   Set
+   :option:`-X thread_inherit_context <-X>` or
+   :envvar:`PYTHON_THREAD_INHERIT_CONTEXT`, or pass an explicit *context*, to
+   select the behavior now and suppress this warning.
+
    If the subclass overrides the constructor, it must make sure to invoke the
    base class constructor (``Thread.__init__()``) before doing anything else to
    the thread.

Doc/using/cmdline.rst

@@ -675,11 +675,12 @@ Miscellaneous options
      .. versionadded:: 3.13
 
    * :samp:`-X thread_inherit_context={0,1}` causes :class:`~threading.Thread`
-     to, by default, use a copy of context of the caller of
-     ``Thread.start()`` when starting.  Otherwise, threads will start
-     with an empty context.  If unset, the value of this option defaults
-     to ``1`` on free-threaded builds and to ``0`` otherwise.  See also
-     :envvar:`PYTHON_THREAD_INHERIT_CONTEXT`.
+     to, by default, use a copy of the context of the caller of
+     ``Thread.start()`` when starting.  Otherwise, threads start with a context
+     containing only the caller's current bindings for context variables
+     created by :meth:`~contextvars.ContextVar.thread_inheritable`.  If unset, the
+     value of this option defaults to ``1`` on free-threaded builds and to ``0``
+     otherwise.  See also :envvar:`PYTHON_THREAD_INHERIT_CONTEXT`.
 
      .. versionadded:: 3.14
 
@@ -1367,10 +1368,12 @@ conflict.
 .. envvar:: PYTHON_THREAD_INHERIT_CONTEXT
 
    If this variable is set to ``1`` then :class:`~threading.Thread` will,
-   by default, use a copy of context of the caller of ``Thread.start()``
-   when starting.  Otherwise, new threads will start with an empty context.
-   If unset, this variable defaults to ``1`` on free-threaded builds and to
-   ``0`` otherwise.  See also :option:`-X thread_inherit_context<-X>`.
+   by default, use a copy of the context of the caller of ``Thread.start()``
+   when starting.  Otherwise, new threads start with a context containing only
+   the caller's current bindings for context variables created by
+   :meth:`~contextvars.ContextVar.thread_inheritable`.  If unset, this variable
+   defaults to ``1`` on free-threaded builds and to ``0`` otherwise.  See also
+   :option:`-X thread_inherit_context<-X>`.
 
    .. versionadded:: 3.14
 

Include/cpython/context.h

@@ -68,6 +68,13 @@ PyAPI_FUNC(int) PyContext_ClearWatcher(int watcher_id);
 PyAPI_FUNC(PyObject *) PyContextVar_New(
     const char *name, PyObject *default_value);
 
+/* Create a new thread-inheritable context variable.
+
+   default_value can be NULL.
+*/
+PyAPI_FUNC(PyObject *) PyContextVar_NewThreadInheritable(
+    const char *name, PyObject *default_value);
+
 
 /* Get a value for the variable.
 

Include/internal/pycore_context.h

@@ -26,13 +26,26 @@ struct _pycontextobject {
     PyHamtObject *ctx_vars;
     PyObject *ctx_weakreflist;
     int ctx_entered;
+    // Redundant subset of ctx_vars holding only the bindings of
+    // thread-inheritable context variables (see
+    // ContextVar.thread_inheritable()).  Used to efficiently create the
+    // starting context of a new thread.
+    PyHamtObject *ctx_thread_inheritable_vars;
+    // used to emit warnings about thread_inherit_context
+    PyHamtObject *ctx_starter_vars;
+    // Nesting depth in the context inheritance tree.  Assigned at creation
+    // time: an empty/base context has depth 0, and a context produced by
+    // copying another (copy_context(), Context.copy(), thread/async context
+    // inheritance) has depth one greater than its source.
+    uint64_t ctx_depth;
 };
 
 
 struct _pycontextvarobject {
     PyObject_HEAD
     PyObject *var_name;
     PyObject *var_default;
+    char var_thread_inheritable;
 #ifndef Py_GIL_DISABLED
     PyObject *var_cached;
     uint64_t var_cached_tsid;
@@ -54,9 +67,14 @@ struct _pycontexttokenobject {
 // _testinternalcapi.hamt() used by tests.
 // Export for '_testcapi' shared extension
 PyAPI_FUNC(PyObject*) _PyContext_NewHamtForTests(void);
+PyAPI_FUNC(PyObject*) _PyContext_NewForThread(void);
 
 PyAPI_FUNC(int) _PyContext_Enter(PyThreadState *ts, PyObject *octx);
 PyAPI_FUNC(int) _PyContext_Exit(PyThreadState *ts, PyObject *octx);
 
+/* Return the depth (see struct _pycontextobject.ctx_depth) of the current
+   context, or 0 if there is no current context. */
+PyAPI_FUNC(uint64_t) _PyContext_CurrentDepth(void);
+
 
 #endif /* !Py_INTERNAL_CONTEXT_H */

Include/internal/pycore_initconfig.h

@@ -179,6 +179,7 @@ extern PyStatus _PyConfig_SetPyArgv(
     PyConfig *config,
     const _PyArgv *args);
 extern PyObject* _PyConfig_CreateXOptionsDict(const PyConfig *config);
+extern int _PyConfig_ThreadInheritContextWarn(const PyConfig *config);
 
 extern void _Py_DumpPathConfig(PyThreadState *tstate);
 

Include/internal/pycore_interp_structs.h

@@ -984,6 +984,9 @@ struct _is {
     // One bit is set for each non-NULL entry in code_watchers
     uint8_t active_code_watchers;
     uint8_t active_context_watchers;
+    // True if we should warn about threads not inheriting context from the
+    // starting thread.
+    bool thread_inherit_context_warn;
 
     struct _py_object_state object_state;
     struct _Py_unicode_state unicode;

Lib/_py_warnings.py

@@ -83,7 +83,8 @@ def _filters(self):
 _global_context = _GlobalContext()
 
 
-_warnings_context = _contextvars.ContextVar('warnings_context')
+_warnings_context = _contextvars.ContextVar.thread_inheritable(
+    'warnings_context')
 
 
 def _get_context():

Lib/_pydecimal.py

@@ -348,8 +348,9 @@ class FloatOperation(DecimalException, TypeError):
 # current context.
 
 import contextvars
+from _contextvars import _current_context_depth
 
-_current_context_var = contextvars.ContextVar('decimal_context')
+_current_context_var = contextvars.ContextVar.thread_inheritable('decimal_context')
 
 _context_attributes = frozenset(
     ['prec', 'Emin', 'Emax', 'capitals', 'clamp', 'rounding', 'flags', 'traps']
@@ -362,18 +363,32 @@ def getcontext():
     a new context and sets this thread's context.
     New contexts are copies of DefaultContext.
     """
+    cur_depth = _current_context_depth()
     try:
-        return _current_context_var.get()
+        context = _current_context_var.get()
     except LookupError:
         context = Context()
+        context._local_depth = cur_depth
         _current_context_var.set(context)
         return context
+    if context._local_depth != cur_depth:
+        # The context value was inherited from another task/thread.  Because
+        # the Context() instance is mutable, copy it to ensure that if it is
+        # changed, those changes are isolated from other tasks/threads.
+        context = context.copy()
+        context._local_depth = cur_depth
+        _current_context_var.set(context)
+    return context
+
 
 def setcontext(context):
     """Set this thread's context to context."""
     if context in (DefaultContext, BasicContext, ExtendedContext):
         context = context.copy()
         context.clear_flags()
+    # Mark the context as owned by the current context scope, so a following
+    # getcontext() returns this very object rather than a copy.
+    context._local_depth = _current_context_depth()
     _current_context_var.set(context)
 
 del contextvars        # Don't contaminate the namespace
@@ -3869,6 +3884,10 @@ class Context(object):
     clamp -  If 1, change exponents if too high (Default 0)
     """
 
+    # Depth of the contextvars context this object was bound into by
+    # getcontext()/setcontext().
+    _local_depth = 0
+
     def __init__(self, prec=None, rounding=None, Emin=None, Emax=None,
                        capitals=None, clamp=None, flags=None, traps=None,
                        _ignored_flags=None):
@@ -3951,6 +3970,8 @@ def __setattr__(self, name, value):
             return self._set_signal_dict(name, value)
         elif name == '_ignored_flags':
             return object.__setattr__(self, name, value)
+        elif name == '_local_depth':
+            return object.__setattr__(self, name, value)
         else:
             raise AttributeError(
                 "'decimal.Context' object has no attribute '%s'" % name)