Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race between PyMember_GetOne and PyMember_SetOne for _Py_T_OBJECT members under freethreading #128144

Open
hawkinsp opened this issue Dec 21, 2024 · 1 comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-bug An unexpected behavior, bug, or error

Comments

@hawkinsp
Copy link
Contributor

hawkinsp commented Dec 21, 2024

Bug report

Bug description:

PyMember_GetOne does not use an atomic read when reading a _Py_T_OBJECT member, even though PyMember_SetOne uses an atomic write when updating such a member.

To demonstrate this, I'm using builtin_function_or_method's __module__, which is an example of a writeable _Py_T_OBJECT member in CPython's codebase. I don't know what field exactly was the source of the race when I found it in another codebase; it may or may not have been the same.

To reproduce, run the following code under CPython 3.13.1t with thread-sanitizer enabled:

import concurrent.futures
import functools
import threading

num_threads = 10

def closure(b, c):
  b.wait()
  for _ in range(100):
    x = c.__module__
    c.__module__ = 42

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for i in range(100):
    b = threading.Barrier(num_threads)
    c = [].append
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b, c))

and you'll get this race:

WARNING: ThreadSanitizer: data race (pid=350546)
  Atomic write of size 8 at 0x7f03d4a7d1e0 by thread T1:
    #0 _Py_atomic_store_ptr_release /usr/local/google/home/phawkins/p/cpython/./Include/cpython/pyatomic_gcc.h:501:3 (python3.13+0x4a73a9) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 PyMember_SetOne /usr/local/google/home/phawkins/p/cpython/Python/structmember.c:308:9 (python3.13+0x4a73a9)
    #2 member_set /usr/local/google/home/phawkins/p/cpython/Objects/descrobject.c:238:12 (python3.13+0x1ff85a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #3 _PyObject_GenericSetAttrWithDict /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1748:19 (python3.13+0x297b4e) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #4 PyObject_GenericSetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1819:12 (python3.13+0x298667) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 PyObject_SetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1385:15 (python3.13+0x294f8a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #6 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:5488:27 (python3.13+0x3f3f6d) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #9 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #11 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #12 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #14 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #15 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #16 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #17 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #18 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #20 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #21 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #23 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #24 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #25 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

  Previous read of size 8 at 0x7f03d4a7d1e0 by thread T10:
    #0 PyMember_GetOne /usr/local/google/home/phawkins/p/cpython/Python/structmember.c:86:13 (python3.13+0x4a6c64) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 member_get /usr/local/google/home/phawkins/p/cpython/Objects/descrobject.c:179:12 (python3.13+0x1ff6ca) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #2 _PyObject_GenericGetAttrWithDict /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1635:19 (python3.13+0x295de1) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #3 PyObject_GenericGetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1717:12 (python3.13+0x295c32) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #4 PyObject_GetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1231:18 (python3.13+0x294597) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:3766:28 (python3.13+0x3eddf0) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #6 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #8 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #9 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #11 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #12 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #13 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #14 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #15 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #16 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #17 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #20 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #21 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #22 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #23 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #24 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

The fix is presumably to make the read atomic as well.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

@hawkinsp hawkinsp added the type-bug An unexpected behavior, bug, or error label Dec 21, 2024
@picnixz picnixz added topic-free-threading interpreter-core (Objects, Python, Grammar, and Parser dirs) infra CI, GitHub Actions, buildbots, Dependabot, etc. and removed infra CI, GitHub Actions, buildbots, Dependabot, etc. labels Dec 21, 2024
@colesbury
Copy link
Contributor

I think this is fixed in main with #123211.

It hasn't been backported to 3.13 because the change is fairly large and I haven't seen any crashes in real code due to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-free-threading type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants