Use of ffi_type_complex_double is unguarded

[zanieb]

Bug report

Bug description:

e.g.

   -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -Werror=unguarded-availability-new  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include     -c ./Modules/_ctypes/cfield.c -o Modules/_ctypes/cfield.o
./Modules/_ctypes/cfield.c:1610:36: error: 'ffi_type_complex_double' is only available on macOS 10.15 or newer [-Werror,-Wunguarded-availability-new]
        case 'C': fd->pffi_type = &ffi_type_complex_double; break;
                                   ^~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ffi/ffi.h:236:46: note: 'ffi_type_complex_double' has been marked as being introduced in macOS 10.15 here, but the deployment target is macOS 10.9.0
FFI_AVAILABLE_APPLE_2019 FFI_EXTERN ffi_type ffi_type_complex_double;
                                             ^
./Modules/_ctypes/cfield.c:1610:36: note: enclose 'ffi_type_complex_double' in a __builtin_available check to silence this warning
        case 'C': fd->pffi_type = &ffi_type_complex_double; break;
                                   ^~~~~~~~~~~~~~~~~~~~~~~
./Modules/_ctypes/cfield.c:1611:36: error: 'ffi_type_complex_float' is only available on macOS 10.15 or newer [-Werror,-Wunguarded-availability-new]
        case 'E': fd->pffi_type = &ffi_type_complex_float; break;
                                   ^~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ffi/ffi.h:235:46: note: 'ffi_type_complex_float' has been marked as being introduced in macOS 10.15 here, but the deployment target is macOS 10.9.0
FFI_AVAILABLE_APPLE_2019 FFI_EXTERN ffi_type ffi_type_complex_float;
                                             ^
./Modules/_ctypes/cfield.c:1611:36: note: enclose 'ffi_type_complex_float' in a __builtin_available check to silence this warning
        case 'E': fd->pffi_type = &ffi_type_complex_float; break;
                                   ^~~~~~~~~~~~~~~~~~~~~~
./Modules/_ctypes/cfield.c:1612:36: error: 'ffi_type_complex_double' is only available on macOS 10.15 or newer [-Werror,-Wung
uarded-availability-new]
        case 'F': fd->pffi_type = &ffi_type_complex_longdouble; break;
                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ffi/ffi.h:240:37: note: expanded from macro 'ffi_type_complex_longdouble'
#define ffi_type_complex_longdouble ffi_type_complex_double
                                    ^~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ffi/ffi.h:236:46: note: 'ffi_type_complex_double' has been marked as being introduced in macOS 10.15 here, but the deployment target is macOS 10.9.0
FFI_AVAILABLE_APPLE_2019 FFI_EXTERN ffi_type ffi_type_complex_double;
                                             ^
./Modules/_ctypes/cfield.c:1612:36: note: enclose 'ffi_type_complex_double' in a __builtin_available check to silence this warning
        case 'F': fd->pffi_type = &ffi_type_complex_longdouble; break;
                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/ffi/ffi.h:240:37: note: expanded from macro 'ffi_type_complex_longdouble'
#define ffi_type_complex_longdouble ffi_type_complex_double
                                    ^~~~~~~~~~~~~~~~~~~~~~~

See #100384 for context and similar issues.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

[zanieb]

I started taking a look at a patch for this, but it doesn't look trivial.

The existing checks use defined(Py_HAVE_C_COMPLEX) && defined(Py_FFI_SUPPORT_C_COMPLEX) and I think it needs to check defined(USING_APPLE_OS_LIBFFI) && defined(HAVE_FFI_TYPE_COMPLEX_DOUBLE_RUNTIME) where the latter is defined with

#ifdef HAVE_BUILTIN_AVAILABLE
#  define HAVE_FFI_TYPE_COMPLEX_DOUBLE_RUNTIME __builtin_available(macOS 10.15, *)
#endif

The ergonomic way to do this is beyond my knowledge, but I'm very happy to submit a patch if someone wants to provide some guidance.

[picnixz]

cc @skirpichev

[zanieb]

@picnixz you can also apply the OS-mac label, there are no runtime symbol checks for Linux yet.

Thank you for your quick triage :)

[skirpichev]

Looks as a false positive to me. I don't understand why we need MacOS-specific workaround when nothing is broken.

[zanieb]

@skirpichev won't CPython segfault when this function is called on macOS older than 10.15?

The typical pattern on macOS is to guard usage, e.g., in these similar situations:

• Availability of ptsname_r is not checked at runtime on macOS #123797
• Availability of utimensat, futimens not checked correctly on macOS #75782

This is also blocking adding a guard against future regressions, see #128155 (comment).

cc @ned-deily

[ned-deily]

won't CPython segfault when this function is called on macOS older than 10.15?

Context: this is for cases where CPython is built to run on a range of macOS releases, like we do for the python.org macOS installer builds. This involves building on a newer (current) version of macOS (like 15) but with MACOSX_DEPLOYMENT_TARGET set to an earlier version, say, 10.13.

[skirpichev]

won't CPython segfault when this function is called on macOS older than 10.15?

How? If libffi has no proper support for complex types, then configure script will not define Py_FFI_SUPPORT_C_COMPLEX.

Edit:

This involves building on a newer (current) version of macOS (like 15) but with MACOSX_DEPLOYMENT_TARGET set to an earlier version, say, 10.13.

I see. So, for such builds you want to revert compile-time decision on Py_FFI_SUPPORT_C_COMPLEX at runtime, right?

That will require substantial rewrite of complex numbers support in ctypes.

[skirpichev]

I would suggest rather unset Py_FFI_SUPPORT_C_COMPLEX for such builds at compile time.

[zanieb]

So, for such builds you want to revert compile-time decision on Py_FFI_SUPPORT_C_COMPLEX at runtime, right?

Basically, yes.

I would suggest rather unset Py_FFI_SUPPORT_C_COMPLEX for such builds at compile time.

That's a possibility, but weak symbols allow for portable builds which retain the functionality when available. This is common on macOS because there's robust support for it, but there's also interest in using weak symbols for Linux targets.

That will require substantial rewrite of complex numbers support in ctypes.

This is unfortunate. I'm not sure how to weigh the trade-offs here — but I'm curious if the idea that it's not macOS-specific changes your opinion?

Anyway, I don't think it's a great outcome to need to rewrite this module. I think an "easy" path forward here is to bump the minimum supported macOS version to 10.15 — it's not a big bump and @ned-deily had previously suggested this. There's some data on usage here.

We may need to revisit this module when adding weak symbols for Linux, but I have no clue how modern of library versions the functionality here depends on yet.

[skirpichev]

I'm curious if the idea that it's not macOS-specific

So far it doesn't look relevant for Linux builds. Unfortunately, here we can't rely just on existence of libffi type at runtime. See #125206.

an "easy" path forward here is to bump the minimum supported macOS version to 10.15

If this is an option - looks great.

[zanieb]

We've set the minimum in CI to 10.15 but formalizing that in a policy will be a separate task (see #128155 (comment)).

I can't tell if it makes sense to leave this issue open since ffi_type_complex_double usage is unguarded (and all of the other macOS 10.15 built-in usage is guarded). It sounds quite challenging to do anything about it though and we won't be enforcing guards for <=10.15 in the foreseeable future.