'This environment is externally managed' python error

[TheMostDiligent]

Description

In MacOS-12 runner version 20231017.6, CMake configuration steps in our CI builds started to fail with the following error:

-- The ASM compiler identification is AppleClang
-- Found assembler: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Symbol prefix: 
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-brew-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-brew packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Notice:  A new release of pip is available: 23.2.1 -> 23.3
Notice:  To update, run: python3.12 -m pip install --upgrade pip
CMake Error at DiligentTools/RenderStateNotation/CMakeLists.txt:32 (message):
  /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12 -m pip
  install libclang failed with 1

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Failing runner:

Current runner version: '2.310.2'
Operating System
  macOS
  12.6.9
  21G726
Runner Image
  Image: macos-12
  Version: 20231017.6
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20231017.6/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20231017.6
Runner Image Provisioner
  2.0.312.1

OK runner:

Current runner version: '2.310.2'
Operating System
  macOS
  12.7
  21G816
Runner Image
  Image: macos-12
  Version: 20230921.4
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20230921.4/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230921.4
Runner Image Provisioner
  2.0.310.1

Is it regression?

Yes, 20231017.6. Worked OK in 20230921.4

Expected behavior

CMake configuration completes successfully, as it used to.

Actual behavior

CMake configuration fails.

Repro steps

For us the build fails on this commit: DiligentGraphics/DiligentFX@d544c7c

[sergei-pyshnoi]

Hello @TheMostDiligent , We will take a look

[TheMostDiligent]

After some digging I figured that this happens because python was updated to version 12 and it changed how packages are installed.
In particular, --break-system-packages command line option is required to use the old behavior.
Here are the links to relevant discussions:

How do I solve "error: externally-managed-environment" every time I use pip 3?
pip install -r requirements.txt is failing: "This environment is externally managed"

I don't think there is anything that can be done on the runner side, so I am closing this issue.
I changed the title so that anyone having the same issue can find it.

[AbdelmajidBen]

Homebrew wants you to create a virtual env so that pip-installed modules don’t break something in your system. Same is true if working with python installed by apt-get in Ubuntu for same reason.

[janbrasna]

Question is whether the marker file should be present at all in this context, see:

• Python 3.12 on Ubuntu 24.04 is Externally Managed - PIP is broken #10781 (comment)