Rscript is missing an essential piece of the PYTHONPATH on ubuntu-latest with Python 3.12

[glatterf42]

Sorry if this is the wrong place to ask about this, please let me know of a better one if it is.

Describe the bug
A few days ago, GitHub switched the ubuntu-latest pointer of their GitHub Actions runner images from Ubuntu 22.04 to 24.04 for our CI. Among other things, the 24.04 version does not come with R pre-installed anymore. Thus, we now rely entirely on the setup-r action that you provide, thanks for that :)
Our CI matrix runs on Macos, Ubuntu, and Windows and Python versions 3.8 through 3.12 and they all still work fine, except for the test on Ubuntu with Python 3.12. This test originally failed with the following error in a step run via Rscript:

 Error in IRkernel::installspec() : 
  jupyter-client has to be installed but “jupyter kernelspec --version” exited with code 1.

However, through our debugging PR, we found that the issue is not about IRkernel, after all. Instead, when examining the paths Python uses to find packages, we see that just before the Rscript step, these paths include '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages', but inside Rscript, this directory is missing from the list of paths. Since this is the location where we install various python packages, the python version inside Rscript fails to run since it can't find the packages it needs.

To Reproduce
To see a current example, pick any of the Tests here and take a look at the ubuntu-latest-py3.12 test.
The PR I mentioned above might also be useful because it provides some extra debugging information in its CI runs and comments.

To reproduce from scratch, run a GitHub Actions workflow on ubuntu-latest using Python 3.12, use actions/setup-python@v5 to install Python, use r-lib/actions/setup-r@v2, install some Python packages using pip, and run a step with shell: Rscript {0} using these packages (and/or pip).

Instead of doing that from scratch, feel free to open a PR to https://github.com/iiasa/ixmp, which will trigger all of these steps.

Expected behavior
The Python path inside Rscript should be identical to the one outside of it for all operating systems and Python versions.

Additional context
I find it curious that this issue only exists on Ubuntu, and only for Python 3.12. The other Ubuntu workflows also use ubuntu-latest, but a different Python version, which is handled correctly from Rscript (i.e. the Python path doesn't change).

[gaborcsardi]

What is the Python path with the bash shell?

[glatterf42]

Thanks for the quick reply :)

In the bash shell, we have

> python -c "import sys; print(sys.executable, sys.path)"
/opt/hostedtoolcache/Python/3.12.7/x64/bin/python ['', '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python312.zip', '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12', '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/lib-dynload', '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/site-packages']

whereas in Rscript, this becomes

> system2('python', '-c "import sys; print(sys.executable, sys.path)"', TRUE, TRUE)
[1] "/opt/hostedtoolcache/Python/3.12.7/x64/bin/python ['', '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python312.zip', '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12', '/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/lib-dynload']"

[gaborcsardi]

I suspect that this problem does not have much to do with r-lib/actions, see e.g. actions/runner-images#10781

[gaborcsardi]

But you were probably aware of that already. In any case, we don't remove anything from the path, and we don't do anything Python related at all, so I am a bit unsure how we could fix this. The Rscript path is whatever the system path is.

Maybe actions/setup-python@v5 sets up different paths or env vars for bash and other programs? Or there is a difference between login and non-login shells somewhere. I am not sure.

But I am also reluctant to hardcode some PYTHON env var or path into the setup-r action, that does not sound like a good solution.

[glatterf42]

Thanks for digging in, I wasn't aware of the issue you linked, in fact. However, this issue seems to be distinct from ours since we are already using a setup-python step.
Maybe the issue is hidden with these externally-managed Python installs, though. Maybe pip from our setup-python action only adds the site-packages directory to the Python path temporarily or so, I'll try to read up :)

[glatterf42]

Agreed that you shouldn't have to hardcode some env var or path into your action :)

[glatterf42]

As per actions/runner-images#10781 (comment), GitHub is rolling back the change of ubuntu-latest to ubuntu 24, so that should fix the issue on our side, at least for the time being. It seems to break several other things, unfortunately, but that would be a new issue if it relates to setup-r. Here's to hoping things go better when they actually switch to ubuntu 24. Thanks for your help!

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue and include a link to this issue