[ci] upgrade R to 4.0.5 in CI

[jameslamb]

R 4.0.5 ("Shake and Throw") was released two days ago (https://stat.ethz.ch/pipermail/r-announce/2021/000668.html).

This PR updates this project's continuous integration environments to use the new version. Linking #3763.

Release notes: https://cran.r-project.org/bin/windows/base/NEWS.R-4.0.5.html

[jameslamb]

On Mac, this is failing with the error observed in #4147 (comment)

==> Downloading http://mirror.ctan.org/systems/mac/mactex/mactex-basictex-20200407.pkg
==> Downloading from http://ctan.math.utah.edu/ctan/tex-archive/systems/mac/mactex/mactex-basictex-20200407.pkg
curl: (22) The requested URL returned error: 404 Not Found
Error: Download failed on Cask 'basictex' with message: Download failed: http://mirror.ctan.org/systems/mac/mactex/mactex-basictex-20200407.pkg

On ubuntu, looks I'm not sure what the root problem is. I tried removing --no-install-recommends and installing r-base-core explicitly, neither worked. Will have to experiment this weekend.

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 r-base-dev : Depends: r-base-core (>= 4.0.5-1.1804.0) but 4.0.4-1.2004.0 is to be installed
E: Unable to correct problems, you have held broken packages.

[StrikerRUS]

@jameslamb

On ubuntu, looks I'm not sure what the root problem is.

This is the same issue I faced during restricting CMake version recently.
#4104 (comment)
You have to (hope temporarily) explicitly specify dependency (r-base-core in this case) with version to resolve the issue.

[jameslamb]

You have to (hope temporarily) explicitly specify dependency (r-base-core in this case) with version to resolve the issue.

Right, I tried this and it did not work. Let me add more details on what I meant by "doesn't work".

Adding r-base-core 4.0.5 explicitly yields this:

The following packages have unmet dependencies:
 r-base-core : Depends: libicu60 (>= 60.1-1~) but it is not installable
               Depends: libreadline7 (>= 6.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

Trying to add libicu60 and libreadline7 explicitly yields this

Package libreadline7 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package libicu60 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libicu60' has no installation candidate
E: Package 'libreadline7' has no installation candidate

So maybe we just cannot update to R 4.0.5 on Ubuntu 20.04 yet. libicu60 and libreadline7 are not available on Ubuntu 20.04 (https://packages.ubuntu.com/search?suite=all&searchon=names&keywords=libicu60, https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libreadline7&searchon=names).

Hopefully if we check back in a few days, it will be possible.

[jameslamb]

Ok I just retried this locally and it seems like just installing r-base-dev should now work without any workarounds! Let's see if CI agrees with me.

To test locally, I replicated LightGBM's CI environment with the commands below. It isn't identical (since R jobs run in the Azure DevOps ubuntu-latest VM image) which is very different from the ubuntu:latest docker image, but I'm hoping that those differences are not meaningful for the purpose of apt resolving dependencies.

Pasting my testing setup here just in case it's useful as a reference in the future.

docker run \
    -v $(pwd):/opt/LightGBM \
    -w /opt/LightGBM \
    --env BUILD_DIRECTORY=/opt/LightGBM \
    --env COMPILER=gcc \
    --env GITHUB_ACTIONS="true" \
    --env IN_UBUNTU_LATEST_CONTAINER="true" \
    --env OS_NAME=linux \
    --env R_VERSION=4.0 \
    --env R_BUILD_TYPE=cran \
    --env SETUP_CONDA="true" \
    --env TASK=r-package \
    -it \
    ubuntu:latest \
    /bin/bash

apt update -y
apt install -y sudo
sudo apt-get install -y dirmngr
./.ci/setup.sh
./.ci/test.sh

[jameslamb]

Sad, this is still failing with the same error. Ok, I'll keep investigating locally.

[jameslamb]

Ok, I think I got this working! The trick was to switch the repository for R 4.0.5 to focal-cran40/.

---

All 3 CUDA jobs failed with no logs after running for more than hour. I just restarted them.

[StrikerRUS]

@jameslamb What do you think about the following trick?
BayesWitnesses/m2cgen@6eabad1
It allows to not change codename of Ubuntu in repo address each time we bump/downgrade/switch it.

[jameslamb]

@jameslamb What do you think about the following trick?
BayesWitnesses/m2cgen@6eabad1
It allows to not change codename of Ubuntu in repo address each time we bump/downgrade/switch it.

Nice trick! But that only works if you are not pinning the version of R (further down in that file, you have an unpinned r-base). That's problematic if you want to test on environments other than the most recent R, I think, since the version numbers have distribution information in them.

For example. 3.6.3-1bionic

LightGBM/.ci/test_r_package.sh

Line 33 in b674439

export R_LINUX_VERSION="3.6.3-1bionic"

So I think it makes sense for mc2gen since there's only a single R job and it always wants to get the newest-released R, but I don't think it would work cleanly here in LightGBM.

[jameslamb]

ok merging this one with admin rights, since lack of it is starting to cause others to fail (#4167 (comment)) and since if we rebuild it, it will fail on the check-docs task (fixed in #4168)

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.