Move rules_scala_dependencies to scala/deps.bzl

[mbland]

Description

Moves rules_scala_dependencies to scala/deps.bzl and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1.

• bazel_skylib: 1.4.1 => 1.7.1
• go: 1.19.5 => 1.23.4
• rules_cc: 0.0.6 => 0.0.9
• rules_go: 0.39.1 => 0.50.1
• rules_java: 7.9.0 => 7.12.3
• rules_python: 0.36.0 => 0.38.0

The rules_java 7.12.13 bump precipitated the following changes:

• Adds the WORKSPACE stanza for rules_java in every WORKSPACE file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate rules_java repos and to register rules_java toolchains.

• Rearranges the other WORKSPACE setup macros for dependencies to come before the rules_scala setup macros.

• Updates almost all dependencies on @bazel_tools//tools/jdk: toolchain targets and .bzl files to corresponding targets and files in @rules_java//toolchains:.

• Removes several deprecated flags from the scrooge_compile_with_jdk_11 test case from test/shell/test_twitter_scrooge.sh, and one obsolete flag that caused it to break.

Motivation

Moving rules_scala_dependencies to scala/deps.bzl ensures we get all the versions of dependencies we want in WORKSPACE, while providing a new API to consumers. It also prevents WORKSPACE from transitively loading any .bzl files that load @rules_java, ensuring Bazel 8 compatibility per #1652.

Reasons for the other rules_java related changes include:

• The WORKSPACE stanza for rules_java should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds.

• Having the other WORKSPACE setup macros for dependencies come before the rules_scala setup macros helps ensure consistent, correct initialization before rules_scala initialization.

• Updating the toolchain specifiers to use @rules_java//toolchains fixed WORKSPACE build breakages when updating to rules_java 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (@bazel_tools//tools/jdk:toolchain_type dependencies remain, as there's not yet a corresponding @rules_java//toolchains target.)

The commit message contains extensive notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes.

As always, I'm happy to break this down further if desired. I've got a couple of branches with my original commits, so breaking it down wouldn't be a problem.

[hvadehra]

However, rules_cc 0.0.9 => 0.0.10 requires Bazel 7, which defines CcSharedLibraryHintInfo:

Note that recent rules_cc changes/releases were not monotonic wrt Bazel compatibility. This particular issue should be fixed with rules_cc 0.0.13 onwards (i.e after bazelbuild/rules_cc@56f4a8b#diff-8235959fa11cd01645fa8636fd5409e9d31ba9c1b3bcad48778106a195410aa9)

[mbland]

However, rules_cc 0.0.9 => 0.0.10 requires Bazel 7, which defines CcSharedLibraryHintInfo:

Note that recent rules_cc changes/releases were not monotonic wrt Bazel compatibility. This particular issue should be fixed with rules_cc 0.0.13 onwards (i.e after bazelbuild/rules_cc@56f4a8b#diff-8235959fa11cd01645fa8636fd5409e9d31ba9c1b3bcad48778106a195410aa9)

@hvadehra Thanks for the tip. I did create a commit to bump to rules_cc 0.0.13 and rules_python 0.40.0, and it passes all tests. But I'm not going to submit it, and here's why.

I noticed that bumping to rules_cc 0.0.14 failed with the following, because it removes the cc_proto_library symbol in bazelbuild/rules_cc@b15fed2:

$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: Traceback (most recent call last):
  File ".../external/remote_java_tools/BUILD",
  line 7, column 60, in <toplevel>
    load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library", "cc_proto_library")
Error: file '@rules_cc//cc:defs.bzl' does not contain symbol 'cc_proto_library'
[ ...snip... ]

But the interesting thing is that one of the removed lines was to cc_proto_library from @protobuf, not @com_google_protobuf, that was added in 0.0.13 by bazelbuild/rules_cc@013a082:

load("@protobuf//bazel:cc_proto_library.bzl", _cc_proto_library = "cc_proto_library")

The WORKSPACE in rules_scala imports the protobuf library as com_google_protobuf, not protobuf. So how does this work? Especially because applying rules_cc 0.0.13 and rules_python 0.0.38 in my Bzlmod branch does _not_work:

$ bazel build //{src,test,jmh,third_party,scala_proto}/...

ERROR: Skipping '//test/...':
  error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python~/python/defs.bzl:17:6:
  at .../external/rules_python~/python/py_binary.bzl:18:6:
  at .../external/rules_python~/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python~/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc~/cc/defs.bzl:16:6:
Label '@@protobuf~//bazel:cc_proto_library.bzl' is invalid
  because 'bazel' is not a package;
  perhaps you meant to put the colon here:
'@@protobuf~//:bazel/cc_proto_library.bzl'?

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python~/python/defs.bzl:17:6:
  at .../external/rules_python~/python/py_binary.bzl:18:6:
  at .../external/rules_python~/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python~/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc~/cc/defs.bzl:16:6:
Label '@@protobuf~//bazel:cc_proto_library.bzl' is invalid
  because 'bazel' is not a package;
  perhaps you meant to put the colon here:
'@@protobuf~//:bazel/cc_proto_library.bzl'?

The answer is because:

• Under Bzlmod, all the protobuf instances resolve to a single canonical instance, and I'm using single_version_override to pin it to 21.7.
• Under WORKSPACE, rules_scala_dependencies() imports version 21.7 as com_google_protobuf, and py_repositories() imports version 27.0 as protobuf: https://github.com/bazelbuild/rules_python/blob/0.38.0/python/private/py_repositories.bzl#L62-L67

It's also telling that rules_cc 0.0.15 and above break because it can't find @com_google_protobuf//bazel, because it's now looking in the v21.7 repo, not the v27.0 repo:

$ bazel build //{src,jmh,test,third_party,scala_proto}/...

ERROR: Skipping '//test/...':
  error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:23:6:
  at .../external/rules_python/python/py_test.bzl:18:6:
  at .../external/rules_python/python/private/py_test_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:cc_proto_library.bzl' is invalid
  because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/cc_proto_library.bzl'?
WARNING: Target pattern parsing failed.

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:23:6:
  at .../external/rules_python/python/py_test.bzl:18:6:
  at .../external/rules_python/python/private/py_test_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:cc_proto_library.bzl' is invalid
  because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/cc_proto_library.bzl'?

So for the time being, I'd like to stick to rules_cc 0.0.9 and rules_python 0.38.0, since that combination will work under both WORKSPACE and Bzlmod while we're using protobuf 21.7. After Bzlmodification lands in the next major release, the following Bazel 8 compatible release (which should land very quickly) can bump protobuf to >= 29.1 and update rules_cc, rules_python, rules_java, etc.

[mbland]

FYI @simuons and @liucijus: It looks like Bazel 8 became the latest release an hour ago. As a result, ./test_rules_scala (latest Bazel) on :ubuntu: Ubuntu 20.04 LTS is failing this PR after the commit I just pushed.

I'll file a PR I've filed #1671 to set that CI job to use to 7.x for now.

[mbland]

I just rebased this on #1671 to get the build passing again. That PR should get merged first.

[mbland]

Just rebased this after #1671 and #1672.