Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Bzlmod and add rules_scala to bazel-central-registry #1482

Open
sfc-gh-pbennes opened this issue Mar 16, 2023 · 40 comments · Fixed by #1619
Open

Support Bzlmod and add rules_scala to bazel-central-registry #1482

sfc-gh-pbennes opened this issue Mar 16, 2023 · 40 comments · Fixed by #1619

Comments

@sfc-gh-pbennes
Copy link

sfc-gh-pbennes commented Mar 16, 2023

Greetings!

It looks like rules_scala isn't part of the Bzlmod effort or added to bazel-central-registry yet. I've opened an issue both here and in https://github.com/bazelbuild/bazel-central-registry to request it: bazelbuild/bazel-central-registry#522

@chrislovecnm
Copy link

Just a note to anyone that starts the work native.register_toolchains is not supported when using bzlmod.

@sluongng
Copy link

@chrislovecnm
Copy link

chrislovecnm commented Jul 11, 2023

@sluongng
Copy link

Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.

So native.register_toolchains call from the extension's starlark does not work, but having the extension create @pythons_hub//:all and then MODULE.bazel calling register_toolchains("@pythons_hub//:all") worked. 👍

@mateuszkuta256
Copy link
Contributor

I prepared a minimal bzlmod support here and gonna split it into a few PRs
TODOs after merging it:

  • write documentation on how to use rules_scala with bzlmod (some properties like SCALA_VERSION will probably be configurable via repo-env)
  • prepare release notes
  • contribute repo to the Bazel Central Registry
  • add tests that the project builds using both WORKSPACE and MODULE.bazel
  • migrate test/external workspaces to blzmod too

@pcj
Copy link
Member

pcj commented Jun 30, 2024

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

@mateuszkuta256
Copy link
Contributor

@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?

hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR

@mbland
Copy link
Contributor

mbland commented Oct 2, 2024

I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.

I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)

There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain WORKSPACE compatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintain WORKSPACE and cross-version compatibility.

For an example of what it looks like from a client perspective, here's what the MODULE.bazel stanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):

bazel_dep(name = "rules_scala", repo_name = "io_bazel_rules_scala")
local_path_override(
    module_name = "rules_scala",
    path = "../../bazelbuild/rules_scala"
)   

scala_dev_deps = use_extension(
    "@io_bazel_rules_scala//scala/extensions:deps.bzl",
    "scala_deps",
    dev_dependency = True,
)   
scala_dev_deps.toolchains(
    scalatest = True,
)

So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.

cc: @BillyAutrey @jayconrod @benjaminp @TheGrizzlyDev

mbland added a commit to mbland/rules_scala that referenced this issue Oct 3, 2024
This begins the Bzlmod compatibility migration by updating Bazel to
version 7.3.2 and adding initial `MODULE.bazel` and `WORKSPACE.bzlmod`
files.

Part of: bazelbuild#1482

Though Bzlmod remains disabled, updating to Bazel 7.3.2 requred updating
or adding the following packages to maintain `WORKSPACE` compatibility.

In `rules_scala_setup()`:

- bazel_skylib: 1.4.1 => 1.7.1
- rules_cc: 0.0.6 => 0.0.10
- rules_java: 5.4.1 => 7.9.0
- rules_proto: 5.3.0-21.7 => 6.0.2

Dev dependencies in `WORKSPACE`:

- com_google_protobuf: 28.2
- rules_pkg: 1.0.1
- rules_jvm_external: 6.4
- com_google_absl: abseil-cpp-20240722.0
- zlib: 1.3.1

Of all of the new, explicit dev dependencies, only `com_google_protobuf`
will be necessary to include in `MODULE.bazel`. The Bzlmod mechanism
will discover these other transitive dev dependencies automatically.

Also removed the `rules_java_extra` repo from `WORKSPACE`, which
appeared unused.

---

Though the current `rules_java` version is 7.12.1, and largely works
with this repo, it requires a few temporary workarounds. Rather than
commit the workarounds, upgrading only to 7.9.0 now seems less crufty.

What follows is a very detailed explanation of what happens with 7.12.1
with Bazel 7.3.2, just to have it on the record.

---

The workaround is to change a few toolchain and macro file targets from
`@bazel_tools//tools/jdk:` to `@rules_java//toolchains:`. This isn't a
terribly bad or invasive workaround, but `@bazel_tools//tools/jdk:` is
clearly the canonical path. Best to keep it that way, lest we build up
technical debt.

Without the workaround, these targets would fail:

- //test/src/main/resources/java_sources:CompiledWithJava11
- //test/src/main/resources/java_sources:CompiledWithJava8
- //test/toolchains:java21_toolchain
- //test:JunitRuntimePlatform
- //test:JunitRuntimePlatform_test_runner
- //test:scala_binary_jdk_11

with this error:

```txt
ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:

While resolving toolchains for target
@@rules_java_builtin//toolchains:platformclasspath (096dcc8):

No matching toolchains found for types
@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.
```

This appears to be a consequence of both upgrading the Bazel version
from 6.3.0 to 7.3.2 and updating `rules_java` to 7.12.1. The
`rules_java_builtin` repo is part of the `WORKSPACE` prefix that adds
implicit dependencies:

- https://bazel.build/external/migration#builtin-default-deps

This repo was added to 7.0.0-pre.20231011.2 in the following change,
mapped to `@rules_java` within the scope of the `@bazel_tools` repo:

- bazelbuild/bazel: Add rules_java_builtin to the users of Java modules
  bazelbuild/bazel@ff1abb2

This change tried to ensure `rules_java` remained compatible with
earlier Bazel versions. However, it changed all instances of
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` to
`//toolchains:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java: Make rules_java backwards compatible with Bazel
  6.3.0
  bazelbuild/rules_java@30ecf3f

Bazel has bumped `rules_java` in its `workspace_deps.bzl` from 7.9.0 to
7.11.0, but it's only available as of 8.0.0-pre.20240911.1.

- bazelbuild/bazel: Update rules_java 7.11.1 / java_tools 13.8
  bazelbuild/bazel#23571
  bazelbuild/bazel@f92124a

---

What I believe is happening is, under Bazel 7.3.2 and `rules_java`
7.12.1:

- Bazel creates `rules_java` 7.9.0 as `@rules_java_builtin` in the
  `WORKSPACE` prefix.

- `@bazel_tools` has `@rules_java` mapped to `@rules_java_builtin` when
  initialized during the `WORKSPACE` prefix, during which
  `@bazel_tools//tools/jdk` registers `alias()` targets to
  `@rules_java` toolchain targets. These aliased toolchains specify
  `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` in their
  `toolchains` attribute.

- `WORKSPACE` loads `@rules_java` 7.12.1 and registers all its
  toolchains with type
  `@rules_java//toolchains:bootstrap_runtime_toolchain_type`.

- Some `@rules_java` rules explicitly specifying toolchains from
  `@bazel_tools//tools/jdk` can't find them, because the
  `@bazel_tools//tools/jdk` toolchain aliases expect toolchains of type
  `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`.

This has broken other projects in the same way:

- bazelbuild/bazel: [Bazel CI] Downstream project broken by rules_java
  upgrade #23619
  bazelbuild/bazel#23619

These problems don't appear under Bzlmod, and `@rules_java_builtin` was
never required. This is because `WORKSPACE` executes its statements
sequentially, while Bzlmod builds the module dependency graph _before_
instantiating repositories (within module extensions).

It seems a fix is on the way that removes `@rules_java_builtin` from the
`WORKSPACE` prefix, and adds `@rules_java` to the suffix. At this
moment, though, it's not even in a prerelase:

- bazelbuild/bazel: Remove rules_java_builtin in WORKSPACE prefix
  bazelbuild/bazel@7506690

---

Note that the error message matches that from the following resolved
issue, but that issue was for non-Bzlmod child repos when `WORKSPACE`
was disabled.

- bazelbuild/bazel: Undefined @@rules_java_builtin repository with
  --noenable_workspace option
  bazelbuild/bazel#22754
mbland added a commit to mbland/rules_scala that referenced this issue Oct 5, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 5, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 6, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 7, 2024
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation
documented at:

- bazelbuild#1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into bazelbuild#1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
mbland added a commit to mbland/rules_scala that referenced this issue Oct 7, 2024
Part of bazelbuild#1482.

Splits the last component off of canonical repo names to produce the
expected repo name.

Without Bzlmod, it returns the original name. With Bzlmod enabled, it
avoids generating output like:

    scala_import(
        name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler",
        jars = ["scala-compiler-2.12.18.jar"],
    )

resulting in errors like:

```
ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD:
no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler':
target 'io_bazel_rules_scala_scala_compiler' not declared in package ''
defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD
and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider'
```

Also fixes the following error when attaching resources from custom repos to
targets under Bzlmod:

```txt
$ bazel test //test/src/main/scala/scalarules/test/resources:all

1) Scala library depending on resources from external resource-only
  jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest)
  java.lang.NullPointerException
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11)
```

Can be replaced with a future bazel-skylib implementation, if accepted
into that repo.

---

We can't rely on the specific canonical repository name format:

> Repos generated by extensions have canonical names in the form of
> `module_repo_canonical_name+extension_name+repo_name`. For extensions
> hosted in the root module, the `module_repo_canonical_name` part is
> replaced with the string `_main`. Note that the canonical name format is
> not an API you should depend on — it's subject to change at any time.
>
> - https://bazel.build/external/extension#repository_names_and_visibility

The change to no longer encode module versions in canonical repo names in
Bazel 7.1.0 is a recent example of Bazel maintainers altering the format:

- bazelbuild/bazel#21316

And the maintainers recently replaced the `~` delimiter with `+` in the
upcoming Bazel 8 release due to build performance issues on Windows:

- bazelbuild/bazel#22865

This function assumes the only valid `repo_name` characters are letters,
numbers, '_', '-', and '.'. It finds the last character not in this set, and
returns the contents of `name` following this character. This is valid so
long as this condition holds:

- https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
mbland added a commit to mbland/rules_scala that referenced this issue Oct 7, 2024
Part of bazelbuild#1482.

Splits the last component off of canonical repo names to produce the
expected repo name.

Without Bzlmod, it returns the original name. With Bzlmod enabled, it
avoids generating output like:

    scala_import(
        name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler",
        jars = ["scala-compiler-2.12.18.jar"],
    )

resulting in errors like:

```
ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD:
no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler':
target 'io_bazel_rules_scala_scala_compiler' not declared in package ''
defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD
and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider'
```

Also fixes the following error when attaching resources from custom repos to
targets under Bzlmod:

```txt
$ bazel test //test/src/main/scala/scalarules/test/resources:all

1) Scala library depending on resources from external resource-only
  jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest)
  java.lang.NullPointerException
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11)
```

Can be replaced with a future bazel-skylib implementation, if accepted
into that repo.

---

We can't rely on the specific canonical repository name format:

> Repos generated by extensions have canonical names in the form of
> `module_repo_canonical_name+extension_name+repo_name`. For extensions
> hosted in the root module, the `module_repo_canonical_name` part is
> replaced with the string `_main`. Note that the canonical name format is
> not an API you should depend on — it's subject to change at any time.
>
> - https://bazel.build/external/extension#repository_names_and_visibility

The change to no longer encode module versions in canonical repo names in
Bazel 7.1.0 is a recent example of Bazel maintainers altering the format:

- bazelbuild/bazel#21316

And the maintainers recently replaced the `~` delimiter with `+` in the
upcoming Bazel 8 release due to build performance issues on Windows:

- bazelbuild/bazel#22865

This function assumes the only valid `repo_name` characters are letters,
numbers, '_', '-', and '.'. It finds the last character not in this set, and
returns the contents of `name` following this character. This is valid so
long as this condition holds:

- https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
liucijus pushed a commit that referenced this issue Oct 8, 2024
Related to #1482, #1618, and #1619. Results from the investigation
documented at:

- #1619 (comment)

Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle
differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources`
values between Bazel 6 and Bazel 7.

Without this change, `_import_paths()` emits incorrect values under
Bazel 7, causing targets containing generated `.proto` inputs to fail,
e.g.  `//test/proto3:test_generated_proto`.

See also:

- Fix paths for sibling repository setup and generated .proto files
  bazelbuild/bazel@6c6c196

- The docstring for `ProtoInfo.proto_source_root` in the Bazel sources:
  https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172

- Remove incompatible_generated_protos_in_virtual_imports
  bazelbuild/bazel@3efaa32

- Comment from: Generated Protos are no longer considered as
  virtual_imports in Bazel 7
  bazelbuild/bazel#21075 (comment)

---

I cherrypicked this commit into #1618. While it fixed the
`//test/proto3` build failure, it does _not_ fix the hanging
scalapb_workers from the ProtoScalaPBRule aspect.

I'll have to investiate further whether than hang is related to Bazel,
rules_proto, com_google_protobuf, or some mixture thereof. Still,
progress!
@simuons simuons reopened this Oct 8, 2024
mbland added a commit to mbland/rules_scala that referenced this issue Oct 8, 2024
Part of bazelbuild#1482.

Splits the last component off of canonical repo names to produce the
expected repo name.

Without Bzlmod, it returns the original name. With Bzlmod enabled, it
avoids generating output like:

    scala_import(
        name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler",
        jars = ["scala-compiler-2.12.18.jar"],
    )

resulting in errors like:

```
ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD:
no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler':
target 'io_bazel_rules_scala_scala_compiler' not declared in package ''
defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD
and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider'
```

Also fixes the following error when attaching resources from custom repos to
targets under Bzlmod:

```txt
$ bazel test //test/src/main/scala/scalarules/test/resources:all

1) Scala library depending on resources from external resource-only
  jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest)
  java.lang.NullPointerException
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11)
    at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11)
```

Can be replaced with a future bazel-skylib implementation, if accepted
into that repo.

---

We can't rely on the specific canonical repository name format:

> Repos generated by extensions have canonical names in the form of
> `module_repo_canonical_name+extension_name+repo_name`. For extensions
> hosted in the root module, the `module_repo_canonical_name` part is
> replaced with the string `_main`. Note that the canonical name format is
> not an API you should depend on — it's subject to change at any time.
>
> - https://bazel.build/external/extension#repository_names_and_visibility

The change to no longer encode module versions in canonical repo names in
Bazel 7.1.0 is a recent example of Bazel maintainers altering the format:

- bazelbuild/bazel#21316

And the maintainers recently replaced the `~` delimiter with `+` in the
upcoming Bazel 8 release due to build performance issues on Windows:

- bazelbuild/bazel#22865

This function assumes the only valid `repo_name` characters are letters,
numbers, '_', '-', and '.'. It finds the last character not in this set, and
returns the contents of `name` following this character. This is valid so
long as this condition holds:

- https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
@mbland
Copy link
Contributor

mbland commented Oct 11, 2024

A quick update for visibility: I'm very close to having the Bzlmodified rules_scala passing 100% of the tests, down to the last couple of failures:

  • I need to replace the bind() calls from twitter_scrooge with alias() or some such to get test_version.sh to pass.
  • I need to fix the Scala 3.1.2 and Scala 3.3.3 failures for the Scalafmt targets from test/shell/test_cross_build.sh (corresponding to the test3 and library3 test cases in test_scalafmt()).

I also need to get test_lint.sh working again, and update the dt_patches repos for Bzlmod. But the following all pass:

  • test_rules_scala.sh
  • test_reproducibility.sh
  • test_examples.sh
  • test_coverage.sh

Hopefully I can get these fixed up today, and I'll start peeling off the next pull request or two. And thanks to @simuons and @liucijus for reviewing and merging #1619 and #1620 already.

mbland added a commit to mbland/rules_scala that referenced this issue Dec 5, 2024
Removes `native.{,new_}local_repository()` calls in macros in favor of
`local_repository` calls from `WORKSPACE`. Part of bazelbuild#1482.

`native.{,new_}local_repository()` isn't available under Bzlmod,
`@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified
definitions isn't available under Bazel 6, and Bazel 8 compatibility
work is imminent. Redefining the repositories in this way will be
compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod.
(`MODULE.bazel` will use a combination of `bazel_dep()` and
`local_path_override()`.)
mbland added a commit to mbland/rules_scala that referenced this issue Dec 5, 2024
This is part of the Bzlmod prep work from bazelbuild#1482, but also reduces
duplication between the `dt_patches/test_dt_patches*/WORKSPACE` files.

The structure of the `dt_patches/compiler_sources/extensions.bzl`
declarations accommodates the fact that, unlike `WORKSPACE,
`MODULE.bazel` files cannot import constants or contain conditional
statements. That is to say, there's no way to port the `if
SCALA_VERSION.startswith("2.")` expressions from the previous
`WORKSPACE` files to Bzlmod.

The new `import_compiler_source_repos` macro, however, works for both
`WORKSPACE` and Bzlmod builds, as it's trivial to wrap
`import_compiler_source_repos` in a module extension.
liucijus pushed a commit that referenced this issue Dec 8, 2024
This is part of the Bzlmod prep work from #1482, but also reduces
duplication between the `dt_patches/test_dt_patches*/WORKSPACE` files.

The structure of the `dt_patches/compiler_sources/extensions.bzl`
declarations accommodates the fact that, unlike `WORKSPACE,
`MODULE.bazel` files cannot import constants or contain conditional
statements. That is to say, there's no way to port the `if
SCALA_VERSION.startswith("2.")` expressions from the previous
`WORKSPACE` files to Bzlmod.

The new `import_compiler_source_repos` macro, however, works for both
`WORKSPACE` and Bzlmod builds, as it's trivial to wrap
`import_compiler_source_repos` in a module extension.
liucijus pushed a commit that referenced this issue Dec 8, 2024
Every Scala version now uses com.google.protobuf:protobuf-java:4.29.0.
Part of #1482 and #1652.

This is necessary, but not sufficient, to make `rules_scala` Bazel 8
compatible by enabling an update to `protobuf` v29.0. Building with
Bazel 6.5.0 and `protobuf` v21.7 continues to pass all tests after this
change.
liucijus pushed a commit that referenced this issue Dec 8, 2024
…e` (#1656)

* Extract dev_deps_repositories() from WORKSPACE

`dev_deps_repositories()` encapsulates the instantiation of repositories
used only for `rules_scala` development. Also removes the unused
`//private` package and its `WORKSPACE` statements. Part of #1482.

Both `WORKSPACE` and Bzlmod builds can use this macro, though how Bzlmod
will use it will depend on whether we continue building with Bazel 6.

`@bazel_tools//tools/build_defs/repo:local.bzl` isn't available under
Bazel 6. To continue building with Bazel 6 under Bzlmod, we will need to
call `dev_deps_repositories()` from `WORKSPACE.bzlmod` to continue using
`native.{,new_}local_repository()`.

If we switch to Bazel 7, we can load `local.bzl` and strip the `native.`
prefix from the `{,new_}local_repository()` calls. Then we can call
`dev_deps_repositories()` from a module extension instead of from
`WORKSPACE.bzlmod`.

Another alternative would be updating the local repositories to become
proper nested repositories. Then we can call `local_repository()` from
`WORKSPACE` and call `bazel_dep()` and `local_path_override()` from
`MODULE.bazel`. In that case, we'd remove the `{,new_}local_repository`
calls from `dev_deps_dependencies()`, and remove
`proto_cross_repo_boundary_repository()` entirely.

* Add test local_repository calls to WORKSPACE

Removes `native.{,new_}local_repository()` calls in macros in favor of
`local_repository` calls from `WORKSPACE`. Part of #1482.

`native.{,new_}local_repository()` isn't available under Bzlmod,
`@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified
definitions isn't available under Bazel 6, and Bazel 8 compatibility
work is imminent. Redefining the repositories in this way will be
compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod.
(`MODULE.bazel` will use a combination of `bazel_dep()` and
`local_path_override()`.)
mbland added a commit to mbland/rules_scala that referenced this issue Dec 8, 2024
Moves the Scalatest, JUnit, and Specs2 toolchains into
`@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482.

Updates all `WORKSPACE` files to set the appropriate `scala_toolchains`
parameters and to remove the unnecessary repository import and toolchain
registration macros.

Adds a `fetch_sources_by_id` parameter to `repositories` from
`third_party/repositories/repositories.bzl`. This enables
`scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping
from artifact ID lists returned by new macros extracted from `WORKSPACE`
macros. The values assigned to each id match the original
`fetch_sources` settings in the corresponding original `WORKSPACE`
macros.

Updates `scala/scala_maven_import_external.bzl` to generate a `load`
line for `//scala:scala_import.bzl` based on the repo's canonical name,
not `@io_bazel_rules_scala`.

As usual, includes several other opportunistic removals of the
`@io_bazel_rules_scala` repo name prefix to avoid an internal dependency
on that name. This means Bzlmod users won't necessarily have to set the
`repo_name` parameter of `bazel_dep` when using `rules_scala`.

---

Introduces more macros to return a framework's Maven artifact
dependencies, rather than inlining them in a `repositories` call. These
inlined lists are replaced by macro invocations, and now the
`scala_toolchains` macro can invoke these macros to collect artifact IDs
to pass to `repositories`. This also allows for future changes to
introduce a `scala_version` parameter if necessary, similar to how
`scalafmt_artifact_ids` already works.

This is important to avoid collisions when creating repositories for
artifacts upon which more than one framework depends under Bzlmod.
`WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will
produce errors like the following, where both `scala_proto` and
`twitter_scrooge` depend upon `io_bazel_rules_scala_guava`:

```txt
$ bazel build //src/...

ERROR: .../scala/scala_maven_import_external.bzl:299:24:
  Traceback (most recent call last):
    File ".../scala/extensions/deps.bzl",
      line 140, column 21, in _scala_deps_impl
        scala_toolchains(
    File ".../scala/private/macros/toolchains.bzl",
      line 140, column 17, in scala_toolchains
        _scrooge(
    File ".../twitter_scrooge/twitter_scrooge.bzl",
      line 96, column 17, in twitter_scrooge
        repositories(
    File ".../third_party/repositories/repositories.bzl",
      line 113, column 37, in repositories
        _scala_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 263, column 30, in scala_maven_import_external
        jvm_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 299, column 24, in jvm_maven_import_external
        jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs)

Error in repository_rule: A repo named
  io_bazel_rules_scala_guava_2_13_15
  is already generated by this module extension at
  .../scala/scala_maven_import_external.bzl:299:24

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/worker:worker_test'
  failed; build aborted:
  error evaluating module extension scala_deps in
  //scala/extensions:deps.bzl
```

Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it
easy to emit full direct dependency lists for artifacts included in
`third_party/repositories/scala_*.bzl`. This increases the likelihood of
collisions, since this expanded metadata forces the macros that
instantiate artifact repos to instantiate even more repos.

By fetching list of artifact IDs from these macros, `scala_toolchains`
can now consolidate them into dictionary keys. Then it passes these
unique keys to `repositories` directly, avoiding the problem of
instantiating the same repo multiple times in the same module extension.

This, in turn, also avoids the need to add parameters to the original
`WORKSPACE` macros that instantiate dependencies to avoid collisions
under Bzlmod. The `scala_toolchains` macro never needs to call these
original macros, under either `WORKSPACE` or Bzlmod.

Finally, it also reduces duplication between these artifact ID lists and
the `_*_DEPS` symbols originally from `testing/BUILD` (and now in
`testing/deps.bzl`). The dependency labels are now generated
programatically.

(Aside: As I mentioned, we may eventually need to pass a Scala version
argument to these macros. It will be possible to cross that bridge
without too much trouble if and when that day comes. Or I can try to
future proof it in a follow up pull request.)
mbland added a commit to mbland/rules_scala that referenced this issue Dec 8, 2024
Removes this symbol from `scala/scala.bzl` as well as
`setup_scala_testing_toolchain`, and deletes
`scala/private/macros/toolchains.bzl`. Part of bazelbuild#1482 and bazelbuild#1652.

This is required for Bazel 8 and `rules_java` 8 compatibility, but is
also compatible with Bazel 6 and 7. In bazelbuild#1652, @hvadehra suggested
partitioning the `.bzl` files such that `WORKSPACE` doesn't `load` a
file that tries to `load` symbols from `rules_java`. I successfully did
so in a separate branch, and along with other minor changes, got
`rules_scala` to build with `rules_java` 8.5.1.

The other changes will come in separate pull requests, but it makes
sense to land this change now before adding any other toolchains to
`scala_toolchains`.

---

Arguably, we should remove all macros exported from `scala/scala.bzl`
that only instantiate toolchain dependencies and define toolchains. That
may be a breaking change for some users, but will ultimately be
necessary for these macros to remain compatible with Bazel 8.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 8, 2024
Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain
targets and `.bzl` files to corresponding targets and files in
`@rules_java//toolchains:`. Part of bazelbuild#1482 and bazelbuild#1652.

All dependencies on `@bazel_tools//tools/jdk:toolchain_type` remain,
however, as there is not yet a corresponding target in
`@rules_java//toolchains:`.

Adds the `WORKSPACE` stanza recommended in the `rules_java` release
notes, and removes our own calls to instantiate `rules_java` repos and
to register `rules_java` toolchains.

These changes are required to avoid build breakages when updating to
`rules_scala` 7.10.0 and higher. All targets build and all tests pass
under Bazel 6.5.0 and 7.4.1.

---

I was on the right track in my analysis from bazelbuild#1619 ("Bump to
rules_java 7.9.0 for Bazel 7 compatibility" in the message for
commit cd22d88). However, I thought we
_shouldn't_ update any targets from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:`. That's why I thought we were stuck on
`rules_java` 7.9.0.

However, this comment from @fmeum made me think switching to
`@rules_java//toolchains:` actually is the preferred approach:

- bazelbuild/rules_java#214 (comment)

So this is a potentially breaking change, but in the good kind of way,
in that it requires an easy, future proof update.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 9, 2024
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several
dependencies as high as they can go and still be compatible with Bazel
6.5.0 and 7.4.1.

- `bazel_skylib`: 1.4.1 => 1.7.1
- `go`: 1.19.5 => 1.23.4
- `rules_cc`: 0.0.6 => 0.0.9
- `rules_go`: 0.39.1 => 0.50.1
- `rules_java`: 7.9.0 => 7.12.3
- `rules_python`: 0.36.0 => 0.38.0

The `rules_java` 7.12.13 bump precipitated the following changes:

- Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file
  per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3.
  This replaces previous calls to instantiate `rules_java` repos and to
  register `rules_java` toolchains.

- Rearranges the other `WORKSPACE` setup macros for dependencies to come
  before the `rules_scala` setup macros.

- Updates almost all dependencies on `@bazel_tools//tools/jdk:`
  toolchain targets and `.bzl` files to corresponding targets and files
  in `@rules_java//toolchains:`.

- Removes several deprecated flags from the
  `scrooge_compile_with_jdk_11` test case from
  `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that
  caused it to break.

---

Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all
the versions of dependencies we want in `WORKSPACE`, while providing a
new API to consumers. It also prevents `WORKSPACE` from transitively
loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8
compatibility per bazelbuild#1652.

Reasons for the other `rules_java` related changes include:

- The `WORKSPACE` stanza for `rules_java` should've already been present
  while using the existing version 7.9.0. However, doing so would've
  broken Bazel 6 builds, as described in detail below.

- Having the other `WORKSPACE` setup macros for dependencies come before
  the `rules_scala` setup macros helps ensure consistent, correct
  initialization before `rules_scala` initialization.

- Updating the toolchain specifiers to use `@rules_java//toolchains`
  fixed `WORKSPACE` build breakages when updating to `rules_java`
  7.10.0 and later. This is a potentially breaking change for consumers,
  but in the good kind of way, in that it requires an easy, futureproof
  update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain,
  as there's not yet a corresponding `@rules_java//toolchains` target.)

What follows are detailed notes on why some dependency versions are
capped where they are, and on some breakages fixed by these changes.

---

`abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7,
respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to
v25.5 is compatible with Bazel 6 provided users set the compiler flags
mentioned in that issue:

```txt
build:linux --cxxopt=-std=c++17
build:linux --host_cxxopt=-std=c++17
build:macos --cxxopt=-std=c++17
build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std=c++17
build:windows --host_cxxopt=/std=c++17
```

---

`rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10,
which introduced `cc/common/cc_info.bzl`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:19:6:
  Label '@rules_cc//cc/common:cc_info.bzl' is invalid
    because 'cc/common' is not a package;
    perhaps you meant to put the colon here:
    '@rules_cc//cc:common/cc_info.bzl'?
```

However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines
`CcSharedLibraryHintInfo`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33:
  name 'CcSharedLibraryHintInfo' is not defined (did you mean
  'CcSharedLibraryInfo'?)

[ ...snip... ]

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/common/cc_common.bzl:17:6:
  compilation of module 'cc/private/rules_impl/native.bzl' failed
```

---

This section provides a methodical description of the errors encountered
and resolved during each step of the `rules_java` update.

The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't
originally have in place, is:

```py
load(
    "@rules_java//java:repositories.bzl",
    "rules_java_dependencies",
    "rules_java_toolchains",
)

rules_java_dependencies()

rules_java_toolchains()
```

Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0
didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build.
Note the `CompiledWithJava{8,11}_java.jar` references:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: .../test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---

ERROR: .../test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---
```

Another variation of this I also saw was:

```txt
$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]
```

Updating the toolchains from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:` for targets in `test/BUILD` and
`test/src/main/resources/java_sources/BUILD` fixed this particular
issue. (More on updating other `@bazel_tools//tools/jdk` toolchain
targets below.) Bazel 6.5.0 then failed with the following expected
issue, fixed by `rules_java` 7.10.0 via:

- bazelbuild/rules_java#210
- bazelbuild/rules_java@30ecf3f

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34:
  Use of Starlark transition without allowlist attribute
  '_allowlist_function_transition'.
  See Starlark transitions documentation for details and usage:
  @rules_java//toolchains:java_toolchain_alias.bzl NORMAL

ERROR:
  .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12:
  While resolving toolchains for target
  //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter:
  invalid registered toolchain '//toolchains:all':
  while parsing '//toolchains:all':
  error loading package '@rules_java//toolchains':
  initialization of module 'toolchains/java_toolchain_alias.bzl' failed

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter'
  failed; build aborted:
```

Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused
Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619.
For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in
the message for commit cd22d88.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:
  While resolving toolchains for target
  @@rules_java_builtin//toolchains:platformclasspath (096dcc8):
  No matching toolchains found for types
  @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.

To debug, rerun with
  --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage
  reading https://bazel.build/concepts/platforms-intro.

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed
```

My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0
built into Bazel's `WORKSPACE` preamble registered
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains
that later versions of `rules_java` couldn't resolve. This was due to
the very same change that fixed 6.5.0, bazelbuild/rules_java#210.

However, I mistakenly thought we needed to keep
`@bazel_tools//tools/jdk:` as the canonical Java toolchains package
instead switching to `@rules_java//toolchains:`. That's why I originally
thought we were stuck on `rules_java` 7.9.0.

I eventually rediscovered the following issue, which surfaced this
problem a month before bazelbuild#1619. The conversation helped me realize that
switching to `@rules_java//toolchains:` actually is the preferred,
futureproof approach (short of migrating to Bzlmod):

- bazelbuild/rules_java: Regression with
  `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0
  bazelbuild/rules_java#214

Switching all `@bazel_tools//tools/jdk:` toolchains to
`@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and
7.4.1 to successfully build using `rules_java` 7.12.2. I then updated
`rules_java` to 7.12.3, which returns to registering some toolchains as
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java#246
- bazelbuild/rules_java@b64eb7d

@hvadehra requested that I try 7.12.3 to see if it resolved the issue.
I was able to build successfully using this version in a branch without
the toolchain updates from this commit.

- bazelbuild#1652 (comment)

However, the changes from this commit should remain futureproof when
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to
`@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one
day.

---

Several of the flags removed from the `scrooge_compile_with_jdk_11` test
case have been no-ops for years, so removing them eliminated their
corresponding deprecation warnings. `--javacopt='--release 11'`,
however, caused this breakage after originally bumping to `rules_java`
7.12.2:

```txt
$ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh

running test scrooge_compile_with_jdk_11
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured).
INFO: Found 64 targets...

ERROR:
  .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13:
  Compiling Java headers
  src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
  (1 source file) failed: (Exit 1): turbine_direct_graal failed:
  error executing command (from target
  //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions)

  external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal
    --output
    bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
    ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  java.lang.NullPointerException:
  attempted to use --release, but JAVA_HOME is not set
    at [email protected]/java.util.Objects.requireNonNull(Objects.java:259)
    at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55)
    at com.google.turbine.main.Main.release(Main.java:318)
    at com.google.turbine.main.Main.bootclasspath(Main.java:304)
    at com.google.turbine.main.Main.compile(Main.java:142)
    at com.google.turbine.main.Main.compile(Main.java:133)
    at com.google.turbine.main.Main.main(Main.java:89)
    at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

INFO: Elapsed time: 0.325s, Critical Path: 0.12s
INFO: 11 processes: 10 internal, 1 worker.
FAILED: Build did NOT complete successfully
 Test "scrooge_compile_with_jdk_11" failed  (0 sec)
```

See also:

- Bazel Blog: Bazel 5.0:
  https://blog.bazel.build/2022/01/19/bazel-5.0.html#java

- bazelbuild/bazel:
  `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain
  resolution for Java rules #7849
  bazelbuild/bazel#7849

---

Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8
will likely happen in a new major release _after_ a release containing
the Bzlmod changes. This is because:

- The Bzlmod changes alone will work with Bazel 6 without requiring that
  users update `protobuf` beyond version 21.7.

- Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add
  the afforementioned C++ compiler flags to support the newer
  `abseil-cpp` versions required by newer `protobuf` versions.

- `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or
  higher to support `protobuf` >= v28.0, but this ScalaPB version will
  _not_ support `protobuf` < v28.0.

The idea is that the next major release of `rules_scala` will help users
migrate to Bazel 7 and Bzlmod, in either order. Then the next major
release after that will enable them to migrate to Bazel 8, with all the
requisite dependency upgrades. (Technically it will still support
`WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See:

- bazelbuild#1482 (comment)

---

Here are the details explaining how `rules_java` 8 currently breaks
`rules_scala`, up until the point that upgrading to `protobuf` >= v29.0
would fix it.

`rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule`
API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java`
8.3.2.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/java/common/rules/android_lint.bzl:142:24:
  name 'subrule' is not defined (did you mean 'rule'?)

ERROR: Error computing the main repository mapping:
  at .../scala/private/extensions/dev_deps.bzl:8:6:
  at .../external/rules_java/java/repositories.bzl:20:6:
  at .../external/rules_java/toolchains/local_java_repository.bzl:17:6:
  at .../external/rules_java/java/defs.bzl:18:6:
  at .../external/rules_java/java/java_library.bzl:16:6:
  at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6:
  compilation of module 'java/common/rules/android_lint.bzl' failed
```

`rules_java` 8.3.0 is broken, as it can't find its own
`@compatibility_proxy` repo:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: error loading package under directory 'src':
  error loading package 'src/protobuf/io/bazel/rules_scala':
  at .../external/rules_java/java/defs.bzl:22:6:
  at .../external/rules_java/java/java_test.bzl:16:6:
  Unable to find package for @compatibility_proxy//:proxy.bzl:
  The repository '@compatibility_proxy' could not be resolved:
  Repository '@compatibility_proxy' is not defined.
```

`rules_java` 8.3.1 seems to fix this, presumably by importing the
`protobuf` repo as `com_google_protobuf`. However, it now requires at
least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`.
Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support
`protobuf` v28, abandoning users of previous `protobuf` versions or
forcing them to upgrade.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[...snip...]
ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/worker':
  at .../external/rules_java/java/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl'
  is invalid because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'?
```
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
This is a pretty straightforward and easy update on top of the previous
`setup_toolchains` and `scala_toolchains_repo` changes. Part of bazelbuild#1482.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482.

The most significant part of the change is moving all the toolchain
rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in
`scala_proto/toolchains.bzl`.

Adds the `scala_proto_deps_providers()` macro to replace
`//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in
the `dep_providers` parameter of `scala_proto_deps_toolchain()`.
Examples of this are in `test/proto/custom_generator/BUILD`.

Excludes `@scala_proto_rules_scalapb_protoc_gen` from
`DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl`
for Scala 2.11. For other Scala versions, this repo name will have the
Scala version appended. This is to avoid build failures under Bzlmod,
since:

- This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at
  ScalaPB 0.9.8.

- Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under
  Scala 2.11 results in an error under Bzlmod, as does importing it
  multiple times when configuring multiple Scala versions.

- `MODULE.bazel` can iterate over a list of Scala versions, filtering
  out Scala 2.11, and call `use_repo()` on each version specific repo.

A lot of the other changes are more opportunistic removals of
`@io_bazel_rules_scala` label prefixes and application of `Label()`
where appropriate. Doing this will allow Bzlmod users to use
`rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in
`bazel_dep()`.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Adds the jmh toolchain to `scala_toolchains()` and moves
`jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8
compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
This is the last of the toolchain to receive the "toolchainization"
treatment prior to Bzlmodification, and moves `twitter_scrooge()` to
`twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8
compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several
dependencies as high as they can go and still be compatible with Bazel
6.5.0 and 7.4.1.

- `bazel_skylib`: 1.4.1 => 1.7.1
- `go`: 1.19.5 => 1.23.4
- `rules_cc`: 0.0.6 => 0.0.9
- `rules_go`: 0.39.1 => 0.50.1
- `rules_java`: 7.9.0 => 7.12.3
- `rules_python`: 0.36.0 => 0.38.0

The `rules_java` 7.12.13 bump precipitated the following changes:

- Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file
  per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3.
  This replaces previous calls to instantiate `rules_java` repos and to
  register `rules_java` toolchains.

- Rearranges the other `WORKSPACE` setup macros for dependencies to come
  before the `rules_scala` setup macros.

- Updates almost all dependencies on `@bazel_tools//tools/jdk:`
  toolchain targets and `.bzl` files to corresponding targets and files
  in `@rules_java//toolchains:`.

- Removes several deprecated flags from the
  `scrooge_compile_with_jdk_11` test case from
  `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that
  caused it to break.

---

Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all
the versions of dependencies we want in `WORKSPACE`, while providing a
new API to consumers. It also prevents `WORKSPACE` from transitively
loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8
compatibility per bazelbuild#1652.

Reasons for the other `rules_java` related changes include:

- The `WORKSPACE` stanza for `rules_java` should've already been present
  while using the existing version 7.9.0. However, doing so would've
  broken Bazel 6 builds, as described in detail below.

- Having the other `WORKSPACE` setup macros for dependencies come before
  the `rules_scala` setup macros helps ensure consistent, correct
  initialization before `rules_scala` initialization.

- Updating the toolchain specifiers to use `@rules_java//toolchains`
  fixed `WORKSPACE` build breakages when updating to `rules_java`
  7.10.0 and later. This is a potentially breaking change for consumers,
  but in the good kind of way, in that it requires an easy, futureproof
  update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain,
  as there's not yet a corresponding `@rules_java//toolchains` target.)

What follows are detailed notes on why some dependency versions are
capped where they are, and on some breakages fixed by these changes.

---

`abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7,
respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to
v25.5 is compatible with Bazel 6 provided users set the compiler flags
mentioned in that issue:

```txt
build:linux --cxxopt=-std=c++17
build:linux --host_cxxopt=-std=c++17
build:macos --cxxopt=-std=c++17
build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std=c++17
build:windows --host_cxxopt=/std=c++17
```

---

`rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10,
which introduced `cc/common/cc_info.bzl`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:19:6:
  Label '@rules_cc//cc/common:cc_info.bzl' is invalid
    because 'cc/common' is not a package;
    perhaps you meant to put the colon here:
    '@rules_cc//cc:common/cc_info.bzl'?
```

However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines
`CcSharedLibraryHintInfo`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33:
  name 'CcSharedLibraryHintInfo' is not defined (did you mean
  'CcSharedLibraryInfo'?)

[ ...snip... ]

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/common/cc_common.bzl:17:6:
  compilation of module 'cc/private/rules_impl/native.bzl' failed
```

---

This section provides a methodical description of the errors encountered
and resolved during each step of the `rules_java` update.

The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't
originally have in place, is:

```py
load(
    "@rules_java//java:repositories.bzl",
    "rules_java_dependencies",
    "rules_java_toolchains",
)

rules_java_dependencies()

rules_java_toolchains()
```

Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0
didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build.
Note the `CompiledWithJava{8,11}_java.jar` references:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: .../test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---

ERROR: .../test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---
```

Another variation of this I also saw was:

```txt
$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]
```

Updating the toolchains from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:` for targets in `test/BUILD` and
`test/src/main/resources/java_sources/BUILD` fixed this particular
issue. (More on updating other `@bazel_tools//tools/jdk` toolchain
targets below.) Bazel 6.5.0 then failed with the following expected
issue, fixed by `rules_java` 7.10.0 via:

- bazelbuild/rules_java#210
- bazelbuild/rules_java@30ecf3f

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34:
  Use of Starlark transition without allowlist attribute
  '_allowlist_function_transition'.
  See Starlark transitions documentation for details and usage:
  @rules_java//toolchains:java_toolchain_alias.bzl NORMAL

ERROR:
  .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12:
  While resolving toolchains for target
  //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter:
  invalid registered toolchain '//toolchains:all':
  while parsing '//toolchains:all':
  error loading package '@rules_java//toolchains':
  initialization of module 'toolchains/java_toolchain_alias.bzl' failed

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter'
  failed; build aborted:
```

Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused
Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619.
For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in
the message for commit cd22d88.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:
  While resolving toolchains for target
  @@rules_java_builtin//toolchains:platformclasspath (096dcc8):
  No matching toolchains found for types
  @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.

To debug, rerun with
  --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage
  reading https://bazel.build/concepts/platforms-intro.

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed
```

My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0
built into Bazel's `WORKSPACE` preamble registered
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains
that later versions of `rules_java` couldn't resolve. This was due to
the very same change that fixed 6.5.0, bazelbuild/rules_java#210.

However, I mistakenly thought we needed to keep
`@bazel_tools//tools/jdk:` as the canonical Java toolchains package
instead switching to `@rules_java//toolchains:`. That's why I originally
thought we were stuck on `rules_java` 7.9.0.

I eventually rediscovered the following issue, which surfaced this
problem a month before bazelbuild#1619. The conversation helped me realize that
switching to `@rules_java//toolchains:` actually is the preferred,
futureproof approach (short of migrating to Bzlmod):

- bazelbuild/rules_java: Regression with
  `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0
  bazelbuild/rules_java#214

Switching all `@bazel_tools//tools/jdk:` toolchains to
`@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and
7.4.1 to successfully build using `rules_java` 7.12.2. I then updated
`rules_java` to 7.12.3, which returns to registering some toolchains as
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java#246
- bazelbuild/rules_java@b64eb7d

@hvadehra requested that I try 7.12.3 to see if it resolved the issue.
I was able to build successfully using this version in a branch without
the toolchain updates from this commit.

- bazelbuild#1652 (comment)

However, the changes from this commit should remain futureproof when
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to
`@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one
day.

---

Several of the flags removed from the `scrooge_compile_with_jdk_11` test
case have been no-ops for years, so removing them eliminated their
corresponding deprecation warnings. `--javacopt='--release 11'`,
however, caused this breakage after originally bumping to `rules_java`
7.12.2:

```txt
$ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh

running test scrooge_compile_with_jdk_11
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured).
INFO: Found 64 targets...

ERROR:
  .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13:
  Compiling Java headers
  src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
  (1 source file) failed: (Exit 1): turbine_direct_graal failed:
  error executing command (from target
  //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions)

  external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal
    --output
    bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
    ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  java.lang.NullPointerException:
  attempted to use --release, but JAVA_HOME is not set
    at [email protected]/java.util.Objects.requireNonNull(Objects.java:259)
    at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55)
    at com.google.turbine.main.Main.release(Main.java:318)
    at com.google.turbine.main.Main.bootclasspath(Main.java:304)
    at com.google.turbine.main.Main.compile(Main.java:142)
    at com.google.turbine.main.Main.compile(Main.java:133)
    at com.google.turbine.main.Main.main(Main.java:89)
    at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

INFO: Elapsed time: 0.325s, Critical Path: 0.12s
INFO: 11 processes: 10 internal, 1 worker.
FAILED: Build did NOT complete successfully
 Test "scrooge_compile_with_jdk_11" failed  (0 sec)
```

See also:

- Bazel Blog: Bazel 5.0:
  https://blog.bazel.build/2022/01/19/bazel-5.0.html#java

- bazelbuild/bazel:
  `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain
  resolution for Java rules #7849
  bazelbuild/bazel#7849

---

Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8
will likely happen in a new major release _after_ a release containing
the Bzlmod changes. This is because:

- The Bzlmod changes alone will work with Bazel 6 without requiring that
  users update `protobuf` beyond version 21.7.

- Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add
  the afforementioned C++ compiler flags to support the newer
  `abseil-cpp` versions required by newer `protobuf` versions.

- `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or
  higher to support `protobuf` >= v28.0, but this ScalaPB version will
  _not_ support `protobuf` < v28.0.

The idea is that the next major release of `rules_scala` will help users
migrate to Bazel 7 and Bzlmod, in either order. Then the next major
release after that will enable them to migrate to Bazel 8, with all the
requisite dependency upgrades. (Technically it will still support
`WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See:

- bazelbuild#1482 (comment)

---

Here are the details explaining how `rules_java` 8 currently breaks
`rules_scala`, up until the point that upgrading to `protobuf` >= v29.0
would fix it.

`rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule`
API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java`
8.3.2.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/java/common/rules/android_lint.bzl:142:24:
  name 'subrule' is not defined (did you mean 'rule'?)

ERROR: Error computing the main repository mapping:
  at .../scala/private/extensions/dev_deps.bzl:8:6:
  at .../external/rules_java/java/repositories.bzl:20:6:
  at .../external/rules_java/toolchains/local_java_repository.bzl:17:6:
  at .../external/rules_java/java/defs.bzl:18:6:
  at .../external/rules_java/java/java_library.bzl:16:6:
  at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6:
  compilation of module 'java/common/rules/android_lint.bzl' failed
```

`rules_java` 8.3.0 is broken, as it can't find its own
`@compatibility_proxy` repo:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: error loading package under directory 'src':
  error loading package 'src/protobuf/io/bazel/rules_scala':
  at .../external/rules_java/java/defs.bzl:22:6:
  at .../external/rules_java/java/java_test.bzl:16:6:
  Unable to find package for @compatibility_proxy//:proxy.bzl:
  The repository '@compatibility_proxy' could not be resolved:
  Repository '@compatibility_proxy' is not defined.
```

`rules_java` 8.3.1 seems to fix this, presumably by importing the
`protobuf` repo as `com_google_protobuf`. However, it now requires at
least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`.
Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support
`protobuf` v28, abandoning users of previous `protobuf` versions or
forcing them to upgrade.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[...snip...]
ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/worker':
  at .../external/rules_java/java/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl'
  is invalid because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'?
```
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several
dependencies as high as they can go and still be compatible with Bazel
6.5.0 and 7.4.1.

- `bazel_skylib`: 1.4.1 => 1.7.1
- `go`: 1.19.5 => 1.23.4
- `rules_cc`: 0.0.6 => 0.0.9
- `rules_go`: 0.39.1 => 0.50.1
- `rules_java`: 7.9.0 => 7.12.3
- `rules_python`: 0.36.0 => 0.38.0

The `rules_java` 7.12.13 bump precipitated the following changes:

- Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file
  per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3.
  This replaces previous calls to instantiate `rules_java` repos and to
  register `rules_java` toolchains.

- Rearranges the other `WORKSPACE` setup macros for dependencies to come
  before the `rules_scala` setup macros.

- Updates almost all dependencies on `@bazel_tools//tools/jdk:`
  toolchain targets and `.bzl` files to corresponding targets and files
  in `@rules_java//toolchains:`.

- Removes several deprecated flags from the
  `scrooge_compile_with_jdk_11` test case from
  `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that
  caused it to break.

---

Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all
the versions of dependencies we want in `WORKSPACE`, while providing a
new API to consumers. It also prevents `WORKSPACE` from transitively
loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8
compatibility per bazelbuild#1652.

Reasons for the other `rules_java` related changes include:

- The `WORKSPACE` stanza for `rules_java` should've already been present
  while using the existing version 7.9.0. However, doing so would've
  broken Bazel 6 builds, as described in detail below.

- Having the other `WORKSPACE` setup macros for dependencies come before
  the `rules_scala` setup macros helps ensure consistent, correct
  initialization before `rules_scala` initialization.

- Updating the toolchain specifiers to use `@rules_java//toolchains`
  fixed `WORKSPACE` build breakages when updating to `rules_java`
  7.10.0 and later. This is a potentially breaking change for consumers,
  but in the good kind of way, in that it requires an easy, futureproof
  update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain,
  as there's not yet a corresponding `@rules_java//toolchains` target.)

What follows are detailed notes on why some dependency versions are
capped where they are, and on some breakages fixed by these changes.

---

`abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7,
respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to
v25.5 is compatible with Bazel 6 provided users set the compiler flags
mentioned in that issue:

```txt
build:linux --cxxopt=-std=c++17
build:linux --host_cxxopt=-std=c++17
build:macos --cxxopt=-std=c++17
build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std=c++17
build:windows --host_cxxopt=/std=c++17
```

---

`rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10,
which introduced `cc/common/cc_info.bzl`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:19:6:
  Label '@rules_cc//cc/common:cc_info.bzl' is invalid
    because 'cc/common' is not a package;
    perhaps you meant to put the colon here:
    '@rules_cc//cc:common/cc_info.bzl'?
```

However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines
`CcSharedLibraryHintInfo`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33:
  name 'CcSharedLibraryHintInfo' is not defined (did you mean
  'CcSharedLibraryInfo'?)

[ ...snip... ]

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/common/cc_common.bzl:17:6:
  compilation of module 'cc/private/rules_impl/native.bzl' failed
```

---

This section provides a methodical description of the errors encountered
and resolved during each step of the `rules_java` update.

The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't
originally have in place, is:

```py
load(
    "@rules_java//java:repositories.bzl",
    "rules_java_dependencies",
    "rules_java_toolchains",
)

rules_java_dependencies()

rules_java_toolchains()
```

Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0
didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build.
Note the `CompiledWithJava{8,11}_java.jar` references:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: .../test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---

ERROR: .../test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---
```

Another variation of this I also saw was:

```txt
$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]
```

Updating the toolchains from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:` for targets in `test/BUILD` and
`test/src/main/resources/java_sources/BUILD` fixed this particular
issue. (More on updating other `@bazel_tools//tools/jdk` toolchain
targets below.) Bazel 6.5.0 then failed with the following expected
issue, fixed by `rules_java` 7.10.0 via:

- bazelbuild/rules_java#210
- bazelbuild/rules_java@30ecf3f

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34:
  Use of Starlark transition without allowlist attribute
  '_allowlist_function_transition'.
  See Starlark transitions documentation for details and usage:
  @rules_java//toolchains:java_toolchain_alias.bzl NORMAL

ERROR:
  .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12:
  While resolving toolchains for target
  //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter:
  invalid registered toolchain '//toolchains:all':
  while parsing '//toolchains:all':
  error loading package '@rules_java//toolchains':
  initialization of module 'toolchains/java_toolchain_alias.bzl' failed

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter'
  failed; build aborted:
```

Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused
Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619.
For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in
the message for commit cd22d88.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:
  While resolving toolchains for target
  @@rules_java_builtin//toolchains:platformclasspath (096dcc8):
  No matching toolchains found for types
  @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.

To debug, rerun with
  --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage
  reading https://bazel.build/concepts/platforms-intro.

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed
```

My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0
built into Bazel's `WORKSPACE` preamble registered
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains
that later versions of `rules_java` couldn't resolve. This was due to
the very same change that fixed 6.5.0, bazelbuild/rules_java#210.

However, I mistakenly thought we needed to keep
`@bazel_tools//tools/jdk:` as the canonical Java toolchains package
instead switching to `@rules_java//toolchains:`. That's why I originally
thought we were stuck on `rules_java` 7.9.0.

I eventually rediscovered the following issue, which surfaced this
problem a month before bazelbuild#1619. The conversation helped me realize that
switching to `@rules_java//toolchains:` actually is the preferred,
futureproof approach (short of migrating to Bzlmod):

- bazelbuild/rules_java: Regression with
  `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0
  bazelbuild/rules_java#214

Switching all `@bazel_tools//tools/jdk:` toolchains to
`@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and
7.4.1 to successfully build using `rules_java` 7.12.2. I then updated
`rules_java` to 7.12.3, which returns to registering some toolchains as
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java#246
- bazelbuild/rules_java@b64eb7d

@hvadehra requested that I try 7.12.3 to see if it resolved the issue.
I was able to build successfully using this version in a branch without
the toolchain updates from this commit.

- bazelbuild#1652 (comment)

However, the changes from this commit should remain futureproof when
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to
`@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one
day.

---

Several of the flags removed from the `scrooge_compile_with_jdk_11` test
case have been no-ops for years, so removing them eliminated their
corresponding deprecation warnings. `--javacopt='--release 11'`,
however, caused this breakage after originally bumping to `rules_java`
7.12.2:

```txt
$ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh

running test scrooge_compile_with_jdk_11
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured).
INFO: Found 64 targets...

ERROR:
  .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13:
  Compiling Java headers
  src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
  (1 source file) failed: (Exit 1): turbine_direct_graal failed:
  error executing command (from target
  //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions)

  external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal
    --output
    bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
    ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  java.lang.NullPointerException:
  attempted to use --release, but JAVA_HOME is not set
    at [email protected]/java.util.Objects.requireNonNull(Objects.java:259)
    at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55)
    at com.google.turbine.main.Main.release(Main.java:318)
    at com.google.turbine.main.Main.bootclasspath(Main.java:304)
    at com.google.turbine.main.Main.compile(Main.java:142)
    at com.google.turbine.main.Main.compile(Main.java:133)
    at com.google.turbine.main.Main.main(Main.java:89)
    at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

INFO: Elapsed time: 0.325s, Critical Path: 0.12s
INFO: 11 processes: 10 internal, 1 worker.
FAILED: Build did NOT complete successfully
 Test "scrooge_compile_with_jdk_11" failed  (0 sec)
```

See also:

- Bazel Blog: Bazel 5.0:
  https://blog.bazel.build/2022/01/19/bazel-5.0.html#java

- bazelbuild/bazel:
  `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain
  resolution for Java rules #7849
  bazelbuild/bazel#7849

---

Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8
will likely happen in a new major release _after_ a release containing
the Bzlmod changes. This is because:

- The Bzlmod changes alone will work with Bazel 6 without requiring that
  users update `protobuf` beyond version 21.7.

- Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add
  the afforementioned C++ compiler flags to support the newer
  `abseil-cpp` versions required by newer `protobuf` versions.

- `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or
  higher to support `protobuf` >= v28.0, but this ScalaPB version will
  _not_ support `protobuf` < v28.0.

The idea is that the next major release of `rules_scala` will help users
migrate to Bazel 7 and Bzlmod, in either order. Then the next major
release after that will enable them to migrate to Bazel 8, with all the
requisite dependency upgrades. (Technically it will still support
`WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See:

- bazelbuild#1482 (comment)

---

Here are the details explaining how `rules_java` 8 currently breaks
`rules_scala`, up until the point that upgrading to `protobuf` >= v29.0
would fix it.

`rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule`
API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java`
8.3.2.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/java/common/rules/android_lint.bzl:142:24:
  name 'subrule' is not defined (did you mean 'rule'?)

ERROR: Error computing the main repository mapping:
  at .../scala/private/extensions/dev_deps.bzl:8:6:
  at .../external/rules_java/java/repositories.bzl:20:6:
  at .../external/rules_java/toolchains/local_java_repository.bzl:17:6:
  at .../external/rules_java/java/defs.bzl:18:6:
  at .../external/rules_java/java/java_library.bzl:16:6:
  at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6:
  compilation of module 'java/common/rules/android_lint.bzl' failed
```

`rules_java` 8.3.0 is broken, as it can't find its own
`@compatibility_proxy` repo:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: error loading package under directory 'src':
  error loading package 'src/protobuf/io/bazel/rules_scala':
  at .../external/rules_java/java/defs.bzl:22:6:
  at .../external/rules_java/java/java_test.bzl:16:6:
  Unable to find package for @compatibility_proxy//:proxy.bzl:
  The repository '@compatibility_proxy' could not be resolved:
  Repository '@compatibility_proxy' is not defined.
```

`rules_java` 8.3.1 seems to fix this, presumably by importing the
`protobuf` repo as `com_google_protobuf`. However, it now requires at
least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`.
Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support
`protobuf` v28, abandoning users of previous `protobuf` versions or
forcing them to upgrade.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[...snip...]
ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/worker':
  at .../external/rules_java/java/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl'
  is invalid because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'?
```
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several
dependencies as high as they can go and still be compatible with Bazel
6.5.0 and 7.4.1.

- `bazel_skylib`: 1.4.1 => 1.7.1
- `go`: 1.19.5 => 1.23.4
- `rules_cc`: 0.0.6 => 0.0.9
- `rules_go`: 0.39.1 => 0.50.1
- `rules_java`: 7.9.0 => 7.12.3
- `rules_python`: 0.36.0 => 0.38.0

The `rules_java` 7.12.13 bump precipitated the following changes:

- Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file
  per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3.
  This replaces previous calls to instantiate `rules_java` repos and to
  register `rules_java` toolchains.

- Rearranges the other `WORKSPACE` setup macros for dependencies to come
  before the `rules_scala` setup macros.

- Updates almost all dependencies on `@bazel_tools//tools/jdk:`
  toolchain targets and `.bzl` files to corresponding targets and files
  in `@rules_java//toolchains:`.

- Removes several deprecated flags from the
  `scrooge_compile_with_jdk_11` test case from
  `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that
  caused it to break.

---

Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all
the versions of dependencies we want in `WORKSPACE`, while providing a
new API to consumers. It also prevents `WORKSPACE` from transitively
loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8
compatibility per bazelbuild#1652.

Reasons for the other `rules_java` related changes include:

- The `WORKSPACE` stanza for `rules_java` should've already been present
  while using the existing version 7.9.0. However, doing so would've
  broken Bazel 6 builds, as described in detail below.

- Having the other `WORKSPACE` setup macros for dependencies come before
  the `rules_scala` setup macros helps ensure consistent, correct
  initialization before `rules_scala` initialization.

- Updating the toolchain specifiers to use `@rules_java//toolchains`
  fixed `WORKSPACE` build breakages when updating to `rules_java`
  7.10.0 and later. This is a potentially breaking change for consumers,
  but in the good kind of way, in that it requires an easy, futureproof
  update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain,
  as there's not yet a corresponding `@rules_java//toolchains` target.)

What follows are detailed notes on why some dependency versions are
capped where they are, and on some breakages fixed by these changes.

---

`abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7,
respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to
v25.5 is compatible with Bazel 6 provided users set the compiler flags
mentioned in that issue:

```txt
build:linux --cxxopt=-std=c++17
build:linux --host_cxxopt=-std=c++17
build:macos --cxxopt=-std=c++17
build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std=c++17
build:windows --host_cxxopt=/std=c++17
```

---

`rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10,
which introduced `cc/common/cc_info.bzl`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:19:6:
  Label '@rules_cc//cc/common:cc_info.bzl' is invalid
    because 'cc/common' is not a package;
    perhaps you meant to put the colon here:
    '@rules_cc//cc:common/cc_info.bzl'?
```

However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines
`CcSharedLibraryHintInfo`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33:
  name 'CcSharedLibraryHintInfo' is not defined (did you mean
  'CcSharedLibraryInfo'?)

[ ...snip... ]

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/common/cc_common.bzl:17:6:
  compilation of module 'cc/private/rules_impl/native.bzl' failed
```

---

This section provides a methodical description of the errors encountered
and resolved during each step of the `rules_java` update.

The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't
originally have in place, is:

```py
load(
    "@rules_java//java:repositories.bzl",
    "rules_java_dependencies",
    "rules_java_toolchains",
)

rules_java_dependencies()

rules_java_toolchains()
```

Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0
didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build.
Note the `CompiledWithJava{8,11}_java.jar` references:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: .../test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---

ERROR: .../test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---
```

Another variation of this I also saw was:

```txt
$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]
```

Updating the toolchains from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:` for targets in `test/BUILD` and
`test/src/main/resources/java_sources/BUILD` fixed this particular
issue. (More on updating other `@bazel_tools//tools/jdk` toolchain
targets below.) Bazel 6.5.0 then failed with the following expected
issue, fixed by `rules_java` 7.10.0 via:

- bazelbuild/rules_java#210
- bazelbuild/rules_java@30ecf3f

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34:
  Use of Starlark transition without allowlist attribute
  '_allowlist_function_transition'.
  See Starlark transitions documentation for details and usage:
  @rules_java//toolchains:java_toolchain_alias.bzl NORMAL

ERROR:
  .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12:
  While resolving toolchains for target
  //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter:
  invalid registered toolchain '//toolchains:all':
  while parsing '//toolchains:all':
  error loading package '@rules_java//toolchains':
  initialization of module 'toolchains/java_toolchain_alias.bzl' failed

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter'
  failed; build aborted:
```

Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused
Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619.
For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in
the message for commit cd22d88.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:
  While resolving toolchains for target
  @@rules_java_builtin//toolchains:platformclasspath (096dcc8):
  No matching toolchains found for types
  @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.

To debug, rerun with
  --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage
  reading https://bazel.build/concepts/platforms-intro.

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed
```

My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0
built into Bazel's `WORKSPACE` preamble registered
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains
that later versions of `rules_java` couldn't resolve. This was due to
the very same change that fixed 6.5.0, bazelbuild/rules_java#210.

However, I mistakenly thought we needed to keep
`@bazel_tools//tools/jdk:` as the canonical Java toolchains package
instead switching to `@rules_java//toolchains:`. That's why I originally
thought we were stuck on `rules_java` 7.9.0.

I eventually rediscovered the following issue, which surfaced this
problem a month before bazelbuild#1619. The conversation helped me realize that
switching to `@rules_java//toolchains:` actually is the preferred,
futureproof approach (short of migrating to Bzlmod):

- bazelbuild/rules_java: Regression with
  `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0
  bazelbuild/rules_java#214

Switching all `@bazel_tools//tools/jdk:` toolchains to
`@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and
7.4.1 to successfully build using `rules_java` 7.12.2. I then updated
`rules_java` to 7.12.3, which returns to registering some toolchains as
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java#246
- bazelbuild/rules_java@b64eb7d

@hvadehra requested that I try 7.12.3 to see if it resolved the issue.
I was able to build successfully using this version in a branch without
the toolchain updates from this commit.

- bazelbuild#1652 (comment)

However, the changes from this commit should remain futureproof when
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to
`@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one
day.

---

Several of the flags removed from the `scrooge_compile_with_jdk_11` test
case have been no-ops for years, so removing them eliminated their
corresponding deprecation warnings. `--javacopt='--release 11'`,
however, caused this breakage after originally bumping to `rules_java`
7.12.2:

```txt
$ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh

running test scrooge_compile_with_jdk_11
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured).
INFO: Found 64 targets...

ERROR:
  .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13:
  Compiling Java headers
  src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
  (1 source file) failed: (Exit 1): turbine_direct_graal failed:
  error executing command (from target
  //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions)

  external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal
    --output
    bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
    ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  java.lang.NullPointerException:
  attempted to use --release, but JAVA_HOME is not set
    at [email protected]/java.util.Objects.requireNonNull(Objects.java:259)
    at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55)
    at com.google.turbine.main.Main.release(Main.java:318)
    at com.google.turbine.main.Main.bootclasspath(Main.java:304)
    at com.google.turbine.main.Main.compile(Main.java:142)
    at com.google.turbine.main.Main.compile(Main.java:133)
    at com.google.turbine.main.Main.main(Main.java:89)
    at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

INFO: Elapsed time: 0.325s, Critical Path: 0.12s
INFO: 11 processes: 10 internal, 1 worker.
FAILED: Build did NOT complete successfully
 Test "scrooge_compile_with_jdk_11" failed  (0 sec)
```

See also:

- Bazel Blog: Bazel 5.0:
  https://blog.bazel.build/2022/01/19/bazel-5.0.html#java

- bazelbuild/bazel:
  `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain
  resolution for Java rules #7849
  bazelbuild/bazel#7849

---

Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8
will likely happen in a new major release _after_ a release containing
the Bzlmod changes. This is because:

- The Bzlmod changes alone will work with Bazel 6 without requiring that
  users update `protobuf` beyond version 21.7.

- Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add
  the afforementioned C++ compiler flags to support the newer
  `abseil-cpp` versions required by newer `protobuf` versions.

- `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or
  higher to support `protobuf` >= v28.0, but this ScalaPB version will
  _not_ support `protobuf` < v28.0.

The idea is that the next major release of `rules_scala` will help users
migrate to Bazel 7 and Bzlmod, in either order. Then the next major
release after that will enable them to migrate to Bazel 8, with all the
requisite dependency upgrades. (Technically it will still support
`WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See:

- bazelbuild#1482 (comment)

---

Here are the details explaining how `rules_java` 8 currently breaks
`rules_scala`, up until the point that upgrading to `protobuf` >= v29.0
would fix it.

`rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule`
API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java`
8.3.2.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/java/common/rules/android_lint.bzl:142:24:
  name 'subrule' is not defined (did you mean 'rule'?)

ERROR: Error computing the main repository mapping:
  at .../scala/private/extensions/dev_deps.bzl:8:6:
  at .../external/rules_java/java/repositories.bzl:20:6:
  at .../external/rules_java/toolchains/local_java_repository.bzl:17:6:
  at .../external/rules_java/java/defs.bzl:18:6:
  at .../external/rules_java/java/java_library.bzl:16:6:
  at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6:
  compilation of module 'java/common/rules/android_lint.bzl' failed
```

`rules_java` 8.3.0 is broken, as it can't find its own
`@compatibility_proxy` repo:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: error loading package under directory 'src':
  error loading package 'src/protobuf/io/bazel/rules_scala':
  at .../external/rules_java/java/defs.bzl:22:6:
  at .../external/rules_java/java/java_test.bzl:16:6:
  Unable to find package for @compatibility_proxy//:proxy.bzl:
  The repository '@compatibility_proxy' could not be resolved:
  Repository '@compatibility_proxy' is not defined.
```

`rules_java` 8.3.1 seems to fix this, presumably by importing the
`protobuf` repo as `com_google_protobuf`. However, it now requires at
least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`.
Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support
`protobuf` v28, abandoning users of previous `protobuf` versions or
forcing them to upgrade.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[...snip...]
ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/worker':
  at .../external/rules_java/java/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl'
  is invalid because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'?
```
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Moves the Scalatest, JUnit, and Specs2 toolchains into
`@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482.

Updates all `WORKSPACE` files to set the appropriate `scala_toolchains`
parameters and to remove the unnecessary repository import and toolchain
registration macros.

Adds a `fetch_sources_by_id` parameter to `repositories` from
`third_party/repositories/repositories.bzl`. This enables
`scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping
from artifact ID lists returned by new macros extracted from `WORKSPACE`
macros. The values assigned to each id match the original
`fetch_sources` settings in the corresponding original `WORKSPACE`
macros.

Updates `scala/scala_maven_import_external.bzl` to generate a `load`
line for `//scala:scala_import.bzl` based on the repo's canonical name,
not `@io_bazel_rules_scala`.

As usual, includes several other opportunistic removals of the
`@io_bazel_rules_scala` repo name prefix to avoid an internal dependency
on that name. This means Bzlmod users won't necessarily have to set the
`repo_name` parameter of `bazel_dep` when using `rules_scala`.

---

Introduces more macros to return a framework's Maven artifact
dependencies, rather than inlining them in a `repositories` call. These
inlined lists are replaced by macro invocations, and now the
`scala_toolchains` macro can invoke these macros to collect artifact IDs
to pass to `repositories`. This also allows for future changes to
introduce a `scala_version` parameter if necessary, similar to how
`scalafmt_artifact_ids` already works.

This is important to avoid collisions when creating repositories for
artifacts upon which more than one framework depends under Bzlmod.
`WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will
produce errors like the following, where both `scala_proto` and
`twitter_scrooge` depend upon `io_bazel_rules_scala_guava`:

```txt
$ bazel build //src/...

ERROR: .../scala/scala_maven_import_external.bzl:299:24:
  Traceback (most recent call last):
    File ".../scala/extensions/deps.bzl",
      line 140, column 21, in _scala_deps_impl
        scala_toolchains(
    File ".../scala/private/macros/toolchains.bzl",
      line 140, column 17, in scala_toolchains
        _scrooge(
    File ".../twitter_scrooge/twitter_scrooge.bzl",
      line 96, column 17, in twitter_scrooge
        repositories(
    File ".../third_party/repositories/repositories.bzl",
      line 113, column 37, in repositories
        _scala_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 263, column 30, in scala_maven_import_external
        jvm_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 299, column 24, in jvm_maven_import_external
        jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs)

Error in repository_rule: A repo named
  io_bazel_rules_scala_guava_2_13_15
  is already generated by this module extension at
  .../scala/scala_maven_import_external.bzl:299:24

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/worker:worker_test'
  failed; build aborted:
  error evaluating module extension scala_deps in
  //scala/extensions:deps.bzl
```

Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it
easy to emit full direct dependency lists for artifacts included in
`third_party/repositories/scala_*.bzl`. This increases the likelihood of
collisions, since this expanded metadata forces the macros that
instantiate artifact repos to instantiate even more repos.

By fetching list of artifact IDs from these macros, `scala_toolchains`
can now consolidate them into dictionary keys. Then it passes these
unique keys to `repositories` directly, avoiding the problem of
instantiating the same repo multiple times in the same module extension.

This, in turn, also avoids the need to add parameters to the original
`WORKSPACE` macros that instantiate dependencies to avoid collisions
under Bzlmod. The `scala_toolchains` macro never needs to call these
original macros, under either `WORKSPACE` or Bzlmod.

Finally, it also reduces duplication between these artifact ID lists and
the `_*_DEPS` symbols originally from `testing/BUILD` (and now in
`testing/deps.bzl`). The dependency labels are now generated
programatically.

(Aside: As I mentioned, we may eventually need to pass a Scala version
argument to these macros. It will be possible to cross that bridge
without too much trouble if and when that day comes. Or I can try to
future proof it in a follow up pull request.)
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Removes this symbol from `scala/scala.bzl` as well as
`setup_scala_testing_toolchain`, and deletes
`scala/private/macros/toolchains.bzl`. Part of bazelbuild#1482 and bazelbuild#1652.

This is required for Bazel 8 and `rules_java` 8 compatibility, but is
also compatible with Bazel 6 and 7. In bazelbuild#1652, @hvadehra suggested
partitioning the `.bzl` files such that `WORKSPACE` doesn't `load` a
file that tries to `load` symbols from `rules_java`. I successfully did
so in a separate branch, and along with other minor changes, got
`rules_scala` to build with `rules_java` 8.5.1.

The other changes will come in separate pull requests, but it makes
sense to land this change now before adding any other toolchains to
`scala_toolchains`.

---

Arguably, we should remove all macros exported from `scala/scala.bzl`
that only instantiate toolchain dependencies and define toolchains. That
may be a breaking change for some users, but will ultimately be
necessary for these macros to remain compatible with Bazel 8.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
This is a pretty straightforward and easy update on top of the previous
`setup_toolchains` and `scala_toolchains_repo` changes. Part of bazelbuild#1482.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482.

The most significant part of the change is moving all the toolchain
rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in
`scala_proto/toolchains.bzl`.

Adds the `scala_proto_deps_providers()` macro to replace
`//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in
the `dep_providers` parameter of `scala_proto_deps_toolchain()`.
Examples of this are in `test/proto/custom_generator/BUILD`.

Excludes `@scala_proto_rules_scalapb_protoc_gen` from
`DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl`
for Scala 2.11. For other Scala versions, this repo name will have the
Scala version appended. This is to avoid build failures under Bzlmod,
since:

- This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at
  ScalaPB 0.9.8.

- Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under
  Scala 2.11 results in an error under Bzlmod, as does importing it
  multiple times when configuring multiple Scala versions.

- `MODULE.bazel` can iterate over a list of Scala versions, filtering
  out Scala 2.11, and call `use_repo()` on each version specific repo.

A lot of the other changes are more opportunistic removals of
`@io_bazel_rules_scala` label prefixes and application of `Label()`
where appropriate. Doing this will allow Bzlmod users to use
`rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in
`bazel_dep()`.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Adds the jmh toolchain to `scala_toolchains()` and moves
`jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8
compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
This is the last of the toolchain to receive the "toolchainization"
treatment prior to Bzlmodification, and moves `twitter_scrooge()` to
`twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8
compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several
dependencies as high as they can go and still be compatible with Bazel
6.5.0 and 7.4.1.

- `bazel_skylib`: 1.4.1 => 1.7.1
- `go`: 1.19.5 => 1.23.4
- `rules_cc`: 0.0.6 => 0.0.9
- `rules_go`: 0.39.1 => 0.50.1
- `rules_java`: 7.9.0 => 7.12.3
- `rules_python`: 0.36.0 => 0.38.0

The `rules_java` 7.12.13 bump precipitated the following changes:

- Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file
  per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3.
  This replaces previous calls to instantiate `rules_java` repos and to
  register `rules_java` toolchains.

- Rearranges the other `WORKSPACE` setup macros for dependencies to come
  before the `rules_scala` setup macros.

- Updates almost all dependencies on `@bazel_tools//tools/jdk:`
  toolchain targets and `.bzl` files to corresponding targets and files
  in `@rules_java//toolchains:`.

- Removes several deprecated flags from the
  `scrooge_compile_with_jdk_11` test case from
  `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that
  caused it to break.

---

Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all
the versions of dependencies we want in `WORKSPACE`, while providing a
new API to consumers. It also prevents `WORKSPACE` from transitively
loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8
compatibility per bazelbuild#1652.

Reasons for the other `rules_java` related changes include:

- The `WORKSPACE` stanza for `rules_java` should've already been present
  while using the existing version 7.9.0. However, doing so would've
  broken Bazel 6 builds, as described in detail below.

- The `rules_java_toolchains()` call follows the `protobuf_deps()` call
  instead of immediately following `rules_java_dependencies()` because
  upgrading to `rules_java` >= 8.5.0 will require this. It has no
  adverse impact to do it now, amidst the other `WORKSPACE` changes, and
  will make the eventual `rules_java` >= 8.5.0 diff smaller.

- Having the other `WORKSPACE` setup macros for dependencies come before
  the `rules_scala` setup macros helps ensure consistent, correct
  initialization before `rules_scala` initialization.

- Updating the toolchain specifiers to use `@rules_java//toolchains`
  fixed `WORKSPACE` build breakages when updating to `rules_java`
  7.10.0 and later. This is a potentially breaking change for consumers,
  but in the good kind of way, in that it requires an easy, futureproof
  update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain,
  as there's not yet a corresponding `@rules_java//toolchains` target.)

What follows are detailed notes on why some dependency versions are
capped where they are, and on some breakages fixed by these changes.

---

`abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7,
respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to
v25.5 is compatible with Bazel 6 provided users set the compiler flags
mentioned in that issue:

```txt
build:linux --cxxopt=-std=c++17
build:linux --host_cxxopt=-std=c++17
build:macos --cxxopt=-std=c++17
build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std=c++17
build:windows --host_cxxopt=/std=c++17
```

---

`rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10,
which introduced `cc/common/cc_info.bzl`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:19:6:
  Label '@rules_cc//cc/common:cc_info.bzl' is invalid
    because 'cc/common' is not a package;
    perhaps you meant to put the colon here:
    '@rules_cc//cc:common/cc_info.bzl'?
```

However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines
`CcSharedLibraryHintInfo`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33:
  name 'CcSharedLibraryHintInfo' is not defined (did you mean
  'CcSharedLibraryInfo'?)

[ ...snip... ]

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/common/cc_common.bzl:17:6:
  compilation of module 'cc/private/rules_impl/native.bzl' failed
```

---

This section provides a methodical description of the errors encountered
and resolved during each step of the `rules_java` update.

The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't
originally have in place, is:

```py
load(
    "@rules_java//java:repositories.bzl",
    "rules_java_dependencies",
    "rules_java_toolchains",
)

rules_java_dependencies()

rules_java_toolchains()
```

Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0
didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build.
Note the `CompiledWithJava{8,11}_java.jar` references:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: .../test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---

ERROR: .../test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---
```

Another variation of this I also saw was:

```txt
$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]
```

Updating the toolchains from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:` for targets in `test/BUILD` and
`test/src/main/resources/java_sources/BUILD` fixed this particular
issue. (More on updating other `@bazel_tools//tools/jdk` toolchain
targets below.) Bazel 6.5.0 then failed with the following expected
issue, fixed by `rules_java` 7.10.0 via:

- bazelbuild/rules_java#210
- bazelbuild/rules_java@30ecf3f

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34:
  Use of Starlark transition without allowlist attribute
  '_allowlist_function_transition'.
  See Starlark transitions documentation for details and usage:
  @rules_java//toolchains:java_toolchain_alias.bzl NORMAL

ERROR:
  .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12:
  While resolving toolchains for target
  //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter:
  invalid registered toolchain '//toolchains:all':
  while parsing '//toolchains:all':
  error loading package '@rules_java//toolchains':
  initialization of module 'toolchains/java_toolchain_alias.bzl' failed

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter'
  failed; build aborted:
```

Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused
Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619.
For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in
the message for commit cd22d88.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:
  While resolving toolchains for target
  @@rules_java_builtin//toolchains:platformclasspath (096dcc8):
  No matching toolchains found for types
  @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.

To debug, rerun with
  --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage
  reading https://bazel.build/concepts/platforms-intro.

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed
```

My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0
built into Bazel's `WORKSPACE` preamble registered
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains
that later versions of `rules_java` couldn't resolve. This was due to
the very same change that fixed 6.5.0, bazelbuild/rules_java#210.

However, I mistakenly thought we needed to keep
`@bazel_tools//tools/jdk:` as the canonical Java toolchains package
instead switching to `@rules_java//toolchains:`. That's why I originally
thought we were stuck on `rules_java` 7.9.0.

I eventually rediscovered the following issue, which surfaced this
problem a month before bazelbuild#1619. The conversation helped me realize that
switching to `@rules_java//toolchains:` actually is the preferred,
futureproof approach (short of migrating to Bzlmod):

- bazelbuild/rules_java: Regression with
  `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0
  bazelbuild/rules_java#214

Switching all `@bazel_tools//tools/jdk:` toolchains to
`@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and
7.4.1 to successfully build using `rules_java` 7.12.2. I then updated
`rules_java` to 7.12.3, which returns to registering some toolchains as
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java#246
- bazelbuild/rules_java@b64eb7d

@hvadehra requested that I try 7.12.3 to see if it resolved the issue.
I was able to build successfully using this version in a branch without
the toolchain updates from this commit.

- bazelbuild#1652 (comment)

However, the changes from this commit should remain futureproof when
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to
`@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one
day.

---

Several of the flags removed from the `scrooge_compile_with_jdk_11` test
case have been no-ops for years, so removing them eliminated their
corresponding deprecation warnings. `--javacopt='--release 11'`,
however, caused this breakage after originally bumping to `rules_java`
7.12.2:

```txt
$ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh

running test scrooge_compile_with_jdk_11
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured).
INFO: Found 64 targets...

ERROR:
  .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13:
  Compiling Java headers
  src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
  (1 source file) failed: (Exit 1): turbine_direct_graal failed:
  error executing command (from target
  //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions)

  external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal
    --output
    bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
    ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  java.lang.NullPointerException:
  attempted to use --release, but JAVA_HOME is not set
    at [email protected]/java.util.Objects.requireNonNull(Objects.java:259)
    at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55)
    at com.google.turbine.main.Main.release(Main.java:318)
    at com.google.turbine.main.Main.bootclasspath(Main.java:304)
    at com.google.turbine.main.Main.compile(Main.java:142)
    at com.google.turbine.main.Main.compile(Main.java:133)
    at com.google.turbine.main.Main.main(Main.java:89)
    at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

INFO: Elapsed time: 0.325s, Critical Path: 0.12s
INFO: 11 processes: 10 internal, 1 worker.
FAILED: Build did NOT complete successfully
 Test "scrooge_compile_with_jdk_11" failed  (0 sec)
```

See also:

- Bazel Blog: Bazel 5.0:
  https://blog.bazel.build/2022/01/19/bazel-5.0.html#java

- bazelbuild/bazel:
  `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain
  resolution for Java rules #7849
  bazelbuild/bazel#7849

---

Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8
will likely happen in a new major release _after_ a release containing
the Bzlmod changes. This is because:

- The Bzlmod changes alone will work with Bazel 6 without requiring that
  users update `protobuf` beyond version 21.7.

- Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add
  the afforementioned C++ compiler flags to support the newer
  `abseil-cpp` versions required by newer `protobuf` versions.

- `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or
  higher to support `protobuf` >= v28.0, but this ScalaPB version will
  _not_ support `protobuf` < v28.0.

The idea is that the next major release of `rules_scala` will help users
migrate to Bazel 7 and Bzlmod, in either order. Then the next major
release after that will enable them to migrate to Bazel 8, with all the
requisite dependency upgrades. (Technically it will still support
`WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See:

- bazelbuild#1482 (comment)

---

Here are the details explaining how `rules_java` 8 currently breaks
`rules_scala`, up until the point that upgrading to `protobuf` >= v29.0
would fix it.

`rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule`
API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java`
8.3.2.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/java/common/rules/android_lint.bzl:142:24:
  name 'subrule' is not defined (did you mean 'rule'?)

ERROR: Error computing the main repository mapping:
  at .../scala/private/extensions/dev_deps.bzl:8:6:
  at .../external/rules_java/java/repositories.bzl:20:6:
  at .../external/rules_java/toolchains/local_java_repository.bzl:17:6:
  at .../external/rules_java/java/defs.bzl:18:6:
  at .../external/rules_java/java/java_library.bzl:16:6:
  at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6:
  compilation of module 'java/common/rules/android_lint.bzl' failed
```

`rules_java` 8.3.0 is broken, as it can't find its own
`@compatibility_proxy` repo:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: error loading package under directory 'src':
  error loading package 'src/protobuf/io/bazel/rules_scala':
  at .../external/rules_java/java/defs.bzl:22:6:
  at .../external/rules_java/java/java_test.bzl:16:6:
  Unable to find package for @compatibility_proxy//:proxy.bzl:
  The repository '@compatibility_proxy' could not be resolved:
  Repository '@compatibility_proxy' is not defined.
```

`rules_java` 8.3.1 seems to fix this, presumably by importing the
`protobuf` repo as `com_google_protobuf`. However, it now requires at
least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`.
Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support
`protobuf` v28, abandoning users of previous `protobuf` versions or
forcing them to upgrade.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[...snip...]
ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/worker':
  at .../external/rules_java/java/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl'
  is invalid because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'?
```
mbland added a commit to mbland/rules_scala that referenced this issue Dec 10, 2024
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several
dependencies as high as they can go and still be compatible with Bazel
6.5.0 and 7.4.1.

- `bazel_skylib`: 1.4.1 => 1.7.1
- `go`: 1.19.5 => 1.23.4
- `rules_cc`: 0.0.6 => 0.0.9
- `rules_go`: 0.39.1 => 0.50.1
- `rules_java`: 7.9.0 => 7.12.3
- `rules_python`: 0.36.0 => 0.38.0

The `rules_java` 7.12.13 bump precipitated the following changes:

- Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file
  per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3.
  This replaces previous calls to instantiate `rules_java` repos and to
  register `rules_java` toolchains.

- Rearranges the other `WORKSPACE` setup macros for dependencies to come
  before the `rules_scala` setup macros.

- Updates almost all dependencies on `@bazel_tools//tools/jdk:`
  toolchain targets and `.bzl` files to corresponding targets and files
  in `@rules_java//toolchains:`.

- Removes several deprecated flags from the
  `scrooge_compile_with_jdk_11` test case from
  `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that
  caused it to break.

---

Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all
the versions of dependencies we want in `WORKSPACE`, while providing a
new API to consumers. It also prevents `WORKSPACE` from transitively
loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8
compatibility per bazelbuild#1652.

Reasons for the other `rules_java` related changes include:

- The `WORKSPACE` stanza for `rules_java` should've already been present
  while using the existing version 7.9.0. However, doing so would've
  broken Bazel 6 builds, as described in detail below.

- The `rules_java_toolchains()` call follows the `protobuf_deps()` call
  instead of immediately following `rules_java_dependencies()` because
  upgrading to `rules_java` >= 8.5.0 will require this. It has no
  adverse impact to do it now, amidst the other `WORKSPACE` changes, and
  will make the eventual `rules_java` >= 8.5.0 diff smaller.

- Having the other `WORKSPACE` setup macros for dependencies come before
  the `rules_scala` setup macros helps ensure consistent, correct
  initialization before `rules_scala` initialization.

- Updating the toolchain specifiers to use `@rules_java//toolchains`
  fixed `WORKSPACE` build breakages when updating to `rules_java`
  7.10.0 and later. This is a potentially breaking change for consumers,
  but in the good kind of way, in that it requires an easy, futureproof
  update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain,
  as there's not yet a corresponding `@rules_java//toolchains` target.)

What follows are detailed notes on why some dependency versions are
capped where they are, and on some breakages fixed by these changes.

---

`abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7,
respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to
v25.5 is compatible with Bazel 6 provided users set the compiler flags
mentioned in that issue:

```txt
build:linux --cxxopt=-std=c++17
build:linux --host_cxxopt=-std=c++17
build:macos --cxxopt=-std=c++17
build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std=c++17
build:windows --host_cxxopt=/std=c++17
```

---

`rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10,
which introduced `cc/common/cc_info.bzl`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:19:6:
  Label '@rules_cc//cc/common:cc_info.bzl' is invalid
    because 'cc/common' is not a package;
    perhaps you meant to put the colon here:
    '@rules_cc//cc:common/cc_info.bzl'?
```

However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines
`CcSharedLibraryHintInfo`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33:
  name 'CcSharedLibraryHintInfo' is not defined (did you mean
  'CcSharedLibraryInfo'?)

[ ...snip... ]

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/common/cc_common.bzl:17:6:
  compilation of module 'cc/private/rules_impl/native.bzl' failed
```

---

This section provides a methodical description of the errors encountered
and resolved during each step of the `rules_java` update.

The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't
originally have in place, is:

```py
load(
    "@rules_java//java:repositories.bzl",
    "rules_java_dependencies",
    "rules_java_toolchains",
)

rules_java_dependencies()

rules_java_toolchains()
```

Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0
didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build.
Note the `CompiledWithJava{8,11}_java.jar` references:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: .../test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---

ERROR: .../test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---
```

Another variation of this I also saw was:

```txt
$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]
```

Updating the toolchains from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:` for targets in `test/BUILD` and
`test/src/main/resources/java_sources/BUILD` fixed this particular
issue. (More on updating other `@bazel_tools//tools/jdk` toolchain
targets below.) Bazel 6.5.0 then failed with the following expected
issue, fixed by `rules_java` 7.10.0 via:

- bazelbuild/rules_java#210
- bazelbuild/rules_java@30ecf3f

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34:
  Use of Starlark transition without allowlist attribute
  '_allowlist_function_transition'.
  See Starlark transitions documentation for details and usage:
  @rules_java//toolchains:java_toolchain_alias.bzl NORMAL

ERROR:
  .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12:
  While resolving toolchains for target
  //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter:
  invalid registered toolchain '//toolchains:all':
  while parsing '//toolchains:all':
  error loading package '@rules_java//toolchains':
  initialization of module 'toolchains/java_toolchain_alias.bzl' failed

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter'
  failed; build aborted:
```

Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused
Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619.
For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in
the message for commit cd22d88.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:
  While resolving toolchains for target
  @@rules_java_builtin//toolchains:platformclasspath (096dcc8):
  No matching toolchains found for types
  @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.

To debug, rerun with
  --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage
  reading https://bazel.build/concepts/platforms-intro.

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed
```

My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0
built into Bazel's `WORKSPACE` preamble registered
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains
that later versions of `rules_java` couldn't resolve. This was due to
the very same change that fixed 6.5.0, bazelbuild/rules_java#210.

However, I mistakenly thought we needed to keep
`@bazel_tools//tools/jdk:` as the canonical Java toolchains package
instead switching to `@rules_java//toolchains:`. That's why I originally
thought we were stuck on `rules_java` 7.9.0.

I eventually rediscovered the following issue, which surfaced this
problem a month before bazelbuild#1619. The conversation helped me realize that
switching to `@rules_java//toolchains:` actually is the preferred,
futureproof approach (short of migrating to Bzlmod):

- bazelbuild/rules_java: Regression with
  `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0
  bazelbuild/rules_java#214

Switching all `@bazel_tools//tools/jdk:` toolchains to
`@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and
7.4.1 to successfully build using `rules_java` 7.12.2. I then updated
`rules_java` to 7.12.3, which returns to registering some toolchains as
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java#246
- bazelbuild/rules_java@b64eb7d

@hvadehra requested that I try 7.12.3 to see if it resolved the issue.
I was able to build successfully using this version in a branch without
the toolchain updates from this commit.

- bazelbuild#1652 (comment)

However, the changes from this commit should remain futureproof when
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to
`@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one
day.

---

Several of the flags removed from the `scrooge_compile_with_jdk_11` test
case have been no-ops for years, so removing them eliminated their
corresponding deprecation warnings. `--javacopt='--release 11'`,
however, caused this breakage after originally bumping to `rules_java`
7.12.2:

```txt
$ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh

running test scrooge_compile_with_jdk_11
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured).
INFO: Found 64 targets...

ERROR:
  .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13:
  Compiling Java headers
  src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
  (1 source file) failed: (Exit 1): turbine_direct_graal failed:
  error executing command (from target
  //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions)

  external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal
    --output
    bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
    ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  java.lang.NullPointerException:
  attempted to use --release, but JAVA_HOME is not set
    at [email protected]/java.util.Objects.requireNonNull(Objects.java:259)
    at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55)
    at com.google.turbine.main.Main.release(Main.java:318)
    at com.google.turbine.main.Main.bootclasspath(Main.java:304)
    at com.google.turbine.main.Main.compile(Main.java:142)
    at com.google.turbine.main.Main.compile(Main.java:133)
    at com.google.turbine.main.Main.main(Main.java:89)
    at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

INFO: Elapsed time: 0.325s, Critical Path: 0.12s
INFO: 11 processes: 10 internal, 1 worker.
FAILED: Build did NOT complete successfully
 Test "scrooge_compile_with_jdk_11" failed  (0 sec)
```

See also:

- Bazel Blog: Bazel 5.0:
  https://blog.bazel.build/2022/01/19/bazel-5.0.html#java

- bazelbuild/bazel:
  `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain
  resolution for Java rules #7849
  bazelbuild/bazel#7849

---

Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8
will likely happen in a new major release _after_ a release containing
the Bzlmod changes. This is because:

- The Bzlmod changes alone will work with Bazel 6 without requiring that
  users update `protobuf` beyond version 21.7.

- Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add
  the afforementioned C++ compiler flags to support the newer
  `abseil-cpp` versions required by newer `protobuf` versions.

- `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or
  higher to support `protobuf` >= v28.0, but this ScalaPB version will
  _not_ support `protobuf` < v28.0.

The idea is that the next major release of `rules_scala` will help users
migrate to Bazel 7 and Bzlmod, in either order. Then the next major
release after that will enable them to migrate to Bazel 8, with all the
requisite dependency upgrades. (Technically it will still support
`WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See:

- bazelbuild#1482 (comment)

---

Here are the details explaining how `rules_java` 8 currently breaks
`rules_scala`, up until the point that upgrading to `protobuf` >= v29.0
would fix it.

`rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule`
API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java`
8.3.2.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/java/common/rules/android_lint.bzl:142:24:
  name 'subrule' is not defined (did you mean 'rule'?)

ERROR: Error computing the main repository mapping:
  at .../scala/private/extensions/dev_deps.bzl:8:6:
  at .../external/rules_java/java/repositories.bzl:20:6:
  at .../external/rules_java/toolchains/local_java_repository.bzl:17:6:
  at .../external/rules_java/java/defs.bzl:18:6:
  at .../external/rules_java/java/java_library.bzl:16:6:
  at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6:
  compilation of module 'java/common/rules/android_lint.bzl' failed
```

`rules_java` 8.3.0 is broken, as it can't find its own
`@compatibility_proxy` repo:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: error loading package under directory 'src':
  error loading package 'src/protobuf/io/bazel/rules_scala':
  at .../external/rules_java/java/defs.bzl:22:6:
  at .../external/rules_java/java/java_test.bzl:16:6:
  Unable to find package for @compatibility_proxy//:proxy.bzl:
  The repository '@compatibility_proxy' could not be resolved:
  Repository '@compatibility_proxy' is not defined.
```

`rules_java` 8.3.1 seems to fix this, presumably by importing the
`protobuf` repo as `com_google_protobuf`. However, it now requires at
least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`.
Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support
`protobuf` v28, abandoning users of previous `protobuf` versions or
forcing them to upgrade.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[...snip...]
ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/worker':
  at .../external/rules_java/java/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl'
  is invalid because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'?
```
@mbland
Copy link
Contributor

mbland commented Dec 11, 2024

Weekly updates!

I'm a little late, but I've been busy making improvements to my branches and pull requests in the wake of the Bazel 8.0.0 release.

Merged

Open

Bazel 8

These latter two pull requests, #1653 and #1660, are related to #1652, which is especially relevant given the brand new Bazel 8.0.0 release. I've rebased both of those pull requests onto #1671 to get them to pass for now.

As I mentioned in an update to the previous update, I already had rules_scala building successfully with Bazel 8.0.0rc6 and rules_java 7.12.2, as well as the less strict 8.0.0rc7 and rules_java 7.12.3. But since the 8.0.0 release, I've updated the following branches, cleaning them up and ensuring they build with Bazel 8.0.0:

With these changes:

These branches bump these dependencies to these very recent versions:

  • ScalaPB jars: 0.11.17 => 1.0.0-alpha.1
  • rules_python: 0.38.0 => 1.0.0
  • rules_cc: 0.0.9 => 0.1.0
  • abseil-cpp: 20220623.1 => 20240722.0
  • rules_java: 7.12.3 => 8.6.2
  • protobuf: 21.7 => 29.1
  • rules_proto: 6.0.2 => 7.0.2

Bzlmod

I've also updated my bzlmod working branch quite a bit. This is the branch that still supports Bazel 6.5.0 and 7.4.1, but not 8.0.0.

To see the differences between what will be the next major version with Bzlmod and the following major version with Bazel 8 compatibility (though .bazelversion may well remain 7.4.1): git diff bzlmod bzlmod-bazel-8

Amongst the changes, which I've also merged into the bzlmod-bazel-8 branch:

  • Refactored module extension code for (hopefully) improved readability.
  • Forced rules_cc to version 0.0.9 for Bazel 6 compatibility, as explained in the top level MODULE.bazel file.
  • Fixed a few tests that broke under 6.5.0 using a regex from Update test scripts for Bzlmod compatibility #1622 for optional canonical repo name prefixes appearing in the expected output.
  • Added compatibility_level and bazel_compatibility to explicitly specify and enforce Bazel version compatibility and to alert users to potential breakage when upgrading. See also: The compatibility_level conundrum bazel#24302.
  • Added WORKSPACE.bzlmod files to ensure that no Bzlmod builds load any WORKSPACE content.

I've also got these branches ready to go after #1653 and #1660, as mentioned previously:

I'll also likely split the following commits into pull requests to land before the next major release:

mbland added a commit to mbland/rules_scala that referenced this issue Dec 16, 2024
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several
dependencies as high as they can go and still be compatible with Bazel
6.5.0 and 7.4.1.

- `bazel_skylib`: 1.4.1 => 1.7.1
- `go`: 1.19.5 => 1.23.4
- `rules_cc`: 0.0.6 => 0.0.9
- `rules_go`: 0.39.1 => 0.50.1
- `rules_java`: 7.9.0 => 7.12.3
- `rules_python`: 0.36.0 => 0.38.0

The `rules_java` 7.12.13 bump precipitated the following changes:

- Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file
  per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3.
  This replaces previous calls to instantiate `rules_java` repos and to
  register `rules_java` toolchains.

- Rearranges the other `WORKSPACE` setup macros for dependencies to come
  before the `rules_scala` setup macros.

- Updates almost all dependencies on `@bazel_tools//tools/jdk:`
  toolchain targets and `.bzl` files to corresponding targets and files
  in `@rules_java//toolchains:`.

- Removes several deprecated flags from the
  `scrooge_compile_with_jdk_11` test case from
  `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that
  caused it to break.

---

Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all
the versions of dependencies we want in `WORKSPACE`, while providing a
new API to consumers. It also prevents `WORKSPACE` from transitively
loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8
compatibility per bazelbuild#1652.

Reasons for the other `rules_java` related changes include:

- The `WORKSPACE` stanza for `rules_java` should've already been present
  while using the existing version 7.9.0. However, doing so would've
  broken Bazel 6 builds, as described in detail below.

- The `rules_java_toolchains()` call follows the `protobuf_deps()` call
  instead of immediately following `rules_java_dependencies()` because
  upgrading to `rules_java` >= 8.5.0 will require this. It has no
  adverse impact to do it now, amidst the other `WORKSPACE` changes, and
  will make the eventual `rules_java` >= 8.5.0 diff smaller.

- Having the other `WORKSPACE` setup macros for dependencies come before
  the `rules_scala` setup macros helps ensure consistent, correct
  initialization before `rules_scala` initialization.

- Updating the toolchain specifiers to use `@rules_java//toolchains`
  fixed `WORKSPACE` build breakages when updating to `rules_java`
  7.10.0 and later. This is a potentially breaking change for consumers,
  but in the good kind of way, in that it requires an easy, futureproof
  update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain,
  as there's not yet a corresponding `@rules_java//toolchains` target.)

What follows are detailed notes on why some dependency versions are
capped where they are, and on some breakages fixed by these changes.

---

`abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7,
respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to
v25.5 is compatible with Bazel 6 provided users set the compiler flags
mentioned in that issue:

```txt
build:linux --cxxopt=-std=c++17
build:linux --host_cxxopt=-std=c++17
build:macos --cxxopt=-std=c++17
build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std=c++17
build:windows --host_cxxopt=/std=c++17
```

---

`rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10,
which introduced `cc/common/cc_info.bzl`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]
ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:19:6:
  Label '@rules_cc//cc/common:cc_info.bzl' is invalid
    because 'cc/common' is not a package;
    perhaps you meant to put the colon here:
    '@rules_cc//cc:common/cc_info.bzl'?
```

However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines
`CcSharedLibraryHintInfo`:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33:
  name 'CcSharedLibraryHintInfo' is not defined (did you mean
  'CcSharedLibraryInfo'?)

[ ...snip... ]

ERROR: error loading package under directory 'test':
  error loading package 'test':
  at .../external/rules_python/python/defs.bzl:17:6:
  at .../external/rules_python/python/py_binary.bzl:18:6:
  at .../external/rules_python/python/private/py_binary_macro.bzl:16:6:
  at .../external/rules_python/python/private/common_bazel.bzl:18:6:
  at .../external/rules_cc/cc/common/cc_common.bzl:17:6:
  compilation of module 'cc/private/rules_impl/native.bzl' failed
```

---

This section provides a methodical description of the errors encountered
and resolved during each step of the `rules_java` update.

The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't
originally have in place, is:

```py
load(
    "@rules_java//java:repositories.bzl",
    "rules_java_dependencies",
    "rules_java_toolchains",
)

rules_java_dependencies()

rules_java_toolchains()
```

Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0
didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build.
Note the `CompiledWithJava{8,11}_java.jar` references:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: .../test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---

ERROR: .../test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.NoSuchMethodError:
  'java.lang.Iterable
   com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)'
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142)
    [ ...snip... ]
---8<---8<--- End of log ---8<---8<---
```

Another variation of this I also saw was:

```txt
$ bazel build //{src,jmh,test,third_party,scala_proto}/...

[ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:11:14:
  Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]

ERROR: test/src/main/resources/java_sources/BUILD:5:14:
  Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar
  (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<---
Error thrown by worker thread, shutting down worker.
  java.lang.UnsupportedClassVersionError:
  com/google/errorprone/ErrorProneError has been compiled by a more
  recent version of the Java Runtime (class file version 61.0), this
  version of the Java Runtime only recognizes class file versions up to
  55.0
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    [ ...snip... ]
```

Updating the toolchains from `@bazel_tools//tools/jdk:` to
`@rules_java//toolchains:` for targets in `test/BUILD` and
`test/src/main/resources/java_sources/BUILD` fixed this particular
issue. (More on updating other `@bazel_tools//tools/jdk` toolchain
targets below.) Bazel 6.5.0 then failed with the following expected
issue, fixed by `rules_java` 7.10.0 via:

- bazelbuild/rules_java#210
- bazelbuild/rules_java@30ecf3f

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34:
  Use of Starlark transition without allowlist attribute
  '_allowlist_function_transition'.
  See Starlark transitions documentation for details and usage:
  @rules_java//toolchains:java_toolchain_alias.bzl NORMAL

ERROR:
  .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12:
  While resolving toolchains for target
  //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter:
  invalid registered toolchain '//toolchains:all':
  while parsing '//toolchains:all':
  error loading package '@rules_java//toolchains':
  initialization of module 'toolchains/java_toolchain_alias.bzl' failed

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter'
  failed; build aborted:
```

Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused
Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619.
For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in
the message for commit cd22d88.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14:
  While resolving toolchains for target
  @@rules_java_builtin//toolchains:platformclasspath (096dcc8):
  No matching toolchains found for types
  @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type.

To debug, rerun with
  --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage
  reading https://bazel.build/concepts/platforms-intro.

ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed;
  build aborted: Analysis failed
```

My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0
built into Bazel's `WORKSPACE` preamble registered
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains
that later versions of `rules_java` couldn't resolve. This was due to
the very same change that fixed 6.5.0, bazelbuild/rules_java#210.

However, I mistakenly thought we needed to keep
`@bazel_tools//tools/jdk:` as the canonical Java toolchains package
instead switching to `@rules_java//toolchains:`. That's why I originally
thought we were stuck on `rules_java` 7.9.0.

I eventually rediscovered the following issue, which surfaced this
problem a month before bazelbuild#1619. The conversation helped me realize that
switching to `@rules_java//toolchains:` actually is the preferred,
futureproof approach (short of migrating to Bzlmod):

- bazelbuild/rules_java: Regression with
  `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0
  bazelbuild/rules_java#214

Switching all `@bazel_tools//tools/jdk:` toolchains to
`@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and
7.4.1 to successfully build using `rules_java` 7.12.2. I then updated
`rules_java` to 7.12.3, which returns to registering some toolchains as
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`:

- bazelbuild/rules_java#246
- bazelbuild/rules_java@b64eb7d

@hvadehra requested that I try 7.12.3 to see if it resolved the issue.
I was able to build successfully using this version in a branch without
the toolchain updates from this commit.

- bazelbuild#1652 (comment)

However, the changes from this commit should remain futureproof when
`@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to
`@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one
day.

---

Several of the flags removed from the `scrooge_compile_with_jdk_11` test
case have been no-ops for years, so removing them eliminated their
corresponding deprecation warnings. `--javacopt='--release 11'`,
however, caused this breakage after originally bumping to `rules_java`
7.12.2:

```txt
$ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh

running test scrooge_compile_with_jdk_11
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
WARNING: Option 'javabase' is deprecated
WARNING: Option 'host_javabase' is deprecated
WARNING: Option 'host_java_toolchain' is deprecated
WARNING: Option 'java_toolchain' is deprecated
INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured).
INFO: Found 64 targets...

ERROR:
  .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13:
  Compiling Java headers
  src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
  (1 source file) failed: (Exit 1): turbine_direct_graal failed:
  error executing command (from target
  //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions)

  external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal
    --output
    bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar
    ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
  java.lang.NullPointerException:
  attempted to use --release, but JAVA_HOME is not set
    at [email protected]/java.util.Objects.requireNonNull(Objects.java:259)
    at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55)
    at com.google.turbine.main.Main.release(Main.java:318)
    at com.google.turbine.main.Main.bootclasspath(Main.java:304)
    at com.google.turbine.main.Main.compile(Main.java:142)
    at com.google.turbine.main.Main.compile(Main.java:133)
    at com.google.turbine.main.Main.main(Main.java:89)
    at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH)

INFO: Elapsed time: 0.325s, Critical Path: 0.12s
INFO: 11 processes: 10 internal, 1 worker.
FAILED: Build did NOT complete successfully
 Test "scrooge_compile_with_jdk_11" failed  (0 sec)
```

See also:

- Bazel Blog: Bazel 5.0:
  https://blog.bazel.build/2022/01/19/bazel-5.0.html#java

- bazelbuild/bazel:
  `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain
  resolution for Java rules #7849
  bazelbuild/bazel#7849

---

Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8
will likely happen in a new major release _after_ a release containing
the Bzlmod changes. This is because:

- The Bzlmod changes alone will work with Bazel 6 without requiring that
  users update `protobuf` beyond version 21.7.

- Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add
  the afforementioned C++ compiler flags to support the newer
  `abseil-cpp` versions required by newer `protobuf` versions.

- `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or
  higher to support `protobuf` >= v28.0, but this ScalaPB version will
  _not_ support `protobuf` < v28.0.

The idea is that the next major release of `rules_scala` will help users
migrate to Bazel 7 and Bzlmod, in either order. Then the next major
release after that will enable them to migrate to Bazel 8, with all the
requisite dependency upgrades. (Technically it will still support
`WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See:

- bazelbuild#1482 (comment)

---

Here are the details explaining how `rules_java` 8 currently breaks
`rules_scala`, up until the point that upgrading to `protobuf` >= v29.0
would fix it.

`rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule`
API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java`
8.3.2.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR:
  .../external/rules_java/java/common/rules/android_lint.bzl:142:24:
  name 'subrule' is not defined (did you mean 'rule'?)

ERROR: Error computing the main repository mapping:
  at .../scala/private/extensions/dev_deps.bzl:8:6:
  at .../external/rules_java/java/repositories.bzl:20:6:
  at .../external/rules_java/toolchains/local_java_repository.bzl:17:6:
  at .../external/rules_java/java/defs.bzl:18:6:
  at .../external/rules_java/java/java_library.bzl:16:6:
  at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6:
  compilation of module 'java/common/rules/android_lint.bzl' failed
```

`rules_java` 8.3.0 is broken, as it can't find its own
`@compatibility_proxy` repo:

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

ERROR: error loading package under directory 'src':
  error loading package 'src/protobuf/io/bazel/rules_scala':
  at .../external/rules_java/java/defs.bzl:22:6:
  at .../external/rules_java/java/java_test.bzl:16:6:
  Unable to find package for @compatibility_proxy//:proxy.bzl:
  The repository '@compatibility_proxy' could not be resolved:
  Repository '@compatibility_proxy' is not defined.
```

`rules_java` 8.3.1 seems to fix this, presumably by importing the
`protobuf` repo as `com_google_protobuf`. However, it now requires at
least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`.
Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support
`protobuf` v28, abandoning users of previous `protobuf` versions or
forcing them to upgrade.

```txt
$ bazel build //{src,test,third_party,scala_proto}/...

[...snip...]
ERROR: error loading package under directory 'src':
  error loading package 'src/java/io/bazel/rulesscala/worker':
  at .../external/rules_java/java/defs.bzl:16:6:
  Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl'
  is invalid because 'bazel' is not a package;
  perhaps you meant to put the colon here:
  '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'?
```
mbland added a commit to mbland/rules_scala that referenced this issue Dec 16, 2024
Moves the Scalatest, JUnit, and Specs2 toolchains into
`@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482.

Updates all `WORKSPACE` files to set the appropriate `scala_toolchains`
parameters and to remove the unnecessary repository import and toolchain
registration macros.

Adds a `fetch_sources_by_id` parameter to `repositories` from
`third_party/repositories/repositories.bzl`. This enables
`scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping
from artifact ID lists returned by new macros extracted from `WORKSPACE`
macros. The values assigned to each id match the original
`fetch_sources` settings in the corresponding original `WORKSPACE`
macros.

Updates `scala/scala_maven_import_external.bzl` to generate a `load`
line for `//scala:scala_import.bzl` based on the repo's canonical name,
not `@io_bazel_rules_scala`.

As usual, includes several other opportunistic removals of the
`@io_bazel_rules_scala` repo name prefix to avoid an internal dependency
on that name. This means Bzlmod users won't necessarily have to set the
`repo_name` parameter of `bazel_dep` when using `rules_scala`.

---

Introduces more macros to return a framework's Maven artifact
dependencies, rather than inlining them in a `repositories` call. These
inlined lists are replaced by macro invocations, and now the
`scala_toolchains` macro can invoke these macros to collect artifact IDs
to pass to `repositories`. This also allows for future changes to
introduce a `scala_version` parameter if necessary, similar to how
`scalafmt_artifact_ids` already works.

This is important to avoid collisions when creating repositories for
artifacts upon which more than one framework depends under Bzlmod.
`WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will
produce errors like the following, where both `scala_proto` and
`twitter_scrooge` depend upon `io_bazel_rules_scala_guava`:

```txt
$ bazel build //src/...

ERROR: .../scala/scala_maven_import_external.bzl:299:24:
  Traceback (most recent call last):
    File ".../scala/extensions/deps.bzl",
      line 140, column 21, in _scala_deps_impl
        scala_toolchains(
    File ".../scala/private/macros/toolchains.bzl",
      line 140, column 17, in scala_toolchains
        _scrooge(
    File ".../twitter_scrooge/twitter_scrooge.bzl",
      line 96, column 17, in twitter_scrooge
        repositories(
    File ".../third_party/repositories/repositories.bzl",
      line 113, column 37, in repositories
        _scala_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 263, column 30, in scala_maven_import_external
        jvm_maven_import_external(
    File ".../scala/scala_maven_import_external.bzl",
      line 299, column 24, in jvm_maven_import_external
        jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs)

Error in repository_rule: A repo named
  io_bazel_rules_scala_guava_2_13_15
  is already generated by this module extension at
  .../scala/scala_maven_import_external.bzl:299:24

ERROR: Analysis of target
  '//src/java/io/bazel/rulesscala/worker:worker_test'
  failed; build aborted:
  error evaluating module extension scala_deps in
  //scala/extensions:deps.bzl
```

Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it
easy to emit full direct dependency lists for artifacts included in
`third_party/repositories/scala_*.bzl`. This increases the likelihood of
collisions, since this expanded metadata forces the macros that
instantiate artifact repos to instantiate even more repos.

By fetching list of artifact IDs from these macros, `scala_toolchains`
can now consolidate them into dictionary keys. Then it passes these
unique keys to `repositories` directly, avoiding the problem of
instantiating the same repo multiple times in the same module extension.

This, in turn, also avoids the need to add parameters to the original
`WORKSPACE` macros that instantiate dependencies to avoid collisions
under Bzlmod. The `scala_toolchains` macro never needs to call these
original macros, under either `WORKSPACE` or Bzlmod.

Finally, it also reduces duplication between these artifact ID lists and
the `_*_DEPS` symbols originally from `testing/BUILD` (and now in
`testing/deps.bzl`). The dependency labels are now generated
programatically.

(Aside: As I mentioned, we may eventually need to pass a Scala version
argument to these macros. It will be possible to cross that bridge
without too much trouble if and when that day comes. Or I can try to
future proof it in a follow up pull request.)
mbland added a commit to mbland/rules_scala that referenced this issue Dec 16, 2024
Removes this symbol from `scala/scala.bzl` as well as
`setup_scala_testing_toolchain`, and deletes
`scala/private/macros/toolchains.bzl`. Part of bazelbuild#1482 and bazelbuild#1652.

This is required for Bazel 8 and `rules_java` 8 compatibility, but is
also compatible with Bazel 6 and 7. In bazelbuild#1652, @hvadehra suggested
partitioning the `.bzl` files such that `WORKSPACE` doesn't `load` a
file that tries to `load` symbols from `rules_java`. I successfully did
so in a separate branch, and along with other minor changes, got
`rules_scala` to build with `rules_java` 8.5.1.

The other changes will come in separate pull requests, but it makes
sense to land this change now before adding any other toolchains to
`scala_toolchains`.

---

Arguably, we should remove all macros exported from `scala/scala.bzl`
that only instantiate toolchain dependencies and define toolchains. That
may be a breaking change for some users, but will ultimately be
necessary for these macros to remain compatible with Bazel 8.
@mbland
Copy link
Contributor

mbland commented Dec 17, 2024

Weekly updates!

Things haven't changed much from last week. Both Bzlmod and Bazel 8 compatibility are in good shape, and we're all collectively slowing down for the holidays. See last week's update for details regarding the state of these changes.

Note that I'm taking care to keep my bzlmod working branch and my bzlmod-bazel-8 working branch in a working state. You may choose to use one of these branches to prepare your own migration, weeding out potential issues ahead of time, until the changes land here in bazelbuild/rules_scala. For example, @meteorcloudy is using the bzlmod branch to migrate rules_webtesting in bazelbuild/rules_webtesting#478.

I'm also occasionally checking to ensure the bzlmod-bazel-8 branch remains compatible with the latest Bazel rolling release and Bazel last_green build.

Update 2024-12-17 7:20pm Eastern:

From scalapb/ScalaPB#1771 (comment) and #1647 (comment): I've confirmed in my replace-custom-scalapb-wrappers-with-protoc-bridge-0.9.8 branch that the recently released protoc-bridge 0.9.8 resolves scalapb/ScalaPB#1771 (i.e., protoc-bridge catches scalapb.ScalaPbCodeGenerator errors, so we don't have to). I've since closed that issue, and will send a pull request containing the changes from this branch eventually.

Mentioning here since the Bzlmodification work spawned the protobuf side quest (#1647), which ultimately provided insights affecting the Bazel 8 compatibility work.

Merged

Open

Bzlmod pending changes

Bazel 8 pending changes (#1652)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment