-
Notifications
You must be signed in to change notification settings - Fork 278
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Bzlmod and add rules_scala to bazel-central-registry #1482
Comments
Just a note to anyone that starts the work |
@chrislovecnm it's still available here no? https://bazel.build/rules/lib/globals/module#register_toolchains |
@sluongng When I last tried it, it did not work when using it inside an extension. In rules_python we are skipping the native call: And build the toolchains here: We build the toolchains and use the following: |
Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file. So |
I prepared a minimal bzlmod support here and gonna split it into a few PRs
|
@mateuszkuta256 thanks for getting this started! Any progress updates or blockers? |
hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future |
I'd like to take on the task of Bzlmodifying this repo through a series of pull requests. I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.) There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain For an example of what it looks like from a client perspective, here's what the bazel_dep(name = "rules_scala", repo_name = "io_bazel_rules_scala")
local_path_override(
module_name = "rules_scala",
path = "../../bazelbuild/rules_scala"
)
scala_dev_deps = use_extension(
"@io_bazel_rules_scala//scala/extensions:deps.bzl",
"scala_deps",
dev_dependency = True,
)
scala_dev_deps.toolchains(
scalatest = True,
) So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process. |
This begins the Bzlmod compatibility migration by updating Bazel to version 7.3.2 and adding initial `MODULE.bazel` and `WORKSPACE.bzlmod` files. Part of: bazelbuild#1482 Though Bzlmod remains disabled, updating to Bazel 7.3.2 requred updating or adding the following packages to maintain `WORKSPACE` compatibility. In `rules_scala_setup()`: - bazel_skylib: 1.4.1 => 1.7.1 - rules_cc: 0.0.6 => 0.0.10 - rules_java: 5.4.1 => 7.9.0 - rules_proto: 5.3.0-21.7 => 6.0.2 Dev dependencies in `WORKSPACE`: - com_google_protobuf: 28.2 - rules_pkg: 1.0.1 - rules_jvm_external: 6.4 - com_google_absl: abseil-cpp-20240722.0 - zlib: 1.3.1 Of all of the new, explicit dev dependencies, only `com_google_protobuf` will be necessary to include in `MODULE.bazel`. The Bzlmod mechanism will discover these other transitive dev dependencies automatically. Also removed the `rules_java_extra` repo from `WORKSPACE`, which appeared unused. --- Though the current `rules_java` version is 7.12.1, and largely works with this repo, it requires a few temporary workarounds. Rather than commit the workarounds, upgrading only to 7.9.0 now seems less crufty. What follows is a very detailed explanation of what happens with 7.12.1 with Bazel 7.3.2, just to have it on the record. --- The workaround is to change a few toolchain and macro file targets from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:`. This isn't a terribly bad or invasive workaround, but `@bazel_tools//tools/jdk:` is clearly the canonical path. Best to keep it that way, lest we build up technical debt. Without the workaround, these targets would fail: - //test/src/main/resources/java_sources:CompiledWithJava11 - //test/src/main/resources/java_sources:CompiledWithJava8 - //test/toolchains:java21_toolchain - //test:JunitRuntimePlatform - //test:JunitRuntimePlatform_test_runner - //test:scala_binary_jdk_11 with this error: ```txt ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. ``` This appears to be a consequence of both upgrading the Bazel version from 6.3.0 to 7.3.2 and updating `rules_java` to 7.12.1. The `rules_java_builtin` repo is part of the `WORKSPACE` prefix that adds implicit dependencies: - https://bazel.build/external/migration#builtin-default-deps This repo was added to 7.0.0-pre.20231011.2 in the following change, mapped to `@rules_java` within the scope of the `@bazel_tools` repo: - bazelbuild/bazel: Add rules_java_builtin to the users of Java modules bazelbuild/bazel@ff1abb2 This change tried to ensure `rules_java` remained compatible with earlier Bazel versions. However, it changed all instances of `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` to `//toolchains:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java: Make rules_java backwards compatible with Bazel 6.3.0 bazelbuild/rules_java@30ecf3f Bazel has bumped `rules_java` in its `workspace_deps.bzl` from 7.9.0 to 7.11.0, but it's only available as of 8.0.0-pre.20240911.1. - bazelbuild/bazel: Update rules_java 7.11.1 / java_tools 13.8 bazelbuild/bazel#23571 bazelbuild/bazel@f92124a --- What I believe is happening is, under Bazel 7.3.2 and `rules_java` 7.12.1: - Bazel creates `rules_java` 7.9.0 as `@rules_java_builtin` in the `WORKSPACE` prefix. - `@bazel_tools` has `@rules_java` mapped to `@rules_java_builtin` when initialized during the `WORKSPACE` prefix, during which `@bazel_tools//tools/jdk` registers `alias()` targets to `@rules_java` toolchain targets. These aliased toolchains specify `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` in their `toolchains` attribute. - `WORKSPACE` loads `@rules_java` 7.12.1 and registers all its toolchains with type `@rules_java//toolchains:bootstrap_runtime_toolchain_type`. - Some `@rules_java` rules explicitly specifying toolchains from `@bazel_tools//tools/jdk` can't find them, because the `@bazel_tools//tools/jdk` toolchain aliases expect toolchains of type `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`. This has broken other projects in the same way: - bazelbuild/bazel: [Bazel CI] Downstream project broken by rules_java upgrade #23619 bazelbuild/bazel#23619 These problems don't appear under Bzlmod, and `@rules_java_builtin` was never required. This is because `WORKSPACE` executes its statements sequentially, while Bzlmod builds the module dependency graph _before_ instantiating repositories (within module extensions). It seems a fix is on the way that removes `@rules_java_builtin` from the `WORKSPACE` prefix, and adds `@rules_java` to the suffix. At this moment, though, it's not even in a prerelase: - bazelbuild/bazel: Remove rules_java_builtin in WORKSPACE prefix bazelbuild/bazel@7506690 --- Note that the error message matches that from the following resolved issue, but that issue was for non-Bzlmod child repos when `WORKSPACE` was disabled. - bazelbuild/bazel: Undefined @@rules_java_builtin repository with --noenable_workspace option bazelbuild/bazel#22754
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation documented at: - bazelbuild#1619 (comment) Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources` values between Bazel 6 and Bazel 7. Without this change, `_import_paths()` emits incorrect values under Bazel 7, causing targets containing generated `.proto` inputs to fail, e.g. `//test/proto3:test_generated_proto`. See also: - Fix paths for sibling repository setup and generated .proto files bazelbuild/bazel@6c6c196 - The docstring for `ProtoInfo.proto_source_root` in the Bazel sources: https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172 - Remove incompatible_generated_protos_in_virtual_imports bazelbuild/bazel@3efaa32 - Comment from: Generated Protos are no longer considered as virtual_imports in Bazel 7 bazelbuild/bazel#21075 (comment) --- I cherrypicked this commit into bazelbuild#1618. While it fixed the `//test/proto3` build failure, it does _not_ fix the hanging scalapb_workers from the ProtoScalaPBRule aspect. I'll have to investiate further whether than hang is related to Bazel, rules_proto, com_google_protobuf, or some mixture thereof. Still, progress!
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation documented at: - bazelbuild#1619 (comment) Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources` values between Bazel 6 and Bazel 7. Without this change, `_import_paths()` emits incorrect values under Bazel 7, causing targets containing generated `.proto` inputs to fail, e.g. `//test/proto3:test_generated_proto`. See also: - Fix paths for sibling repository setup and generated .proto files bazelbuild/bazel@6c6c196 - The docstring for `ProtoInfo.proto_source_root` in the Bazel sources: https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172 - Remove incompatible_generated_protos_in_virtual_imports bazelbuild/bazel@3efaa32 - Comment from: Generated Protos are no longer considered as virtual_imports in Bazel 7 bazelbuild/bazel#21075 (comment) --- I cherrypicked this commit into bazelbuild#1618. While it fixed the `//test/proto3` build failure, it does _not_ fix the hanging scalapb_workers from the ProtoScalaPBRule aspect. I'll have to investiate further whether than hang is related to Bazel, rules_proto, com_google_protobuf, or some mixture thereof. Still, progress!
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation documented at: - bazelbuild#1619 (comment) Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources` values between Bazel 6 and Bazel 7. Without this change, `_import_paths()` emits incorrect values under Bazel 7, causing targets containing generated `.proto` inputs to fail, e.g. `//test/proto3:test_generated_proto`. See also: - Fix paths for sibling repository setup and generated .proto files bazelbuild/bazel@6c6c196 - The docstring for `ProtoInfo.proto_source_root` in the Bazel sources: https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172 - Remove incompatible_generated_protos_in_virtual_imports bazelbuild/bazel@3efaa32 - Comment from: Generated Protos are no longer considered as virtual_imports in Bazel 7 bazelbuild/bazel#21075 (comment) --- I cherrypicked this commit into bazelbuild#1618. While it fixed the `//test/proto3` build failure, it does _not_ fix the hanging scalapb_workers from the ProtoScalaPBRule aspect. I'll have to investiate further whether than hang is related to Bazel, rules_proto, com_google_protobuf, or some mixture thereof. Still, progress!
Related to bazelbuild#1482, bazelbuild#1618, and bazelbuild#1619. Results from the investigation documented at: - bazelbuild#1619 (comment) Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources` values between Bazel 6 and Bazel 7. Without this change, `_import_paths()` emits incorrect values under Bazel 7, causing targets containing generated `.proto` inputs to fail, e.g. `//test/proto3:test_generated_proto`. See also: - Fix paths for sibling repository setup and generated .proto files bazelbuild/bazel@6c6c196 - The docstring for `ProtoInfo.proto_source_root` in the Bazel sources: https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172 - Remove incompatible_generated_protos_in_virtual_imports bazelbuild/bazel@3efaa32 - Comment from: Generated Protos are no longer considered as virtual_imports in Bazel 7 bazelbuild/bazel#21075 (comment) --- I cherrypicked this commit into bazelbuild#1618. While it fixed the `//test/proto3` build failure, it does _not_ fix the hanging scalapb_workers from the ProtoScalaPBRule aspect. I'll have to investiate further whether than hang is related to Bazel, rules_proto, com_google_protobuf, or some mixture thereof. Still, progress!
Part of bazelbuild#1482. Splits the last component off of canonical repo names to produce the expected repo name. Without Bzlmod, it returns the original name. With Bzlmod enabled, it avoids generating output like: scala_import( name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler", jars = ["scala-compiler-2.12.18.jar"], ) resulting in errors like: ``` ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD: no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler': target 'io_bazel_rules_scala_scala_compiler' not declared in package '' defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider' ``` Also fixes the following error when attaching resources from custom repos to targets under Bzlmod: ```txt $ bazel test //test/src/main/scala/scalarules/test/resources:all 1) Scala library depending on resources from external resource-only jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest) java.lang.NullPointerException at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11) ``` Can be replaced with a future bazel-skylib implementation, if accepted into that repo. --- We can't rely on the specific canonical repository name format: > Repos generated by extensions have canonical names in the form of > `module_repo_canonical_name+extension_name+repo_name`. For extensions > hosted in the root module, the `module_repo_canonical_name` part is > replaced with the string `_main`. Note that the canonical name format is > not an API you should depend on — it's subject to change at any time. > > - https://bazel.build/external/extension#repository_names_and_visibility The change to no longer encode module versions in canonical repo names in Bazel 7.1.0 is a recent example of Bazel maintainers altering the format: - bazelbuild/bazel#21316 And the maintainers recently replaced the `~` delimiter with `+` in the upcoming Bazel 8 release due to build performance issues on Windows: - bazelbuild/bazel#22865 This function assumes the only valid `repo_name` characters are letters, numbers, '_', '-', and '.'. It finds the last character not in this set, and returns the contents of `name` following this character. This is valid so long as this condition holds: - https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
Part of bazelbuild#1482. Splits the last component off of canonical repo names to produce the expected repo name. Without Bzlmod, it returns the original name. With Bzlmod enabled, it avoids generating output like: scala_import( name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler", jars = ["scala-compiler-2.12.18.jar"], ) resulting in errors like: ``` ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD: no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler': target 'io_bazel_rules_scala_scala_compiler' not declared in package '' defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider' ``` Also fixes the following error when attaching resources from custom repos to targets under Bzlmod: ```txt $ bazel test //test/src/main/scala/scalarules/test/resources:all 1) Scala library depending on resources from external resource-only jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest) java.lang.NullPointerException at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11) ``` Can be replaced with a future bazel-skylib implementation, if accepted into that repo. --- We can't rely on the specific canonical repository name format: > Repos generated by extensions have canonical names in the form of > `module_repo_canonical_name+extension_name+repo_name`. For extensions > hosted in the root module, the `module_repo_canonical_name` part is > replaced with the string `_main`. Note that the canonical name format is > not an API you should depend on — it's subject to change at any time. > > - https://bazel.build/external/extension#repository_names_and_visibility The change to no longer encode module versions in canonical repo names in Bazel 7.1.0 is a recent example of Bazel maintainers altering the format: - bazelbuild/bazel#21316 And the maintainers recently replaced the `~` delimiter with `+` in the upcoming Bazel 8 release due to build performance issues on Windows: - bazelbuild/bazel#22865 This function assumes the only valid `repo_name` characters are letters, numbers, '_', '-', and '.'. It finds the last character not in this set, and returns the contents of `name` following this character. This is valid so long as this condition holds: - https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
Related to #1482, #1618, and #1619. Results from the investigation documented at: - #1619 (comment) Updates `_import_paths()` in `scala_proto_aspect.bzl` to handle differences `ProtoInfo.proto_source_root` and `ProtoInfo.direct_sources` values between Bazel 6 and Bazel 7. Without this change, `_import_paths()` emits incorrect values under Bazel 7, causing targets containing generated `.proto` inputs to fail, e.g. `//test/proto3:test_generated_proto`. See also: - Fix paths for sibling repository setup and generated .proto files bazelbuild/bazel@6c6c196 - The docstring for `ProtoInfo.proto_source_root` in the Bazel sources: https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/starlark/builtins_bzl/common/proto/proto_info.bzl#L155-L172 - Remove incompatible_generated_protos_in_virtual_imports bazelbuild/bazel@3efaa32 - Comment from: Generated Protos are no longer considered as virtual_imports in Bazel 7 bazelbuild/bazel#21075 (comment) --- I cherrypicked this commit into #1618. While it fixed the `//test/proto3` build failure, it does _not_ fix the hanging scalapb_workers from the ProtoScalaPBRule aspect. I'll have to investiate further whether than hang is related to Bazel, rules_proto, com_google_protobuf, or some mixture thereof. Still, progress!
Part of bazelbuild#1482. Splits the last component off of canonical repo names to produce the expected repo name. Without Bzlmod, it returns the original name. With Bzlmod enabled, it avoids generating output like: scala_import( name = "_main~scala_deps~io_bazel_rules_scala_scala_compiler", jars = ["scala-compiler-2.12.18.jar"], ) resulting in errors like: ``` ERROR: .../_main~_repo_rules~io_bazel_rules_scala/scala/BUILD: no such target '@@_main~scala_deps~io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler': target 'io_bazel_rules_scala_scala_compiler' not declared in package '' defined by .../_main~scala_deps~io_bazel_rules_scala_scala_compiler/BUILD and referenced by '@@_main~_repo_rules~io_bazel_rules_scala//scala:default_toolchain_scala_compile_classpath_provider' ``` Also fixes the following error when attaching resources from custom repos to targets under Bzlmod: ```txt $ bazel test //test/src/main/scala/scalarules/test/resources:all 1) Scala library depending on resources from external resource-only jar::allow to load resources(scalarules.test.resources.ScalaLibResourcesFromExternalDepTest) java.lang.NullPointerException at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.get(ScalaLibResourcesFromExternalDepTest.scala:17) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$3(ScalaLibResourcesFromExternalDepTest.scala:11) at scalarules.test.resources.ScalaLibResourcesFromExternalDepTest.$anonfun$new$2(ScalaLibResourcesFromExternalDepTest.scala:11) ``` Can be replaced with a future bazel-skylib implementation, if accepted into that repo. --- We can't rely on the specific canonical repository name format: > Repos generated by extensions have canonical names in the form of > `module_repo_canonical_name+extension_name+repo_name`. For extensions > hosted in the root module, the `module_repo_canonical_name` part is > replaced with the string `_main`. Note that the canonical name format is > not an API you should depend on — it's subject to change at any time. > > - https://bazel.build/external/extension#repository_names_and_visibility The change to no longer encode module versions in canonical repo names in Bazel 7.1.0 is a recent example of Bazel maintainers altering the format: - bazelbuild/bazel#21316 And the maintainers recently replaced the `~` delimiter with `+` in the upcoming Bazel 8 release due to build performance issues on Windows: - bazelbuild/bazel#22865 This function assumes the only valid `repo_name` characters are letters, numbers, '_', '-', and '.'. It finds the last character not in this set, and returns the contents of `name` following this character. This is valid so long as this condition holds: - https://github.com/bazelbuild/bazel/blob/7.3.2/src/main/java/com/google/devtools/build/lib/cmdline/RepositoryName.java#L159-L162
A quick update for visibility: I'm very close to having the Bzlmodified rules_scala passing 100% of the tests, down to the last couple of failures:
I also need to get
Hopefully I can get these fixed up today, and I'll start peeling off the next pull request or two. And thanks to @simuons and @liucijus for reviewing and merging #1619 and #1620 already. |
Removes `native.{,new_}local_repository()` calls in macros in favor of `local_repository` calls from `WORKSPACE`. Part of bazelbuild#1482. `native.{,new_}local_repository()` isn't available under Bzlmod, `@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified definitions isn't available under Bazel 6, and Bazel 8 compatibility work is imminent. Redefining the repositories in this way will be compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod. (`MODULE.bazel` will use a combination of `bazel_dep()` and `local_path_override()`.)
This is part of the Bzlmod prep work from bazelbuild#1482, but also reduces duplication between the `dt_patches/test_dt_patches*/WORKSPACE` files. The structure of the `dt_patches/compiler_sources/extensions.bzl` declarations accommodates the fact that, unlike `WORKSPACE, `MODULE.bazel` files cannot import constants or contain conditional statements. That is to say, there's no way to port the `if SCALA_VERSION.startswith("2.")` expressions from the previous `WORKSPACE` files to Bzlmod. The new `import_compiler_source_repos` macro, however, works for both `WORKSPACE` and Bzlmod builds, as it's trivial to wrap `import_compiler_source_repos` in a module extension.
This is part of the Bzlmod prep work from #1482, but also reduces duplication between the `dt_patches/test_dt_patches*/WORKSPACE` files. The structure of the `dt_patches/compiler_sources/extensions.bzl` declarations accommodates the fact that, unlike `WORKSPACE, `MODULE.bazel` files cannot import constants or contain conditional statements. That is to say, there's no way to port the `if SCALA_VERSION.startswith("2.")` expressions from the previous `WORKSPACE` files to Bzlmod. The new `import_compiler_source_repos` macro, however, works for both `WORKSPACE` and Bzlmod builds, as it's trivial to wrap `import_compiler_source_repos` in a module extension.
Every Scala version now uses com.google.protobuf:protobuf-java:4.29.0. Part of #1482 and #1652. This is necessary, but not sufficient, to make `rules_scala` Bazel 8 compatible by enabling an update to `protobuf` v29.0. Building with Bazel 6.5.0 and `protobuf` v21.7 continues to pass all tests after this change.
…e` (#1656) * Extract dev_deps_repositories() from WORKSPACE `dev_deps_repositories()` encapsulates the instantiation of repositories used only for `rules_scala` development. Also removes the unused `//private` package and its `WORKSPACE` statements. Part of #1482. Both `WORKSPACE` and Bzlmod builds can use this macro, though how Bzlmod will use it will depend on whether we continue building with Bazel 6. `@bazel_tools//tools/build_defs/repo:local.bzl` isn't available under Bazel 6. To continue building with Bazel 6 under Bzlmod, we will need to call `dev_deps_repositories()` from `WORKSPACE.bzlmod` to continue using `native.{,new_}local_repository()`. If we switch to Bazel 7, we can load `local.bzl` and strip the `native.` prefix from the `{,new_}local_repository()` calls. Then we can call `dev_deps_repositories()` from a module extension instead of from `WORKSPACE.bzlmod`. Another alternative would be updating the local repositories to become proper nested repositories. Then we can call `local_repository()` from `WORKSPACE` and call `bazel_dep()` and `local_path_override()` from `MODULE.bazel`. In that case, we'd remove the `{,new_}local_repository` calls from `dev_deps_dependencies()`, and remove `proto_cross_repo_boundary_repository()` entirely. * Add test local_repository calls to WORKSPACE Removes `native.{,new_}local_repository()` calls in macros in favor of `local_repository` calls from `WORKSPACE`. Part of #1482. `native.{,new_}local_repository()` isn't available under Bzlmod, `@bazel_tools//tools/build_defs/repo:local.bzl` with the Starlarkified definitions isn't available under Bazel 6, and Bazel 8 compatibility work is imminent. Redefining the repositories in this way will be compatible with Bazel 6, 7, and 8, both under `WORKSPACE` and Bzlmod. (`MODULE.bazel` will use a combination of `bazel_dep()` and `local_path_override()`.)
Moves the Scalatest, JUnit, and Specs2 toolchains into `@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482. Updates all `WORKSPACE` files to set the appropriate `scala_toolchains` parameters and to remove the unnecessary repository import and toolchain registration macros. Adds a `fetch_sources_by_id` parameter to `repositories` from `third_party/repositories/repositories.bzl`. This enables `scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping from artifact ID lists returned by new macros extracted from `WORKSPACE` macros. The values assigned to each id match the original `fetch_sources` settings in the corresponding original `WORKSPACE` macros. Updates `scala/scala_maven_import_external.bzl` to generate a `load` line for `//scala:scala_import.bzl` based on the repo's canonical name, not `@io_bazel_rules_scala`. As usual, includes several other opportunistic removals of the `@io_bazel_rules_scala` repo name prefix to avoid an internal dependency on that name. This means Bzlmod users won't necessarily have to set the `repo_name` parameter of `bazel_dep` when using `rules_scala`. --- Introduces more macros to return a framework's Maven artifact dependencies, rather than inlining them in a `repositories` call. These inlined lists are replaced by macro invocations, and now the `scala_toolchains` macro can invoke these macros to collect artifact IDs to pass to `repositories`. This also allows for future changes to introduce a `scala_version` parameter if necessary, similar to how `scalafmt_artifact_ids` already works. This is important to avoid collisions when creating repositories for artifacts upon which more than one framework depends under Bzlmod. `WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will produce errors like the following, where both `scala_proto` and `twitter_scrooge` depend upon `io_bazel_rules_scala_guava`: ```txt $ bazel build //src/... ERROR: .../scala/scala_maven_import_external.bzl:299:24: Traceback (most recent call last): File ".../scala/extensions/deps.bzl", line 140, column 21, in _scala_deps_impl scala_toolchains( File ".../scala/private/macros/toolchains.bzl", line 140, column 17, in scala_toolchains _scrooge( File ".../twitter_scrooge/twitter_scrooge.bzl", line 96, column 17, in twitter_scrooge repositories( File ".../third_party/repositories/repositories.bzl", line 113, column 37, in repositories _scala_maven_import_external( File ".../scala/scala_maven_import_external.bzl", line 263, column 30, in scala_maven_import_external jvm_maven_import_external( File ".../scala/scala_maven_import_external.bzl", line 299, column 24, in jvm_maven_import_external jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs) Error in repository_rule: A repo named io_bazel_rules_scala_guava_2_13_15 is already generated by this module extension at .../scala/scala_maven_import_external.bzl:299:24 ERROR: Analysis of target '//src/java/io/bazel/rulesscala/worker:worker_test' failed; build aborted: error evaluating module extension scala_deps in //scala/extensions:deps.bzl ``` Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it easy to emit full direct dependency lists for artifacts included in `third_party/repositories/scala_*.bzl`. This increases the likelihood of collisions, since this expanded metadata forces the macros that instantiate artifact repos to instantiate even more repos. By fetching list of artifact IDs from these macros, `scala_toolchains` can now consolidate them into dictionary keys. Then it passes these unique keys to `repositories` directly, avoiding the problem of instantiating the same repo multiple times in the same module extension. This, in turn, also avoids the need to add parameters to the original `WORKSPACE` macros that instantiate dependencies to avoid collisions under Bzlmod. The `scala_toolchains` macro never needs to call these original macros, under either `WORKSPACE` or Bzlmod. Finally, it also reduces duplication between these artifact ID lists and the `_*_DEPS` symbols originally from `testing/BUILD` (and now in `testing/deps.bzl`). The dependency labels are now generated programatically. (Aside: As I mentioned, we may eventually need to pass a Scala version argument to these macros. It will be possible to cross that bridge without too much trouble if and when that day comes. Or I can try to future proof it in a follow up pull request.)
Removes this symbol from `scala/scala.bzl` as well as `setup_scala_testing_toolchain`, and deletes `scala/private/macros/toolchains.bzl`. Part of bazelbuild#1482 and bazelbuild#1652. This is required for Bazel 8 and `rules_java` 8 compatibility, but is also compatible with Bazel 6 and 7. In bazelbuild#1652, @hvadehra suggested partitioning the `.bzl` files such that `WORKSPACE` doesn't `load` a file that tries to `load` symbols from `rules_java`. I successfully did so in a separate branch, and along with other minor changes, got `rules_scala` to build with `rules_java` 8.5.1. The other changes will come in separate pull requests, but it makes sense to land this change now before adding any other toolchains to `scala_toolchains`. --- Arguably, we should remove all macros exported from `scala/scala.bzl` that only instantiate toolchain dependencies and define toolchains. That may be a breaking change for some users, but will ultimately be necessary for these macros to remain compatible with Bazel 8.
Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. Part of bazelbuild#1482 and bazelbuild#1652. All dependencies on `@bazel_tools//tools/jdk:toolchain_type` remain, however, as there is not yet a corresponding target in `@rules_java//toolchains:`. Adds the `WORKSPACE` stanza recommended in the `rules_java` release notes, and removes our own calls to instantiate `rules_java` repos and to register `rules_java` toolchains. These changes are required to avoid build breakages when updating to `rules_scala` 7.10.0 and higher. All targets build and all tests pass under Bazel 6.5.0 and 7.4.1. --- I was on the right track in my analysis from bazelbuild#1619 ("Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88). However, I thought we _shouldn't_ update any targets from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:`. That's why I thought we were stuck on `rules_java` 7.9.0. However, this comment from @fmeum made me think switching to `@rules_java//toolchains:` actually is the preferred approach: - bazelbuild/rules_java#214 (comment) So this is a potentially breaking change, but in the good kind of way, in that it requires an easy, future proof update.
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
This is a pretty straightforward and easy update on top of the previous `setup_toolchains` and `scala_toolchains_repo` changes. Part of bazelbuild#1482.
Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482. The most significant part of the change is moving all the toolchain rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in `scala_proto/toolchains.bzl`. Adds the `scala_proto_deps_providers()` macro to replace `//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in the `dep_providers` parameter of `scala_proto_deps_toolchain()`. Examples of this are in `test/proto/custom_generator/BUILD`. Excludes `@scala_proto_rules_scalapb_protoc_gen` from `DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl` for Scala 2.11. For other Scala versions, this repo name will have the Scala version appended. This is to avoid build failures under Bzlmod, since: - This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at ScalaPB 0.9.8. - Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under Scala 2.11 results in an error under Bzlmod, as does importing it multiple times when configuring multiple Scala versions. - `MODULE.bazel` can iterate over a list of Scala versions, filtering out Scala 2.11, and call `use_repo()` on each version specific repo. A lot of the other changes are more opportunistic removals of `@io_bazel_rules_scala` label prefixes and application of `Label()` where appropriate. Doing this will allow Bzlmod users to use `rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in `bazel_dep()`.
Adds the jmh toolchain to `scala_toolchains()` and moves `jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
This is the last of the toolchain to receive the "toolchainization" treatment prior to Bzlmodification, and moves `twitter_scrooge()` to `twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
Moves the Scalatest, JUnit, and Specs2 toolchains into `@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482. Updates all `WORKSPACE` files to set the appropriate `scala_toolchains` parameters and to remove the unnecessary repository import and toolchain registration macros. Adds a `fetch_sources_by_id` parameter to `repositories` from `third_party/repositories/repositories.bzl`. This enables `scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping from artifact ID lists returned by new macros extracted from `WORKSPACE` macros. The values assigned to each id match the original `fetch_sources` settings in the corresponding original `WORKSPACE` macros. Updates `scala/scala_maven_import_external.bzl` to generate a `load` line for `//scala:scala_import.bzl` based on the repo's canonical name, not `@io_bazel_rules_scala`. As usual, includes several other opportunistic removals of the `@io_bazel_rules_scala` repo name prefix to avoid an internal dependency on that name. This means Bzlmod users won't necessarily have to set the `repo_name` parameter of `bazel_dep` when using `rules_scala`. --- Introduces more macros to return a framework's Maven artifact dependencies, rather than inlining them in a `repositories` call. These inlined lists are replaced by macro invocations, and now the `scala_toolchains` macro can invoke these macros to collect artifact IDs to pass to `repositories`. This also allows for future changes to introduce a `scala_version` parameter if necessary, similar to how `scalafmt_artifact_ids` already works. This is important to avoid collisions when creating repositories for artifacts upon which more than one framework depends under Bzlmod. `WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will produce errors like the following, where both `scala_proto` and `twitter_scrooge` depend upon `io_bazel_rules_scala_guava`: ```txt $ bazel build //src/... ERROR: .../scala/scala_maven_import_external.bzl:299:24: Traceback (most recent call last): File ".../scala/extensions/deps.bzl", line 140, column 21, in _scala_deps_impl scala_toolchains( File ".../scala/private/macros/toolchains.bzl", line 140, column 17, in scala_toolchains _scrooge( File ".../twitter_scrooge/twitter_scrooge.bzl", line 96, column 17, in twitter_scrooge repositories( File ".../third_party/repositories/repositories.bzl", line 113, column 37, in repositories _scala_maven_import_external( File ".../scala/scala_maven_import_external.bzl", line 263, column 30, in scala_maven_import_external jvm_maven_import_external( File ".../scala/scala_maven_import_external.bzl", line 299, column 24, in jvm_maven_import_external jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs) Error in repository_rule: A repo named io_bazel_rules_scala_guava_2_13_15 is already generated by this module extension at .../scala/scala_maven_import_external.bzl:299:24 ERROR: Analysis of target '//src/java/io/bazel/rulesscala/worker:worker_test' failed; build aborted: error evaluating module extension scala_deps in //scala/extensions:deps.bzl ``` Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it easy to emit full direct dependency lists for artifacts included in `third_party/repositories/scala_*.bzl`. This increases the likelihood of collisions, since this expanded metadata forces the macros that instantiate artifact repos to instantiate even more repos. By fetching list of artifact IDs from these macros, `scala_toolchains` can now consolidate them into dictionary keys. Then it passes these unique keys to `repositories` directly, avoiding the problem of instantiating the same repo multiple times in the same module extension. This, in turn, also avoids the need to add parameters to the original `WORKSPACE` macros that instantiate dependencies to avoid collisions under Bzlmod. The `scala_toolchains` macro never needs to call these original macros, under either `WORKSPACE` or Bzlmod. Finally, it also reduces duplication between these artifact ID lists and the `_*_DEPS` symbols originally from `testing/BUILD` (and now in `testing/deps.bzl`). The dependency labels are now generated programatically. (Aside: As I mentioned, we may eventually need to pass a Scala version argument to these macros. It will be possible to cross that bridge without too much trouble if and when that day comes. Or I can try to future proof it in a follow up pull request.)
Removes this symbol from `scala/scala.bzl` as well as `setup_scala_testing_toolchain`, and deletes `scala/private/macros/toolchains.bzl`. Part of bazelbuild#1482 and bazelbuild#1652. This is required for Bazel 8 and `rules_java` 8 compatibility, but is also compatible with Bazel 6 and 7. In bazelbuild#1652, @hvadehra suggested partitioning the `.bzl` files such that `WORKSPACE` doesn't `load` a file that tries to `load` symbols from `rules_java`. I successfully did so in a separate branch, and along with other minor changes, got `rules_scala` to build with `rules_java` 8.5.1. The other changes will come in separate pull requests, but it makes sense to land this change now before adding any other toolchains to `scala_toolchains`. --- Arguably, we should remove all macros exported from `scala/scala.bzl` that only instantiate toolchain dependencies and define toolchains. That may be a breaking change for some users, but will ultimately be necessary for these macros to remain compatible with Bazel 8.
This is a pretty straightforward and easy update on top of the previous `setup_toolchains` and `scala_toolchains_repo` changes. Part of bazelbuild#1482.
Adds scala_proto toolchains to `scala_toolchains()`. Part of bazelbuild#1482. The most significant part of the change is moving all the toolchain rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in `scala_proto/toolchains.bzl`. Adds the `scala_proto_deps_providers()` macro to replace `//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in the `dep_providers` parameter of `scala_proto_deps_toolchain()`. Examples of this are in `test/proto/custom_generator/BUILD`. Excludes `@scala_proto_rules_scalapb_protoc_gen` from `DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl` for Scala 2.11. For other Scala versions, this repo name will have the Scala version appended. This is to avoid build failures under Bzlmod, since: - This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at ScalaPB 0.9.8. - Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under Scala 2.11 results in an error under Bzlmod, as does importing it multiple times when configuring multiple Scala versions. - `MODULE.bazel` can iterate over a list of Scala versions, filtering out Scala 2.11, and call `use_repo()` on each version specific repo. A lot of the other changes are more opportunistic removals of `@io_bazel_rules_scala` label prefixes and application of `Label()` where appropriate. Doing this will allow Bzlmod users to use `rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in `bazel_dep()`.
Adds the jmh toolchain to `scala_toolchains()` and moves `jmh_repositories()` to `jmh/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
This is the last of the toolchain to receive the "toolchainization" treatment prior to Bzlmodification, and moves `twitter_scrooge()` to `twitter_scrooge/toolchain/toolchain.bzl` for `rules_java` 8 compatibility. Part of bazelbuild#1482 and bazelbuild#1652.
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - The `rules_java_toolchains()` call follows the `protobuf_deps()` call instead of immediately following `rules_java_dependencies()` because upgrading to `rules_java` >= 8.5.0 will require this. It has no adverse impact to do it now, amidst the other `WORKSPACE` changes, and will make the eventual `rules_java` >= 8.5.0 diff smaller. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - The `rules_java_toolchains()` call follows the `protobuf_deps()` call instead of immediately following `rules_java_dependencies()` because upgrading to `rules_java` >= 8.5.0 will require this. It has no adverse impact to do it now, amidst the other `WORKSPACE` changes, and will make the eventual `rules_java` >= 8.5.0 diff smaller. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
Weekly updates! I'm a little late, but I've been busy making improvements to my branches and pull requests in the wake of the Bazel 8.0.0 release. Merged
Open
Bazel 8These latter two pull requests, #1653 and #1660, are related to #1652, which is especially relevant given the brand new Bazel 8.0.0 release. I've rebased both of those pull requests onto #1671 to get them to pass for now. As I mentioned in an update to the previous update, I already had With these changes:
These branches bump these dependencies to these very recent versions:
BzlmodI've also updated my To see the differences between what will be the next major version with Bzlmod and the following major version with Bazel 8 compatibility (though Amongst the changes, which I've also merged into the
I've also got these branches ready to go after #1653 and #1660, as mentioned previously:
I'll also likely split the following commits into pull requests to land before the next major release: |
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - The `rules_java_toolchains()` call follows the `protobuf_deps()` call instead of immediately following `rules_java_dependencies()` because upgrading to `rules_java` >= 8.5.0 will require this. It has no adverse impact to do it now, amidst the other `WORKSPACE` changes, and will make the eventual `rules_java` >= 8.5.0 diff smaller. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
Moves the Scalatest, JUnit, and Specs2 toolchains into `@io_bazel_rules_scala_toolchains//testing`. Part of bazelbuild#1482. Updates all `WORKSPACE` files to set the appropriate `scala_toolchains` parameters and to remove the unnecessary repository import and toolchain registration macros. Adds a `fetch_sources_by_id` parameter to `repositories` from `third_party/repositories/repositories.bzl`. This enables `scala_toolchains` to build the `artifact_ids_to_fetch_sources` mapping from artifact ID lists returned by new macros extracted from `WORKSPACE` macros. The values assigned to each id match the original `fetch_sources` settings in the corresponding original `WORKSPACE` macros. Updates `scala/scala_maven_import_external.bzl` to generate a `load` line for `//scala:scala_import.bzl` based on the repo's canonical name, not `@io_bazel_rules_scala`. As usual, includes several other opportunistic removals of the `@io_bazel_rules_scala` repo name prefix to avoid an internal dependency on that name. This means Bzlmod users won't necessarily have to set the `repo_name` parameter of `bazel_dep` when using `rules_scala`. --- Introduces more macros to return a framework's Maven artifact dependencies, rather than inlining them in a `repositories` call. These inlined lists are replaced by macro invocations, and now the `scala_toolchains` macro can invoke these macros to collect artifact IDs to pass to `repositories`. This also allows for future changes to introduce a `scala_version` parameter if necessary, similar to how `scalafmt_artifact_ids` already works. This is important to avoid collisions when creating repositories for artifacts upon which more than one framework depends under Bzlmod. `WORKSPACE` doesn't seem affected by these collisions, but Bzlmod will produce errors like the following, where both `scala_proto` and `twitter_scrooge` depend upon `io_bazel_rules_scala_guava`: ```txt $ bazel build //src/... ERROR: .../scala/scala_maven_import_external.bzl:299:24: Traceback (most recent call last): File ".../scala/extensions/deps.bzl", line 140, column 21, in _scala_deps_impl scala_toolchains( File ".../scala/private/macros/toolchains.bzl", line 140, column 17, in scala_toolchains _scrooge( File ".../twitter_scrooge/twitter_scrooge.bzl", line 96, column 17, in twitter_scrooge repositories( File ".../third_party/repositories/repositories.bzl", line 113, column 37, in repositories _scala_maven_import_external( File ".../scala/scala_maven_import_external.bzl", line 263, column 30, in scala_maven_import_external jvm_maven_import_external( File ".../scala/scala_maven_import_external.bzl", line 299, column 24, in jvm_maven_import_external jvm_import_external(jar_urls = jar_urls, srcjar_urls = srcjar_urls, coordinates = artifact, **kwargs) Error in repository_rule: A repo named io_bazel_rules_scala_guava_2_13_15 is already generated by this module extension at .../scala/scala_maven_import_external.bzl:299:24 ERROR: Analysis of target '//src/java/io/bazel/rulesscala/worker:worker_test' failed; build aborted: error evaluating module extension scala_deps in //scala/extensions:deps.bzl ``` Recent updates to `scripts/create_repository.py` (bazelbuild#1639, bazelbuild#1642) make it easy to emit full direct dependency lists for artifacts included in `third_party/repositories/scala_*.bzl`. This increases the likelihood of collisions, since this expanded metadata forces the macros that instantiate artifact repos to instantiate even more repos. By fetching list of artifact IDs from these macros, `scala_toolchains` can now consolidate them into dictionary keys. Then it passes these unique keys to `repositories` directly, avoiding the problem of instantiating the same repo multiple times in the same module extension. This, in turn, also avoids the need to add parameters to the original `WORKSPACE` macros that instantiate dependencies to avoid collisions under Bzlmod. The `scala_toolchains` macro never needs to call these original macros, under either `WORKSPACE` or Bzlmod. Finally, it also reduces duplication between these artifact ID lists and the `_*_DEPS` symbols originally from `testing/BUILD` (and now in `testing/deps.bzl`). The dependency labels are now generated programatically. (Aside: As I mentioned, we may eventually need to pass a Scala version argument to these macros. It will be possible to cross that bridge without too much trouble if and when that day comes. Or I can try to future proof it in a follow up pull request.)
Removes this symbol from `scala/scala.bzl` as well as `setup_scala_testing_toolchain`, and deletes `scala/private/macros/toolchains.bzl`. Part of bazelbuild#1482 and bazelbuild#1652. This is required for Bazel 8 and `rules_java` 8 compatibility, but is also compatible with Bazel 6 and 7. In bazelbuild#1652, @hvadehra suggested partitioning the `.bzl` files such that `WORKSPACE` doesn't `load` a file that tries to `load` symbols from `rules_java`. I successfully did so in a separate branch, and along with other minor changes, got `rules_scala` to build with `rules_java` 8.5.1. The other changes will come in separate pull requests, but it makes sense to land this change now before adding any other toolchains to `scala_toolchains`. --- Arguably, we should remove all macros exported from `scala/scala.bzl` that only instantiate toolchain dependencies and define toolchains. That may be a breaking change for some users, but will ultimately be necessary for these macros to remain compatible with Bazel 8.
Weekly updates! Things haven't changed much from last week. Both Bzlmod and Bazel 8 compatibility are in good shape, and we're all collectively slowing down for the holidays. See last week's update for details regarding the state of these changes. Note that I'm taking care to keep my I'm also occasionally checking to ensure the Update 2024-12-17 7:20pm Eastern: From scalapb/ScalaPB#1771 (comment) and #1647 (comment): I've confirmed in my replace-custom-scalapb-wrappers-with-protoc-bridge-0.9.8 branch that the recently released protoc-bridge 0.9.8 resolves scalapb/ScalaPB#1771 (i.e., Mentioning here since the Bzlmodification work spawned the MergedOpen
Bzlmod pending changes
Bazel 8 pending changes (#1652)
|
Greetings!
It looks like rules_scala isn't part of the Bzlmod effort or added to bazel-central-registry yet. I've opened an issue both here and in https://github.com/bazelbuild/bazel-central-registry to request it: bazelbuild/bazel-central-registry#522
The text was updated successfully, but these errors were encountered: