Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Move
rules_scala_dependencies to scala/deps.bzl
Moves `rules_scala_dependencies` to `scala/deps.bzl` and bumps several dependencies as high as they can go and still be compatible with Bazel 6.5.0 and 7.4.1. - `bazel_skylib`: 1.4.1 => 1.7.1 - `go`: 1.19.5 => 1.23.4 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_go`: 0.39.1 => 0.50.1 - `rules_java`: 7.9.0 => 7.12.3 - `rules_python`: 0.36.0 => 0.38.0 The `rules_java` 7.12.13 bump precipitated the following changes: - Adds the `WORKSPACE` stanza for `rules_java` in every `WORKSPACE` file per https://github.com/bazelbuild/rules_java/releases/tag/7.12.3. This replaces previous calls to instantiate `rules_java` repos and to register `rules_java` toolchains. - Rearranges the other `WORKSPACE` setup macros for dependencies to come before the `rules_scala` setup macros. - Updates almost all dependencies on `@bazel_tools//tools/jdk:` toolchain targets and `.bzl` files to corresponding targets and files in `@rules_java//toolchains:`. - Removes several deprecated flags from the `scrooge_compile_with_jdk_11` test case from `test/shell/test_twitter_scrooge.sh`, and one obsolete flag that caused it to break. --- Moving `rules_scala_dependencies` to `scala/deps.bzl` ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. It also prevents `WORKSPACE` from transitively loading any `.bzl` files that load `@rules_java`, ensuring Bazel 8 compatibility per bazelbuild#1652. Reasons for the other `rules_java` related changes include: - The `WORKSPACE` stanza for `rules_java` should've already been present while using the existing version 7.9.0. However, doing so would've broken Bazel 6 builds, as described in detail below. - The `rules_java_toolchains()` call follows the `protobuf_deps()` call instead of immediately following `rules_java_dependencies()` because upgrading to `rules_java` >= 8.5.0 will require this. It has no adverse impact to do it now, amidst the other `WORKSPACE` changes, and will make the eventual `rules_java` >= 8.5.0 diff smaller. - Having the other `WORKSPACE` setup macros for dependencies come before the `rules_scala` setup macros helps ensure consistent, correct initialization before `rules_scala` initialization. - Updating the toolchain specifiers to use `@rules_java//toolchains` fixed `WORKSPACE` build breakages when updating to `rules_java` 7.10.0 and later. This is a potentially breaking change for consumers, but in the good kind of way, in that it requires an easy, futureproof update. (`@bazel_tools//tools/jdk:toolchain_type` dependencies remain, as there's not yet a corresponding `@rules_java//toolchains` target.) What follows are detailed notes on why some dependency versions are capped where they are, and on some breakages fixed by these changes. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v21.7, respectively, for Bazel 6 compatibility per bazelbuild#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue: ```txt build:linux --cxxopt=-std=c++17 build:linux --host_cxxopt=-std=c++17 build:macos --cxxopt=-std=c++17 build:macos --host_cxxopt=-std=c++17 build:windows --cxxopt=/std=c++17 build:windows --host_cxxopt=/std=c++17 ``` --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` However, `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- This section provides a methodical description of the errors encountered and resolved during each step of the `rules_java` update. The proper `WORKSPACE` setup stanza for `rules_java`, which we didn't originally have in place, is: ```py load( "@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains", ) rules_java_dependencies() rules_java_toolchains() ``` Adding this stanza to `WORKSPACE` while still using `rules_java` 7.9.0 didn't break the Bazel 7.4.1 build, but it broke the Bazel 6.5.0 build. Note the `CompiledWithJava{8,11}_java.jar` references: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) [ ...snip... ] ---8<---8<--- End of log ---8<---8<--- ``` Another variation of this I also saw was: ```txt $ bazel build //{src,jmh,test,third_party,scala_proto}/... [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ERROR: test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-33-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.UnsupportedClassVersionError: com/google/errorprone/ErrorProneError has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0 at java.base/java.lang.ClassLoader.defineClass1(Native Method) [ ...snip... ] ``` Updating the toolchains from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:` for targets in `test/BUILD` and `test/src/main/resources/java_sources/BUILD` fixed this particular issue. (More on updating other `@bazel_tools//tools/jdk` toolchain targets below.) Bazel 6.5.0 then failed with the following expected issue, fixed by `rules_java` 7.10.0 via: - bazelbuild/rules_java#210 - bazelbuild/rules_java@30ecf3f ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` Updating to `rules_java` 7.10.0 fixed the Bazel 6.5.0 build, but caused Bazel 7.4.1 to fail with the error I originally described in bazelbuild#1619. For details, see "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` My analysis from bazelbuild#1619 _was_ on the right track. The `rules_java` 7.9.0 built into Bazel's `WORKSPACE` preamble registered `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` toolchains that later versions of `rules_java` couldn't resolve. This was due to the very same change that fixed 6.5.0, bazelbuild/rules_java#210. However, I mistakenly thought we needed to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package instead switching to `@rules_java//toolchains:`. That's why I originally thought we were stuck on `rules_java` 7.9.0. I eventually rediscovered the following issue, which surfaced this problem a month before bazelbuild#1619. The conversation helped me realize that switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach (short of migrating to Bzlmod): - bazelbuild/rules_java: Regression with `@@rules_java//toolchains:bootstrap_runtime_toolchain_type` in 7.10.0 bazelbuild/rules_java#214 Switching all `@bazel_tools//tools/jdk:` toolchains to `@rules_java//toolchains:` resolved this issue, enabling Bazel 6.5.0 and 7.4.1 to successfully build using `rules_java` 7.12.2. I then updated `rules_java` to 7.12.3, which returns to registering some toolchains as `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java#246 - bazelbuild/rules_java@b64eb7d @hvadehra requested that I try 7.12.3 to see if it resolved the issue. I was able to build successfully using this version in a branch without the toolchain updates from this commit. - bazelbuild#1652 (comment) However, the changes from this commit should remain futureproof when `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` moves to `@rules_java//toolchains:bootstrap_runtime_toolchain_type` again one day. --- Several of the flags removed from the `scrooge_compile_with_jdk_11` test case have been no-ops for years, so removing them eliminated their corresponding deprecation warnings. `--javacopt='--release 11'`, however, caused this breakage after originally bumping to `rules_java` 7.12.2: ```txt $ RULES_SCALA_TEST_ONLY="scrooge_compile_with_jdk_11" bash ./test/shell/test_twitter_scrooge.sh running test scrooge_compile_with_jdk_11 WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated WARNING: Option 'javabase' is deprecated WARNING: Option 'host_javabase' is deprecated WARNING: Option 'host_java_toolchain' is deprecated WARNING: Option 'java_toolchain' is deprecated INFO: Analyzed 64 targets (0 packages loaded, 0 targets configured). INFO: Found 64 targets... ERROR: .../src/java/io/bazel/rulesscala/scalac/compileoptions/BUILD:3:13: Compiling Java headers src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar (1 source file) failed: (Exit 1): turbine_direct_graal failed: error executing command (from target //src/java/io/bazel/rulesscala/scalac/compileoptions:compileoptions) external/remote_java_tools_darwin_arm64/java_tools/turbine_direct_graal --output bazel-out/darwin_arm64-fastbuild/bin/src/java/io/bazel/rulesscala/scalac/compileoptions/libcompileoptions-hjar.jar ... (remaining 32 arguments skipped) Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging java.lang.NullPointerException: attempted to use --release, but JAVA_HOME is not set at [email protected]/java.util.Objects.requireNonNull(Objects.java:259) at com.google.turbine.binder.CtSymClassBinder.bind(CtSymClassBinder.java:55) at com.google.turbine.main.Main.release(Main.java:318) at com.google.turbine.main.Main.bootclasspath(Main.java:304) at com.google.turbine.main.Main.compile(Main.java:142) at com.google.turbine.main.Main.compile(Main.java:133) at com.google.turbine.main.Main.main(Main.java:89) at [email protected]/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) INFO: Elapsed time: 0.325s, Critical Path: 0.12s INFO: 11 processes: 10 internal, 1 worker. FAILED: Build did NOT complete successfully Test "scrooge_compile_with_jdk_11" failed (0 sec) ``` See also: - Bazel Blog: Bazel 5.0: https://blog.bazel.build/2022/01/19/bazel-5.0.html#java - bazelbuild/bazel: `incompatible_use_toolchain_resolution_for_java_rules`: use toolchain resolution for Java rules #7849 bazelbuild/bazel#7849 --- Bazel 8 requires `rules_java` 8, per bazelbuild#1652. Updating to `rules_java` 8 will likely happen in a new major release _after_ a release containing the Bzlmod changes. This is because: - The Bzlmod changes alone will work with Bazel 6 without requiring that users update `protobuf` beyond version 21.7. - Bazel 8 requires `protobuf` >= 29.0. Bazel 6 users would have to add the afforementioned C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - `rules_scala` itself needs to update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` >= v28.0, but this ScalaPB version will _not_ support `protobuf` < v28.0. The idea is that the next major release of `rules_scala` will help users migrate to Bazel 7 and Bzlmod, in either order. Then the next major release after that will enable them to migrate to Bazel 8, with all the requisite dependency upgrades. (Technically it will still support `WORKSPACE`, but hopefully they'll make the jump to Bzlmod first.) See: - bazelbuild#1482 (comment) --- Here are the details explaining how `rules_java` 8 currently breaks `rules_scala`, up until the point that upgrading to `protobuf` >= v29.0 would fix it. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazelbuild#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
- Loading branch information
