From a30ce08c64ca9fbd9a41ffcb571949de7e851740 Mon Sep 17 00:00:00 2001 From: Mike Bland Date: Fri, 29 Nov 2024 18:18:31 -0500 Subject: [PATCH] Move `rules_scala_dependencies` to scala/deps.bzl Ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. Part of #1652. Bumps several packages as high as they can go and still be compatible with Bazel 6 and 7: - `bazel_skylib`: 1.4.1 => 1.7.1 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_python`: 0.36.0 => 0.38.0 - `rules_go`: 0.50.1 The following packages are at the maximum version to prevent breakages under Bazel 6.5.0. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v27.1, respectively, for Bazel 6 compatibility per #1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue. --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` --- `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- `rules_java` remains at 7.9.0 due to the more complicated situation it presents. The current version that could be compatible with Bazel 6 and 7 is 7.12.2. As it turns out, we could include `rules_java` versions up to and including 7.12.2 if we don't call the following in `WORKSPACE`: ```py load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") rules_java_dependencies() rules_java_toolchains() ``` In fact, __we don't do that today__. When we do, Bazel 7.4.1 builds successfully, but Bazel 6.5.2 fails with: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` When we bump to rules_java 7.10.0, which contains https://github.com/bazelbuild/rules_java/pull/210, Bazel 7.4.1 fails with: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` This is the error I described in #1619, under "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d8896f209c37be8622f1dcc9770c2dc18f48. Today I rediscovered: - bazelbuild/rules_java: Regression with @@rules_java//toolchains:bootstrap_runtime_toolchain_type in 7.10.0 https://github.com/bazelbuild/rules_java/issues/214 But even worse, Bazel 6.5.0 fails with: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:57) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:110) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118) at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:111) at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:91) at com.google.devtools.build.buildjar.BazelJavaBuilder.lambda$main$0(BazelJavaBuilder.java:52) at com.google.devtools.build.lib.worker.WorkRequestHandler$WorkRequestCallback.apply(WorkRequestHandler.java:252) at com.google.devtools.build.lib.worker.WorkRequestHandler.respondToRequest(WorkRequestHandler.java:480) at com.google.devtools.build.lib.worker.WorkRequestHandler.lambda$startResponseThread$1(WorkRequestHandler.java:433) at java.base/java.lang.Thread.run(Thread.java:829) ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:57) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:110) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118) at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:111) at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:91) at com.google.devtools.build.buildjar.BazelJavaBuilder.lambda$main$0(BazelJavaBuilder.java:52) at com.google.devtools.build.lib.worker.WorkRequestHandler$WorkRequestCallback.apply(WorkRequestHandler.java:252) at com.google.devtools.build.lib.worker.WorkRequestHandler.respondToRequest(WorkRequestHandler.java:480) at com.google.devtools.build.lib.worker.WorkRequestHandler.lambda$startResponseThread$1(WorkRequestHandler.java:433) at java.base/java.lang.Thread.run(Thread.java:829) ---8<---8<--- End of log ---8<---8<--- ``` In #1619, I thought we wanted to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package, instead of `@rules_java//toolchains:`. But based on this comment from @fmeum, I'm inclined to believe switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach: - https://github.com/bazelbuild/rules_java/issues/214#issuecomment-2331630136 --- Bazel 8 requires `rules_java` 8, per #1652. We can't update to `rules_java` 8 right now, unless we decided to do the following: - Tell Bazel 6 users to add C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - Abandon Bazel 6 support in favor of supporting Bazel 7 at a minimum. - In addition to either of the above cases, update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` v28. Here are the details explaining why `rules_java` 8 currently breaks. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per #1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ``` --- WORKSPACE | 54 +++++++++---------- scala/deps.bzl | 58 +++++++++++++++++++++ scala/private/macros/scala_repositories.bzl | 58 ++------------------- 3 files changed, 85 insertions(+), 85 deletions(-) create mode 100644 scala/deps.bzl diff --git a/WORKSPACE b/WORKSPACE index b111cf67e..9b938f061 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -1,15 +1,9 @@ workspace(name = "io_bazel_rules_scala") load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") +load("//scala:deps.bzl", "rules_scala_dependencies") -http_archive( - name = "bazel_skylib", - sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7", - urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz", - "https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz", - ], -) +rules_scala_dependencies() load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace") @@ -17,29 +11,18 @@ bazel_skylib_workspace() http_archive( name = "rules_python", - sha256 = "ca77768989a7f311186a29747e3e95c936a41dffac779aff6b443db22290d913", - strip_prefix = "rules_python-0.36.0", - url = "https://github.com/bazelbuild/rules_python/releases/download/0.36.0/rules_python-0.36.0.tar.gz", + sha256 = "ca2671529884e3ecb5b79d6a5608c7373a82078c3553b1fa53206e6b9dddab34", + strip_prefix = "rules_python-0.38.0", + url = "https://github.com/bazelbuild/rules_python/releases/download/0.38.0/rules_python-0.38.0.tar.gz", ) load("@rules_python//python:repositories.bzl", "py_repositories") py_repositories() -load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config") - -scala_config(enable_compiler_dependency_tracking = True) - -load("//scala:scala.bzl", "scala_toolchains") - -scala_toolchains(fetch_sources = True) +load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps") -register_toolchains( - "//testing:testing_toolchain", - "//scala:unused_dependency_checker_error_toolchain", - "//test/proto:scalapb_toolchain", - "@io_bazel_rules_scala_toolchains//...:all", -) +protobuf_deps() load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies") @@ -53,9 +36,20 @@ load("@rules_proto//proto:toolchains.bzl", "rules_proto_toolchains") rules_proto_toolchains() -load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps") +load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config") -protobuf_deps() +scala_config(enable_compiler_dependency_tracking = True) + +load("//scala:scala.bzl", "scala_toolchains") + +scala_toolchains(fetch_sources = True) + +register_toolchains( + "//testing:testing_toolchain", + "//scala:unused_dependency_checker_error_toolchain", + "//test/proto:scalapb_toolchain", + "@io_bazel_rules_scala_toolchains//...:all", +) load("//twitter_scrooge:twitter_scrooge.bzl", "twitter_scrooge") @@ -101,10 +95,10 @@ local_repository( http_archive( name = "io_bazel_rules_go", - sha256 = "6dc2da7ab4cf5d7bfc7c949776b1b7c733f05e56edc4bcd9022bb249d2e2a996", + sha256 = "f4a9314518ca6acfa16cc4ab43b0b8ce1e4ea64b81c38d8a3772883f153346b8", urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.39.1/rules_go-v0.39.1.zip", - "https://github.com/bazelbuild/rules_go/releases/download/v0.39.1/rules_go-v0.39.1.zip", + "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.50.1/rules_go-v0.50.1.zip", + "https://github.com/bazelbuild/rules_go/releases/download/v0.50.1/rules_go-v0.50.1.zip", ], ) @@ -116,7 +110,7 @@ load( go_rules_dependencies() -go_register_toolchains(version = "1.19.5") +go_register_toolchains(version = "1.23.0") http_archive( name = "bazelci_rules", diff --git a/scala/deps.bzl b/scala/deps.bzl new file mode 100644 index 000000000..4809a560b --- /dev/null +++ b/scala/deps.bzl @@ -0,0 +1,58 @@ +"""Macro for instantiating repos required for core functionality.""" + +load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") +load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe") + +def rules_scala_dependencies(): + """Instantiates repos needed by rules provided by `rules_scala`.""" + maybe( + http_archive, + name = "bazel_skylib", + sha256 = "bc283cdfcd526a52c3201279cda4bc298652efa898b10b4db0837dc51652756f", + urls = [ + "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz", + "https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz", + ], + ) + + maybe( + http_archive, + name = "rules_cc", + urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.9/rules_cc-0.0.9.tar.gz"], + sha256 = "2037875b9a4456dce4a79d112a8ae885bbc4aad968e6587dca6e64f3a0900cdf", + strip_prefix = "rules_cc-0.0.9", + ) + + # Needed by protobuf-21.7 and Bazel 6.5.0, as later versions require C++14. + maybe( + http_archive, + name = "com_google_absl", + sha256 = "91ac87d30cc6d79f9ab974c51874a704de9c2647c40f6932597329a282217ba8", + strip_prefix = "abseil-cpp-20220623.1", + url = "https://github.com/abseil/abseil-cpp/archive/refs/tags/20220623.1.tar.gz", + ) + + maybe( + http_archive, + name = "rules_java", + urls = [ + "https://github.com/bazelbuild/rules_java/releases/download/7.9.0/rules_java-7.9.0.tar.gz", + ], + sha256 = "41131de4417de70b9597e6ebd515168ed0ba843a325dc54a81b92d7af9a7b3ea", + ) + + maybe( + http_archive, + name = "com_google_protobuf", + sha256 = "75be42bd736f4df6d702a0e4e4d30de9ee40eac024c4b845d17ae4cc831fe4ae", + strip_prefix = "protobuf-21.7", + url = "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v21.7.tar.gz", + ) + + maybe( + http_archive, + name = "rules_proto", + sha256 = "6fb6767d1bef535310547e03247f7518b03487740c11b6c6adb7952033fe1295", + strip_prefix = "rules_proto-6.0.2", + url = "https://github.com/bazelbuild/rules_proto/releases/download/6.0.2/rules_proto-6.0.2.tar.gz", + ) diff --git a/scala/private/macros/scala_repositories.bzl b/scala/private/macros/scala_repositories.bzl index 8e02640ef..81b6c9cbf 100644 --- a/scala/private/macros/scala_repositories.bzl +++ b/scala/private/macros/scala_repositories.bzl @@ -1,3 +1,4 @@ +load("//scala:deps.bzl", "rules_scala_dependencies") load( "//scala:scala_cross_version.bzl", "extract_major_version", @@ -117,59 +118,6 @@ def dt_patched_compiler_setup(scala_version, scala_compiler_srcjar = None): integrity = srcjar.get("integrity"), ) -def load_rules_dependencies(): - if not native.existing_rule("bazel_skylib"): - http_archive( - name = "bazel_skylib", - sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7", - urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz", - "https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz", - ], - ) - - # Needed by protobuf-21.7 and Bazel 6.5.0, as later versions require C++14. - if not native.existing_rule("com_google_absl"): - http_archive( - name = "com_google_absl", - sha256 = "91ac87d30cc6d79f9ab974c51874a704de9c2647c40f6932597329a282217ba8", - strip_prefix = "abseil-cpp-20220623.1", - url = "https://github.com/abseil/abseil-cpp/archive/refs/tags/20220623.1.tar.gz", - ) - - if not native.existing_rule("com_google_protobuf"): - http_archive( - name = "com_google_protobuf", - sha256 = "75be42bd736f4df6d702a0e4e4d30de9ee40eac024c4b845d17ae4cc831fe4ae", - strip_prefix = "protobuf-21.7", - url = "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v21.7.tar.gz", - ) - - if not native.existing_rule("rules_cc"): - http_archive( - name = "rules_cc", - urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.6/rules_cc-0.0.6.tar.gz"], - sha256 = "3d9e271e2876ba42e114c9b9bc51454e379cbf0ec9ef9d40e2ae4cec61a31b40", - strip_prefix = "rules_cc-0.0.6", - ) - - if not native.existing_rule("rules_java"): - http_archive( - name = "rules_java", - urls = [ - "https://github.com/bazelbuild/rules_java/releases/download/7.9.0/rules_java-7.9.0.tar.gz", - ], - sha256 = "41131de4417de70b9597e6ebd515168ed0ba843a325dc54a81b92d7af9a7b3ea", - ) - - if not native.existing_rule("rules_proto"): - http_archive( - name = "rules_proto", - sha256 = "6fb6767d1bef535310547e03247f7518b03487740c11b6c6adb7952033fe1295", - strip_prefix = "rules_proto-6.0.2", - url = "https://github.com/bazelbuild/rules_proto/releases/download/6.0.2/rules_proto-6.0.2.tar.gz", - ) - def setup_scala_compiler_sources(srcjars = {}): """Generates Scala compiler source repos used internally by rules_scala. @@ -188,7 +136,7 @@ def setup_scala_compiler_sources(srcjars = {}): ) def rules_scala_setup(scala_compiler_srcjar = None): - load_rules_dependencies() + rules_scala_dependencies() setup_scala_compiler_sources({ version: scala_compiler_srcjar for version in SCALA_VERSIONS @@ -260,7 +208,7 @@ def scala_repositories( scala_compiler_srcjars = {}): if load_dep_rules: # When `WORKSPACE` goes away, so can this case. - load_rules_dependencies() + rules_scala_dependencies() setup_scala_compiler_sources(scala_compiler_srcjars)