Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: Upgrade spotless version to 2.43.0 #85

Merged
merged 2 commits into from
Feb 22, 2024
Merged

build: Upgrade spotless version to 2.43.0 #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e195cf7
build: Upgrade spotless version to 2.43.0
viirya Feb 22, 2024
69eb711
Fix
viirya Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .github/actions/java-test/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,17 @@ runs:
shell: bash
run: |
echo "JAVA_VERSION=${JAVA_VERSION}"
./mvnw -B compile test-compile scalafix:scalafix -Psemanticdb -Djava.version=${JAVA_VERSION}
if [ $JAVA_VERSION == "1.8" ]; then
./mvnw -B compile test-compile scalafix:scalafix -Psemanticdb -Djava.version=${JAVA_VERSION} -Dspotless.version=2.29.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it OK to use 2.29.0 for Java 8?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line uses 2.29.0 for Java 8, isn't?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean whether we'll run into the issue mentioned in the PR description if we use 2.29 for Java 8

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is no.

This issue is particular for Java 21: diffplug/spotless#1791

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh cool.

else
./mvnw -B compile test-compile scalafix:scalafix -Psemanticdb -Djava.version=${JAVA_VERSION}
fi

- name: Run tests
shell: bash
run: |
SPARK_HOME=`pwd` ./mvnw -B clean install -Djava.version=${JAVA_VERSION}
if [ $JAVA_VERSION == "1.8" ]; then
SPARK_HOME=`pwd` ./mvnw -B clean install -Djava.version=${JAVA_VERSION} -Dspotless.version=2.29.0
else
SPARK_HOME=`pwd` ./mvnw -B clean install -Djava.version=${JAVA_VERSION}
fi
6 changes: 5 additions & 1 deletion .github/actions/rust-test/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,11 @@ runs:
shell: bash
run: |
cd common
../mvnw -B clean compile -DskipTests -Djava.version=${JAVA_VERSION}
if [ $JAVA_VERSION == "1.8" ]; then
../mvnw -B clean compile -DskipTests -Djava.version=${JAVA_VERSION} -Dspotless.version=2.29.0
else
../mvnw -B clean compile -DskipTests -Djava.version=${JAVA_VERSION}
fi

- name: Run Cargo test
shell: bash
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ under the License.
<parquet.maven.scope>provided</parquet.maven.scope>
<arrow.version>14.0.2</arrow.version>
<codehaus.jackson.version>1.9.13</codehaus.jackson.version>
<spotless.version>2.29.0</spotless.version>
<spotless.version>2.43.0</spotless.version>
<jni.dir>${project.basedir}/../core/target/debug</jni.dir>
<platform>darwin</platform>
<arch>x86_64</arch>
Expand Down
1 change: 1 addition & 0 deletions spark/src/main/java/org/apache/spark/shuffle/comet/CometShuffleMemoryAllocator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ public final class CometShuffleMemoryAllocator extends MemoryConsumer {

/** The number of bits used to address the page table. */
private static final int PAGE_NUMBER_BITS = 13;

/** The number of entries in the page table. */
private static final int PAGE_TABLE_SIZE = 1 << PAGE_NUMBER_BITS;

Expand Down
1 change: 1 addition & 0 deletions .../src/main/java/org/apache/spark/sql/comet/execution/shuffle/CometUnsafeShuffleWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ public class CometUnsafeShuffleWriter<K, V> extends ShuffleWriter<K, V> {
private long peakMemoryUsedBytes = 0;
private ExposedByteArrayOutputStream serBuffer;
private SerializationStream serOutputStream;

/**
* Are we in the process of stopping? Because map tasks can call stop() with success = true and
* then call stop() with success = false if they get an exception, we want to make sure we don't
Expand Down