[branch-4.15] Fix CI in branch-4.15 (#4207)

* Add support for Junit5 tests and fix CI that has been broken (#4197)

* Add support for Junit5 tests

- upgrade surefire plugin
- replace <forkMode>always</forkMode> with supported way in newer maven-surefire-plugin
- maven-failsafe-plugin version must match maven-surefire-plugin version
- remove invalid test case. Log4j is in the classpath for other reasons
  checked by running
  "mvn -pl tests/integration-tests-utils/pom.xml test -X"
  also verified by debugging that it's not in the system classpath, it is loaded by the
  MavenClassLoader
- add assertj-core for fluent assertions in tests

* Upgrade arquillian-junit-standalone

* Fix error in pom.xml

* Fix java.lang.ClassNotFoundException: javax.ws.rs.ProcessingException

* Pick docker-java-api version that is compatible with arquillian-cube-docker

* Fix snakeyaml issue

* Fix running of bookkeeper in Docker in tests

* Improve GitHub Actions CI workflow logging

* Increase client tests timeout

* Add script for picking fast ubuntu mirror for docker build

* upgrade maven-dependency-plugin version (used for copying .tar.gz for docker image building)

* Fix building of the docker image

* Increase timeouts for integration tests

* Use eclipse-temurin base image

* Replace "FROM ubuntu:22.10" with "FROM ubuntu:22.04" to use LTS version of ubuntu

22.10 (kinetic) is EOL since July 20 2023

* Build docker image also for backward-compatibility-tests

* Install gpg that is required by the scripts

* Add UBUNTU_MIRROR

* Remove redundancy from integration tests

* Fix missing dependency

* Fix classpath issues for arquillian-cube

* Upgrade actions versions

* Fix invalid references in native-io/pom.xml

* Fix circe-checksum native library compilation

* Migrate Gradle specific settings to Maven

* Another attempt to disable backward-compat tests in integration test build job

* Add working solution for skipping backward compat tests in the int tests

* Attempt to fix issue in locating the file

* Fix tar file extraction

* Improve solution

* Backward compat tests build commands without int tests

* Copy tune-runner-vm and clean-disk from Pulsar CI and clean disk

* Revisit skipping of backward compat tests once again

* Use similar MAVEN_OPTS as there is in Pulsar

* Fix directory

* Add way to find git root directory

* Revisit one more time

* Add current-version-image back

* Enable tests for backwardCompatTests

* Don't create /ledgers in ZK if it already exists

* Add uncaughtexceptionhandler

* Add exclusions for reload4j

* Use maven-surefire 2.8.1 for certain tests

* Add failing test back in, also fails with surefire 2.8.1

* Exclude logging implementations

* Revert "Use maven-surefire 2.8.1 for certain tests"

This reverts commit 94e2b8e.

* Exclude log4j

* Add instructions how to run on Mac Apple Silicon

* Find slf4j version

* Block loading some classes

* Change context classloader while creating new bookkeeper

* Route commons logging to slf4j

* Update shrinkwrap.version

* Wait for ZK client to shutdown before returning from close

* Upgrade Groovy version

* Remove arquillian-junit-container which causes tests to be run twice

* Skip backward compat tests unless -DbackwardCompatTests is passed

* Delay closing of classloaders by 5 seconds

* Fix checkstyle

* Improve instructions for running tests

* Upgrade Testcontainers and import the bom in dependencyManagement

* Fix synchronization bug pointed out by spotbugs

* upgrade markdown-link-check

* Ignore checking Google calendar for dead links

* Ignore anchors

* Add 2 more patterns to dead link checker exclusions

* Collect logs

* Log integration tests to console to observe progress, disable retries

* Use maven version provided by GitHub Action runner

* Improve instructions for docker socket proxy

* Add test timeout of 5 minutes for BookieShellTestBase

* Fix test to match changes made in e41d672

* Upgrade mockito to 4.11.0

* Align hamcrest with junit

* Revert "Upgrade mockito to 4.11.0"

This reverts commit 55b7fcd.

* Make JAVA_HOME optional

* Tune Java's DNS cache timeouts for docker images

* Use different directory for JDK 8

* Add tooling to containers that help debugging

* Upgrade JMH version and configuration

* Specify maven.compiler.release to fix issues with building with Java 11

* Remove gmavenplus plugin since there's already groovy-eclipse compiler in use

* Upgrade apache-rat-plugin

* Fix validation step

* Don't import shaded class

* Move freebuilder out of distributed libraries (there was invalid config for the annotation processor)

* Remove use of shaded class in test

* Upgrade junit5, assertj-core & awaitility to latest stable

* Fix simpletest logic

- LAC is updated asynchronously. Entries might be unconfirmed when read before ledger is closed.
- The last entry id passed to the read method was invalid

* Fix SimpleTestCommandTest

---------

Co-authored-by: Matteo Merli <[email protected]>
(cherry picked from commit 7bd91a7)

* Sync bk-ci.yml from master and delete other workflows

* Don't backport windows build changes to the build or CI

* Upgrade jna to 5.13.0

* migrate apply-config-from-env.py from python2 to python3 (#3614)

Fix #3549

Descriptions of the changes in this PR:
migrate apply-config-from-env.py from python2 to python3
fix docker/Dockerfile

### Motivation

Issue 3549: migrate apply-config-from-env.py from python2 to python3

### Changes

add newline character, and reformat for docker/Dockerfile
change print to print()

Master Issue: 3549

* Fix logging configurations are broken in docker image (#4137)

---------

Co-authored-by: xiang092689 <[email protected]>
Co-authored-by: 萧易客 <[email protected]>

.github/ISSUE_TEMPLATE/bp.md

@@ -9,7 +9,7 @@ assignees: ''
 
 **BP**
 
-> Follow the instructions at http://bookkeeper.apache.org/community/bookkeeper_proposals/ to create a proposal.
+> Follow the instructions at https://bookkeeper.apache.org/community/bookkeeper-proposals/ to create a proposal.
 
 This is the master ticket for tracking BP-xyz :
 

.github/actions/clean-disk/action.yml

@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: clean disk
+description: makes some more space available on the disk by removing files
+inputs:
+  mode:
+    description: "Use 'full' to clean as much as possible"
+    required: false
+runs:
+  using: composite
+  steps:
+    - run: |
+        if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+          directories=(/usr/local/lib/android /opt/ghc)
+          if [[ "${{ inputs.mode }}" == "full" ]]; then
+            # remove these directories only when mode is 'full'
+            directories+=(/usr/share/dotnet /opt/hostedtoolcache/CodeQL)
+          fi
+          emptydir=/tmp/empty$$/
+          mkdir $emptydir
+          echo "::group::Available diskspace"
+          time df -BM / /mnt
+          echo "::endgroup::"
+          for directory in "${directories[@]}"; do
+            echo "::group::Removing $directory"
+            # fast way to delete a lot of files on linux
+            time sudo eatmydata rsync -a --delete $emptydir ${directory}/
+            time sudo eatmydata rm -rf ${directory}
+            time df -BM / /mnt
+            echo "::endgroup::"
+          done
+          echo "::group::Cleaning apt state"
+          time sudo bash -c "apt-get clean; apt-get autoclean; apt-get -y --purge autoremove"
+          time df -BM / /mnt
+          echo "::endgroup::"
+        fi
+        echo "::group::Available diskspace"
+        time df -BM / /mnt
+        echo "::endgroup::"
+      shell: bash

.github/actions/tune-runner-vm/action.yml

@@ -24,8 +24,79 @@ runs:
   steps:
     - run: |
         if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+            echo "::group::Configure and tune OS"
             # Ensure that reverse lookups for current hostname are handled properly
             # Add the current IP address, long hostname and short hostname record to /etc/hosts file
             echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts
+
+            # The default vm.swappiness setting is 60 which has a tendency to start swapping when memory
+            # consumption is high.
+            # Set vm.swappiness=1 to avoid swapping and allow high RAM usage
+            echo 1 | sudo tee /proc/sys/vm/swappiness
+            (
+              shopt -s nullglob
+              # Set swappiness to 1 for all cgroups and sub-groups
+              for swappiness_file in /sys/fs/cgroup/memory/*/memory.swappiness /sys/fs/cgroup/memory/*/*/memory.swappiness; do
+                echo 1 | sudo tee $swappiness_file > /dev/null
+              done
+            ) || true
+
+            # use "madvise" Linux Transparent HugePages (THP) setting
+            # https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html
+            # "madvise" is generally a better option than the default "always" setting
+            # Based on Azul instructions from https://docs.azul.com/prime/Enable-Huge-Pages#transparent-huge-pages-thp
+            echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
+            echo advise | sudo tee /sys/kernel/mm/transparent_hugepage/shmem_enabled
+            echo defer+madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
+            echo 1 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
+    
+            # tune filesystem mount options, https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
+            # commit=999999, effectively disables automatic syncing to disk (default is every 5 seconds)
+            # nobarrier/barrier=0, loosen data consistency on system crash (no negative impact to empheral CI nodes)
+            sudo mount -o remount,nodiscard,commit=999999,barrier=0 /
+            sudo mount -o remount,nodiscard,commit=999999,barrier=0 /mnt
+            # disable discard/trim at device level since remount with nodiscard doesn't seem to be effective
+            # https://www.spinics.net/lists/linux-ide/msg52562.html
+            for i in /sys/block/sd*/queue/discard_max_bytes; do
+              echo 0 | sudo tee $i
+            done
+            # disable any background jobs that run SSD discard/trim
+            sudo systemctl disable fstrim.timer || true
+            sudo systemctl stop fstrim.timer || true
+            sudo systemctl disable fstrim.service || true
+            sudo systemctl stop fstrim.service || true
+
+            # stop php-fpm
+            sudo systemctl stop php8.0-fpm.service || true
+            sudo systemctl stop php7.4-fpm.service || true
+            # stop mono-xsp4
+            sudo systemctl disable mono-xsp4.service || true
+            sudo systemctl stop mono-xsp4.service || true
+            sudo killall mono || true
+
+            # stop Azure Linux agent to save RAM
+            sudo systemctl stop walinuxagent.service || true
+          
+            # enable docker experimental mode which is
+            # required for using "docker build --squash" / "-Ddocker.squash=true"
+            daemon_json="$(sudo cat /etc/docker/daemon.json  | jq '.experimental = true')"
+            echo "$daemon_json" | sudo tee /etc/docker/daemon.json
+            # restart docker daemon
+            sudo systemctl restart docker
+            echo '::endgroup::'
+
+            # show memory
+            echo "::group::Available Memory"
+            free -m
+            echo '::endgroup::'
+            # show disk
+            echo "::group::Available diskspace"
+            df -BM
+            echo "::endgroup::"
+            # show cggroup
+            echo "::group::Cgroup settings for current cgroup $CURRENT_CGGROUP"
+            CURRENT_CGGROUP=$(cat /proc/self/cgroup | grep '0::' | awk -F: '{ print $3 }')
+            sudo cgget -a $CURRENT_CGGROUP || true
+            echo '::endgroup::'
         fi
-      shell: bash
+      shell: bash

.github/changes-filter.yaml

@@ -0,0 +1,16 @@
+# contains pattern definitions used in workflows "changes" step
+# pattern syntax: https://github.com/micromatch/picomatch
+all:
+  - '**'
+docs:
+  - 'site3/**'
+  - '.asf.yaml'
+  - '*.md'
+  - '**/*.md'
+  - '.github/changes-filter.yaml'
+  - '.github/ISSUE_TEMPLATE/**'
+need_owasp:
+  - 'pom.xml'
+  - '**/pom.xml'
+  - 'src/owasp-dependency-check-false-positives.xml'
+  - 'src/owasp-dependency-check-suppressions.xml'

.github/workflows/README.md

@@ -0,0 +1,60 @@
+<!--
+
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+-->
+
+## GitHub Workflows
+
+This directory contains all BookKeeper CI checks.
+
+### Required Workflows
+
+When adding new CI workflows, please update the [.asf.yaml](../../.asf.yaml) if the workflow is required to pass before
+a PR can be merged. Instructions on how to update the file are below.
+
+This project uses the [.asf.yaml](../../.asf.yaml) to configure which workflows are required to pass before a PR can
+be merged. In the `.asf.yaml`, the required contexts are defined in the `github.protected_branches.*.required_status_checks.contexts.[]`
+where * is any key in the `protected_branches` map.
+
+You can view the currently required status checks by running the following command:
+
+```shell
+curl -s -H 'Accept: application/vnd.github.v3+json' https://api.github.com/repos/apache/bookkeeper/branches/master | \
+jq .protection
+```
+
+These contexts get their names in one of two ways depending on how the workflow file is written in this directory. The
+following command will print out the names of each file and the associated with the check. If the `name` field is `null`,
+the context will be named by the `id`.
+
+```shell
+for f in .github/workflows/*.yaml .github/workflows/*.yml; \
+do FILE=$f yq eval -o j '.jobs | to_entries | {"file": env(FILE),"id":.[].key, "name":.[].value.name}' $f; \
+done
+```
+
+Duplicate names are allowed, and all checks with the same name will be treated the same (required or not required).
+
+When working on workflow changes, one way to find out the names of the status checks is to retrieve the names
+from the PR build run. The "check-runs" can be found by commit id. Here's an example:
+
+```shell
+curl -s "https://api.github.com/repos/apache/bookkeeper/commits/$(git rev-parse HEAD)/check-runs" | \
+  jq -r '.check_runs | .[] | .name' |sort
+```