Skip to content

Commit

Permalink
fix(github): fix redundant hadoop-bin while uploading artifact and …
Browse files Browse the repository at this point in the history
…build `thirdparties-src` image on each OS version (#2134)

Resolve #2136:

- move `hadoop-bin` dir into the specified position at the stage of rebuilding/unpacking
`thirdparty` instead of uploading artifact;  
- build `thirdparties-src` image on Ubuntu instead of EOL CentOS 7 to make downstream
workflows work.
  • Loading branch information
empiredan authored Oct 21, 2024
1 parent 73d4bf7 commit d91a6e7
Show file tree
Hide file tree
Showing 11 changed files with 51 additions and 34 deletions.
2 changes: 2 additions & 0 deletions .github/actions/rebuild_thirdparty_if_needed/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,6 @@ runs:
../admin_tools/download_zk.sh zookeeper-bin
rm -rf hadoop-bin/share/doc
rm -rf zookeeper-bin/docs
mv hadoop-bin ..
mv zookeeper-bin ..
shell: bash
2 changes: 2 additions & 0 deletions .github/actions/unpack_prebuilt_thirdparties/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,6 @@ runs:
find ./thirdparty -name '*CMakeFiles*' -type d -exec rm -rf "{}" +
rm -rf ./thirdparty/hadoop-bin/share/doc
rm -rf ./thirdparty/zookeeper-bin/docs
mv thirdparty/hadoop-bin ./
mv thirdparty/zookeeper-bin ./
shell: bash
2 changes: 0 additions & 2 deletions .github/actions/upload_artifact/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ runs:
steps:
- name: Tar files
run: |
mv thirdparty/hadoop-bin ./
mv thirdparty/zookeeper-bin ./
rm -rf thirdparty
# The following operations are tricky, these directories and files don't exist if not build with '--test'.
# When build binaries for client tests, it's not needed to add '--test'.
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/lint_and_test_cpp.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ on:
- ci-test # testing branch for github action
- '*dev'
paths:
- .github/actions
- .github/actions/**
- .github/workflows/lint_and_test_cpp.yaml
- .github/workflows/thirdparty-regular-push.yml
- build_tools/pack_server.sh
Expand Down
34 changes: 21 additions & 13 deletions .github/workflows/thirdparty-regular-push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@ on:
jobs:
build_push_src_docker_images:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
osversion:
- ubuntu1804
- ubuntu2004
- ubuntu2204
# TODO(wangdan): disable centos7 temporarily before image build-env-centos7-* is fixed.
# - centos7
steps:
- uses: actions/checkout@v4
- name: Set up QEMU
Expand All @@ -60,11 +69,12 @@ jobs:
file: ./docker/thirdparties-src/Dockerfile
push: true
tags: |
apache/pegasus:thirdparties-src-${{ github.ref_name }}
apache/pegasus:thirdparties-src-${{ matrix.osversion }}-${{ github.ref_name }}
build-args: |
GITHUB_BRANCH=${{ github.ref_name }}
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
OS_VERSION=${{ matrix.osversion }}
HADOOP_BIN_PATH=hadoop-bin
ZOOKEEPER_BIN_PATH=zookeeper-bin
build_push_bin_docker_images:
runs-on: ubuntu-latest
Expand All @@ -82,7 +92,8 @@ jobs:
- ubuntu1804
- ubuntu2004
- ubuntu2204
- centos7
# TODO(wangdan): disable centos7 temporarily before image build-env-centos7-* is fixed.
# - centos7
steps:
# The glibc version on ubuntu1804 and centos7 is lower than the actions/checkout@v4 required, so
# we need to force to use actions/checkout@v3.
Expand All @@ -107,8 +118,8 @@ jobs:
build-args: |
GITHUB_BRANCH=${{ github.ref_name }}
OS_VERSION=${{ matrix.osversion }}
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
HADOOP_BIN_PATH=hadoop-bin
ZOOKEEPER_BIN_PATH=zookeeper-bin
build_push_bin_jemalloc_docker_images:
runs-on: ubuntu-latest
Expand All @@ -126,7 +137,8 @@ jobs:
- ubuntu1804
- ubuntu2004
- ubuntu2204
- centos7
# TODO(wangdan): disable centos7 temporarily before image build-env-centos7-* is fixed.
# - centos7
steps:
# The glibc version on ubuntu1804 and centos7 is lower than the actions/checkout@v4 required, so
# we need to force to use actions/checkout@v3.
Expand All @@ -152,8 +164,8 @@ jobs:
GITHUB_BRANCH=${{ github.ref_name }}
OS_VERSION=${{ matrix.osversion }}
USE_JEMALLOC=ON
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
HADOOP_BIN_PATH=hadoop-bin
ZOOKEEPER_BIN_PATH=zookeeper-bin
build_push_bin_test_docker_images:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -188,8 +200,6 @@ jobs:
ROCKSDB_PORTABLE=1
HADOOP_BIN_PATH=hadoop-bin
ZOOKEEPER_BIN_PATH=zookeeper-bin
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
build_push_bin_test_jemalloc_docker_images:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -225,5 +235,3 @@ jobs:
USE_JEMALLOC=ON
HADOOP_BIN_PATH=hadoop-bin
ZOOKEEPER_BIN_PATH=zookeeper-bin
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
12 changes: 8 additions & 4 deletions admin_tools/download_hadoop.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,19 @@ set -e

CWD=$(cd "$(dirname "$0")" && pwd)

if [ $# -ge 1 ]; then
HADOOP_BIN_PATH=$1
if [ $# -lt 1 ]; then
echo "Invalid arguments !"
echo "USAGE: $0 <HADOOP_BIN_PATH>"
exit 1
fi

HADOOP_BIN_PATH=$1

HADOOP_VERSION="hadoop-3.3.6"
arch_output=$(arch)
if [ "$arch_output"x == "aarch64"x ]; then
HADOOP_PACKAGE_MD5="369f899194a920e0d1c3c3bc1718b3b5"
HADOOP_BASE_NAME=${HADOOP_VERSION}-"$(arch)"
HADOOP_PACKAGE_MD5="369f899194a920e0d1c3c3bc1718b3b5"
HADOOP_BASE_NAME=${HADOOP_VERSION}-"$(arch)"
else
if [ "$arch_output"x != "x86_64"x ]; then
echo "WARNING: unrecognized CPU architecture '$arch_output', use 'x86_64' as default"
Expand Down
9 changes: 2 additions & 7 deletions admin_tools/download_package.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,15 @@

set -e

if [ $# -lt 2 ]; then
if [ $# -lt 3 ]; then
echo "Invalid arguments !"
echo "USAGE: $0 <PACKAGE_BASE_NAME> <PACKAGE_MD5> [TARGET_PATH]"
echo "USAGE: $0 <PACKAGE_BASE_NAME> <PACKAGE_MD5> <TARGET_PATH>"
exit 1
fi

PACKAGE_BASE_NAME=$1
PACKAGE_MD5=$2

if [ $# -lt 3 ]; then
echo "TARGET_PATH is not provided, thus do not try to download ${PACKAGE_BASE_NAME}"
exit 0
fi

TARGET_PATH=$3
if [ -d "${TARGET_PATH}" ]; then
echo "TARGET_PATH ${TARGET_PATH} has existed, thus do not try to download ${PACKAGE_BASE_NAME}"
Expand Down
8 changes: 6 additions & 2 deletions admin_tools/download_zk.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,14 @@ set -e

CWD=$(cd $(dirname $0) && pwd)

if [ $# -ge 1 ]; then
ZOOKEEPER_BIN_PATH=$1
if [ $# -lt 1 ]; then
echo "Invalid arguments !"
echo "USAGE: $0 <ZOOKEEPER_BIN_PATH>"
exit 1
fi

ZOOKEEPER_BIN_PATH=$1

ZOOKEEPER_VERSION=3.7.0
ZOOKEEPER_DIR_NAME=apache-zookeeper-${ZOOKEEPER_VERSION}-bin
ZOOKEEPER_PACKAGE_MD5="8ffa97e7e6b0b2cf1d022e5156a7561a"
Expand Down
5 changes: 4 additions & 1 deletion docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,10 @@ It packages the downloaded sources into a zip in the container, so that
other repos can easily extract third-parties from the container (via `docker cp`),
without downloading from the cloud object storage.

- `apache/pegasus:thirdparties-src-<branch>`
- `apache/pegasus:thirdparties-src-centos7-<branch>`
- `apache/pegasus:thirdparties-src-ubuntu1804-<branch>`
- `apache/pegasus:thirdparties-src-ubuntu2004-<branch>`
- `apache/pegasus:thirdparties-src-ubuntu2204-<branch>`

## thirdparties-bin

Expand Down
6 changes: 3 additions & 3 deletions docker/thirdparties-bin/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

ARG GITHUB_BRANCH=master
ARG OS_VERSION=centos7
FROM apache/pegasus:thirdparties-src-${GITHUB_BRANCH} as builder
FROM apache/pegasus:thirdparties-src-${OS_VERSION}-${GITHUB_BRANCH} as builder
FROM apache/pegasus:build-env-${OS_VERSION}-${GITHUB_BRANCH}

WORKDIR /root
Expand All @@ -28,8 +28,8 @@ ARG GITHUB_BRANCH
ARG GITHUB_REPOSITORY_URL=https://github.com/apache/incubator-pegasus.git
ARG ROCKSDB_PORTABLE=native
ARG USE_JEMALLOC=OFF
ARG HADOOP_BIN_PATH
ARG ZOOKEEPER_BIN_PATH
ARG HADOOP_BIN_PATH=hadoop-bin
ARG ZOOKEEPER_BIN_PATH=zookeeper-bin
RUN git clone --depth=1 --branch=${GITHUB_BRANCH} ${GITHUB_REPOSITORY_URL} \
&& cd incubator-pegasus/thirdparty \
&& unzip /root/thirdparties-src.zip -d . \
Expand Down
3 changes: 2 additions & 1 deletion docker/thirdparties-src/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@
# under the License.

ARG GITHUB_BRANCH=master
FROM apache/pegasus:build-env-centos7-${GITHUB_BRANCH} as builder
ARG OS_VERSION=centos7
FROM apache/pegasus:build-env-${OS_VERSION}-${GITHUB_BRANCH} as builder

WORKDIR /root

Expand Down

0 comments on commit d91a6e7

Please sign in to comment.