build: Fix potential libcrypto lib loading issue for X86 mac runners

[advancedxy]

Which issue does this PR close?

Closes #41

Rationale for this change

Another attempt to fix libcrypto loading issues.
Currently, CIs are still failing randomly with the following logging, for example:
WARNING: /Users/runner/hostedtoolcache/Java_Adopt_jdk/17.0.10-7/x64/Contents/Home/bin/java is loading libcrypto in an unsafe way

What changes are included in this PR?

install openssl for mac runners and setup corresponding DYLD_LIBRARY_PATH

How are these changes tested?

Existed CI workflow

[viirya]

Looks like it still fails:

WARNING: /Users/runner/hostedtoolcache/Java_Adopt_jdk/17.0.10-7/x64/Contents/Home/bin/java is loading libcrypto in an unsafe way

And, it is in x64 Mac, not aarch64.

[advancedxy]

And, it is in x64 Mac, not aarch64.

Ah, yes. The failures observed for now are all on X86 macs.

Let me see if there's other ways to investigate this.

[advancedxy]

Ah, yes. The failures observed for now are all on X86 macs.
Let me see if there's other ways to investigate this.

I believe I have found the root cause to x86 mac is failing loading libcrypto.dylib.

Solutions:

1. install openssl and setup DYLD_LIBRARY_PATH with openssl's actual lib path (chosen by this PR)
2. or passing property commons.crypto.cipher.classes=org.apache.commons.crypto.cipher.JceCipher to loading JCE cipher only for apache commons' crypto lib.

Detailed investigations

Q1: why mac runners are failing with loading libcrypto in an unsafe way?
A1: See this comment: cl-plus-ssl/cl-plus-ssl#114 (comment)

Q2: why libcrypto.dylib is loaded? Or who is loading it?
A2: It's loaded by Apache's commons-crypto lib, which is used by Spark to enable IO encryption. After enabling DYLD_PRINT_SEARCHING=true for mac runners, it shows it's loaded by commons-crypto's native lib

Log path: https://github.com/apache/arrow-datafusion-comet/actions/runs/7971374594/job/21760923967

Q3: why only X86 runners are failing?
A3: It's because commons-crypto only bundled with x86's native lib.

When running under Silicon Macs, there's no native lib and fails loading. However the native code is to provide OpenSslCipher only. Spark's IO encryption doesn't require OpenSSL. JceCipher is sufficient.

Q4: why is it flaky?
A4: Not sure. Maybe the dynamic libs are loaded in different order for each run. And the correct libcrypto.dylib has a chance to be loaded by the JVM.

[advancedxy]

Close and reopen to trigger another run to make sure it actually works.

cc @viirya @sunchao

[sunchao]

LGTM, thanks for the detailed investigation @advancedxy !

[advancedxy]

I'm still observing this issue when running my own PR, https://github.com/advancedxy/arrow-datafusion-comet/actions/runs/7982531503/job/21796212861 which is already rebased with origin/main and include this fix.

I'm not observing this issue in this repo's PRs though after this commit. Looks like that this PR also don't fix it completely, but it reduces the chances. It might be still possible to load the incorrect libcrypto by the JVM.

Let's see how frequent this issue happens. If it's still annoying, we can try to investigate further and try option 2.
Or we can disable X86 mac runners as last resort since it took much longer time than other runners and slows down the CI execution.