omit tests related legacy provider

[hsbt]

Tests of legacy provider failed with recent update of FreeBSD

https://rubyci.s3.amazonaws.com/freebsd13/ruby-master/log/20240207T023002Z.fail.html.gz

I make to skip them at ruby/ruby repo. please let me know if you have any idea about that.

Thanks ❤️

[junaruga]

Seeing the freebsd13 logs, it is a success at the latest 78898c53c76 . Did you change anything on the server?
https://rubyci.s3.amazonaws.com/freebsd13/ruby-master/recent.html

On the freebsd13 server, it seems the used openssl version is 1.1.1t, right?

$ which openssl
/usr/bin/openssl

$ openssl version
OpenSSL 1.1.1t-freebsd  7 Feb 2023

I found a document for OpenSSL 3. But if the used OpenSSL is 1.1 series, the document is not related to it.

https://www.practicalnetworking.net/practical-tls/openssl-3-and-legacy-providers/

[hsbt]

Did you change anything on the server?

No, tests after 78898c53c76 are skipped.

[junaruga]

Did you change anything on the server?

No, tests after 78898c53c76 are skipped.

Thanks for explaining it. I compared between 8bd83bb133b (failure) and 78898c53c76 (success) in the history of the freebsd13.

$ git log 8bd83bb133b..78898c53c76

Then I found this commit ruby/ruby@d95d348 adding to skip the tests between the 2 commits above.

Now I am comparing the 86547fd69d9 (success) and 8041b7d9677 (failure) to find what triggered the failure. But I am still not sure what trigger the failure.

$ git diff 86547fd69d9..8041b7d9677
$ git log 86547fd69d9..8041b7d9677

[junaruga]

In the history of the logs, the first failure 8041b7d9677 comes from the no enough disk space. So, we can not judge it. Then the 2nd failure 8ba8e979c8b I found an interesting log below.

https://rubyci.s3.amazonaws.com/freebsd13/ruby-master/log/20240201T083002Z.diff.html.gz

@@ 22565 @@
 [n/n] Psych::Visitors::TestEmitter#test_document = <elapsed> s
 [n/n] Psych::Visitors::TestEmitter#test_stream = <elapsed> s
 [n/n] Psych::Visitors::TestEmitter#test_scalar_with_tag = <elapsed> s
+[n/n] OpenSSL::TestProvider#test_openssl_provider_names = <elapsed> s
+[n/n] OpenSSL::TestProvider#test_openssl_legacy_provider = <elapsed> s
+[n/n] OpenSSL::TestProvider#test_openssl_provider_name_inspect = <elapsed> s
+[n/n] OpenSSL::TestProvider#test_unloaded_openssl_provider = <elapsed> s
 [n/n] TestThreadGroup#test_thread_init = <elapsed> s
 [n/n] TestThreadGroup#test_frozen_thgroup = <elapsed> s
 [n/n] TestThreadGroup#test_enclosed_thgroup = <elapsed> s
@@ 32014 @@
 [n/n] Test_TypedData#test_wrong_argtype = <elapsed> s
 [n/n] Test_TypedData#test_deferred_free = <elapsed> s
 
+ <n>) Error:
+OpenSSL::TestProvider#test_openssl_provider_names:
+OpenSSL::Provider::ProviderError: Failed to load legacy provider: (null) (name=legacy)
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `load'
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `<main>'
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `with_openssl'
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `test_openssl_provider_names'
 
+ <n>) Error:
+OpenSSL::TestProvider#test_openssl_legacy_provider:
+OpenSSL::Provider::ProviderError: Failed to load legacy provider: (null) (name=legacy)
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `load'
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `<main>'
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `with_openssl'
+    <build-dir>/ruby/test/openssl/test_provider.rb:<line_a>:in `test_openssl_legacy_provider'

 + ldd .ext/amd64-freebsd13.2/openssl.so
 .ext/amd64-freebsd13.2/openssl.so:
-	libssl.so.11 => /usr/local/lib/libssl.so.11 (<address>)
-	libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (<address>)
 	libm.so.5 => /lib/libm.so.5 (<address>)
 	libthr.so.3 => /lib/libthr.so.3 (<address>)
 	libc.so.7 => /lib/libc.so.7 (<address>)

And now the OpenSSL library doesn't exist on the freebsd13 server.

$ ls -l /usr/local/lib/libssl.so.11
ls: /usr/local/lib/libssl.so.11: No such file or directory
$ ls -l /usr/local/lib/libcrypto.so.11
ls: /usr/local/lib/libcrypto.so.11: No such file or directory

In this log, the OpenSSL::TestProvider tests appeared newly. Because the tests are executed only if OpenSSL version is 3 or later version where the provider features are implemented. The provider feature is new in OpenSSL 3.0.0.

https://github.com/ruby/ruby/blob/1983949811607fb87584943749a0cf9056a49c01/test/openssl/test_provider.rb#L3

So, the log shows that the previously before the failures, the Ruby OpenSSL was built with OpenSSL 1.1. That's why the tests were not executed. Then somehow the freebsd's used OpenSSL was upgraded from OpenSSL 1.1 to OpenSSL 3.

I found another OpenSSL 3.0.13 which was installed on 1st February 2024.

$ /usr/local/bin/openssl version
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

$ ls -l /usr/local/bin/openssl
-rwxr-xr-x  1 root  wheel  847008 Feb  1 01:18 /usr/local/bin/openssl

$ /usr/local/bin/openssl version -d
OPENSSLDIR: "/usr/local/openssl"

And I can see the lib/ossl-modules/legacy.so in my local OpenSSL 3.0 directory installed from the source on my local Linux. But I was not able to find the legacy.so in the freebsd13's /usr/local/openssl. I assume that is the reason why Ruby OpenSSL can not load the legacy provider.

[junaruga]

$ /usr/local/bin/openssl version -a
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
built on: Thu Feb  1 01:17:17 2024 UTC
platform: BSD-x86_64
options:  bn(64,64)
compiler: cc -fPIC -pthread -Wa,--noexecstack -Qunused-arguments -O2 -pipe  -fstack-protector-strong -fno-strict-aliasing -DL_ENDIAN -DOPENSSL_PIC -D_THREAD_SAFE -D_REENTRANT -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
OPENSSLDIR: "/usr/local/openssl"
ENGINESDIR: "/usr/local/lib/engines-12"
MODULESDIR: "/usr/local/lib/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0xfed83203178bffff:0x400000219c01a9

I found the fips.so for fips provider, and no legacy.so for legacy provider.

$ ls -l /usr/local/lib/ossl-modules
total 1792
-rwxr-xr-x  1 root  wheel  1776024 Feb  1 01:18 fips.so

I suspect that the OpenSSL was configured, built and installed with no-legacy option.
https://github.com/openssl/openssl/blob/openssl-3.0/INSTALL.md#no-legacy

[junaruga]

I am starting to think we should adjust the test test/openssl/test_provider.rb to pass in the case that the legacy provider is not enabled. Because the case can be common more and more. People may want to avoid the legacy API forcely for their strategy. And people may use running the tests to verify the build on their environment.

[MSP-Greg]

I am starting to think we should adjust the test test/openssl/test_provider.rb to pass in the case that the legacy provider is not enabled

Maybe 'skip' or 'pend', rather than 'pass'?

[junaruga]

I am starting to think we should adjust the test test/openssl/test_provider.rb to pass in the case that the legacy provider is not enabled

Maybe 'skip' or 'pend', rather than 'pass'?

Yeah I think the pend is better. I will send a PR for that.

[junaruga]

I am starting to think we should adjust the test test/openssl/test_provider.rb to pass in the case that the legacy provider is not enabled

Maybe 'skip' or 'pend', rather than 'pass'?

Yeah I think the pend is better. I will send a PR for that.

The choice is omit (skip) or pend. I would prefer the omit because omit is used to skip the FIPS provider. And pend is used for pending or WIP test that can be improved in the future.

[rhenium]

I found the fips.so for fips provider, and no legacy.so for legacy provider.
[...]
I suspect that the OpenSSL was configured, built and installed with no-legacy option.
https://github.com/openssl/openssl/blob/openssl-3.0/INSTALL.md#no-legacy

That seems to be the case in the openssl port: https://www.freshports.org/security/openssl/

I agree these test cases should be skipped.

[junaruga]

That seems to be the case in the openssl port: https://www.freshports.org/security/openssl/

I agree these test cases should be skipped.

Thanks for providing the info! I can see the legacy provider is off in the port.

====> Provider modules
     FIPS=on: Build FIPS provider
     LEGACY=off: Older algorithms

[junaruga]

... I will send a PR for that.

I sent the PR #721.

[junaruga]

... I will send a PR for that.

I sent the PR #721.

After the #721 will be merged to the master branch in this repository ruby/openssl, I want to revert the ruby/ruby@d95d348, and synchronize a commit by the #721 into the ruby/ruby at the same time.

[hsbt]

I merged this because ruby/ruby already skipped them. Feel free to improve this.

[junaruga]

All right.
I initially wondered why you didn't squash this PR's 2 commits to the 1 commit to merge it to the master branch on ruby/openssl. But I noticed and understood that the 1st commit already existed ruby/ruby@d95d348 on the ruby/ruby, I understood that you wanted to add the 2nd commit separately and sync it into the ruby/ruby.