[BUG] The node certificate in the demo certificates does not include the IPv6 loopback address ::1

[cwperks]

The demo esnode.pem certificate (defined here) does not include ::1 as a subject alternate name which has caused issues running the backwards compatibility tests with the security plugin installed.

> openssl x509 -noout -text -in 'config/esnode.pem'

X509v3 Subject Alternative Name:
                Registered ID:1.2.3.4.5.5, DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1

The IPv6 loopback address can be added in an ext file using openssl like this:

openssl genrsa -out esnode-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in esnode-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out esnode-key.pem
openssl req -new -key esnode-key.pem -subj "/C=de/L=test/O=node/OU=node/CN=node-0.example.com" -out esnode.csr
openssl x509 -req -days 3650 -extfile <(printf "subjectAltName=DNS:node-0.example.com,DNS:localhost,IP:::1,IP:127.0.0.1,RID:1.2.3.4.5.5") -in esnode.csr -out esnode.pem -CA root-ca.pem -CAkey root-ca-key.pem

[stephen-crawford]

[Triage] The outcome of this would require generating a new set of certificates. The steps to generate new certificates can be found online and in past PRs to generate certs.

[cwperks]

FYI if the Demo Certificates are replaced then new hashes need to be added here: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java#L305-L319

Edit: The history of demo certs should be kept since any demo cert is widely known. Anyone can easily lookup old demo certs through git history.

[cwperks]

@DarshitChanpura Can you reply back with the commands to generate the cert checksums when you figure it out?

[DarshitChanpura]

Solution Approach:

NOTE: Since this required adding a new value to SAN for node certificates, I had to end up generating a new set of demo certs.

1. Generate a new root certificate as Signing authority while ensuring all X509 extensions are preserved.
2. Generate a new node certificate + key signed with root-ca while ensuring all X509 extensions are preserved.
3. Generate a new admin certificate + key signed with root-ca while preserving all X509 extensions.
4. Update these new certs in install_demo_configuration scripts.
5. Tested that the new demo certificates work as expected
   a. curl IPv4 - curl -XGET https://admin:admin@localhost:9200/ -k
   b. curl IPv6 - curl -XGET https://admin:admin@\[::1\]:9200/ -k
6. Tested that these new demo certificates work with BWC test being fixed via [2.x] Enable security for bwc tests #3257 (Exit criteria: Tests should no longer fail with: No subject alternative names matching IP address ::1 found)
   a. Created a zip for 2.10.0.0-SNAPSHOT for security and placed it under bwc-test/src/test/resources/2.10.0.0/
   b. Replaced kirk.pem, kirk-key.pem, esnode.pem, esnode-key.pem and root-ca.pem under bwc-test/src/test/resources/security/ with newly generated certificates
   c. Ran bwcTestSuite task
7. Created a checksum calculator tools to generate SHA256 hash. Update these hashes in OpenSearchSecurityPlugin#demoCertHashes
8. Added a shell and a batch script for this new tool.
9. Updated certificates used by sanity-tests.

---

Appendix

Steps to generate certs.

## ROOT

openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/DC=com/DC=example/O=Example Com Inc./OU=Example Com Inc. Root CA/CN=Example Com Inc. Root CA" -addext "basicConstraints = critical,CA:TRUE" -addext "keyUsage = critical, digitalSignature, keyCertSign, cRLSign" -addext "subjectKeyIdentifier = hash" -addext "authorityKeyIdentifier = keyid:always,issuer:always" -out root-ca.pem


## NODE

openssl genrsa -out esnode-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in esnode-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out esnode-key.pem
openssl req -new -key esnode-key.pem -subj "/C=de/L=test/O=node/OU=node/CN=node-0.example.com" -out esnode.csr
openssl x509 -req -in esnode.csr -out esnode.pem -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -days 3650 -extfile <(printf "subjectAltName = RID:1.2.3.4.5.5, DNS:node-0.example.com, DNS:localhost, IP:::1, IP:127.0.0.1\nkeyUsage = digitalSignature, nonRepudiation, keyEncipherment\nextendedKeyUsage = serverAuth, clientAuth\nbasicConstraints = critical,CA:FALSE")


#ADMIN

openssl req -new -newkey rsa:2048 -keyout kirk-key.pem -out kirk.csr -nodes -subj "/C=de/L=test/O=client/OU=client/CN=kirk"
openssl x509 -req -in kirk.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -out kirk.pem -days 3650 -extfile <(printf "basicConstraints = critical,CA:FALSE\nkeyUsage = critical,digitalSignature,nonRepudiation,keyEncipherment\nextendedKeyUsage = critical,clientAuth\nauthorityKeyIdentifier = keyid,issuer:always\nsubjectKeyIdentifier = hash")

cURL command execution

➜  ~ curl -XGET https://admin:admin@localhost:9200/ -k
{
  "name" : "3c06300b34da.ant.amazon.com",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "tt8Ya64RTO6svp7Zxi8jtA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.0.0-SNAPSHOT",
    "build_type" : "tar",
    "build_hash" : "8cfde6ca4031047b4f7e5ecf706e434e1c535567",
    "build_date" : "2023-08-25T19:24:44.964931Z",
    "build_snapshot" : true,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "2.10.0",
    "minimum_index_compatibility_version" : "2.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
➜  ~ curl -XGET https://admin:admin@\[::1\]:9200/ -k
{
  "name" : "3c06300b34da.ant.amazon.com",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "tt8Ya64RTO6svp7Zxi8jtA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.0.0-SNAPSHOT",
    "build_type" : "tar",
    "build_hash" : "8cfde6ca4031047b4f7e5ecf706e434e1c535567",
    "build_date" : "2023-08-25T19:24:44.964931Z",
    "build_snapshot" : true,
    "lucene_version" : "9.8.0",
    "minimum_wire_compatibility_version" : "2.10.0",
    "minimum_index_compatibility_version" : "2.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

BWC test run

## Modify `network.host` setting to `::1`  in bwc-test/build.gradle and run this:

./gradlew clean bwcTestSuite -Dtests.security.manager=false -Dtests.opensearch.secure=true -Dtests.opensearch.username=admin -Dtests.opensearch.password=admin -Dbwc.version.previous=2.10.0.0 -Dbwc.version.next=2.10.0.0

## Should not see any failures like: `No subject alternative names matching IP address ::1 found`

Example run of checksum calculator tool

➜  tools ./checksum_calculator.sh ~/Documents/temp/ipv6-certificate pem

Certificate Hash (SHA-256):
File: kirk.pem, Hash (SHA-256): 069beaf566b9cf631e3676b82da8c60a191c4d4ab2832ad18efe3a5bd2a875d6
File: esnode-key.pem, Hash (SHA-256): ba9c5a61065f7f6115188128ffbdaa18fca34562b78b811f082439e2bef1d282
File: root-ca.pem, Hash (SHA-256): 9948688bc4c7a198f2a0db1d91f4f54499b8626902d03361b6d43e822d3691e4
File: esnode-key-temp.pem, Hash (SHA-256): ba9c5a61065f7f6115188128ffbdaa18fca34562b78b811f082439e2bef1d282
File: kirk-key.pem, Hash (SHA-256): 25e34a9a5d4f1dceed1666eb624397bf3fe5787a7133cd32838ace0381bce1f7
File: esnode.pem, Hash (SHA-256): dd3cf88e72e9e1a803bd12f4bafb4f29e642110db26c39ed5f2ef2e9351bc61c
File: root-ca-key.pem, Hash (SHA-256): c3f7b6cbf0392cde35d6164222746a13dc7227136cfc54d02abc2fa297543af3

Sanity tests JKS

## replace root-ca with the new root

openssl pkcs12 -export -in kirk.pem -inkey kirk-key.pem -certfile root-ca.pem -out kirk.p12 -name client

keytool -importkeystore -srckeystore kirk.p12 -srcstoretype PKCS12 -destkeystore kirk-test.jks -deststoretype PKCS12

[DarshitChanpura]

Update: I was able to have plugin-install running for Ubuntu. but still facing trouble with windows

[DarshitChanpura]

Update2: I was able to solve windows issue by running: openssl s_client -connect localhost:9200 and verifying that the node certificate returned by windows machine was not correct. As a fix, I modified demo install script to replace with the correct certificates.