Errors in FTSAgent after upgrading from v7.3.18 to v8.0.24 #7165

zhangxiaomei · 2023-08-08T15:24:00Z

zhangxiaomei
Aug 8, 2023

FTSAgent got the following failure when upgrading from v7.3.18 to v8.0.24. DIRAC failed to submit jobs to FTS server:

"2023-08-08 15:11:00 UTC DataManagement/FTS3Agent/treatOperation/1963837 ERROR: Could not get context {'OK': False, 'Errno': 0, 'Message': 'BadEndpoint("https://fts3.ihep.ac.cn:8446 (HTTPSConnectionPool(host='fts3.ihep.ac.cn', port=8446): Max retries exceeded with url: // (Caused by SSLError(SSLError(399, '[SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3900)'))))")', 'CallStack': [' File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/threading.py", line 995, in _bootstrap\n self._bootstrap_inner()\n', ' File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/threading.py", line 1038, in _bootstrap_inner\n self.run()\n', ' File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/threading.py", line 975, in run\n self._target(*self._args, **self._kwargs)\n', ' File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/multiprocessing/pool.py", line 125, in worker\n result = (True, func(*args, **kwds))\n', ' File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/DIRAC/DataManagementSystem/Agent/FTS3Agent.py", line 481, in _treatOperation\n res = self.getFTS3Context(ftsJob.username, ftsJob.userGroup, ftsServer, threadID=threadID)\n', ' File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/DIRAC/DataManagementSystem/Agent/FTS3Agent.py", line 203, in getFTS3Context\n res = FTS3Job.generateContext(ftsServer, proxyFile, lifetime=self.proxyLifetime)\n', ' File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/DIRAC/DataManagementSystem/Client/FTS3Job.py", line 779, in generateContext\n return S_ERROR(repr(e))\n']}
2023-08-08 15:11:00 UTC DataManagement/FTS3Agent ERROR: Error generating context BadEndpoint("https://fts3.ihep.ac.cn:8446 (HTTPSConnectionPool(host='fts3.ihep.ac.cn', port=8446): Max retries exceeded with url: // (Caused by SSLError(SSLError(399, '[SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3900)'))))")
Traceback (most recent call last):
File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1040, in validate_conn
conn.connect()
File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/urllib3/connection.py", line 416, in connect
self.sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/opt/dirac/versions/v2.2.5-1690312593/Linux-x86_64/lib/python3.11/site-packages/urllib3/util/ssl.py", line 418, in ssl_wrap_socket
context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3900)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
"

Answered by chrisburr

Aug 10, 2023

To be more specific, the problem is that the certifcate used by fts3.ihep.ac.cn is only 1024-bit.

$ echo | openssl s_client fts3.ihep.ac.cn:8446 2>/dev/null | grep 'Server public key is'
Server public key is 1024 bit

For reference, 1024-bit RSA was first factorised in 2007 and there is little reason not to be using 4096-bit RSA. Many CAs only sign RSA 2048-bit+ or ECDSA keys.

View full answer

fstagni · 2023-08-08T15:56:57Z

fstagni
Aug 8, 2023
Maintainer

At a first approximation, it seems to me that the certificate used does not have the required strength.
Which version of openssl is used by fts3.ihep.ac.cn ?

And for info look also at #6851

0 replies

atsareg · 2023-08-08T17:26:38Z

atsareg
Aug 8, 2023
Maintainer

The server cert is 1024 bits I guess:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            97:ee:a6:d7:f7:4e:e1:e7:29:4d
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, O=HEP, CN=Institute of High Energy Physics Certification Authority
        Validity
            Not Before: Feb 23 07:15:01 2023 GMT
            Not After : Feb 23 07:15:01 2024 GMT
        Subject: C=CN, O=HEP, O=IHEP, OU=CC, CN=fts3.ihep.ac.cn
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:bd:71:9f:3b:5f:4a:90:8f:20:0e:24:08:91:1f:
                    18:13:ae:32:e6:f0:f0:a8:7f:73:db:dd:ac:88:99:
                    b5:2f:98:44:1a:8a:b4:bf:a1:b8:1f:a7:3a:c1:44:
                    fc:7b:2c:24:13:8f:44:e6:6b:6e:99:03:13:08:4d:
                    df:24:19:d2:9d:04:50:63:00:47:3f:ec:8a:53:2f:
                    54:fb:aa:55:71:c1:88:31:27:d7:aa:73:f5:2a:e6:
                    9d:7b:67:cc:89:6c:5e:17:cb:35:02:50:aa:01:d0:
                    6e:bc:72:33:d9:61:3c:48:8c:72:ee:ed:27:0c:e6:
                    c7:88:29:51:73:f4:03:46:9d
                Exponent: 65537 (0x10001)

Is it too small ?

1 reply

marianne013 Aug 8, 2023

I think the short answer to this is yes. Ours are all 2048.

zhangxiaomei · 2023-08-09T06:41:29Z

zhangxiaomei
Aug 9, 2023
Author

I see. The certs of our server hasn't changed to 2048. I try to do it as soon as possible.

0 replies

zhangxiaomei · 2023-08-10T04:35:57Z

zhangxiaomei
Aug 10, 2023
Author

I have changed server cers to 2048, but the problems are still there. Is it because user proxy is still 1024?
Our fts server openssl version:
[root@fts3 ~]# rpm -qa |grep "openssl"
openssl-libs-1.0.2k-26.el7_9.x86_64
globus-gsi-openssl-error-4.4-1.el7.x86_64
globus-openssl-module-5.2-1.el7.x86_64
openssl-devel-1.0.2k-26.el7_9.x86_64
openssl-1.0.2k-26.el7_9.x86_64

0 replies

zhangxiaomei · 2023-08-10T08:20:27Z

zhangxiaomei
Aug 10, 2023
Author

The new errors after updating certs to 2048:
2023-08-10 06:05:01 UTC Configuration/VOMS2CSAgent/VOMS2CSSynchronizer ERROR: Could not retrieve user information from VOMS Unauthorized query ( 1111 : Failed to contact the VOMS server: https://voms.ihep.ac.cn:8443/voms/bes/apiv2/users:SSLError(MaxRetryError("HTTPSConnectionPool(host='voms.ihep.ac.cn', port=8443): Max retries exceeded with url: /voms/bes/apiv2/users?startIndex=0&pageSize=100 (Caused by SSLError(SSLError(397, '[SSL: CA_KEY_TOO_SMALL] ca key too small

2 replies

chrisburr Aug 10, 2023
Maintainer

This is a different issue. I think the problem is that you're using a 1024-bit proxy. I can reproduce your error in LHCb with:

from DIRAC.Core.Security import Locations
import requests

dirac-proxy-init --strength 2048

requests.get("https://lcg-voms2.cern.ch:8443/voms/lhcb/user/home.action", cert=("/tmp/x509up_u1000", "/tmp/x509up_u1000"), verify=Locations.getCAsLocation())
# All okay

dirac-proxy-init --strength 1024

requests.get("https://lcg-voms2.cern.ch:8443/voms/lhcb/user/home.action", cert=("/tmp/x509up_u1000", "/tmp/x509up_u1000"), verify=Locations.getCAsLocation())
# SSLError: HTTPSConnectionPool(host='lcg-voms2.cern.ch', port=8443): Max retries exceeded with url: /voms/lhcb/user/home.action (Caused by SSLError(SSLError(399, '[SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3900)')))

chrisburr Aug 10, 2023
Maintainer

By 1024-bit proxy, I mean the entire chain must be 2048-bits (or more) as the strength is computed as the weakest certificate in the chain.

chrisburr · 2023-08-10T08:37:19Z

chrisburr
Aug 10, 2023
Maintainer

The issue you're seeing is the same as #6299 that the default OpenSSL security level was increased to 2 (see python/cpython#88164).

The best option would be to get a new certificate for voms.ihep.ac.cn.

1 reply

chrisburr Aug 10, 2023
Maintainer

To be more specific, the problem is that the certifcate used by fts3.ihep.ac.cn is only 1024-bit.

$ echo | openssl s_client fts3.ihep.ac.cn:8446 2>/dev/null | grep 'Server public key is'
Server public key is 1024 bit

For reference, 1024-bit RSA was first factorised in 2007 and there is little reason not to be using 4096-bit RSA. Many CAs only sign RSA 2048-bit+ or ECDSA keys.

Answer selected by chrisburr

zhangxiaomei · 2023-08-10T09:21:07Z

zhangxiaomei
Aug 10, 2023
Author

The above error is from the Configuration_VOMS2CSAgent agent. The VOMS2CS agent has the same problem.
(base) [zhangxm@prod-dirac vomses]$ echo | openssl s_client voms.ihep.ac.cn:8443 2>/dev/null | grep 'Server public key is'
Server public key is 2048 bit

0 replies

zhangxiaomei · 2023-08-10T09:26:40Z

zhangxiaomei
Aug 10, 2023
Author

The fts server has not been restarted correctly. Now it is also:
(base) [zhangxm@prod-dirac vomses]$ echo | openssl s_client fts3.ihep.ac.cn:8446 2>/dev/null | grep 'Server public key is'
Server public key is 2048 bit

0 replies

zhangxiaomei · 2023-08-10T09:27:22Z

zhangxiaomei
Aug 10, 2023
Author

You are right. We need to push our CA to use 4096-bit cert. Now it is 2048 bit. But it looks like not easy. To use the latest version of DIRAC, it is a must to update CAs to 4096-bit? @chrisburr

1 reply

chrisburr Aug 10, 2023
Maintainer

2048 should be fine

zhangxiaomei · 2023-08-10T10:00:04Z

zhangxiaomei
Aug 10, 2023
Author

Updating user certs to be 2048 bit (not only 2048 proxy) seems working.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Errors in FTSAgent after upgrading from v7.3.18 to v8.0.24 #7165

{{title}}

Replies: 10 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Errors in FTSAgent after upgrading from v7.3.18 to v8.0.24 #7165

zhangxiaomei Aug 8, 2023

Replies: 10 comments · 5 replies

fstagni Aug 8, 2023 Maintainer

atsareg Aug 8, 2023 Maintainer

marianne013 Aug 8, 2023

zhangxiaomei Aug 9, 2023 Author

zhangxiaomei Aug 10, 2023 Author

zhangxiaomei Aug 10, 2023 Author

chrisburr Aug 10, 2023 Maintainer

chrisburr Aug 10, 2023 Maintainer

chrisburr Aug 10, 2023 Maintainer

chrisburr Aug 10, 2023 Maintainer

zhangxiaomei Aug 10, 2023 Author

zhangxiaomei Aug 10, 2023 Author

zhangxiaomei Aug 10, 2023 Author

chrisburr Aug 10, 2023 Maintainer

zhangxiaomei Aug 10, 2023 Author

zhangxiaomei
Aug 8, 2023

Replies: 10 comments 5 replies

fstagni
Aug 8, 2023
Maintainer

atsareg
Aug 8, 2023
Maintainer

zhangxiaomei
Aug 9, 2023
Author

zhangxiaomei
Aug 10, 2023
Author

zhangxiaomei
Aug 10, 2023
Author

chrisburr Aug 10, 2023
Maintainer

chrisburr Aug 10, 2023
Maintainer

chrisburr
Aug 10, 2023
Maintainer

chrisburr Aug 10, 2023
Maintainer

zhangxiaomei
Aug 10, 2023
Author

zhangxiaomei
Aug 10, 2023
Author

zhangxiaomei
Aug 10, 2023
Author

chrisburr Aug 10, 2023
Maintainer

zhangxiaomei
Aug 10, 2023
Author