PYTHON-2560 Retry KMS requests on transient errors

[ShaneHarvey]

PYTHON-2560 Retry KMS requests on transient errors

Depends on:

• DRIVERS-1541 Add kms_failpoint_server.py to start-servers.sh for KMS retry testing mongodb-labs/drivers-evergreen-tools#554
• PYTHON-4992 Support retrying KMS requests libmongocrypt#920

[blink1073]

Failures look relevant: "FAILURE: AssertionError: "kms.us-east-1.amazonaws.com:12345" does not match "KMS request failed after 3 retries due to a network error" ()"

[ShaneHarvey]

Ah this is caused by the problem described in: https://jira.mongodb.org/browse/MONGOCRYPT-752

With this new feature, the user gets a too vague error message that just says "KMS request failed after 3 retries due to a network error" so we can't assert on the error message for those other tests now. Without libmongocrypt changes, we'll need to introduce a new context object to store the previous exception error message.

[ShaneHarvey]

It turns out to be quite simple to include the last error message. I didn't realize the error comes from mongocrypt_kms_ctx_fail. Tests are passing locally now.

[blink1073]

test.test_encryption.TestCustomEndpoint.test_04_aws_endpoint_invalid_port is still failing with:

[2024/12/03 16:50:30.134] FAILURE: AssertionError: "kms.us-east-1.amazonaws.com:12345" does not match "KMS request failed after 3 retries due to a network error, last attempt failed with: [Errno 111] Connection refused" ()

[blink1073]

LGTM!