SQS Response Protocol Migrated from XML to JSON

[mkeemon]

Moto (installed via Docker, pip)

• Python version: 3.12.0
• Moto Version: 4.2.9

AWS Ruby SDK

• aws-sdk-core: 3.187.1
• aws-sdk-sqs: 1.67.0

According to AWS documentation, the most recent version of their SDKs utilize JSON instead of XML for SQS responses. It appears this is a backward incompatible change, so migrating to the most recent version of the various AWS SDKs will cause cause issues with Moto.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-json-faqs.html

Tests are failing in the ruby-shoryuken/shoryuken repository due to the XML responses in Moto

Action run in ruby-shoryuken/shoryuken:

JSON::ParserError:
          #   859: unexpected token at '<CreateQueueResponse><CreateQueueResult><QueueUrl>http://localhost:5000/123456789012/shoryuken-travis-StandardWorker-455c8b98-cade-4220-a7cd-8b9e5bdfed27</QueueUrl></CreateQueueResult><ResponseMetadata><RequestId>F4FSDWJ3GAHTAWXHEQ11QOW56ZOVVDXRLHTUKZLB10T1XSI17DSA</RequestId></ResponseMetadata></CreateQueueResponse>'
          #   ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/json/json_engine.rb:8:in `load'

     2.2) Failure/Error:
            queue_url = Shoryuken::Client.sqs.get_queue_url(
              queue_name: StandardWorker.get_shoryuken_options['queue']
            ).queue_url

          Aws::SQS::Errors::NotFound:
            Aws::SQS::Errors::NotFound
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-core-3.187.1/lib/seahorse/client/request.rb:72:in `send_request'
          # ./vendor/bundle/ruby/3.1.0/gems/aws-sdk-sqs-1.67.0/lib/aws-sdk-sqs/client.rb:1475:in `get_queue_url'
          # ./spec/integration/launcher_spec.rb:56:in `block (3 levels) in <top (required)>'

Dockerfile for Moto

FROM python:3.12

RUN pip install 'moto[server,sqs]'
CMD moto_server sqs -H 0.0.0.0

HEALTHCHECK --interval=5s --timeout=3s \
  CMD curl --fail http://localhost:5000/?Action=ListQueues || exit 1

EXPOSE 5000

[bblommers]

Interesting - that should have been fixed by #6331 already. Maybe the Ruby SDK uses a different header than the other SDK's to flag that we should return JSON. Will look into it later.

[bblommers]

I can't reproduce this with the latest version of Moto.

Looking at the repository that you linked, you are using a custom Docker image:

https://github.com/ruby-shoryuken/shoryuken/blob/d65636ebacd6a33b68cdc7f539aa9ce6a50eef9f/.github/workflows/specs.yml#L17

And that Docker image was updated 3 years ago - which makes me think that you're actually running against an old(er) version of Moto. (https://quay.io/repository/cjlarose/moto-sqs-server)

[mkeemon]

Hey @bblommers I was able to replicate this using the Dockerfile I provided in the initial post, which uses the most up to date version of Python and Moto

I've created a very basic app that uses the most up to date versions of Moto and the AWS Ruby SDK. Interestingly, the create_queue function runs just fine, but delete_queue fails due to the JSON/XML mismatch.

https://github.com/mkeemon/moto-aws-ruby-debug

Edit: To validate this further, modifying the Gemfile to downgrade aws-sdk-sqs to 1.65.0 and re-running the test works.

gem 'aws-sdk-sqs', '1.65.0'

aws-sdk-sqs 1.67.0

aws-sdk-sqs 1.65.0

[bblommers]

Thanks for the investigation @mkeemon. I had only tested with the operations that happened to work, that's why I couldn't reproduce.

The problem is that there are a few operations that do not return any useful data (such as delete_queue), where Moto only returns XML data. The Python SDK completely ignores the response, so then it doesn't matter that it's in the wrong format, but the Ruby SDK does try to parse it as JSON.

I'll raise a PR to fix this + add a test using the Ruby SDK.