[Bug]: bootstrap pipeline BUILD phase fails after sam/build-python3.12:1.132.0 update

[niklaswesterstrahleknowit]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Noticed that after the latest update to upstream sam/build-python3.12 the build phase of aws-deployment-framework-bootstrap-pipeline fails immediately when sam is called.

[Container] 2024/12/19 13:43:05.532623 Entering phase BUILD
[Container] 2024/12/19 13:43:33.317976 Phase complete: BUILD State: FAILED

Our builds prior to this update went through ok. We didn't go into reviewing what has changed between the versions.

Expected Behavior

Expected the build to run through ok with latest (default for adf).

Tested rolling back to 1.131.0 and that runs ok.

Current Behavior

Container] 2024/12/19 13:43:05.532623 Entering phase BUILD
[Container] 2024/12/19 13:43:05.533546 Running command sam build --use-container --template adf-bootstrap/deployment/global.yml
SAM CLI now collects telemetry to better understand customer needs.
You can OPT OUT and disable telemetry collection by setting the
environment variable SAM_CLI_TELEMETRY=0 in your shell.
Thanks for your help!
Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html
Starting Build inside a container
Building layer 'ADFSharedPythonLambdaLayerVersion'
Fetching public.ecr.aws/sam/build-python3.12:latest-arm64 Docker container image..............................................................................................................................................................................................................................................................................................................
Mounting /codebuild/output/src789601948/src/adf-build/shared/python as /tmp/samcli/source:ro,delegated, inside runtime container
exec /usr/local/opt/lambda-builders/bin/lambda-builders: exec format error
Builder crashed:
Error: Expecting value: line 1 column 1 (char 0)
Traceback:
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/cli/cli_config_file.py", line 347, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/telemetry/metric.py", line 185, in wrapped
raise exception # pylint: disable=raising-bad-type
^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/telemetry/metric.py", line 150, in wrapped
return_value = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/utils/version_checker.py", line 43, in wrapped
actual_result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/cli/main.py", line 95, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/commands/build/command.py", line 175, in cli
do_cli(
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/commands/build/command.py", line 264, in do_cli
ctx.run()
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/commands/build/build_context.py", line 284, in run
self._build_result = builder.build()
^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 225, in build
return ApplicationBuildResult(build_graph, build_strategy.build())
^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/build_strategy.py", line 82, in build
result.update(self._build_layers(self._build_graph))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/build_strategy.py", line 110, in _build_layers
layer_build_results.update(self.build_single_layer_definition(layer_definition))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/build_strategy.py", line 239, in build_single_layer_definition
layer.full_path: self._build_layer(
^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 566, in _build_layer
self._build_function_on_container(
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 963, in _build_function_on_container
response = self._parse_builder_response(stdout_data, container.image)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/site-packages/samcli/lib/build/app_builder.py", line 984, in _parse_builder_response
response = json.loads(stdout_data)
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/json/init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.12.4/lib/python3.12/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
An unexpected error was encountered while executing "sam build".
Search for an existing issue:
https://github.com/aws/aws-sam-cli/issues?q=is%3Aissue+is%3Aopen+Bug%3A%20sam%20build%20-%20JSONDecodeError
Or create a bug report:
https://github.com/aws/aws-sam-cli/issues/new?template=Bug_report.md&title=Bug%3A%20sam%20build%20-%20JSONDecodeError
[Container] 2024/12/19 13:43:33.314007 Command did not exit successfully sam build --use-container --template adf-bootstrap/deployment/global.yml exit status 1
[Container] 2024/12/19 13:43:33.317976 Phase complete: BUILD State: FAILED

Steps To Reproduce

No response

Possible Solution

[Container] 2024/12/19 14:32:14.096712 Running command sam build --use-container --template adf-bootstrap/deployment/global.yml --build-image public.ecr.aws/sam/build-python3.12:1.131.0-20241122183713

We told the build to use an older image by editing template.yml, and deploying that.

Additional Information/Context

No response

ADF Version

4.0.0

Contributing a fix?

• Yes, I am working on a fix to resolve this issue

[stemons]

Hello, we temporarily solved by modifying the buildspec file of the bootstrap pipeline adding the older version of python ecr image

sam build
--use-container
--build-image public.ecr.aws/sam/build-python3.12:1.131.0
--template adf-bootstrap/deployment/global.yml

[niklaswesterstrahleknowit]

Exactly what we ended doing, that’s a possible solution - but doesn’t sound smart long term.