In Kubernetes driver, how can I reduce the memory (docker layer cache) within a Pod?

[githubeto]

I am using ECR as the storage and usage destination for caches with the Kubernetes driver.
However, after a build without cache is executed, I have confirmed that some memory within the Pod remains occupied and does not get freed up.
Since the memory within the node continues to be used for an extended period, I am looking to reduce memory usage outside of build execution.

example.

docker buildx build . \
  -t xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/sample-cache:latest  \
  --cache-to compression=zstd,mode=max,image-manifest=true,oci-mediatypes=true,type=registry,ref=xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/sample-cache:latest \
  --cache-from type=registry,ref=xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/sample-cache:latest \
  --push

This issue seems to occur not only because the cache is pushed to ECR but also because the layer cache remains locally, and I have confirmed that it can be released by executing docker buildx prune --builder <builder-name>.

I looked into the GC documentation, and while it seems possible to adjust the timing, I couldn't find a setting to not leave the cache locally.

https://docs.docker.com/build/cache/garbage-collection/

When using ECR's Cache with the Kubernetes driver, what should I do about the local cache? Is setting the GC timing to a very short interval the only option, or is there another approach?