x/build: improve the container deployment process

[cagedmantis]

I propose changing how services are deployed from the x/build repository. I would like to continue/finalize the migration from building and deploying containers on local machines to doing most of the work on Cloud Build. I propose we:

Current State: Services are built and deployed as containers primarily on local machines.

Create a trigger in Cloud Build for each service, The trigger will take a x/build repo commit hash as input and:

• Checkout the commit from go.googlesource.com.
• Check if a container for the requested commit already exists.
• If a container image for the requested commit does not exist, it will build and push the container image.
• Once the container image exists, it will deploy the service using the container image.

When a service is being deployed, the user can initiate a deployment by running a make target which:

• If the service being deployed is from a published commit (the commit ID is available on the remote Gerrit server) and the local work tree has no uncommitted or staged changes, it will initiate a cloud build trigger with a commit hash (see previously mentioned trigger).
• If the service being deployed contains uncommitted work or staged changes (a dirty commit):
  • A locally defined cloud build job will be initiated (much like many of the services are currently deployed now)
  • It will upload the local cloud build definition and checked out repository.
  • Cloud build will create the image and publish it.
  • The XB command will be called to deploy the newly created container.

Additional changes:

• There should be a mechanism for redeploying an existing deployment. Possible solutions include:
  • Each Kubernetes manifest should contain a GO_IMAGE_DEPLOYMENT_TIMESTAMP environmental variable that is updated with a timestamp of the requested deployment time. This should ensure that the deployment manifests will always be unique.
  • Deleting the pods for the deployment and waiting for the Kubernetes reconciliation loop to recreate them again.
• The canonical source for cloud build definitions will be stored in the x/build repository. There should be make targets added which will set the locally defined cloud build definitions to the existing cloud build triggers. This models work initiated by @dmitshur. How and where the the various elements are stored is completely open to discussion. We should use this issue to resolve those questions.
• The deployment make files should share as much logic as possible since most of these jobs will use the same logic.

Possible Benefits:

• A more centralized build and deployment process.
• Reduced reliance on local machines. Which is great for security and reducing errors due to bespoke local machine configurations.

@golang/release

[gabyhelp]

Related Issues

• x/build: use cloud build to deploy x/build services #57852

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}