Migrate Prometheus Deployments to be co-located in environments

[ChrisMcGowan]

To keep current with the Prometheus Bosh releases and support newer versions of CF exports, the existing Prometheus Deployments need to be upgraded as follows:

• Turn the two existing pipeline jobs for Production and Stage into four: Dev, Stage, Tooling, Prod
• For the initial rollout - leave the existing deployments in tooling alone
• Each deployment will be co-located on each BOSH Director
• For the new tooling deployment - just colocate the needed BOSH and other exporters to support tooling alone - no CF needed
• Remove the legacy cf-attic exporter and dashboards
• Use the current supported cf-firehose export and dashboard
• Migrate to using the existing upstream manifest
• Add back in CG customizations to the deploy including making the new cf-firehose job be co-located on the prometheus VM instead of a dedicated VM
• Very all the dashboards are getting the new data and working as expected
• 14 days after prod and staging are deployed, remove the legacy deployments in tooling only leaving the new tooling specific Prometheus behind

Notes

ADR on the approach with approval: https://github.com/cloud-gov/internal-docs/blob/main/docs/ADRs/ADR-0015-prometheus-location-change.md

Acceptance Criteria

• Make sure deployments work while leaving existing alone
• Update the pipeline to reflect the env changes
• Update documentation and diagrams to reflect the change
• Leave the legacy deployments up and running at the same time for two weeks post prod sign-off before deleting legacy deployments in tooling.