Deadman alerts for missing data ingested through pushgateway

[bengerman13]

We have a several metrics running through pushgateway, but because of the way pushgateway is architected, we wouldn't know if these alerts stopped publishing. We should set up alerts for missing data on all our metrics that come through pushgateway.

[karareinsel]

@bengerman13 can this be fleshed out a bit more? What are the metrics that will need alerts? Removing from triage for now - alert to the engineers to revisit it during the engineering huddle after Thanksgiving to reassess priority.

[bengerman13]

This ticket, I think, is about:

• figuring out how to do this
• starting on the enumeration of metrics.
  We have tons of metrics in prometheus. I'd guess most but not quite all need this check.