Tracking "real" resource utilization metrics

[drmorr0]

Right now, Virtual Kubelet (and, I assume, KWOK) don't report any metrics about "real" resource utilization of the nodes or pods running on them. You can get information about pod requests/limits via, e.g., kube-state-metrics, but if you wanted to know (or replay) actual CPU utilization of a pod via cAdvisor metrics, you can't.

In theory sk-tracer could capture this data, but there are a bunch of problems to solve:

1. How do you store the data efficiently/at what granularity?
2. How do you communicate that data efficiently to the simulated pod?
3. How does the simulation actually report this data in a way that's meaningful?

Solving this problem is critical before we can do things like simulating HPA or KEDA behaviour, for example.