Add a metric of AppWrappers counts per state

[ronensc]

Issue link

Closes #674

What changes have been made

This PR adds a custom metric to expose how many AppWrappers are per state.
It is built on top of #651. To review only the delta, skip the first 3 commits in this PR.

Verification steps

Same as mentioned in #651, I checked that the metrics are available in the codeflare-operator operator by running it:

# in codeflare-operator root
go mod edit -replace github.com/project-codeflare/multi-cluster-app-dispatcher=../multi-cluster-app-dispatcher
make build
NAMESPACE=default go run ./main.go -kubeconfig <path-to-kubeconfig>

and in a new shell:

curl localhost:8080/metrics | grep appwrappers_count


# HELP mcad_appwrappers_count AppWrappers count per state
# TYPE mcad_appwrappers_count gauge
mcad_appwrappers_count{state="Completed"} 0
mcad_appwrappers_count{state="Deleted"} 0
mcad_appwrappers_count{state="Failed"} 0
mcad_appwrappers_count{state="Pending"} 0
mcad_appwrappers_count{state="Running"} 1
mcad_appwrappers_count{state="RunningHoldCompletion"} 0

To see the counts change, I created a sample AppWrapper:

# in MCAD root
kubectl apply -f doc/usage/aw-01.yaml

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign asm582 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ronensc]

In case of error, should we update the counters to 0 or leave them as is?

[mamy-CS]

I think we should leave them as is and retry to update with new value.

[ronensc]

This update is done periodically. So instead of retrying, we can simply wait for the next update on the next tick.
I dug deeper into the implementation of controller.appWrapperLister.List(labels.Everything()), and it seems that since labels.Everything().Empty() returns true, there is no way that controller.appWrapperLister.List() will return an error. So the discussion seems to be purely theoretical.