Fixed logic to give the correct AppWrapper status transition

[metalcycling]

What changes have been made

When an AppWrapper has multiple generic items that finish at different times, transitioning to Complete state depends on the status conditions of each generic item. If no generic item has reached any of its specified completionstatus conditions, then the AppWrapper remains in Running state after dispatching. If some but not all of the generic items reach their completionstatus conditions, then the AppWrapper state becomes RunninHoldCompletion. Once all the generic items reach their completionstatus conditions, then the AppWrapper becomes Completed.

Verification steps

To observe the transitions use the following AppWrapper and monitor the .status.state value for the AppWrapper as time goes by. Without these changes, the AppWrapper goes from Running to Completed despite the fact that some generic items complete in between the two states. The correct transition should be Running->RunningHoldCompletion->Completed:

apiVersion: mcad.ibm.com/v1beta1
kind: AppWrapper
metadata:
    name: appwrapper-jobs
    namespace: default
spec:
    priority: 5
    priorityslope: 0.0
    schedulingSpec:
        minAvailable: 2
        requeuing:
            timeInSeconds: 300
    resources:
        GenericItems:
            - replicas: 1
              completionstatus: Complete
              custompodresources:
                  - replicas: 1
                    requests:
                        cpu: 1
                        nvidia.com/gpu: 0
                        memory: 1Gi
                    limits:
                        cpu: 1
                        nvidia.com/gpu: 0
                        memory: 1Gi
              generictemplate:
                  apiVersion: batch/v1
                  kind: Job
                  metadata:
                      name: job-1
                      namespace: default
                      labels:
                          appwrapper.mcad.ibm.com: appwrapper-jobs
                  spec:
                      parallelism: 1
                      completions: 1
                      backoffLimit: 0
                      template:
                          metadata:
                              name: job-1
                              namespace: default
                              labels:
                                  appwrapper.mcad.ibm.com: appwrapper-jobs
                          spec:
                              priorityClassName: "default-priority"
                              terminationGracePeriodSeconds: 1
                              restartPolicy: Never
                              containers:
                                  - name: pytorch
                                    image: ubuntu
                                    imagePullPolicy: IfNotPresent
                                    command:
                                        - sh
                                        - -c
                                        - |
                                          sleep 15
                                    resources:
                                        requests:
                                            cpu: 1
                                            nvidia.com/gpu: 0
                                            memory: 1Gi
                                        limits:
                                            cpu: 1
                                            nvidia.com/gpu: 0
                                            memory: 1Gi
            - replicas: 1
              completionstatus: Complete
              custompodresources:
                  - replicas: 1
                    requests:
                        cpu: 1
                        nvidia.com/gpu: 0
                        memory: 1Gi
                    limits:
                        cpu: 1
                        nvidia.com/gpu: 0
                        memory: 1Gi
              generictemplate:
                  apiVersion: batch/v1
                  kind: Job
                  metadata:
                      name: job-2
                      namespace: default
                      labels:
                          appwrapper.mcad.ibm.com: appwrapper-jobs
                  spec:
                      parallelism: 1
                      completions: 1
                      backoffLimit: 0
                      template:
                          metadata:
                              name: job-2
                              namespace: default
                              labels:
                                  appwrapper.mcad.ibm.com: appwrapper-jobs
                          spec:
                              priorityClassName: "default-priority"
                              terminationGracePeriodSeconds: 1
                              restartPolicy: Never
                              containers:
                                  - name: pytorch
                                    image: ubuntu
                                    imagePullPolicy: IfNotPresent
                                    command:
                                        - sh
                                        - -c
                                        - |
                                          sleep 30
                                    resources:
                                        requests:
                                            cpu: 1
                                            nvidia.com/gpu: 0
                                            memory: 1Gi
                                        limits:
                                            cpu: 1
                                            nvidia.com/gpu: 0
                                            memory: 1Gi

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from metalcycling. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[asm582]

@MichaelClifford @Maxusmusti @anishasthana The above change will require all genericItems to have completion status. Some intelligence would need to be added to fill completionstatus for deployment genericItem to None as deployment does not have completion, do you agree this change will impact SDK?