[FEAT] add driverPod/executorPod in Spark

[machichima]

Tracking issue

#4105

Why are the changes needed?

Enable setting K8sPod separately for Spark Driver and Executor pods.

What changes were proposed in this pull request?

Add driverPod and executorPod field with type K8sPod in SparkJob. Uses existing mergePodSpecs to merge default podSpec with our driverPod or executorPod.

How was this patch tested?

Unit tests

I extended the existing Spark unit test TestBuildResourceContainer and TestBuildResourcePodTemplate and create a new test named TestBuildResourceCustomK8SPod for testing.

Test with my_spark example

Modified the @task for hello_spark function in ``my_spark` example here as follow to set the driver_pod and executor_pod.

driver_pod_spec = V1PodSpec(
    containers=[
        V1Container(
            name="primary",
            image="ghcr.io/machichima",
            command=["echo"],
            args=["wow"],
            env=[V1EnvVar(name="x/custom-driver", value="driver")]
        ),
    ],
    tolerations=[
        V1Toleration(
            key="x/custom-driver",
            operator="Equal",
            value="foo-driver",
            effect="NoSchedule",
        ),
    ],
)

executor_pod_spec = V1PodSpec(
    containers=[
        V1Container(
            name="primary",
            image="ghcr.io/machichima",
            command=["echo"],
            args=["wow"],
            env=[V1EnvVar(name="x/custom-executor", value="executor")]
        ),
    ],
    tolerations=[
        V1Toleration(
            key="x/custom-executor",
            operator="Equal",
            value="foo-executor",
            effect="NoSchedule",
        ),
    ],
)

@task(
    task_config=Spark(
        # This configuration is applied to the Spark cluster
        spark_conf={
            "spark.executor.cores": "1",
            "spark.executor.instances": "2",
            "spark.driver.cores": "1",
            "spark.jars": "https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar",
        },
        driver_pod=K8sPod(pod_spec=driver_pod_spec.to_dict()),
        executor_pod=K8sPod(pod_spec=executor_pod_spec.to_dict()),
    ),
    container_image=custom_image,
    pod_template=PodTemplate(primary_container_name="primary"),
)

Verify the pods have Tolerations and EnvVar set.

❯ kubectl describe sparkapplications.sparkoperator.k8s.io -n flytesnacks-development acsqt4vd4pctzvp8t4cz-n0-0 | grep "Tolerations:" -A 4
    Tolerations:
      Effect:    NoSchedule
      Key:       x/custom-driver
      Operator:  Equal
      Value:     foo-driver
--
    Tolerations:
      Effect:             NoSchedule
      Key:                x/custom-executor
      Operator:           Equal
      Value:              foo-executor

❯ kubectl describe sparkapplications.sparkoperator.k8s.io -n flytesnacks-development acsqt4vd4pctzvp8t4cz-n0-0 | grep "Name:        x/custom-executor" -A 1
      Name:        x/custom-executor
      Value:       executor

❯ kubectl describe sparkapplications.sparkoperator.k8s.io -n flytesnacks-development acsqt4vd4pctzvp8t4cz-n0-0 | grep "Name:        x/custom-driver" -A 1
      Name:        x/custom-driver
      Value:       driver

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

flyteorg/flytekit#3016

Docs link

[codecov]

Codecov Report

Attention: Patch coverage is 58.82353% with 28 lines in your changes missing coverage. Please review.

Project coverage is 36.98%. Comparing base (ba331fd) to head (70cfdff).
Report is 28 commits behind head on master.

Files with missing lines	Patch %	Lines
flyteplugins/go/tasks/plugins/k8s/spark/spark.go	72.00%	10 Missing and 4 partials ⚠️
flyteidl/gen/pb-go/flyteidl/plugins/spark.pb.go	0.00%	10 Missing ⚠️
...ns/go/tasks/pluginmachinery/flytek8s/pod_helper.go	50.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6085      +/-   ##
==========================================
- Coverage   37.10%   36.98%   -0.13%     
==========================================
  Files        1318     1318              
  Lines      132331   132570     +239     
==========================================
- Hits        49097    49026      -71     
- Misses      78961    79280     +319     
+ Partials     4273     4264       -9     

Flag	Coverage Δ
unittests-datacatalog	51.58% <ø> (ø)
unittests-flyteadmin	54.05% <ø> (-0.05%)	⬇️
unittests-flytecopilot	30.99% <ø> (ø)
unittests-flytectl	62.29% <ø> (ø)
unittests-flyteidl	7.23% <0.00%> (-0.01%)	⬇️
unittests-flyteplugins	53.86% <68.96%> (+0.03%)	⬆️
unittests-flytepropeller	42.59% <ø> (-0.04%)	⬇️
unittests-flytestdlib	55.18% <ø> (-2.36%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.