Curl Plugin leaks curl processes when timing out.

[kshave]

Description

When the a curl synthetic-test reaches the "timeouts.run" timeout and the curl process is still running, the curl process remains after the plugin is killed and restarted causing processes to be leaked. This eventually causes the following error for all synthetic-tests on the same agent:

runtime: failed to create new OS thread (have 9 already; errno=11) runtime: may need to increase max user processes (ulimit -u) fatal error: newosproc

Below is the output from running a "pstree" command on one of the synthetic-agents with the Curl plugin failing:

/app/synthetic-heart $ pstree
agent-+-4894*[curl]
      `-14*[{agent}]
/app/synthetic-heart $ 

This behaviour was found when a curl synthetic-test was provided with a target domain which was not-responsive, causing the curl process to hang indefinitely waiting for a response. Each time the plugin was restarted, the previous curl process would remain on the container, eventually reaching the mac user processes limit and causing other plugins to fail.

Expected Behavior

After a curl synthetic-test has reached the "timeouts.run" timeout, during plugin cleanup the curl process is killed.

Actual Behavior

After a curl synthetic-test has reached the "timeouts.run" timeout, the curl process is not cleaned up.

Affected Version

v1.2.3

Steps to Reproduce

1. Setup local synthetic-heart with podman following steps outlined in Readme's
2. Install the following curl synthetic-test CRD

apiVersion: synheart.infra.webex.com/v1
kind: SyntheticTest
metadata:
  name: curl-amazon
  labels:
    infra: "true"
spec:
  plugin: curl
  node: $
  displayName: Curl Metrics (CI FLS)
  description: Test to get curl metrics for CI FLS
  timeouts:
    run: 100ms
    repeat: 1m
  config: |
    url: https://www.amazon.com

Very small timeout will cause the plugin to exit before curl command can resolve and request domain.
3. Run "pstree" on the synthetic-agent container where the plugin is run, number of curl processes slowly increasing after each plugin restart.

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

[subbaksh]

Thanks @kshave , thats a good issue. Will investigate.