Propose a feasible way to add/remove plugins

[gameloser]

Problem Statement

From the roadmap

• Plugin versions
• Dynamically add or remove plugins at build time (to reduce binary size)

Proposed Solution

Instead of packing plugins to binary, making per plugin per image, refer to kuberhealthy's daemonset-check. The plugin manager in Agent will also need some adjustments.

Alternatives Considered

Additional Context

[subbaksh]

Yeah, The design decision not to create new pods for every test is deliberate. This is because when there are 1000s of tests, it puts a lot of load on the apiserver. So we wanted each test run and plugin to have relatively small costs in terms of compute and memory, as well as the number of requests to the api server. But there could be a compromise where you deploy a new plugin as a new pod, which gets invoked by pluginmanager at set intervals. Good suggestion!