Add pants-plugins/sample_conf to streamline regenerating conf/st2.conf.sample

[cognifloyd]

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

• pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
• pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

This PR improves the DX (developer experience) by wiring up the fmt and lint goals to generate conf/st2.conf.sample if needed (or for lint, warn that it needs to be done).

This includes creating a sample_conf plugin for pants that adds a sample_conf target.

Developer Experience

Today, if someone changes one of the files used to generate conf/st2.conf.sample, they will probably forget to run the Makefile target the regenerates them: make configgen.

I've missed running that several times. CI complains as part of the ci-checks Makefile target, with instructions to Please run "make configgen". This also gets regenerated any time someone runs the lint or tests Makefile targets, which might be surprising (tests generally shouldn't handle codegen).

With this PR, we add st2.conf.sample generation into the fmt goal. This also adds it to the lint goal which will tell people if fmt needs to run to update any code (or in this case, regenerate the sample conf file). Then, we only need simple instructions that say something like:

Please run ./pants fmt lint :: before submitting your PR. This will reformat code, if needed, and warn you about any other errors.

Fine grained results cache

After someone runs ./pants fmt :: once, they will benefit from the pants' results cache. For the sample_conf plugin, here's what that means:

First, the sample_conf target in conf/ depends on tools/config_gen.py:

st2/conf/BUILD

Lines 6 to 9 in 1faea8f

	sample_conf( # defined in pants-plugins/sample_conf
	name="st2.conf.sample",
	source="st2.conf.sample",
	dependencies=[

If the config_gen script changes, then pants now knows that the sample conf file need to be regenerated. Or, in other words, the config_gen.py file is part of the cache key for conf/st2.conf.sample. If that file changes, then pants will re-run the generator.

For many of our scripts, we can depend on pants' python dependency inference to grab the other files. But, this script uses importlib instead of standard imports, so we need to add the dependencies manually (aside: there is a feature that tries to use strings for dep inference, but it is not enabled by default, and I have not tested to see how many false positives it will find across our repo). In the future, hopefully we can remove or simplify this.

st2/tools/BUILD

Lines 1 to 24 in 1faea8f

	python_sources(
	overrides={
	"config_gen.py": {
	"dependencies": [
	# the auth backends get listed in the conf file
	"//:auth_backends",
	# the following should match CONFIGS in config_gen.py
	# grep -rl '^def register_opts(ignore_errors=False):' st2*
	"st2actions/st2actions/scheduler/config.py",
	"st2actions/st2actions/workflows/config.py",
	"st2actions/st2actions/notifier/config.py",
	"st2actions/st2actions/config.py",
	"st2api/st2api/config.py",
	"st2auth/st2auth/config.py",
	"st2common/st2common/config.py",
	"st2reactor/st2reactor/garbage_collector/config.py",
	"st2reactor/st2reactor/timer/config.py",
	"st2reactor/st2reactor/sensor/config.py",
	"st2reactor/st2reactor/rules/config.py",
	"st2stream/st2stream/config.py",
	]
	},
	},
	)

Please note the explicit dependency on //:auth_backends. Nothing in our code base directly imports from the auth backends. But, we ship with a default set of backends, so we need to make sure that their settings are included in the sample conf file's auth section. This is defined here:

st2/BUILD

Lines 43 to 49 in 1faea8f

	target(
	name="auth_backends",
	dependencies=[
	"//:reqs#st2-auth-backend-flat-file",
	"//:reqs#st2-auth-ldap",
	],
	)

Then, I ran into one more dependency issue. st2-auth-ldap does not declare a dependency on our st2auth module, in part because that is an implicit dep for auth modules. We should probably make that explicit at some point, but that might require publishing more wheels on pypi. That's messy, but pants meets us where we are in this mess. 😄 So, we can tell pants that any time we grab the st2-auth-ldap package, it requires something in the st2auth module, like this:

st2/BUILD

Lines 34 to 39 in 1faea8f

	# make sure anything that uses st2-auth-ldap gets the st2auth constant
	"st2-auth-ldap": {
	"dependencies": [
	"st2auth/st2auth/backends/constants.py",
	]
	}

Then, once all of those deps that pants can't infer (for now) are recorded, pants is able to invalidate the st2.conf.sample cache if any of the dependency python source files (including all that get imported directly or transitively by tools/config_gen.py).

This PR actually only wires up the fmt goal and took advantage of the fact that pants also runs formatters whenever it runs linters, which are side-effect free. And because pants runs the formatters in a sandbox, it doesn't have to materialize any changes during lint.

Continuing the example, since pants cached the results when running the lint goal, running fmt will be very fast. Pants will just materialize the already-generated schemas from its cache.

Developing pants plugins

Pants has extensive documentation on the plugin API, including how to create targets, how to write rules, and there's even a mini tutorial about how to add a formatter (adding formatters is a very common thing for plugins to do).

• Plugins overview
• The Target API
• The Rules API
• Processes (running subprocesses in pants plugins)
• Add a formatter
• Testing Plugins

[amanda11]

LGTM

[rush-skills]

LGTM