feat: Introduce ecosystem tests for popular plugins

[JoshuaKGoldberg]

Summary

Adding an CI job to the eslint/eslint repo that checks changes against a small selection of third-party plugins.

Related Issues

eslint/eslint#19139

[voxpelli]

That none of the eslint-community plugins are included feels a bit odd? Since it's somewhat operated under the same umbrella as ESLint itself?

At least one of eg eslint-plugin-n, eslint-plugin-promise or eslint-plugin-security would be good to include?

I also note that the selection criteria is similar to those outlined in the suggested eslint-community governance, and which have gotten feedback / objections / concerns there: eslint-community/governance#1 (comment)

[nzakas]

My apologies, I started a review last week, left a bunch of comments, and then forgot to hit "Submit Review".

I think this outlines a good infrastructure, just looking for some clarifications.

[aladdin-add]

Looks great to me! This is an excellent starting point. We can proceed to run it and see how it performs. 🚀🚀🚀

[nzakas]

I just realized we're not really defining what "breakage" is anywhere. Are we just running npm test on each package with the local ESLint changes?

[JoshuaKGoldberg]

I wasn't planning on suggesting including source code from plugins in this proposal. To start I was thinking of just verifying that npm lint doesn't error out - regardless of rule reports. Meanwhile, the instigating issues mentioned in the RFC were all runtime crashes that would be caught by it.

Relying on tests makes me a little nervous. It'd be a lot slower -especially if the plugins have build steps- and we'd need to make sure none of them have tests that rely on specifics of rule reports.

My vote would be to just use the plugins as end-users until we have a breakage that would have been caught by source code level checks.

[nzakas]

To start I was thinking of just verifying that npm lint doesn't error out - regardless of rule reports.

Can you explain how that would work? Are you running npm lint in the ESLint repo? If so, does that mean we can only test plugins that ESLint itself uses?

[JoshuaKGoldberg]

That's what I intend by this:

rfcs/designs/2024-repo-ecosystem-plugin-tests/README.md

Lines 30 to 35 in 8fd5da4

	1. Create a new directory containing:
	- `package.json`
	- `eslint.config.js` with the closest equivalent to an _"enable all rules"_ preset from the plugin
	- A small set of files known to be parsed and not cause lint reports with the plugin
	2. Run a lint command (i.e. `npx eslint .`) in that directory
	3. Assert that the lint command passed with 0 lint reports.

Is that not clear? Is there a different phrasing you'd suggest?