Initialize pantsbuild

[cognifloyd]

Background

This is the first part of introducing pants, as discussed in the TSC Meetings on 12 July 2022 and 02 Aug 2022. Pants has fine-grained per-file caching of results for lint, fmt (like black), test, etc. It also has lockfiles that work well for monorepos that have multiple python packages. With these lockfiles CI should not break when any of our dependencies or our transitive dependencies release new versions, because CI will continue to use the locked version until we explicitly relock with updates.

To keep PRs as manageable/reviewable as possible, introducing pants will take a series of PRs. I do not know yet how many PRs; I will break this up into logical steps with these goals:

• introduce pants to the st2 repo, and
• teach TSC members about pants step-by-step.

So, I want as many TSC members as possible to review each of the pants PRs. Please ask questions so I can help explain what each part does.

Overview of this PR

There is very little to review in this PR. Effectively, there are only two lines of config (selecting the version of pants to use) that are unique to this repo.

This walks through a basic pants installation using:

• https://www.pantsbuild.org/docs/manual-installation
  • record [GLOBAL].pants_version in the root pants.toml file
  • Download the ./pants launch script from https://static.pantsbuild.org/setup/pants

pants components

./pants launch script

./pants is the primary entry point for accessing pants functionality. Updating this script does not happen very often, and it is designed to be checked into git.

Pants has a launch script (called ./pants) that handles downloading, bootstrapping, and upgrading Pants, which you need to save at the root of your repository.

https://www.pantsbuild.org/docs/installation

Examples of usage (:: tells pants to select everything - you can also pass a path to a file or directory instead.):

• to run linters like flake8 and shellcheck we will run ./pants lint ::
• to run formatters like black we will run ./pants fmt ::
• to run tests via pytest (once that works) we will run ./pants test ::

Note that this PR does not actually hook up linters/formatters/etc, so these will not do anything yet. That will happen in follow-up PRs.

pants.toml config file

Pants configuration lives in a file called pants.toml in the root of the repo.

https://www.pantsbuild.org/docs/initial-configuration#create-pantstoml

Many important things get configured in the pants.toml config file, but this PR only configures [GLOBAL].pants_version.

[GLOBAL].pants_version

As the name suggests, we use [GLOBAL].pants_version to switch to a newer version of pants. When someone updates, make sure to do one minor version at a time (ie to go from 2.9 to 2.11, you would do 2.9-2.10 and then 2.10-2.11) so that pants can handle migrating any config or metadata. Pants provides tooling to handle most of the required changes to config/metadata, and for everything else, they make sure to issue warnings when something is deprecated and how to update it.

Things you can do with pantsbuild

This PR does not wire up any lockfiles, formatters, linters, etc, yet. But you can test that pants works by running ./pants --version.

./pants --version creates a virtualenv for running pants and then shows the version

$ ./pants --version
Bootstrapping Pants using /opt/local/bin/python3.9
Installing pantsbuild.pants==2.13.0rc2 into a virtual environment at /.../.cache/pants/setup/.../2.13.0rc2_py39
New virtual environment successfully created at /.../.cache/pants/setup/.../2.13.0rc2_py39.
17:01:04.02 [INFO] Initializing scheduler...
17:01:04.08 [INFO] Scheduler initialized.
# ... some warnings that we can ignore for now
2.13.0rc2

Note that if you do it again, it returns immediately because the virtualenv does not have to be recreated:

$ ./pants --version
2.13.0rc2

We can also use ./pants count-loc :: without any other config. The count-loc goal counts the lines of code that pants sees in the repo.

$ ./pants count-loc ::
───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
Python                    1304    180480    20902     25682   133896       6453
YAML                       870     20484      693       294    19497          0
Plain Text                  66       972       11         0      961          0
JSON                        35     21364        1         0    21363          0
Shell                       34      1202      191       264      747        110
Markdown                    31       623      159         0      464          0
Autoconf                    18       190        0        54      136          0
gitignore                   13       176       21        45      110          0
Makefile                     9      1862      264       196     1402        136
BASH                         8      1641      222       182     1237        150
ReStructuredText             4      4791     1119         0     3672          0
Jinja                        3      5655       72         1     5582         15
License                      3       605       98         0      507          0
Powershell                   2        32        3         0       29          0
TOML                         2        20        0         0       20          0
HTML                         1         3        0         0        3          0
JavaScript                   1        19        3         6       10          0
───────────────────────────────────────────────────────────────────────────────
Total                     2404    240119    23759     26724   189636       6864
───────────────────────────────────────────────────────────────────────────────
Estimated Cost to Develop (organic) $6,659,048
Estimated Schedule Effort (organic) 28.269015 months
Estimated People Required (organic) 20.927502
───────────────────────────────────────────────────────────────────────────────
Processed 30497469 bytes, 30.497 megabytes (SI)
───────────────────────────────────────────────────────────────────────────────

[cognifloyd]

This file is copied from https://static.pantsbuild.org/setup/pants
I did not change it at all. Any future updates (which should be infrequent) will just replace this file with a downloaded update. We should not edit this file.

[winem]

I guess you don't see there any risk by these "unattended" updates? I guess it's actually a good way to reduce technical debt and "force" us to stay up-to-date.

[cognifloyd]

Updates to pants are not unattended. The version used is pinned in pants.toml:

st2/pants.toml

Line 2 in 92f08eb

pants_version = "2.13.0rc2"

So, we need a PR + reviews whenever we update the version of pants we're using.

If we need to update this pants launch script, then we also have to manually download + commit the update.

[winem]

Ah ok, that sounds good, thanks!

[cognifloyd]

I'm not sure if this should be in the 3.8.0 milestone or not.

[Eric-Arellano]

Hey, Pants maintainer here. Sounds like y'all have talked to @benjyw already - but I want to echo that we're happy to help y'all with adoption and eager for any feedback :)