Create system to define includes / excludes in artifact archives

[deviantintegral]

Once a build is complete, we need some way to indicate what files should be included in an archive or not. This could be something with git and gitignores, or another file format that's compatible with some sort of archiver.

[deviantintegral]

Something I realized; using git as we discussed @justafish doesn't work for compiled projects where the artifacts are never in source control. That may mean that a better / more robust solution that already exists may come from the C, Go, or other ecosystems.

[deviantintegral]

I've been looking into this. It looks like there may be some limitations in using the gitignore format:

1. Both tar and zip can take a text file with a list of paths to include in an archive. Of course, this could be used with git add ... too. I think that's a reasonable base to target - no matter what we use, we want to generate a list of files to include in the archive.
2. Git doesn't have a command that works absent a git index. There's the git check-index command, but it explicitly ignores files tracked in the index. That means you can't exclude a file that you want in the repository.
3. Git doesn't have a command or library separated out from git itself. We could possibly write a tool to do so. The closest code is at https://github.com/git/git/blob/master/builtin/check-ignore.c#L71-L125
4. tar supports "tcsh globs" for file matching. I'm going to investigate to see if task's sh has support for similar, and if it supports excluding files.

[deviantintegral]

I ran into a significant issue with Task; unfortunately, it's shell doesn't support extended globs. It also doesn't support them in sources. Both uses lead to a panic, which can be replicated with the underlying library:

$ gosh -c 'echo !(c*)'
panic: unhandled word part: *syntax.ExtGlob

goroutine 1 [running]:
mvdan.cc/sh/v3/expand.(*Config).wordFields(0xc0000be140, 0xc0000ce010, 0x1, 0x1, 0xc00001c110, 0x4, 0x0, 0x0, 0x0)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/expand/expand.go:617 +0x1290
mvdan.cc/sh/v3/expand.Fields(0xc0000be140, 0xc00006a430, 0x2, 0x2, 0xc00006a430, 0x1015372, 0xc00006a430, 0xc0000c6ab0, 0x10)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/expand/expand.go:392 +0x158
mvdan.cc/sh/v3/interp.(*Runner).fields(0xc0000c4000, 0xc00006a430, 0x2, 0x2, 0x2, 0x2, 0x0)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/interp/runner.go:150 +0x5a
mvdan.cc/sh/v3/interp.(*Runner).cmd(0xc0000c4000, 0x11bf218, 0xc00001c0c0, 0x11beb90, 0xc0000c6a80)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/interp/runner.go:317 +0x118a
mvdan.cc/sh/v3/interp.(*Runner).stmtSync(0xc0000c4000, 0x11bf218, 0xc00001c0c0, 0xc0000c8000)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/interp/runner.go:268 +0x412
mvdan.cc/sh/v3/interp.(*Runner).stmt(0xc0000c4000, 0x11bf218, 0xc00001c0c0, 0xc0000c8000)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/interp/runner.go:249 +0x1e7
mvdan.cc/sh/v3/interp.(*Runner).stmts(0xc0000c4000, 0x11bf218, 0xc00001c0c0, 0xc0000d2000, 0x1, 0x4)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/interp/runner.go:638 +0x57
mvdan.cc/sh/v3/interp.(*Runner).Run(0xc0000c4000, 0x11bf218, 0xc00001c0c0, 0x11bd7f8, 0xc000080100, 0xc000080100, 0x0)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/interp/api.go:502 +0x36b
main.run(0xc0000c4000, 0x11bd2b0, 0xc000074080, 0x0, 0x0, 0x0, 0xc000012420)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/cmd/gosh/main.go:64 +0xed
main.runAll(0xc000082180, 0xc0000121c0)
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/cmd/gosh/main.go:41 +0x349
main.main()
	/Users/andrew/workspace/go/pkg/mod/mvdan.cc/sh/[email protected]/cmd/gosh/main.go:24 +0x8a

• Extended globbing support go-task/task#225
• Add support for exclude rules for sources go-task/task#309

I played around with... using /bin/sh as a workaround, but that also leads to multiple files on a single line, while tar and zip expect one file per line. There's no easy way to deal with spaces and \0 characters either.

In the end, I think I was hoping to have something similar to Python's Manifest.in / Manifest files, in glob or other form: https://setuptools.readthedocs.io/en/latest/deprecated/distutils/sourcedist.html

I search the setuptools code, thinking if manifest generation was cleanly separated from setuptools, perhaps we just use that. Unfortunately, it's not: https://github.com/pypa/setuptools/blob/a4dbe3457d89cf67ee3aa571fdb149e6eb544e88/setuptools/_distutils/command/sdist.py#L324-L349

But wait... we're a PHP library. Perhaps that's how we get around this? Enter https://github.com/webmozarts/glob. We should be able to require it, and then write a very simple PHP CLI tool to generate the list of files to archive. I'll either work on this tomorrow, or we can discuss first.

[justafish]

@deviantintegral I've put up another approach here which creates a snapshot of the folder after processing ignored files and folders #16 - I'm thinking we can then push that up to git or tar the whole thing up depending on the circumstance.
.xignore seems like quite a common format for these things - keeping to something familiar like that is quite appealing to me 🤓