Design of the packaging system – packaging / library / namespace controls

[strangemonad]

These are extra-linguistic concepts and have no bearing on the semantics of definitions. Definitions can be moved from one namesspace / library / package to another transparently (the only changes needed are renames to the respective qualified identifiers and dependencies).

That said, software in the very large requires the help of aditional structure and tooling support.

Guiding principle

Decoupling Package systems are the interface of software development in the very large. This really means they are the interface for different groups of people with different goals – changing different parts at different speeds. Software in the small can be done by a single person (or a small group of people at the cost of mental synchronization). As you scale up, to a larger repo with multiple projects (and hence multiple groups of people) or to external dependencies, the cost of universal agreement increases exponentially. A packaging system, at its core, decouples key work practices (release schedules, code organization, publishing locations).

Goals

• namespaces Some form of explicit namespace control. Symbols defined at the top level are qualified by the namespace (e.g. what Java calls package NOT what SML calls Structure/Signature. SML has a single anonymous package that all modules live in and structures are overused for namespace control). Namespaces are logical (can be separated from actual organization of resources). The system gives an illusion of a single logical namespace tree.
• singular artifacts Most modern build systems (that aim to be performant and reproducible) have taken the stance that a build target or goal yeilds a single artifact (single file or single directory where the equality is checked point-wise)
• libs Specify the unit of separate compilation. Compilation units depend on a certain number of sources internal to it. They can have dependencies (but only in the form of other compilation units / libs). This is the natural link between the module system and the pacakging system (ie: what is known about the type information of the dependency). Libs are a single artifact (in the sense of buck / bazel or llbuild build systems).
• packages internally, organize sources and other resources (e.g. build configs, finary resources etc). Externally, packages provide a unit of publishing something to the outside world (e.g. in github repos or some package manager)

Additional goals

• semantically transparent Packaging and namespaces are just a fancy control over source code organization and naming. They have no semantic effects.
• generality Define things up-to the module language but the design of the package system should accommodate any choice of module language and core language.
• parsimonious The naming conventions for packages, libs and namespaces should all play well together. It’s annoying to search for a package (usually with some cute name) and not know what libraries it might expose or depend on a library and not know what namespaces symbols are defined in.
• flexiblity That said, the world changes and you don’t want to break old software. Even though the various names might agree by default, you might want a mirror to publish your sources (e.g. maven mirrors or http download vs github or vendored sources in a tree). It’s important that these various names can be decoupled without breaking (or with minimal breakage). For example, a personal project becomes popular and moves to a different github org. It would be nice to roll out this change in phases. 1) just update the repo but none of the namespace names in the qualified identifiers (note this is NOT a semantic change). 2) Tag a version in the new repo and update the library / snaspace names to reflect the new repo (again, this is not a semantic change).
• versioning It should be possible to pick a point in time to decouple . Sem ver is kind of a sham so I don’t want to bake that in. Fundamentaly, a library exposes it’s type signature. However, it needs to be possible to 1) pick a “point-in-time” whether it’s a known stable comit or explicitely published version so that you can upgrade separately from the upstream release cycle. 2) in larger projects, deal with transitive diamonds with mismatched version requirements. Bonus points if this can play nicely with lib naming and namespacing.
• mono-repro / multi-repo Simple enough, the repo != the package. Larger company’s often choose a large monolithic repository because it’s convenient to make a single atomic commit with changes across multiple packages / libs.
• interoperability Don’t do anything to make it hard to interop with various build systems.

Non-goals

• visibility control Related to the transparency goal, visibility is a semantic concern and falls to the module system.
• build system integration Actual integration with various build systems is not the responsibility of the packaging system (though it should be designed in a way so that it’s not too hard to build such integrations).