Managing breaking changes in Cue dependencies

[shykes]

Hi all, as the launch of our Cue-based tool approaches, I am thinking more and more about API stability, not of Cue itself, but of user packages. What happens when a cue package changes in backwards-incompatible ways? How should downstream consumers of the package handle it? Are there patterns to follow by the published? Will the cue tooling help in some way?

Context is that we will be publishing a sizeable body of cue packages, and encouraging our users to import them. We are worried that when we inevitably break some of those packages, chaos will ensue.

[myitcv]

@shykes - thank you very much for moving the conversation here from Slack - we will definitely get better visibility on the question and any subsequent answers. I've copied my response from Slack below.

Just to make sure I'm talking about the right thing and understanding the context, I see there are three types of breaking change:

1. CUE breaking changes;
2. your breaking changes, i.e. core schema changes
3. breaking changes in the your package ecosystem, packages that are written against your core schema

All three will impact end-users.

Changes in category 1 will be largely be mitigated be automated fixes to CUE files via cue fmt and friends. Users of the API will need some way to provide similar functionality for their users (which might involve telling them to run cue fmt, that functionality could be provided by a different means). I'm assuming that you will provide some means via your binary to cue fmt equivalent?

Just out of interest, how do you plan to handle category 2 breaking changes?

If I understand correctly the statement "we’re embedding a ready-to-use cue.mod at runtime" then initially you're not going to support users importing third-party dependencies from outside that embedded cue.mod? That way, you cover off category 2 breaking changes if there are going to be any at least for now - because you ship an updated binary with the appropriate cue.mod distributed as part of it. The embedded cue.mod is never going to be broken with respect to the binary it is encapsulated in.

So the breaking changes we're talking about here are, for now, limited to category 3. And specifically how the end user could/should react to, say, a breaking change in a package definition. Assuming I'm holding this right, what exacerbates the situation in this case is that someone using a later version of your binary immediately gets their dependencies bumped by virtue of the embedded cue.mod - their only option is not bumping them is to continue to use the old binary (which might be less than ideal if it contains bug fixes for example). In a world where you are not bundling a cue.mod, then as @verdverm points out the end user declares their dependencies, constraints and some algorithm (e.g. MVS) determines the actual version of a module to use. That way using a later version of your binary does not immediately break a user; they are free to upgrade their dependencies as and when they want. But that is not a panacea, because when they do come to upgrade a dependency (and in some situations they might be forced to upgrade a dependency by virtue of adding another dependency that bumps the required minimal version via MVS) that obviously exposes them to any category 3 breaking changes the dependency author may have made.

AFAICS, there are two options:

a) package authors should not make breaking changes unless they do so via a new major version;
b) package authors provide some automated means by which breaking changes can be healed.

Option a is not available in the embedded cue.mod scenario. Nor is it necessarily that case if that package is still v0, i.e. pre v1. So breaking changes are going to happen (let's assume that post v1 everyone is "good" and plays by the rules... which doesn't really happen 100% of the time in practice, but it's a good aim).

That leaves option b.

At this stage I think it comes down to what kind of breaking change we are talking about, and how you want to handle it. cue fmt and friends that handle category 1 breaking changes use machinery to do automated refactoring, machinery that could well be used by other tools. But they are refactors that need to be specified by the author of the breaking change - it's not possible in the general case to automatically detect the refactor that needs to happen in response to a breaking change.

In the Go world, Russ Cox is working on a tool called rf (https://github.com/rsc/rf) that takes a script that specifies a series of refactors. Such a tool will allow package authors to specify scripts for their users that "heal" breaking changes (or execute any sort of change for that matter). A similar kind of concept could easily exist for CUE generally, with a well defined means by which package authors can specify a set of idempotent "fix" scripts as part of their package (at any given version) which users can run (via some tool) to fix their usage of those packages.

Which I think gets to the crux of the problem you're referring to. Right now, that tool does not exist for CUE, but the machinery that would sit behind it definitely does. Given that you are shipping an embedded cue.mod, you know and are in control of the breaking changes within packages contained in the cue.mod: is it conceivable therefore that your binary could support a "heal" command that would perform the fixes required to end-user code?

[jlongtine]

I think it's probably worth mentioning the work that @sdboyer has done for scuemata. I don't know that it's a solution for all of these problems, but it seems a useful addition + thing to consider as a part of a solution for some of them.

[myitcv]

I'd love to hear more about scuemata, and @sdboyer's thoughts regarding this kind of refactoring.

My feeling/hope is that we can write something like rf (cuerf?), which would mean there is then a generic language for describing CUE refactors regardless of the application of CUE. Ideally that tool would also be usable as a library, so cuerf could be embedded as part of other tools.

[sdboyer]

Sorry, missed this in my notifications!

The possible application of scuemata for this case is interesting. i actually watched your dockercon talk yesterday, @shykes, and (in between "this looks great" feels!) i had the same feeling of foreboding you expressed in OP about what happens when changes arise in the dagger packages you're exposing to the world.

Scuemata is currently designed around a clear division between schema, and concrete objects that pass through the schema. I haven't thought deeply about how it could be used in arbitrary CUE programs - i.e., dagger. That doesn't mean it's impossible, but lemme note a few salient points up front:

• There's a fundamentally different versioning paradigm with scuemata; details here. It's not semver, not SIV, but rather intended to be a lower-level, simpler system that ought to be be composable with those concepts. (though being fundamentally simpler, it could readily compose into such systems)
• Using scuemata in CUE programs more generally would almost certainly entail being able to draw a clear distinction between "data structs" and "behavior/function structs" in your program; i suspect that only the former would be feasibly scuematizable.
• When the invariant of "concrete object defined in JSON as input to scuemata migrations" no longer holds, one obstacle i can anticipate will exist is that handling defaults gracefully will be hard. That is, unifying a concrete object instance with an old version of a schema, then passing it through the migration process where a later schema version specifies a different default, would almost certainly incorrectly result in multiple, ambiguous defaults. Not immediately sure how to fix this without lang capabilities for doing something like excising defaults from a reference
• This would be fundamentally different than e.g. a cuerf-style approach, which if i'm understanding correctly relies on code rewriting on the depender's side. Scuemata is designed around data migration on the dependency's side. That said, my immediate gut feeling is that, if we do find scuemata's migrations at all useful outside of the strict schema/concrete object context, then any approach to cuerf rules would probably be able to use scuemata migrations as rules for rewriting dependers with minimal work, if not almost directly. AIUI, the premise is generally the same - convert an old form of data to a new form. Just, scuemata takes no stance on whether the program continues to work with the on-the-fly migrated data, or if the goal of the problem is to go back and rewrite the inputs with the migrated data. (handwave handwave)

That said it seems plausible that scuemata could be a good fit for creating a solid UX in the face of package changes for dagger. A few assumptions about how dagger's design works, though note that my assumptions not holding does not mean there isn't a possible path, here):

• dagger is written in Go
• It does a lot of loading and translating the user's .cue files to something buildkit (...?) recognizes
• The dagger CLI statically embeds most/all of the core object definitions it uses, similar to how we're doing it in Grafana now, and all your logic/code is written only against the latest version of those core object definitions

A big premise with scuemata is "translating data to canonical form at the process boundary" - with dagger, that might mean a flow like this:

1. You move to declaring dagger's key, core object types in scuemata. I'll use dagger.#Artifact as an example, as i saw it in your demo
2. You expose the latest schema in each lineage for dagger.#Artifact as a conveniently importable object (this could be where SIV-style import paths make sense)
3. User writes their cue files, importing that public-facing schema for dagger.#Artifact
4. User invokes dagger on those file(s)
5. dagger reads the specified files, doing whatever it already does to figure out which fields are supposed to be instances of dagger.#Artifact
6. For each such field, and before dagger does any other core work, the user's instance of dagger.#Artifact is mapped back onto the scuemata family (something like this func, though again that's really only meant for concrete resources), with four possibilities:
   • The instance does not validate wrt any version of the schema -> immediately error out
   • The instance is written against the same version of the schema as what's embedded in that version of the dagger binary -> continue unmodified
   • Instance version is newer than what's in dagger binary -> depends? you could error out, or just warn and continue
   • Instance version is older than what's in dagger binary -> automigrate the instance, which will also emit any "human-readable notes" specified in the scuemata. Depending on the note, you may then do any of
     • swallow it
     • emit it to the user as a warning and continue
     • emit it to the user as a terminal error

Personally, i'd be delighted if someone wanted to kick the tires on making scuemata more usable from CUE - basically, making item 2 in the above list easy. Right now, i'm relying on fugly hacks like this on the consumer side to create a reference to a schema that the scuemata guarantees promise will be durably backwards-compatible. It's no way to live. Codegen might be the answer.

[cueckoo]

This discussion has been migrated to cue-lang/cue#1008.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.