require.context / dynamic filesystem dependencies in Metro

[motiz88]

Hoisting this from #metro on Discord for visibility - I've sketched out an idea for how Metro might support something like Webpack's require.context. This is in response to a question from @EvanBacon about trying to implement require.context in a transformer, and running into cache invalidation issues as a result.

https://discord.com/channels/514829729862516747/514832110595604510/976020483478786100

A proper design might be something like adding a "dynamic dependencies" pass after the transformer. The transformer would record the fact that there's a glob regex dependency, but not expand the regex, so its output would remain cacheable under the existing assumptions. Then in DeltaBundler, after obtaining a transformer result for a module but before recursing into its dependencies, we would call a method on DependencyGraph to expand the regex based on the current state of the filesystem (already in memory via metro-file-map) and store the results in the module object. Finally in the serializer we would inject the dynamic dependencies cheaply, similarly to how we inject the resolved static dependencies.

With the unexpanded regex dependencies thus recorded, we could reasonably re-evaluate filesystem changes against the set of regexes currently "active" in the graph, without invalidating the world on every change. (Handwaving but this seems doable)

This is my back-of-a-napkin design anyway 😋 The idea is to work with the grain of what makes Metro fast, so we can keep it fast while adding this capability. Happy to work with you (or other interested contributors) on fleshing this out.

[EvanBacon]

The transformer would record the fact that there's a glob regex dependency, but not expand the regex, so its output would remain cacheable under the existing assumptions.

Just guessing, I imagine this part could be achieved with unstable_collectDependencies. Matching the regex and require.context then pushing some sort of template require (similar to asyncRequireModulePathStringLiteral).

we would call a method on DependencyGraph to expand the regex based on the current state of the filesystem (already in memory via metro-file-map) and store the results in the module object.

Also guessing, would this be something like DependencyGraph's resolveDependency method, but accepting a RegExp and returning an array of strings?

With the unexpanded regex dependencies thus recorded, we could reasonably re-evaluate filesystem changes against the set of regexes currently "active" in the graph, without invalidating the world on every change. (Handwaving but this seems doable)

If I add a new file, how would the serializer know to update. Is there any existing mechanism for updating the serializer based on a file change? If not, where does this need to go?

Finally in the serializer we would inject the dynamic dependencies cheaply, similarly to how we inject the resolved static dependencies.

Not sure what this means, my guess would be this.

[motiz88]

Re collectDependencies and resolveDependency, yeah, this is pretty much spot on. (We wouldn't use the unstable_collectDependencies extension point but just add to Metro's built-in collectDependencies implementation.)

---

If I add a new file, how would the serializer know to update

The serializer is completely stateless and purely concatenates some strings that (by that point) DeltaBundler has already computed and stored in memory. What we need is to have DeltaBundler respond to the change by processing the modified files and marking their dependents (where the require.context calls are) as also modified, if the set of expanded dependencies for a module has changed.

Specifically, there are a handful of places where at the moment we filter filesystem changes by whether they are in the current known graph. These will need to be amended to account for regex dependencies.

https://github.com/facebook/metro/blob/94e79e0778c8b41224b54c24e48b01e9c53e426a/packages/metro/src/DeltaBundler/DeltaCalculator.js#L225-L228

https://github.com/facebook/metro/blob/94e79e0778c8b41224b54c24e48b01e9c53e426a/packages/metro/src/DeltaBundler/graphOperations.js#L112-L114

---

Re: injecting dependencies in the serializer, you're close with the wrapModule call site, but it would need to happen inside the function.

wrapModule injects parameters that will be passed to the define function at runtime. It would need to work in coordination with the transformed AST resulting from our collectDependencies changes, and with changes we'll need to make in the define function itself. Eventually we want the arrays of expanded module information to be available in module scope so that our transformed require.context calls can access it.

---

There are still lots of details to figure out but I hope this helps! Among other things we should probably have a discussion about the API - i.e. whether require.context make sense to import wholesale from Webpack, or whether there's some more constrained API shape that would serve the use case and not lock us into Webpack compatibility specifically.

All that said: I think it's also worth looking into Webpack's own implementation, as I understand each require.context call creates a virtual "context module". That might be a useful concept to copy as well, instead of storing expanded regex dependencies directly on ordinary modules. AFAICT this would end up being pretty much equivalent in capabilities to the above approach, but possibly more elegant and less invasive.

[EvanBacon]

Ok I opened a rough draft of the first part facebook/metro#821 -- I tried to implement the other parts, but the codebase is a bit too ambiguous to interact with at the moment (ex: DependencyGraph doesn't have unit tests -> not sure how things operate even before I modify them).

[motiz88]

I want to share a quick update on the current progress, and a longer spec / brain dump regarding what's next. tl;dr we are taking the "virtual module" approach I briefly mentioned above.

The transformer would record the fact that there's a glob regex dependency, but not expand the regex

@EvanBacon is implementing this in facebook/metro#821 as a change to collectDependencies, which is part of the transformer. In the PR, a require.context call in the source code (with static arguments) is transformed into a plain require() of a virtual module, with the expectation that the bundler will later generate/update it (with dependencies of its own) based on the arguments collected from the require.context call. This extra information is stored as contextParams in the dependency object corresponding to the require call.

The current implementation is gated by the unstable_allowRequireContext transformer config option. When the feature is stable, we will rename the option to allowRequireContext and document it.

Then in DeltaBundler, after obtaining a transformer result for a module but before recursing into its dependencies, we would call a method on DependencyGraph to expand the regex based on the current state of the filesystem (already in memory via metro-file-map)

@EvanBacon opened facebook/metro#822 to add the regex expansion functionality to DependencyGraph.

---

The big piece left is to teach DeltaBundler to first generate, then update the necessary context modules ( = virtual modules based on contextParams) at the right times. Here I'll describe what changes I currently think are needed, along with code pointers.

NOTE: Code pointers below are based on the branch in facebook/metro#820, not main.

Generating context modules for the first time

Building the graph starts with initialTraverseDependencies. We call processModule on each entry point and recurse from there.

1. When we first encounter a dependency that has contextParams, it will be here in resolveDependencies. Instead of passing it to the module resolver, we should generate a unique fake "path" for it (based on e.g. a hash of contextParams) and make sure the contextParams are recorded in the resolved dependency data structure. We'll probably also want to convert the relative directory path to an absolute path at this point.
2. Inside resolveDependencies, we need to use a compound key instead of just the relative path to dedupe dependencies. This is a lot like the key we derive inside collectDependencies (per feat: add support for capturing require.context in dependency collector facebook/metro#821). Here it's a little subtle, as getting this wrong could mess up our reference counting within the graph :( The invariant is that a dependency with a unique key (meaning unique among its siblings in a given dependencies array) must resolve to a unique absolute path. (Because of this subtlety I'm leaning towards duplicating the key extraction logic inside graphOperations, but we can look at this detail later.)
3. When we traverse a resolved dependency, if it has contextParams, we need to pass those contextParams into processModule so we can use them to generate the module's contents.
4. For a context module, we need almost all of processModule to run as-is, except for the part that calls options.transform to obtain the module's contents and dependencies based on its path. The analogue for context modules should be an options.transformContext function that takes contextParams and returns a transformer result (same as options.transform's return type).
5. The expectation is that transformContext will return a generated JavaScript module that sets module.exports to whatever is needed in order to fulfil the require.context runtime API. In particular, transformContext is responsible for expanding the given contextParams into a list of files (e.g. using feat: add require.context support facebook/metro#822 matchFilesWithContext), generating the require calls for each one, and also returning the appropriate dependencies array reflecting the expanded dependencies (as if collectDependencies was called on the generated code).
6. Caching of transformContext results across multiple graphs (analogous to how the existing transformer cache fits into options.transform) is an external concern from an individual graph / DeltaCalculator's perspective. We should revisit it before productionising this feature, as shipping it without this level of caching can lead to perf problems, but I don't have a clear design for this just yet. Note that within a single graph context modules will be cached, since we will avoid invalidating them unnecessarily (see below).

Updating context modules on relevant file changes

When a file (regular module) that is part of an existing dependency graph changes on disk, we start traversing from that module, which leads to processModule again. Through a process of diffing the previous and current dependencies recursively, processModule ends up being called on exactly (1) all files that have changed, (2) all dependencies that have been added to changed/added files, and nothing else.

NOTE: A design goal here is to avoid both false negatives (not invalidating on a relevant change) and false positives (invalidating on an irrelevant change).

When a context module is added to the graph, it will be because some other module has changed and gone through processModule (e.g. a require.context call was added to it). Likewise, when a context module is deleted from the graph, it will be because a require.context call has been deleted in some other module. Therefore, the changes to processModule described above, that handle the initial traversal, will Just Work for subsequent traversals.

The trick here is to detect when a context module has "changed". Remember that a context module has a fake path, so no filesystem event will directly mention it. Instead, a context module "changes" when the set of expanded files matching its contextParams changes: that is, when a file matching the context is added or deleted.

This all boils down to the following two rules:

1. When a file is added, we need to test it against the contextParams of all context modules currently existing in the graph, mark those as modified, then continue with the normal algorithm for handling modified files.
2. When a file is deleted, if any of its inverse dependencies are context modules, we need to mark those as modified, then continue with the normal algorithm for handling modified files. (If we don't have the inverse dependencies for a deleted file, it means it wasn't in the graph to begin with, which means there are no context modules referencing it, which means no context modules need to be regenerated.)

Code pointers:

1. In DeltaCalculator.prototype. _getChangedDependencies, this code invalidates dependents of deleted files and bails out of retraversing the graph if none of the paths in modifiedFiles are paths of existing modules in the graph. It will need to be updated to mark context modules as modified when files matching their contextParams are added/deleted (not modified), thus implementing the two rules described above.
2. Accordingly, this code in _handleFileChange will probably need to start distinguishing between added and modified files.
3. To implement rule 1 (matching added files against all context modules in the graph), we'll want fast access within the graph to just the set of all context modules and their contextParams. We will want to maintain this as part of graphOperations (e.g. add to this set here and delete from it here). This set is of course still theoretically O(graph) in size, but in practice there should be very few context modules compared to ordinary modules in the graph, so this will be a perf win.
4. Ideally, we'd implement (3) as a piece of private state within the graph. This probably requires refactoring (1) into a new function in graphOperations.js, where the private graph state is accessible.

Drawbacks of this design

If transformContext is responsible for expanding contextParams into a set of paths, we miss out on the ability to do this incrementally: we'll scan the entire file map on every addition/deletion of a file matching some contextParams, which is inefficient. A better design would maintain the last known set of expanded paths for each context directly within the graph. That way we could add/delete just the paths we know have been added/deleted, and regenerate the module from that. Avoiding O(project) operations for incremental file changes is key to Metro's performance, so this will have to be part of the stable version of this feature.

[motiz88]

[NOTE: Longer-term discussion]

I should also add that "virtual modules" or "loaders" aren't currently a thing in Metro, and supporting them more generally isn't a goal of this design, but they're a useful concept that I'm keeping in the back of my mind here.

Rough definition: a virtual module = a module in the bundle that isn't sourced from disk, whose existence can be "discovered" during the build, and whose contents can change from build to build.

My tentative plan is that once we have a couple of other use cases for virtual modules, we'll go back and design a proper API for them, and refactor the handcrafted support for context modules (as described above) to instead use that API. Right now it seems like the hardest things to generalise / build a straightforward API around will be:

1. Propagating changes to the right places in the graph - esp. distinguishing the effect of file existence changes, which context modules care about exclusively, from file content changes, which general virtual modules might care about.
2. Incremental updates - maintaining enough state in each virtual module to handle a filesystem event without necessarily regenerating the module from scratch.
3. Type safety: if we have completely pluggable "loaders" that can generate virtual modules, we need them to fully encapsulate everything from collectDependencies through the resolver and into the bundler (or even serializer) so that Metro's core can treat their internal data as opaque.