Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] npx does not fetch latest possible semvar match #7838

Open
2 tasks done
jeff-an opened this issue Oct 15, 2024 · 29 comments
Open
2 tasks done

[BUG] npx does not fetch latest possible semvar match #7838

jeff-an opened this issue Oct 15, 2024 · 29 comments
Labels
Bug thing that needs fixing

Comments

@jeff-an
Copy link

jeff-an commented Oct 15, 2024

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

When using the syntax npx <package>@<semvar> <command>, npx is always using a local cached version instead of fetching the latest available version that falls within the semvar from the npm registry and prompting for an upgrade.

Running npm cache clean --force does not seem to help.

The issue only seems to be reproducible on some machines. One user even reported that with [email protected] installed locally, npx momentic^1 was still invoking 1.0.11 instead of the newer version.

Expected Behavior

I expect npx to issue a prompt like the one below:

Need to install the following packages:
[email protected]
Ok to proceed? (y)

rather than proceeding with the locally cached version of [email protected], for example.

Steps To Reproduce

  1. Run npx [email protected] init and accept the install prompt. Ignore the output of the program (the program in this case doesn't matter and can be substituted with any other).
  2. Run npx momentic@^1 init. This should be expected to prompt to install 1.0.13 or whatever the latest version is. However, it does not and instead prints the same output as step 1.

Screenshot of what I mean on the turbo repo (the latest turbo version is 2.1.3 at time of writing):
Screenshot 2024-10-15 at 3 56 24 PM

Environment

  • npm: 10.9.0
  • Node.js: v20.9.0
  • OS Name: Mac OS Sonoma 14.4
  • System Model Name: M3 Max MBP
  • npm config:
; "project" config from /Users/jeffan/code/momentic/.npmrc

auto-install-peers = true
public-hoist-pattern = ["*eslint-plugin*","*prisma*","*bull*"]

I confirmed that my npx path is fixed and set to:

which npx
/Users/<REDACTED>/.nvm/versions/node/v20.9.0/bin/npx
@jeff-an jeff-an added Bug thing that needs fixing Needs Triage needs review for next steps labels Oct 15, 2024
@milaninfy
Copy link
Contributor

milaninfy commented Oct 16, 2024

I am getting expected behaviour

~/workarea/rep/test $ npx -ddd [email protected] init
Need to install the following packages:
[email protected]
Ok to proceed? (y) y
~/workarea/rep/test $ npx momentic@^1 init
Need to install the following packages:
[email protected]
Ok to proceed? (y) 

@jeff-an
Copy link
Author

jeff-an commented Oct 16, 2024

Thanks for the responses folks! --no-cache and prefer-online both do not seem to help this case:
Screenshot 2024-10-16 at 4 18 39 PM

We know that it works on some people's machines but not others. How can we debug why? At this point we are thinking of just hitting npm's registry programmatically at startup to figure out what the latest version is.

@ljharb
Copy link
Contributor

ljharb commented Oct 16, 2024

you don't need to do that; do npx foo@latest and you'll get the latest no matter what's locally available.

@jeff-an
Copy link
Author

jeff-an commented Oct 17, 2024

We are aware of that, but we don't want to use @latest because it will automatically install versions that may be backwards incompatible with what the user is currently using.

Besides, it seems like a bug that this behavior is a) non-deterministic across machines and b) different from what is advertised in the official docs:

Package names with a specifier will only be considered a match if they have the exact same name and version as the local dependency.

@milaninfy
Copy link
Contributor

@jeff-an what's the output of npm -v and npm config ls -a

@jeff-an
Copy link
Author

jeff-an commented Oct 17, 2024

I put it in the environment section:

version: 10.9.0

npm config:

auto-install-peers = true
public-hoist-pattern = ["*eslint-plugin*","*prisma*","*bull*"]

@milaninfy
Copy link
Contributor

npx will first check in local project/workspaces from where you are running the command to see if matching range version is found, if not then check globally and then pull from registry. So if you are running npx command in a folder where this package is already installed or part of node_modules then it would use that if it's matching.

@jeff-an
Copy link
Author

jeff-an commented Oct 21, 2024

What constitutes a local project or workspace? We have not installed this package (momentic) anywhere - it is only invoked as a CLI. It never appears as an entry in any package.json in our working tree or above.

@milaninfy
Copy link
Contributor

Project with package.json and dependencies installed or this cli tool installed globally.
unless it's installed locally on project from where you are running the command or globally installed. it should get the correct version based on range or version specified.
However at my end it's not reproducible even with node 20.9.0 and npx 10.9.0. It does fetch correct values
Please provide verbose logs of these runs if possible.

My output

~/workarea/rep $ node -v                               
v20.9.0
~/workarea/rep $ npm -v
10.9.0
~/workarea/rep $ npx -v
10.9.0
~/workarea/rep $ npx [email protected] -V                    
Need to install the following packages:
[email protected]
Ok to proceed? (y) 

 ERROR  unexpected argument '-V' found

  tip: to pass '-V' as a value, use '-- -V'

Usage: turbo [OPTIONS] [COMMAND]

For more information, try '--help'.

~/workarea/rep $ npx turbo@^2 -V                       
Need to install the following packages:
[email protected]
Ok to proceed? (y) 

 ERROR  unexpected argument '-V' found

  tip: to pass '-V' as a value, use '-- -V'

Usage: turbo [OPTIONS] [COMMAND]

For more information, try '--help'.

~/workarea/rep $ npm config ls
; "project" config from /Users/milaninfy/workarea/rep/.npmrc

auto-install-peers = true
public-hoist-pattern = "[\"*eslint-plugin*\",\"*prisma*\",\"*bull*\"]"

~/workarea/rep $ 

@jeff-an
Copy link
Author

jeff-an commented Oct 30, 2024

What kind of debug logs can we provide? Unfortunately it does not appear npx has a --debug or --verbose mode that prints more information about how its resolving. A colleague of ours running on windows just encountered the problem again yesterday. Here's the information from his machine:
Screenshot 2024-10-30 at 11 40 56 AM

We confirmed that there is no package.json in the current directory where he was running the command. Will try to get npm list -g information as well.

@jeff-an
Copy link
Author

jeff-an commented Oct 30, 2024

Screenshot 2024-10-30 at 2 57 11 PM

npm list -g showing nothing installed globally

@milaninfy
Copy link
Contributor

milaninfy commented Oct 31, 2024

you can use command this way npx -ddd turbo@^2 -V to enable silly logs

@jeff-an
Copy link
Author

jeff-an commented Oct 31, 2024

Screenshot 2024-10-31 at 11 53 06 AM

Screenshot 2024-10-31 at 11 57 12 AM

Here's the output from my laptop and a repro of the bug

@jeff-an
Copy link
Author

jeff-an commented Oct 31, 2024

also repros in my tmp folder, where there is no package.json:

Screenshot 2024-10-31 at 11 57 40 AM

@jeff-an
Copy link
Author

jeff-an commented Nov 5, 2024

any other information I can provide here @milaninfy ? only thing that seems to definitively fix the issue for the next invocation is npx clear-npx-cache

@jeff-an
Copy link
Author

jeff-an commented Nov 11, 2024

friendly bump...

@jeff-an
Copy link
Author

jeff-an commented Nov 16, 2024

Some possibly related weird behavior. Here npx asks me if I want to install the same package version twice. And invoking npx [email protected] -V does not work in a folder that contains a package.json with the name momentic, but invoking npx momentic@alpha -V in that same folder does work.

@wraithgar
Copy link
Member

These are different entries in the npx cache. The npx cache is indexed by the entire package arg.

Within a given npx cache entry, if the spec (everything after the @) is a range it will look to see if a newer version exists. It will not consider other entries in its cache.

In order to get the behavior you want you need to give npx the same package arg each time. If you want latest, just give it the package name with no spec. If you want a version, use that version every time. You can also use a dist-tag.

@jeff-an
Copy link
Author

jeff-an commented Nov 26, 2024

Thanks for the response! A few questions:

  1. Using npm-package-arg, these two things seem to have the same "name". What field from the na result is used for the cache key?
> na("[email protected]")
Result {
  type: 'version',
  registry: true,
  where: undefined,
  raw: '[email protected]',
  name: 'momentic',
  escapedName: 'momentic',
  scope: undefined,
  rawSpec: '1.0.12',
  saveSpec: null,
  fetchSpec: '1.0.12',
  gitRange: undefined,
  gitCommittish: undefined,
  gitSubdir: undefined,
  hosted: undefined
}
> na("momentic@^1")
Result {
  type: 'range',
  registry: true,
  where: undefined,
  raw: 'momentic@^1',
  name: 'momentic',
  escapedName: 'momentic',
  scope: undefined,
  rawSpec: '^1',
  saveSpec: null,
  fetchSpec: '^1',
  gitRange: undefined,
  gitCommittish: undefined,
  gitSubdir: undefined,
  hosted: undefined
}
  1. If 1.0.12 and ^1 are different cache "scopes", how can a range ever be useful? Wouldn't it only be applied if the user does not have an existing matching installation?

  2. If the separate scopes is the intended behavior, why does the behavior I assume to be the expected behavior sometimes happen on my machine and others' machine as well (e.g. in @milaninfy 's examples)?

@wraithgar
Copy link
Member

The entire argument as given on the cli is used. If multiple packages are given (i.e. with the -p flag) they are all combined and used.

No cli commands are going to be able to clear the npx cache. It's isolated from npm's normal cache. There is also no way to inspect the npx cache. It's located at ~/.npm/_npx/ and its existence is probably the reason for perceived inconsistencies.

@jeff-an
Copy link
Author

jeff-an commented Nov 26, 2024

OK, but that doesn't seem to explain the original issue, where running npx package@^range did not look for the latest entry against npm's registry? Your comment would seem to imply that would happen:

Within a given npx cache entry, if the spec (everything after the @) is a range it **will look to see if a newer version exists**. It will not consider other entries in its cache.

@wraithgar
Copy link
Member

It does for me locally. We're back to the point where we can't reproduce this locally

$ npx [email protected] init
Need to install the following packages:
[email protected]
Ok to proceed? (y) 

Welcome to the Momentic project setup wizard!
$ npx momentic@^1 init
Need to install the following packages:
[email protected]
Ok to proceed? (y) n

@jeff-an
Copy link
Author

jeff-an commented Nov 27, 2024

Yes, we know it works on some people's machines and not on others'. But as you can see from earlier in this thread, I've provided logs and screenshots from multiple sources that say this doesn't work (I can still in fact reproduce on my laptop). So is there some additional information we can provide to narrow down the problem? Or if you want to point us to where this code is, we're happy to look ourselves as well.

Screenshot 2024-11-26 at 8 59 11 PM

@wraithgar
Copy link
Member

wraithgar commented Nov 27, 2024

Here is where npm determines whether or not it can find the package locally installed.

Here it looks in the global namespace.

Here is where npm looks in the npx cache.

@wraithgar wraithgar added Documentation documentation related issue and removed Bug thing that needs fixing Needs Triage needs review for next steps labels Nov 27, 2024
@wraithgar wraithgar added Bug thing that needs fixing and removed Cannot Reproduce Documentation documentation related issue labels Nov 27, 2024
@wraithgar
Copy link
Member

when using a range, npm is supposed to use whatever it finds, but ONLY if it's present in local or global. The npx cache inspection is supposed to look only at the resolved version.

This line is likely where the bug is. It's supposed to make npm NOT match by range or tag if we're checking the npx cache, and fall through to an identical version

@kyle-blair
Copy link

kyle-blair commented Dec 5, 2024

The behavior requested by @jeff-an is the opposite of the behavior I would expect. The npm package json dependencies documentation says

^version "Compatible with version"

and the npm node-semver package's section on caret ranges says

Allows changes that do not modify the left-most non-zero element in the [major, minor, patch] tuple.

The italicized emphasis is mine in both quotes. Based on the documentation, I would expect npx foo@^1 to use any version it finds where the major version is 1. The semver spec itself doesn't appear to discuss ranges at all.

Neither of the documentation linked above discusses what version to prefer when provided with a range. I think that might be the source of the relevant nuance here.

For example, if I have this in a package.json file:

  "dependencies": {
    "express": "^4"
  }

and express is not installed, the npm install command will favor the latest version that starts with 4. It installs 4.21.1 at time of writing. However, if I already have version 4.0.0 installed, running npm install will leave version 4.0.0. This makes sense because the existing version satisfies ^4. If I run npm install express@^4 it will then upgrade me to the latest 4.21.1 version. I think that is somewhat unexpected but it could be reasonably explained given the context: because I am explicitly asking npm to install an express version compatible with 4, it finds the latest 4.x version and installs it.

Given that npx exists to

run an arbitrary command from an npm package (either one installed locally, or fetched remotely)

and states

If any requested packages are not present in the local project dependencies, then they are installed

It goes on to say, however, that

Package names provided without a specifier will be matched with whatever version exists in the local project. Package names with a specifier will only be considered a match if they have the exact same name and version as the local dependency.

I think this part of the specification (documentation, in this case) violates semver. Assuming the "specifier" can be any valid semver, including a range, then I think this statement is at best incomplete/confusing, and at worst in direct conflict.

I would expect it to use any local package that matches the supplied semver specifier. I would not expect it to install a later version of the package. I could easily see how that behavior would be considered a feature though, since otherwise you may end up "stuck" on an old version as time goes on. But I also feel it is counterintuitive and confusing.

To add to the confusion, one of the top search results is a stack overflow answer that incorrectly states

^version “Compatible with version”, will automatically update you...

I don't have any context as to why npx behaves this way, so forgive my ignorance. But hey, sometimes lack of context can be an advantage.

If the behavior I propose is the expected behavior, that would potentially make the current behavior a bug.

I came across this issue in the context of npx attempting to install the latest version of a package even though both the package.json and npm-shrinkwrap.json specify an exact, earlier version of that dependency. I think that will be a separate, but potentially related, issue.

@jeff-an
Copy link
Author

jeff-an commented Dec 5, 2024

I would be totally OK with the behaviour described by @kyle-blair as well, as long as --no-cache truly busted the cache and always fetched the latest matching version based on the registry as opposed to using any locally installed packages. I am sure there are use cases for both "always get me the latest" and "I don't care which version as long as it satisfies the criteria".

But more importantly it seems like today neither behaviour happens reliably which is the worst of both worlds in my opinion.

@kyle-blair
Copy link

kyle-blair commented Dec 5, 2024

To summarize:

  • Decide/agree that npx should, by default, respect the full semantic versioning range specification when checking for local packages. That is, npx foo@^2 would accept a locally installed [email protected] (because it satisfies the specifier ^2) instead of trying to install [email protected]`.
  • Identify the root cause of the inconsistent behavior and fix it to always behave in the agreed upon manner.
  • Update the documentation to remove ambiguity/confusion around supplying a "specifier" (see excerpt below).

Package names provided without a specifier will be matched with whatever version exists in the local project. Package names with a specifier will only be considered a match if they have the exact same name and version as the local dependency.

@wraithgar
Copy link
Member

wraithgar commented Dec 6, 2024

The italicized emphasis is mine in both quotes. Based on the documentation, I would expect npx foo@^1 to use any version it finds where the major version is 1. The semver spec itself doesn't appear to discuss ranges at all.

Yes, this is the expected behavior everywhere else but the npx cache itself. The user does not have any direct access to that cache, so it needs to be able to update if a newer version is available.

A fix for this should not alter the existing behavior for packages found in the local package, or installed globally.

I still think a fix for this should fetch the highest matching semver version in the absence of that package anywhere except the npx cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing
Projects
None yet
Development

No branches or pull requests

6 participants
@wraithgar @ljharb @kyle-blair @jeff-an @milaninfy and others