Provide a way to link packages to node_modules avoiding symlinks

Is your feature request related to a problem? Please describe.

pnpm is a mature package manager used by a significant amount of companies and it works well with almost all stacks in the ecosystem. However, because the node_modules structure created by pnpm heavily relies on symlinks, pnpm is unusable in environments that do not support symlinks:

  • pnpm doesn’t work with electron apps
  • an app that uses pnpm cannot be deployed to lambda

These issues of symlinks are probably the main reason neither npm nor Yarn considered to create a properly nested node_modules using symlinks.

Describe the solution you’d like

Some way to link a package from a different location using a file, not a symlink. For instance, node_modules/foo can be a text file that contains a relative path to a different location. When require('foo') is searching for the location of foo, the path from the text file is read and used to track down the reallocation of foo (in case of pnpm it will be something like .pnpm/foo@1.0.0/node_modules/foo)

Describe alternatives you’ve considered

  1. I considered Yarn Plug’n’Play. pnpm supports Yarn PnP, there’s an option that makes pnpm create and use a pnp generated via one of Yarn’s packages. I like the concept of Plug’n’Play but it still has many issues. Even more issues than symlinks.

    I think Plug’n’Play would be a good solution if Node.js would natively support providing some import maps. pnpm would then generate such import maps and node would read the files directly from pnpm’s content-addressable store.

  2. I also considered creating dummy redirect files. Instead of creating a symlink, we can create something like node_modules/foo/index.js:

    module.exports = require('.pnpm/foo@1.0.0/node_modules/foo`)

    However, this would not cover cases like require('foo/lib/something')

  3. I also considered replacing the contents of all require/import statements during installation. So if a package is installed, which requires foo, pnpm replaces require('foo') with the reallocation of foo (require('../../foo@1.0.0/node_modules/foo')) in all files.
    This solution would probably work in most cases but it would require a lot of new logic to be added to pnpm.
    Also, it would only work in copy mode, and that would destroy all the great disk saving benefits of pnpm.

cc @shellscape @vjpr @ExE-Boss

2 thoughts on “Provide a way to link packages to node_modules avoiding symlinks

  1. I personally doubt avoiding symlinks would be enough (assuming it could even be done).

    The node_modules algorithm is deeply flawed in its very design, and symlinks (or not-symlinks, as suggested here) only try to workaround the problem – in that you can often make it work-ish, but you see light on the sides. Peer dependencies and workspaces, in particular, are extremely hard (and impossible in their current state) to make work together while being true to both of their contracts, symlinks or no symlinks. Similarly, good hoisting is unbelievably difficult to compute, for no good reason. Add on top of that that the algorithmic complexity is suboptimal, the I/O overhead, the leaky boundaries, etc etc…

    In my opinion, the best option at this stage would be to accept that node_modules will keep being problematic, and open the project to the possibility for having first-class support for an additional, modern, builtin module loading algorithm (PnP being of course the one I would suggest). Loaders are a good step in this direction, but as long as they aren’t first-class they will face unwarranted resistance that will hurt their adoption and therefore make the user experience subpar.

    Fwiw I think we (at Yarn) would be happy to champion builtin PnP support, but we’d need a clear signal that this would be seriously considered: a co-champion would also have to manifest themselves amongst the Node core contributors.

  2. Am I wrong in thinking that implementing import maps would solve the problem? Using import maps, each package manager would create an import map when changing the package.json (install, uninstall, update…), and fully describe to node which modules to load when using bare specifiers. In that way, the module resolution is entirely left to the package manager.

    This is a good way to distribute responsibility: the package manager implement the rules of what to load, and Node implements the loading itself.