yarn install fails with `ENOENT: no such file or directory` occasionally

9 thoughts on “yarn install fails with `ENOENT: no such file or directory` occasionally”

1. @victornoel, can you reproduce the bug with yarn install --network-concurrency 1?

2. @bestander with --network-concurrency 1 the bug does not appear (while without it, it appears every time).
   But what is the default value of this parameter? Whichever value I choose for it (1, 2, 4, 8), it works, while if I don’t put it at all, it fails…

3. The default is 15, I can repro the issue with concurrency 15 with a clean checkout of https://gitlab.com/linagora/petals-cockpit.git#075bac4c54fee466568c000c7ffe8025f593e212.

4. Some results.

   TL;DR I am out of ideas how to properly fix it for good, this needs some deeper Node.js knowledge.

   1. Network can be eliminated from possible issues.
      I’ve setup offline mirror for the .tgz files in yarn.lock and can reproduce the issue with packages installed from disk.

   The issue is in unzip/untar stream in tarball-fetcher code.

   2. I tried a different library that extracts tar – https://github.com/mafintosh/tar-fs vs current https://github.com/npm/node-tar/. They both fail the same way.
      Going a bit deeper – exceptions are happening in node when doing multiple mkdirp operations

   Error: ENOENT: no such file or directory, chmod '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts'
     errno: -2,
     code: 'ENOENT',
     syscall: 'chmod',
     path: '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts' }
   

   I think core-4.0.0 and typescript-2.2.1 fail because they have quite a few files and deep folder structures, and they fail to install while making many concurrent mkdir/copy operations.

   Every time there is a different syscall that fails: chmod, rmdir, mkdir, lstat, utime.

   And it is not something obvious in the libraries’ code.

   3. Fails the same on Node 4, 6 and 7.

   4. I could not reproduce the error with concurrency set to 8, so I’ll send a PR to reduce default network concurrency.

   ---

   5. I was wondering how concurrency affects installation speed.

   5.1. Using offline-mirror (no download), on my MBPro 13″, clean cache and using node-tar to unzip files.
   Concurrency 12 – fails
   Concurrency 8 – 18 seconds
   Concurrency 4 – 18 seconds
   Concurrency 2 – 21 seconds

   5.2. Using offline-mirror (no download), on my MBPro 13″, clean cache and using tar-fs to unzip files.
   Concurrency 12 – 15 seconds
   Concurrency 8 – 15 seconds
   Concurrency 4 – 17 seconds
   Concurrency 2 – 18 seconds

   5.3. Downloading packages from internet, on my MBPro 13″, clean cache and using tar-fs to unzip files.
   Concurrency 12 – failed once
   Concurrency 8 – 21 seconds
   Concurrency 4 – 23 seconds
   Concurrency 2 – 34 seconds

   Looks like setting concurrency to 8 is safe enough, also it makes sense to switch the tar library.
   I’ll follow up with a PR.

5. while ! yarn install; do echo --- ; done

   Really sad that the only fix is brute force … unbelievable that no one fixed this yet.

   cc @arcanis

6. Can confirm still a problem on 1.22.4. Originally, it was mocha, and after making sure all packages were using the same version, I’m now getting errors generated from camelcase which I don’t even use in my project — apparently it’s from yargs, possibly from Lerna.

   error An unexpected error occurred: “ENOENT: no such file or directory, lstat ‘/code/project/src/packages/private-package/node_modules/camelcase'”.

7. May I ask if there is a solution in sight? We keep deleting 20 node_modules and a yarn.lock to fix it.

8. Deleting yarn.lock then yarn install worked for me as well

9. I fixed it by temporarily changing cache directory, that I delete just afterwards.

   For me it’s Docker build:

   RUN yarn install --check-files --cache-folder .ycache && rm -rf .ycache
   

Comments are closed.