yarn install fails with `ENOENT: no such file or directory` occasionally

Running yarn install as part of a build step for a Docker image based on node:7 fails on Travis CI with ENOTEMPTY, EEXISTS errors. It always seems to error on the webdriverio package.

yarn install v0.19.1
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/webdriverio/-/webdriverio-4.6.2.tgz: ENOENT: no such file or directory, open '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol/timeouts.js'".

When Travis runs yarn install as part of the install phase it works just fine. The error only happens when building a Docker image.

Repo which reproduces this issue.

node:7
OS: Docker + Travis CI
yarn: 0.19.1
package.json
yarn.lock

I’ve tried installing yarn both with npm install -g and with apt and both methods cause the failure on Travis.

Weirdly enough, the image builds successfully on my local machine which runs Ubuntu 16.04.1 LTS with Docker version 1.13.0, build 49bf474.

Author: Fantashit

9 thoughts on “yarn install fails with `ENOENT: no such file or directory` occasionally

  1. @bestander with --network-concurrency 1 the bug does not appear (while without it, it appears every time).
    But what is the default value of this parameter? Whichever value I choose for it (1, 2, 4, 8), it works, while if I don’t put it at all, it fails…

  2. The default is 15, I can repro the issue with concurrency 15 with a clean checkout of https://gitlab.com/linagora/petals-cockpit.git#075bac4c54fee466568c000c7ffe8025f593e212.

  3. Some results.

    TL;DR I am out of ideas how to properly fix it for good, this needs some deeper Node.js knowledge.

    1. Network can be eliminated from possible issues.
      I’ve setup offline mirror for the .tgz files in yarn.lock and can reproduce the issue with packages installed from disk.

    The issue is in unzip/untar stream in tarball-fetcher code.

    1. I tried a different library that extracts tar – https://github.com/mafintosh/tar-fs vs current https://github.com/npm/node-tar/. They both fail the same way.
      Going a bit deeper – exceptions are happening in node when doing multiple mkdirp operations
    Error: ENOENT: no such file or directory, chmod '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts'
      errno: -2,
      code: 'ENOENT',
      syscall: 'chmod',
      path: '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts' }
    

    I think core-4.0.0 and typescript-2.2.1 fail because they have quite a few files and deep folder structures, and they fail to install while making many concurrent mkdir/copy operations.

    Every time there is a different syscall that fails: chmod, rmdir, mkdir, lstat, utime.

    And it is not something obvious in the libraries’ code.

    1. Fails the same on Node 4, 6 and 7.

    2. I could not reproduce the error with concurrency set to 8, so I’ll send a PR to reduce default network concurrency.


    1. I was wondering how concurrency affects installation speed.

    5.1. Using offline-mirror (no download), on my MBPro 13″, clean cache and using node-tar to unzip files.
    Concurrency 12 – fails
    Concurrency 8 – 18 seconds
    Concurrency 4 – 18 seconds
    Concurrency 2 – 21 seconds

    5.2. Using offline-mirror (no download), on my MBPro 13″, clean cache and using tar-fs to unzip files.
    Concurrency 12 – 15 seconds
    Concurrency 8 – 15 seconds
    Concurrency 4 – 17 seconds
    Concurrency 2 – 18 seconds

    5.3. Downloading packages from internet, on my MBPro 13″, clean cache and using tar-fs to unzip files.
    Concurrency 12 – failed once
    Concurrency 8 – 21 seconds
    Concurrency 4 – 23 seconds
    Concurrency 2 – 34 seconds

    Looks like setting concurrency to 8 is safe enough, also it makes sense to switch the tar library.
    I’ll follow up with a PR.

  4. Can confirm still a problem on 1.22.4. Originally, it was mocha, and after making sure all packages were using the same version, I’m now getting errors generated from camelcase which I don’t even use in my project — apparently it’s from yargs, possibly from Lerna.

    error An unexpected error occurred: “ENOENT: no such file or directory, lstat ‘/code/project/src/packages/private-package/node_modules/camelcase'”.

  5. I fixed it by temporarily changing cache directory, that I delete just afterwards.

    For me it’s Docker build:

    RUN yarn install --check-files --cache-folder .ycache && rm -rf .ycache
    

Comments are closed.