Workspaces: lock file per workspace

19 thoughts on “Workspaces: lock file per workspace”

1. @connectdotz it may not be needed for a library or published package but for building a docker container you’re going to want to deploy somewhere it definitely would be.

2. So for us, we don’t want to package the whole monorepo into the resulting docker container. We are using docker in production and those images should be as light as possible. Our mono repo is quite big and contains multiple microservices that share code between them using library packages (and some of the libraries are relevant for some of the microservices, but not all). So when we package a microservice, we want the image to contain the files of that microservice and any other dependencies as proper dependencies – downloaded from our private registry, and built for the arch of the docker image.

   So I think the main consideration here is to keep our docker images as light as possible, and packaging the whole monorepo doesn’t fit our needs. Also, when we run “yarn” inside the image of the microservice, we don’t want to have symlinks there, just a normal dependency.

   The solution here doesn’t have to be creating a yarn.lock file per workspace, it could also be a yarn command, that helps in the process of packaging a given workspace, generating a yarn.lock file for a workspace on demand, etc..

   Hope it helped clearify the use case.. 🍻

3. While I can se the use of individual lock files, they are not necessary. If you run docker from the root of the repo with the -f flag pointing to the individual files you’ll have the whole repo as context and can copy in the package.json and yarn.lock from the root.

   You only need the package.json for the packages you will build in the image and yarn will only install packages for those package.json files you have copied in even thou the yarn.lock includes much more.

   EDIT: With that said. it causes docker cache to not be used for package changes in any package even though it is not included in the build

4. #4206 is related/duplicate, and the use-case described there is exactly the problem we’re facing:

   Let’s say we have ten different packages. We want all of them to live in their own repository so that people can work on them independently if they want, but we also want to be able to link them together if we need to. To do this, we have a mega repository with a submodule for each package, and a package.json that references each of these submodule as a workspace.

5. Also struggling with this. We’ve got an Angular CLI project alongside our API so they’re in the same repository and trying to push the frontend to Heroku.

   We’re using a buildpack which tells Heroku to jump up to the frontend repository first: https://github.com/lstoll/heroku-buildpack-monorepo

   Problem is, there’s no yarn.lock inside that nohoist package so Heroku just installs with npm and we end up with all new packages rather than the locked ones

6. You can just use the global yarn.lock file with the individual packages. I’ve recently approached my Dockerfile like this:

   WORKDIR /app
   ENV NODE_ENV=production
   
   ADD yarn.lock /app/
   ADD package.json /app/
   
   # Only copy the packages that I need
   ADD packages/my-first-package /app/packages/my-first-package
   ADD packages/my-second-package /app/packages/my-second-package
   
   RUN cd /app && yarn install --frozen-lockfile
   

   This will install only dependencies that are actually in use by the two packages I copied and not by anyone else.

   I have a build process where first I’d like to create a release artifact from one package and then not have any of its dependencies installed. This is fairly easy with Docker’s multistage build

   1. Add only yarn.lock, package.json and the UI package to docker
   2. run yarn install --frozen-lockfile
   3. run the build process
   4. start a new stage and add yarn.lock, package.json and the necessary runtime packages/workspace folders
   5. Do a COPY --from=<stage> for the built artifact
   6. run yarn install --frozen-lockfile and expose a RUN command.

   And you’ll end up with a small container that only contains the dependencies specified in your yarn.lock file and needed in production.

7. I guess the core question is whether hoisting and a single yarn.lock file are strictly necessary for workspaces. I mean, is is what truly defines them or is it “just” the first feature they historically got?

   For example, in our use case, the best hypothetical behavior of workspaces would be:

   • Hoist node_modules at development time for efficiency.
   • Keep local yarn.lock files for build (we build specific packages in Docker, something that other people mentioned in this thread as well) and also so that packages can lock their specific versions. See also #6563.
   • Run scripts via yarn workspaces run <script> even if you don’t need (or must avoid) hoisting.

   Hoisting can be disabled with nohoist, run can be “disabled” by just not using the command but it’s not possible to “disable” the single yarn.lock file, and I’m not sure if it’s such a core feature that it cannot be disabled or if it just hasn’t been requested enough yet 🙂

8. I’m not exactly sure if separate lockfiles is the answer, but I have a similar problem. I have a monorepo set up with a CLI and a backend. The CLI requires a few packages that are platform-specific and only work on desktop machines with a particular setup. On the other hand I need to be able to build my api into a docker image, which is fundamentally impossible in the current implementation of workspaces.

9. Very similar use case than @samuela here! This one would be massively helpful!

10. My use-case might seem laughable compared to the other, “real” ones. But I have a monorepo for some utils – in ths case react hooks – inside packages/*.

    I have a second workspace next to packages/*, and that is local/*. This is actually on gitignore, and the idea is that developers in the company may do whatever they like in there, for example put create-react-app apps in there and test the hooks during development.

    Now, although the local/* packages are on gitignore, the root yarn.lock is simply bloated and polluted – and checked into git – because of the local workspaces.

    What I would wish for is a way to specify that some workspaces shall use some specific lockfiles, e.g. some mapping like so:

      "workspaces": {
        "packages": [
          "packages/*",
          "local/*"
        ],
        "lockfiles": {
          "local/*": "./local.yarn.lock"
        }
      }
    

    Or even a way to specify “do not put anything from this workspace into the lockfile at all”.

    But yeah, mine is not a serious use-case in the first place 🙂

11. I’m not exactly sure if separate lockfiles is the answer, but I have a similar problem. I have a monorepo set up with a CLI and a backend. The CLI requires a few packages that are platform-specific and only work on desktop machines with a particular setup. On the other hand I need to be able to build my api into a docker image, which is fundamentally impossible in the current implementation of workspaces.

    You nailed it – as I see it, one of the very core benefits of yarn.lock file is for creating frozen-dep production builds! Did the creators of Yarn forget that?

12. @the-spyke but the original issue is about exactly the opposite. A lock file per workspace.

    I fail to understand how you cant have a lockfile per workspace.

    It breaks deps uniformity in the Monorepo? Sure for development. But the whole purpose of shared dependencies goes out the window when you must deploy lightweight micro services from each of your workspaces

    Shared, hoisted deps only makes sense in development.

    For a workspace to live in Docker, it requires a lockfile.

13. @dan-cooke As you can see, I had this issue in 2018 too, but now I have different opinion.

    You’re saying Docker and Microservices. But what if I develop a regular npm package? I have no production dependencies subtree to pin, because they will be provided by end-user according to my dependencies specification. So, what I want is to maximize my development experience, and that what Monorepos and Yarn Workspaces perfectly do.

    Same time, if you’re developing microservices (MSs) there 2 possible situations:

    1. Independent projects. Some MSs are in development, some weren’t touched in years. In this case they are completely independent. It is possible to have UserService using LoggingService@1.2.3 and MessagesService using LoggingService@2.3.4. That’s not that easy world where you just link folders from Workspaces to the root node_modules. So, no point in having root lock-file. Create separate files (roots) and manage them independently. That called a Multirepo in Yarn docs. But now what you’re saying is “I want to run tasks in different folders from the root folder for convenience”. And that’s a completely different topic.

    2. Projects with unified dependencies like Jest/Babel/etc. This is what Workspaces were made for, but in MSs there additional requirements. During CI stages like linting and testing all works fine because it works the same as you do on a developer machine: deps installed by Yarn into root node_modules and flattened out. Just with addition that you probably cache the yarn install phase to speed up concurrent builds.

       In production it’s completely different: starting from that you only need deps for one workspace and ending with how to install that utils package? Should it be linked or downloaded as tarball? So, what you really need is not having lock-files per Workspace, but having a command like yarn install --prod <workspace> that you can run specifying a Workspace and it will install only production deps and same time ignore other not referenced Workspaces. Like if my data WS depends on utils WS, but not on logging WS, then logging itself and its deps should not appear in node_modules. A similar result, but a completely different approach to a “lock-file per workspace”.

       If you’re publishing build packages into a repository (npm, Arifactory, GutHub), you can get similar behavior by just copying lock-file into a Workspace and doing yarn install --prod here. It should warn about outdated file, but instead of recreating if from scratch with fresh versions it should just remove excess deps from it (just tried and looks legit). Should be even better and robust with using Offline Mirror.

       And in the end you have Focused Workspaces implemented exactly for Multirepos.

    So, what I was saying it that maybe the issue doesn’t look like what it is.

14. I’ve changed my stance now and realize that while having an individual lockfile per workspace might be the first thing that comes to mind when managing an entire monorepo with Yarn workspaces, it might not be the right question. A better question might be “Is Yarn workspaces designed to manage a monorepo?”. The answer, as usual, is “it depends”.

    If you’re Babel and you have a single team working on the monorepo and everything is meant to change in lockstep, then yes, this is what Yarn workspaces was designed for. But if you’re an organization with multiple teams and you’re using a monorepo, you likely don’t want to manage the entire monorepo with a single Yarn workspace root. You probably just want to use Yarn’s default behavior or multiple Yarn workspace roots within the monorepo. This will be determined by what apps you’re building, how many teams there are, etc.

    For us, it became clear that for each deployable entity (in our case there’s a Dockerfile), we want to have a separate yarn install done for each one (whether it’s a worksapce root or not). This provides clarity around code ownership, allows for isolated deployments that happen at different cadences, solves caching issues with lockfiles and Docker, etc. There are a few downsides to this, though:

    • What about duplicated node_modules packages? This is a common class of problems with monorepos and while Yarn workspaces help with hoisting, it’s not a general monorepo solution. There are other solutions, though. For example, Yarn PnP takes care of this. You could also use Lerna without Yarn and use the --hoist option.
    • What about the utility of running command across workspaces during development? Again, Yarn workspaces lets you do this, but that doesn’t mean one should make the entire monorepo a Yarn workspace root. Building the necessary tooling and scripts will be different for each team and depends on their monorepo. Yarn workspaces probably wasn’t designed as a monorepo task runner. One might try to bend Yarn workspaces a bit to do this job (i.e., run NPM scripts across the monorepo using yarn workspace ...) but it’s important to keep in mind that a single workspace root for the entire monorepo probably won’t give you what you need unless you’re like Babel, Jest, React, etc.

    There’s a whole other host of problems that come with running a monorepo. For example, what about tracking dependencies and only rebuilding things that changed to save time in CI? Yarn workspaces could help there by letting you query the dependency graph. Lerna does this, for example, to allow topological sorting of commands being run. Yarn v2 actually lets you query the dependency graph as well. The package manager PNPM also does that. But I would argue that depending on the complexity of the monorepo one might want to try tools built for that (not package managers) like Bazel, Pants, Buck, etc.

15. We have the same use case as @migueloller and one possible idea is for Yarn to support multiple sets of workspaces, like this:

    {
      "workspaces": {
        "frontend-app": ["frontend", "common"],
        "backend-app": ["backend", "common"]
      }
    }
    

    Yarn would maintain two additional lock files (I imagine the main yarn.lock would still exist):

    .
    └── monorepo/
        ├── yarn.frontend-app.lock
        ├── yarn.backend-app.lock
        └── packages/
            ├── frontend
            ├── backend
            └── common
    

    When building a Docker image e.g. for frontend, we’d create a context (e.g., via tar) that includes this:

    .
    └── <Docker build context>/
        ├── yarn.frontend-app.lock
        └── packages/
            ├── frontend
            └── common
    

    What I didn’t think about deeply is whether it’s possible to install (link in node_modules) the right versions of dependencies if frontend and backend lock different versions. But purely from the high-level view, two-dimensional Yarn workspaces is probably what we’re after.

    (Something similar was also posted here.)

16. It looks like that you don’t need a lock file per workspace but instead you require node_modules per workspace for deployment

    @gfortaine, if you read the discussion you will realize that that’s actually not the case. The reason for having a separate lockfile has nothing to do with installation, but instead having a lockfile that only changes when a specific package changes. A top-level lockfile will change with every workspace dependency change, but a lockfile scoped to a single package will only change when that package’s dependencies change.

    It might be worth mentioning that that this can be done in user-land. Using the @yarnpkg/lockfile package one can parse the top-level lockfile, and then using yarn workspaces info one can determine workspace dependencies. Using this information, together with the package.json of each workspace, one can generate a lockfile per workspace. Then, one could set up that as a postinstall script in the repo to keep those individual lockfiles in sync with the top-level one.

    I might take a stab at building this and report back with my findings.

17. My need for this behavior (versioning per workspace, but still have lockfiles in each package) is that I have a nested monorepo, where a subtree is exported to another repo entirely, so must remain independent. Right now I’m stuck with lerna/npm and some custom logic to attempt to even out versions. Would be nice if yarn (I guess v2, given that’s where nested support lies?) could manage all of them at once, but leave the correct subset of the global pinning in each.

    A postinstall script could attempt to manage the lock files directly, I suppose. Sounds complicated, but would be nice either way.

18. For anyone looking to generate a lock file from the workspaces lockfile, I have created a CLI that extracts your dependencies from a passed in package.json and spits out a lockfile using only the contents of the workspace lockfile. You can check it out on npm and GitHub

    https://www.npmjs.com/package/generate-lockfile

19. Since the goal here was to create docker deployments i’ll close this as fixed in v2 where you can use the plugin https://yarn.build/ or https://gitlab.com/Larry1123/yarn-contrib/-/tree/master/packages/plugin-production-install to handle that

    https://yarnpkg.com/getting-started/migration

Comments are closed.