Allow subdirectories within git repos in yarn install

Do you want to request a feature or report a bug?
Feature

What is the current behavior?
Unable to install packages that are inside subdirectories in git repos. This is a very common case for monorepos and just repositories that can’t have the node package in the root repo.

If the current behavior is a bug, please provide the steps to reproduce.
Not a but

What is the expected behavior?
There should be an availably syntax for yarn add and yarn global add that allows to add packages that are inside subdirectories in a git repo.

It make it much easier to use unreleased code when testing unreleased versions for issues/regressions or fixes that aren’t live yet like this ticket microsoft/code-push#428

ex. packages affected by this issue
https://github.com/Microsoft/code-push/tree/master/cli
https://github.com/Microsoft/BotBuilder/tree/master/Node
https://github.com/facebook/react-native/tree/master/react-native-cli
https://github.com/babel/babel/tree/master/packages/babel-cli

Related npm issue: npm/npm#2974

Please mention your node.js, yarn and operating system version.
yarn versions v1.2.1
node version v8.7.0
macOS Sierra 10.12.6

Author: Fantashit

23 thoughts on “Allow subdirectories within git repos in yarn install

  1. There are a couple of good ideas for syntax to use to specify subdirectories in the linked issue (npm/npm#2974)

    Use pip’s way of using a query parameter
    git+https://git.repo/some_repo.git?subdirectory=subdir_path#branch
    or
    git+https://git.repo/some_repo.git#branch?path=subdir_path
    Although I would assume most url parsers won’t parse this one.

    Use :: as it is not a valid token for a branch name
    git+https://git.repo/some_repo.git#branch::subdir_path
    git+https://git.repo/some_repo.git#::subdir_path

  2. Another idea: git+https://git.repo/some_repo.git/subdir_path#branch. If I were to assume this feature existed, that’s what I would try first. The other options would require someone to look up the syntax.

  3. That might be ambiguous depending on how the git server urls are setup.

    The url might be something like

    git+https://github.com/yarnpkg/yarn/<subdir_path>#branch

    Then it become hard to say what is part of the repo path and what is the subdir_path

  4. Not sure if this applies to every repo, but with Github both git+https://github.com/yarnpkg/yarn and git+https://github.com/yarnpkg/yarn.git work. You could assume anything after .git/ is a sub-directory.

    Regardless of syntax it would be nice to have this feature either way. It would be cool to make the syntax intuitive though :).

  5. Yeah, rising popularity of monorepo architectures makes it a crucial feature. Is there any workaround for this, for now? Discussion about public API tend to be years long…

  6. My # 1 use-case for this is pulling in bug-fixes from monorepo’s that haven’t been merged or released yet. I often want to keep most of the original packages, but pull in one of the sub-repos with a bug-fix. Currently I am unable to do this, and the workarounds are complex. Being able to reference a Github branch would greatly lower the resistance.

    I know there are some people at npm that are highly resistant to the idea of this, and monorepos in general. However, many of the contributors here work on projects that use monorepos. I would suspect that if yarn were to implement this feature and it became popular enough npm would follow suit. A lot of yarn ideas have been implemented in npm later on. yarn to me is the babel of npm.

  7. @erykpiast couple workarounds that I can think of:

    1. clone the target repo somewhere and then use yarn add /path/to/repo/subdirectory
    2. or possibly (if it’s in your own monorepo) use yarn workspaces?

    Or — this is maybe even crazier — add the remote in question as a submodule and (optionally move your project files into a subdirectory) then use yarn workspaces?

    Disclaimer: I haven’t tried either of these approaches and they both add overhead for new developers & your CI builds…

  8. Or — this is maybe even crazier — add the remote in question as a submodule and (optionally move your project files into a subdirectory) then use yarn workspaces?

    I can confirm this works.

    Upgraded to webpack 4. Create React App doesn’t have support yet but there’s a PR that’s waiting to be merged – unfortunately webpack 4 has changed it’s plugin API. Doubly unfortunate: create-react-app is repo containing multiple packages. Adding the forked PR as a submodule and adding the package using ‘file:’ syntax worked a charm.

    Literally saved an entire day or more of work with this work around.

  9. It’s alarming that this isn’t on the roadmap for either npm or yarn due to the rise in popularity of monorepos. The git+... syntax makes contributing to single-package projects a breeze, and by contrast not having it for subdirectories makes contributing to monorepo projects a total headache.

  10. Seriously, gitpkg works! It creates tags inside the project repository, and can be used with repositories containing multiple packages.

    And this prompted me to ditch npm because only yarn can fetch nicely from Git.

  11. I built a service https://gitpkg.now.sh/ to download the sub folder of a git repo as a .tgz file, so that it can be installed by yarn and npm. With this service, package users can use the subdir directly and there is no need for package developers to publish the subdirs.

    For example, if you want to add babel-core in branch next-8-dev as dependency,
    just use:

    yarn add https://gitpkg.now.sh/babel/babel/packages/babel-core?next-8-dev
    

    also works for npm install.

    Note that I must say sorry to @ramasilveyra because my service’s name conflicts with the tool he develops but I didn’t know that before.

  12. I’m having an issue trying to fork a repo (https://github.com/ianstormtaylor/slate) that uses workspaces and has multiple packages in the same repo. I only want to modify and import one of the packages. Trying to yarn add pointing to the directory that holds the individual package.json file gives an error…

    yarn add https://github.com/kr-project/slate/tree/master/packages/slate#e73db650a332fc5ded3fd72cffb1fcd7279b85d7
    error https://github.com/kr-project/slate/tree/master/packages/slate: Extracting tar content of undefined failed, the file appears to be corrupt: "Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?"
    

    I’ve also tried to check in the published files, but my understanding is this is a problem with yarn looking a tar only found at the root directory of a github repo?

    So is the only way to fork a project that uses workspaces to use something like gitpkg? I’m a bit surprised that there doesn’t seem to be more native support for this.

  13. So @LucidDan and others.

    I’ve found the following to work:

    • You must be running yarn 2.1.X for this to work.
    • See this page for the newer syntax that allows this: https://yarnpkg.com/features/protocols
    • You have to add individual entries for each package in the monorepo to your package.json under either dependencies or, in my case, devDependencies:
    "package-1": "my-org-name/mono-repo-name#head=my-git-branch&workspace=package-1"
    

    It’s disappointing that there isn’t a way to just have yarn install the top-level monorepo and unroll all of the attendant packages into their proper place. Maybe there is and I’m just being stupid here? If so, someone enlighten me 😉

  14. Try https://gitpkg.now.sh/

    yarn add https://gitpkg.now.sh/<user>/<repo>/<subdir>[?<branch>]

    OR

    "@react-native-community/cli-platform-ios":"https://gitpkg.now.sh/nomi9995/cli/packages/platform-ios?fix-ios-getProductName-regex"
    

Comments are closed.