RemoveParentModulesPlugin takes a long time with hundreds of chunks

Do you want to request a feature or report a bug?
I want to report a performance bug.

What is the current behavior?
In projects with a lot of chunks, a lot of time is spent on RemoveParentModulesPlugin.

The original report (angular/angular-cli#5618 (comment)) detailed a 1h build time for a non-production (no uglify etc) build.

The project uses Angular CLI and contains around ~2700 source files. This project has 376 chunks.

A similar sized project (https://github.com/filipesilva/angular-cli-test-repo) takes only a few minutes to build using the same options.

If the current behavior is a bug, please provide the steps to reproduce.

The project is closed source so I do not have a source, but the owner is available for some remote debugging (angular/angular-cli#5618 (comment)).

He provided CPU profiles for the first ~10m with node 8.4.0 (here) and with node 9.3.0 (here).

Both profiles show the majority of time being spent in RemoveParentModulesPlugin, specifically within the hasModule function. Also see the “Other information” bit at the end for a comparison with Webpack 4.

What is the expected behavior?
RemoveParentModulesPlugin is optimized to allow having a lot of chunks without a big performance hit.

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.

Node.js 8.4.0, Webpack 3.10.0. Unknown browser/OS.

Other information.

I’ve been looking at Webpack 4 and thought maybe it could help since there are a lot of performance improvements there. The plugin in question has suffered some changes:

It looks like there’s a fair bit of optimizations changes there. But I’m not sure if the bottleneck has changed.

There’s two places where time is being spent on the CPU profile:

  • a has call inside the hasModule function:
    image
  • a filter call inside the hasModule function:
    image

This function hasn’t changed though so I don’t think it would speed up in this case.

/cc @pleerock @TheLarkInn

Author: Fantashit

5 thoughts on “RemoveParentModulesPlugin takes a long time with hundreds of chunks

  1. For incremental build situations (mode: development) for us, when I put this option in:

    optimization: {
        removeAvailableModules: false,
    }
    

    The initial build time went down from total of 150s to 100s while the incremental went from near the full build time to 24s!! I’m happy, but it looks like I can do even more here by removing more plugins in my dev mode build.

  2. On https://github.com/angular/angular-cli we noticed something similar back on Webpack 3 with TS some time ago. Transpiling to CommonJS modules resulted in MUCH faster rebuilds than ES2015 modules (with ES5 code).

    The reason being imports were analysed for tree shaking with ES2015 modules, but not with CommonJS. This was tracked in #5863 and marked as resolved for Webpack 4.

    Having these granular controls allows to decide what the development builds actually do. Just going through the descriptions in that article it sounds to me that disabling the following for dev would help:

    • optimization.removeAvailableModules
    • optimization.removeEmptyChunks

    I’d like to actually benchmark the impact that each has but haven’t yet.

    I don’t think chunk size is relevant in dev. One could make the argument that smaller chunks take less time for the browser to parse, and that’s true. But the fastest sourcemaps for dev (eval, the default) are already essentially duplicating all of your JS chunks anyway.

Comments are closed.