ModuleConcatenationPlugin memory usage increases dramatically with chunk module count

Do you want to request a feature or report a bug?
I want to report a bug.

What is the current behavior?
ModuleConcatenationPlugin memory usage increases dramatically with chunk module count.

If the current behavior is a bug, please provide the steps to reproduce.

git clone https://github.com/filipesilva/webpack-module-concatenation-memory
cd webpack-module-concatenation-memory
npm install
npm run ngc
npm run webpack

This will result in peak memory usage of about 17GB.

Commenting out new webpack.optimize.ModuleConcatenationPlugin(), in webpack.config.js results in
peak 1.8GB memory usage.

npm run ngc is necessary because it also generates some files in node_modules/ that are
imported by the app.

The app has a 12 lazy loaded chunks by default.
These can be imported into the main chunk by editing src/app/app-routing.module.js (after npm run ngc), commenting out the System.import: and uncommenting import declarations as shown in the repo readme. This will increase memory usage further.

Note: this repo uses a minimal webpack.config.json with no loaders:

const path = require('path');
const webpack = require('webpack');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const UglifyJSPlugin = require('uglifyjs-webpack-plugin');
const ProgressPlugin = require('webpack/lib/ProgressPlugin');

module.exports = {
  entry: './src/main.js',
  output: {
    path: path.resolve(__dirname, './dist'),
    filename: 'bundle.js'
  },
  plugins: [
    new ProgressPlugin(),
    // Ignore `Critical dependency: the request of a dependency is an expression` warnings.
    new webpack.ContextReplacementPlugin(
      /angular(\\|\/)core(\\|\/)esm5/,
      __dirname
    ),
    new HtmlWebpackPlugin({
      template: './src/index.html'
    }),
    new webpack.optimize.ModuleConcatenationPlugin(),
  ]
};

What is the expected behavior?
Memory usage should be more in line with base webpack usage (around 1.8GB instead of 17GB).

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.
No browser was used, node v8.9.1, webpack 3.8.1, Windows 10

Author: Fantashit

1 thought on “ModuleConcatenationPlugin memory usage increases dramatically with chunk module count

  1. I think I found the problem.

    The identifier for a concatenated module is the list of all file paths included in that module:

    identifier() {
    return this._orderedConcatenationList.map(info => {
    switch(info.type) {
    case “concatenated”:
    return info.module.identifier();
    }
    }).filter(Boolean).join(” “);
    }

    This string can get very, very big when there’s a bunch of modules concatenated together. And everywhere that a module is used, if it’s inside the concatenated module, it also uses the same identifier.

    So if you had 350 modules their identifiers were, say, a path of 100 character length each. Now all those modules are inside a single concatenated module. The identifier of that concatenated module is 350 * 100 = 35,000 characters. That identifier is being used in at least 350 places, so at least 350 * 35,000 = 12,250,000 extra characters.

    Changing this identifier to be a small string instead lowers the memory usage for the original repro from 17GB to 2.8GB.

    Will put up a PR.

    Special thanks to @clydin that pointed me at the increased size of the stats files and how it could be related to this problem.

Comments are closed.