Posted in Uncategorized

Webpack dev server takes up a lot of memory (751.9MB) for GitLab CE

   February 23, 2021  7 Comments on Webpack dev server takes up a lot of memory (751.9MB) for GitLab CE

@TheLarkInn @mikegreiling @jschatz1

Do you want to request a feature or report a bug?

Unexpected Behavior

What is the current behavior?

We are unsure if this is a combination of both webpack and webpack-dev-server. So posting here first 😄

The webpack-dev-server process is a memory hog. As of Feb 22nd it is coming in at 751.9MB for the CE branch. This is after having removed the imports-loader plugin in Issue 9123 and before this it was often as high as 850MB.

If the current behavior is a bug, please provide the steps to reproduce.

GitLab Issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/28557

webpack-memory

This seems really excessive and we are trying to ascertain the root cause of it. This issue will serve as a place to discuss and chronicle our attempts to alleviate this memory footprint.

I think there are a few things worth trying:

  • Try exempting node_modules within devServer.watchOptions (suggested by Kees Kluskens)
  • Test out this webpack PR meant to improve memory use (suggested by Kees Kluskens)
  • Implement CommonsChunkPlugin to prevent multiple copies of Vue from being retained in memory
  • Reduce the number of babel plugins we use by replacing babel-preset-es2015 with babel-preset-env (#28026)

What is the expected behavior?

That our dev server should not have such a large footprint 😂

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.

macOS – "webpack": "^2.2.1"

Can ask other people that use the GDK to tell us what their mem footprint is on Linux as well if need be.

Author: Fantashit

7 thoughts on “Webpack dev server takes up a lot of memory (751.9MB) for GitLab CE”

  2. In our project the footprint is 400-500M. In nodejs world I think it is huge. We use only watch & rebuild and javascript files with imports

  4. @terencechow at @GitLab we saved a lot of memory by doing the following:

    • removing all uses of require.context()
    • switching to ES module syntax wherever possible (apparently statically analyzed dependencies must require fewer resources)
    • utilizing CommonsChunkPlugin to remove redundant copies of vendor modules from our bundles (before this we compiled Vue in 5 or 6 separate bundles and I don’t think babel and/or webpack was smart enough to hold this in memory in only one place internally)
    • removing any plugins that aren’t specifically needed in development mode (CompressionPlugin, HashedModuleIdsPlugin, BundleAnalyzerPlugin, etc)
    • making sure to exclude: /node_modules/ in all of our loaders
    • switching away from inline-source-map to cheap-module-eval-source-map

    All that said, it still takes up a good chunk of memory, and the more of webpack/babel’s cool features and plugins you want to use, the more memory it will require. Thankfully when the process has been running a while on macOS, it tends to use compressed memory to reduce this burden a bit.

  6. @evilebottnawi I’ve copied the entire gitlab frontend and webpack config to https://gitlab.com/psimyn/gitlab-webpack-memory (also includes icons, images etc)

    Heap snapshots are around ~1.3GB. The node-inspect command would OOM error unless I increased --max_old_space_size

    git clone https://gitlab.com/psimyn/gitlab-webpack-memory.git
cd gitlab-webpack-memory

yarn
yarn dev-server # this is what we usually run in main repo
yarn node-inspect # runs node with --inspect flag

Comments are closed.