Webpack4 has memory leak in development mode

Do you want to request a feature or report a bug?
bug

What is the current behavior?
The memory increased when running webpack4 in development, and the memory didn’t release. So there was memory leak.
Then node will throw error
JavaScript heap out of memory
I saw the heapsnapshot, there was many repeated String Objects which were compiled by webpack.
If the current behavior is a bug, please provide the steps to reproduce.
run webpack development mode with , update the business code and waiting rebuild, then update the code…you will see the increasing memory.

What is the expected behavior?
Clear memory in time.
If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version, and Operating System.
webpack@4.2.0

Author: Fantashit

22 thoughts on “Webpack4 has memory leak in development mode

  1. I just upgraded from 3.10.0 to 4.5.0 and I’m seeing this sporadically in development:

    i 「wdm」: Compiling...
    webpack building...
    
    <--- Last few GCs --->
    
    [9284:000000000028FA40]  1100280 ms: Mark-sweep 1385.1 (1411.9) -> 1385.1 (1411.9
    ) MB, 292.3 / 0.1 ms  allocation failure GC in old space requested
    [9284:000000000028FA40]  1100624 ms: Mark-sweep 1385.1 (1411.9) -> 1385.1 (1407.9
    ) MB, 343.1 / 0.1 ms  last resort GC in old space requested
    [9284:000000000028FA40]  1100867 ms: Mark-sweep 1385.1 (1407.9) -> 1385.1 (1407.9
    ) MB, 243.6 / 0.1 ms  last resort GC in old space requested
    
    
    <--- JS stacktrace --->
    
    ==== JS stack trace =========================================
    
    Security context: 000003025D1257C1 <JSObject>
        0: builtin exit frame: lastIndexOf(this=000003B48B1F6031 <Very long string[60
    6700]>,00000315AAE8BF89 <String[1]\: \n>)
    
        1: /* anonymous */(aka /* anonymous */) [C:\www\node\poject\node_modu
    les\webpack-sources\node_modules\source-list-map\lib\SourceListMap.js:~100] [pc=0
    000029834AF4F94](this=0000006331D822D1 <undefined>,sln=00000157ABFAF271 <SourceNo
    de map = 0000026155AF2B41>)
     ...
    
    FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memor
    y
     1: node_module_register
     2: v8::internal::FatalProcessOutOfMemory
     3: v8::internal::FatalProcessOutOfMemory
     4: v8::internal::Factory::NewRawTwoByteString
     5: v8::internal::Smi::SmiPrint
     6: v8::internal::StackGuard::HandleInterrupts
     7: v8::internal::SlicedString::SlicedStringGet
     8: v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString
     9: v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString
    10: 0000029833B86B21
    

    The only change I made to my config file was adding mode: 'development'.

  2. We need minimum reproducible repo, without this it makes no sense to write about the fact that you have a problem. Thanks!

  3. Guys please read #6929 (comment) again, we can’t solve this problem because in pure usage webpack no problems, i.e. problem with memory leak in some loader/plugin, please create minimum reproducible test repo if you want to solve problem fastly. Thanks!

  4. Ok, here is a minimal test case for the leak that I am experiencing.

    I extracted the example from a much larger project that I am working on. The problem here seems caused by a third-party plugin, DojoWebpackPlugin, but in some snapshot I recall seeing other plugins that were causing the issue.

    It appears that webpack is keeping Compilation instances around, probably because of how tapable hooks are used. I don’t know the webpack source code well enough to proceed further in the investigation. See my example README for details.

  5. I did some more investigation on my leak involving dojo-webpack-plugin (see the updated test case) and I found the root cause of the leak.

    Long story short: it is probably not webpack’s fault.
    The gist of what dojo-webpack-plugin is doing is the following.

    compiler.hooks.compilation.tap("test", compilation => {
      const context = Object.create(this, {
        compilation: { value: compilation },
      });
      let fn = (function() {
        console.log("Making compilation"); 
      }).bind(context);
      compiler.hooks.make.tap("test", fn);
    });

    As you can see, it is registering a Compiler hook that internally holds a reference to each Compilation object. This leaks all compilations via the tapable hook.
    I also tried the equivalent webpack 3 code and it leaks the same.

    I will be reporting this to the plugin author since I guess this is not something a plugin should do, isn’t it?

  6. After i’ve commented out devtool: 'inline-source-map',
    it worked without memory exception, obviously also without source maps and normal debugging.

    After i’ve tried to add TerserPlugin to optimization (or plugins) section, and it fails again, so obvious that there is a problem with source maps library
    node_modules\source-map\lib\util.js (Terser)
    node_modules\source-map\lib\source-node.js ( devtool: ‘inline-source-map’ )

    so i stick with inline-cheap-source-map option, at least it makes hot reload possible.

  7. Something to note here:
    If you are using a watch mode or dev-server, webpack opts to using an in-memory file system. This means, that anything generating a deterministic results [hashed file names] will persist in the memory file system forever until watch mode is killed and started again.

  8. @evictor Potentially, but as I understand it, it can be non-trivial to tell when things like autogenerated bundles containing split chunks are no longer needed, since these modules may only be named by hash. Or, what if your app is loaded in the browser, a file is edited which causes an async bundle to get renamed, but then in the browser, your code requests the old async chunk? It would error out unexpectedly because the requested bundle was pruned, which could seem more like a misconfiguration than expected behavior.

    It could be a whole lot more effective to allow users to just opt out of using memory-fs, since disk space is much cheaper to consume than memory. The change is only a couple of lines.

  9. @salemhilal That’s not ideal, of course. It is not unreasonable to expect webpack-dev-middleware to prune no longer needed files when their hash changes.

    I think we can fix this behavior for next major release

  10. I close the issue because there are many different problems and it is not possible to track them, many problems have been fixed, if you encounter a problem please create a new issue with a reproducible test repo, thanks

  11. @afwn90cj93201nixr2e1re Нет ничего сложного открыть новую проблему с воспроизводимым примером ошибки/зависания/утечки и тд. Используйте только анлийский язык в будущем.

  12. Мда, то есть тебе не достаточно этих репортов? 90%% которых ссылаются на неверно реализованное кеширование дев сервера…
    И да, это видимо главная проблема, которая до сих пор не исправлена, but issue closed, np,

  13. btw most of them (90%%) related to cache issue’s…
    5%% – to old version, 4%% – to old plugins, 1%% – other stuff.

Comments are closed.