FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory

Description

When running test for 100 to 200 scenarios, I always get this issue. All test suites on my project might take more than 4 hours for complete. But when showing this error, the test process will be stuck and I can’t run completely all test suite.

Expected behavior

Is there a way to increase the node JS timeout in DETOX (–max-old-space-size)? When doing research the node JS script, people ask to change this number, but I’m not sure about that.

Screenshots

Screen Shot 2021-01-13 at 05 55 13

Environment (please complete the following information):

  • Detox: 17.11.4
  • React Native: 0.60.5
  • Node: 10
  • Device: iPhone 11
  • Xcode: 11.7
  • iOS: 13.7
  • macOS: 10.15.6

1 possible answer(s) on “FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory

  1. Thanks for the report! Looks interesting, although it might take time to get to your issue — as far as I could understand, to reproduce it, I might need about 200-300 autogenerated dummy test files and run them all.

    As for workarounds. I can’t tell if this is going to work out, but indeed you could try prepending NODE_OPTIONS environment variable to your detox test command — something like this:

    NODE_OPTIONS="--max-old-space-size=8192" detox test ...
    

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory

  • Version: 7.2.0
  • Platform: MacOS, Ubuntu 16.04, Windows
  • Subsystem:
  testing add new ObjectType 

<--- Last few GCs --->

[8138:0x102801600]   145460 ms: Mark-sweep 1265.6 (1301.6) -> 1265.6 (1308.6) MB, 289.8 / 0.0 ms  allocation failure GC in old space requested
[8138:0x102801600]   145740 ms: Mark-sweep 1265.6 (1308.6) -> 1265.6 (1277.6) MB, 280.6 / 0.0 ms  last resort gc 
[8138:0x102801600]   146035 ms: Mark-sweep 1265.6 (1277.6) -> 1265.6 (1277.6) MB, 295.0 / 0.0 ms  last resort gc 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x39c891dc0d31 <JS Object>
    1: DoJoin(aka DoJoin) [native array.js:~97] [pc=0x5d1facabad4](this=0x39c891d04311 <undefined>,q=0x5a024bf3be1 <JS Array[2241635]>,r=2241635,F=0x39c891d043b1 <true>,B=0x39c891ddafe9 <String[1]\: \n>,A=0x39c891d04421 <false>)
    2: Join(aka Join) [native array.js:~122] [pc=0x5d1fb5cde96](this=0x39c891d04311 <undefined>,q=0x5a024bf3be1 <JS Array[2241635]>,r=2241635,B=0x39c891ddafe9 <String[1...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/erossignon/.nvm/versions/node/v7.2.0/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/erossignon/.nvm/versions/node/v7.2.0/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/erossignon/.nvm/versions/node/v7.2.0/bin/node]
 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/erossignon/.nvm/versions/node/v7.2.0/bin/node]
 5: v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/erossignon/.nvm/versions/node/v7.2.0/bin/node]
 6: 0x5d1faa063a7
Abort trap: 6

  • How to reproduce:

On ubuntu 16.04 for instance, install node 7.20 , and run the following commands

git clone https://github.com/node-opcua/node-opcua
cd node-opcua
npm i
npm test
  • Note:
    Node-opcua only fails with node 7.2 and works perfecrlt with any version of node (0.12,4.xx,6.9.1)

13 thoughts on “FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory

  1. Does it work with v7.1.0? Does it work with --max_old_space_size=2048 or more?

    By the way, the stack trace indicates that the process is running out of memory because it can’t concatenate two (presumably large) strings. You might have just been lucky and skimmed the edge with older versions.

    • Version : 7.10 same error

    With or without

    Command run (to narrow the issue a little bit):

    node  --max_old_space_size=2048  ./node_modules/.bin/mocha test/address_space/test_address_space_add_object_type.js 
    
    <--- Last few GCs --->
    
    [9296:0x103000c00]   139535 ms: Mark-sweep 1286.0 (1315.8) -> 1265.5 (1311.6) MB, 298.8 / 0.0 ms  allocation failure GC in old space requested
    [9296:0x103000c00]   139814 ms: Mark-sweep 1265.5 (1311.6) -> 1265.5 (1277.6) MB, 278.9 / 0.0 ms  last resort gc 
    [9296:0x103000c00]   140104 ms: Mark-sweep 1265.5 (1277.6) -> 1265.5 (1277.6) MB, 290.7 / 0.0 ms  last resort gc 
    
    
    <--- JS stacktrace --->
    
    ==== JS stack trace =========================================
    
    Security context: 0x19c04f0c0d11 <JS Object>
        1: DoJoin(aka DoJoin) [native array.js:~97] [pc=0x3d004bbadbf4](this=0x19c04f004311 <undefined>,q=0x17a8c2085899 <JS Array[2241635]>,r=2241635,F=0x19c04f0043b1 <true>,B=0x19c04f0dafc9 <String[1]\: \n>,A=0x19c04f004421 <false>)
        2: Join(aka Join) [native array.js:~122] [pc=0x3d004c4970b6](this=0x19c04f004311 <undefined>,q=0x17a8c2085899 <JS Array[2241635]>,r=2241635,B=0x19c04f0dafc9 <Stri...
    
    FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
     1: node::Abort() [/Users/erossignon/.nvm/versions/node/v7.1.0/bin/node]
     2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/erossignon/.nvm/versions/node/v7.1.0/bin/node]
     3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/erossignon/.nvm/versions/node/v7.1.0/bin/node]
     4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/erossignon/.nvm/versions/node/v7.1.0/bin/node]
     5: v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/erossignon/.nvm/versions/node/v7.1.0/bin/node]
     6: 0x3d004b9063a7
    Abort trap: 6
    
  2. Ben, I think @abenhamdine issue was different but produced the same effect.
    For the record, here is how fix the issue on my side.
    The “should” package produces rich object output when an assert is raised.
    In my case a error occurred in a unit test in a line looking like

    var obj = getsomeobject();
    should(obj).not.eql(null);

    In my case, the object owns a back pointer to its parent object, which exposes thousand of complex objects.
    When should raises the assert it tries to dump the whole object including the parent and the other sibling objects causing a large string to be constructed with Array.prototype.join.
    The string contains more than 10 millions little fragments and causes the memory exhaustion.

    I have fixed the issue using the following:

    • replace should(obj).not.eql(null); with should.exist(obj)
    • make the back pointer not enumerable using Object.defineProperty(this, "_parent", {configurable: true,value: parent, hidden:true,enumerable: false});

    The problem existed in my original code and has been highlighted by the new behavior in nodeJS 7.x.
    Ben, do you have any idea why the memory behavior has changed between 6.9 and 7.0?

  3. I used –max_old_space_size and I again got the memory error. After doing some online research i realized it not underscore but dash like –max-old-space-size

  4. @erossignon is there a way to use –max_old_space_size with npm run unit: cross-env BABEL_ENV=test karma start test/unit/karma.conf.js –single-run? I have been trying almost all day

  5. how many of the people with this issue are using firebase? I found out in my case issue was with the firebase. as soon as I removed it, my project was built in no time

  6. using the command to build the app is

    “build”: “cross-env NODE_ENV=production webpack NODE_OPTIONS=-max_old_space_size=12000 –watch –progress –config internals/webpack/webpack.prod.babel.js –color -p –progress”,

    still getting the issue.

  7. I have tried 👍 the below code and its working fine.

    • open terminal from project root dir
    • execute the cmd to set new size.
      set NODE_OPTIONS=--max_old_space_size=8172
  8. @bnoordhuis where can I find documentation on how to read the stack trace?

    It isn’t NodeJS’s fault that memory is running out.

    Tho it is hard for JS developers like me to use the V8 trace to find where in their code the string concat (or other) happened.

    Do you have any tech talks, blogs, or docs that would help devs like me correctly read the trace?

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory

Hi,

I use the nuxtent module, when I yarn generate I have this message :

<--- Last few GCs --->

[8503:0x2a784d0]   124609 ms: Mark-sweep 1402.9 (1492.6) -> 1402.9 (1488.1) MB, 1949.4 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1949 ms) last resort 
[8503:0x2a784d0]   126604 ms: Mark-sweep 1402.9 (1488.1) -> 1402.9 (1488.1) MB, 1993.6 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3949dd29cef1 <JSObject>
    0: builtin exit frame: assign(this=0x3949dd29b919 <JSFunction Object (sfi = 0x3949dd29b869)>,0x1400f1560559 <Object map = 0x2313e04a25f9>,0x7863dde1671 <Object map = 0x3ecc755372b9>,0x19704383cf91 <Object map = 0x2313e0484829>)

    1: renderAttrs [/home/nginx/domains/stnetwork.fr/public/test/nuxtent/src/node_modules/vue-server-renderer/build.js:~335] [pc=0xe90ecc5a480](this=0xea43946c941 <J...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x13740dc [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewTransitionArray(int) [node]
 6: v8::internal::TransitionArray::Insert(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [node]
 7: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Handle<v8::internal::LayoutDescriptor>, v8::internal::TransitionFlag, v8::internal::MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [node]
 8: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [node]
 9: v8::internal::Map::CopyWithField(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [node]
10: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Object::StoreFromKeyed) [node]
11: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [node]
12: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [node]
13: v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [node]
14: v8::internal::JSReceiver::SetOrCopyDataProperties(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Object>, v8::internal::ScopedVector<v8::internal::Handle<v8::internal::Object> > const*, bool) [node]
15: 0xc626b7 [node]
16: v8::internal::Builtin_ObjectAssign(int, v8::internal::Object**, v8::internal::Isolate*) [node]
17: 0xe90ec43c8dd
error Command failed with signal "SIGABRT".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

You can reproduce the issue with my repo

I opened an issue in the nuxtent-module, because I thought it was related to the module, but It’s probably related to NuxtJS.

Thanks to help me.

This bug report is available on Nuxt.js community (#c1550)

4 thoughts on “FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed – JavaScript heap out of memory

  1. @STnetwork You are definitely not alone. I also experienced problems with a memory leak this week.

    Disclaimer: My staging environment has been running stable on v0.10.7 since 5 months and I still did not migrate to v1.0.0, because when I tried migrating during the alpha phase, there were too many side effects and now I feel too close to my clients production beta roll out to go through the migration pain. And due to the fact that people here were experiencing leaks with the latest v1.0.0-rc-11 and I didn’t have issue with 0.10.7 until last week, I decided to stay on the deprecated version for my investigations.

    Without exactly knowing what I was searching for (tbh: I did not even know what I was doing at all), I was trying to analyze heapdumps as suggested here: #805

    No deeper knowledge was needed to observe that my dumps went incredibly large (~1,8G after 5 hours of uptime with almost no traffic) and were containing ridiculous amounts of VueComponent objects which were never released by the garbage collector and from my humble evaluation the leak seemed to be related to errors on the __VUE_SSR_CONTEXT__ as you can see in this screen shot:

    screen shot 2017-09-28 at 22 47 01

    It seemed that the memory leak started after I had dockerized my app. The fresh build pipeline introduced a couple of new module versions for dependencies which weren’t issued before when I was manually updating the project with simple npm install executions, so I tried to pin my versions to the same versions that I could find in the old directory for the manual updates to see if the leak was disappearing. It turned out that my app was still crashing, sometimes after uptimes between 10 to 20 hours. The heap profiles also did not improve.

    After 3 days of sweat and tears, I had my heureka moment today, when I realized, that there was a SSR rendering error (caused by testing for this.$route.name.startsWith('something')) on my dev server command line, which I did not pay attention to, because everything looked fine in the client and expecting the root cause for the leak to be deeper inside the framework made me completely blind-eyed for cleaning in front of my own house. I fixed the null pointer error (don’t know if that is the correct term in the Javascript lingua) and now the garbage collector seems to do his job as expected. At least the current heap comparison shows an healthy amount of object deletions now.

    Maybe my story is unique and incredibly stupid, but I wanted to share it, because I found a couple of similar threads on this topic here. Just in case somebody becomes blind-eyed like I did, I recommend to make sure that the dev server output is not showing any weird errors. Even if your front end works fine and you ignore the output to focus on the bigger problem, there might be a good chance, that you introduced the problem yourself.

    @pi0 @Atinux @alexchopin Although my problem was home-made and could be fixed, there might still be a general problem when errors in the space of the __VUE_SSR_CONTEXT__ happen. I am aware that you reuse the context with runInNewContext: false, so I guess the amount of allocated memory is drastically reduced by that alone, but judging from the current complaints there might still be issues (like in my case) where this alone does not help avoiding the sort of memory leak I produced. Does my assumption make any sense? I hope my investigations could still give any lead, considering that I am still running on v0.10.7. People are still experiencing errors on the latest release candidate, so I guess there is still room for improvements? If you are interested I could also share the heapdumps of my malicious version.

  2. Colleagues, sorry for crossposting #1750 (the details are there), but I think it is important to notice, that context copies made with vue-router via nuxt-link.

    I have a heavy page with many-many nuxt-links (catalog), > 200 links. When I apparently changed them to simple anchors, SSR instance memory footprint decreased after load test to initial values in meaningful time.