Isolates and HttpServer.bind(…, shared: true): failed assert, isolate exit and hard crash

  • Dart SDK Version (dart --version): 2.2.0
  • Whether you are using Windows, MacOSX, or Linux (if applicable): MacOSX and Linux

I’ve been using dart server for a few months now and really enjoying it, and have recently rolled it out into production for a project with a relatively small user base. For the most part, things are going smoothly! However, I think I’ve hit what may be a rough edge at the intersection of isolates + “shared” HttpServer functionality (think “cluster” functionality from node.js). The shortest way to put it is as follows: when one isolate (running one of multiple “shared” HttpServer instances) exits, a SEPARATE and seemingly RANDOM other isolate (also running a “shared” HttpServer instance on the same port) dies. With --enable-asserts, it’s possible to see that that other isolate is asserting on _NativeSocket.multiplex (dart:io/runtime/binsocket_patch.dart:906:18).

I’ve created a stripped down repository containing the minimum code necessary for repro: https://github.com/emeyex/dart-iso-server-crash-repro.

I’m not sure if this should be multiple issues, or if that assert failure is the root of the hard crash as well. Please let me know if I can provide any more info.

Author: Fantashit

3 thoughts on “Isolates and HttpServer.bind(…, shared: true): failed assert, isolate exit and hard crash

  1. I can reproduce! Let me take a look later.

    ../../runtime/bin/eventhandler.h: 555: error: expected: entry != NULL
    version=2.6.0-edge.0449905e2de69724e5fe4acc9c7589c8cab4046b (Tue Oct 22 13:40:01 2019 +0000) on "linux_x64"
    thread=257467, isolate=(null)((nil))
      pc 0x0000560a1651590c fp 0x00007ffa7007a730 dart::Profiler::DumpStackTrace(void*)
      pc 0x0000560a16131f32 fp 0x00007ffa7007a810 dart::Assert::Fail(char const*, ...)
    GET / 127.0.0.1 -> Hello from worker: 2
      pc 0x0000560a1611d35c fp 0x00007ffa7007a850 /usr/local/google/home/zichangguo/dart-sdk/sdk/out/DebugX64/dart+0x17f735c
      pc 0x0000560a1611bdb2 fp 0x00007ffa7007ab40 dart::bin::EventHandlerImplementation::HandleInterruptFd()
      pc 0x0000560a1611c5bc fp 0x00007ffa7007ac40 dart::bin::EventHandlerImplementation::HandleEvents(epoll_event*, int)
    GET / 127.0.0.1 -> Hello from worker: 1
      pc 0x0000560a1611c7b1 fp 0x00007ffa7007ae40 dart::bin::EventHandlerImplementation::Poll(unsigned long)
      pc 0x0000560a16175cc5 fp 0x00007ffa7007ae70 /usr/local/google/home/zichangguo/dart-sdk/sdk/out/DebugX64/dart+0x184fcc5
    -- End of DumpStackTrace
    Aborted
    
  2. A brief update for root cause.
    In this case, each isolate will create its own socket to bind to the same address. In VM, we only keep one copy of OS socket if that’s shared(Of course, address and port should be same). This socket will be used by different dart sockets.

    When one of isolates get killed, it calls finalizer to clean up. Finalizer needs to close the socket living inside dead isolate. But it didn’t store all connected dart sockets so ends up closing a random dart socket. This causes

    when one isolate (running one of multiple “shared” HttpServer instances) exits, SEPARATE and seemingly RANDOM other isolate (also running a “shared” HttpServer instance on the same port) dies.

    I will work on a fix. Thanks for patience.

Comments are closed.