Embedded SDK hash in kernel binaries

There are several layers of compatibility in the VM regarding to kernel. One is whether the VM can correctly parse Kernel files. Another is whether the kernel file actually has the right information inside it (e.g. the right async transformer has run, the right classes/fields/variables are in it).

The second point does not have any validation atm and can lead to customers accidentally using kernel files of an older version, which the VM can still parse, but the kernel files might have the wrong information in it (e.g. old async transformer ran instead of new one).

To make such cases not crash the VM and instead give a nicer error message, we want the VM to validate that the Kernel it consumes comes from the same Dart SDK commit as the VM was built at.

We already have support for embedding a hash in the VM which validates that e.g. AOT snapshots given to the VM were produced by gen_snapshot from the same sources.

=> We would like to extend this to kernel files and embed a hash on those.

/cc @mraleph @johnniwinther

Author: Fantashit

1 thought on “Embedded SDK hash in kernel binaries

  1. This is a VM issue. It was an observable change in behavior which has caused ongoing breakage for over 2 months
    after being reported. The existence (and not closure) of these issues has led us to believe we did not need to do anything on
    our end and it would be fixed.

    For the past several years the VM has had predictable behavior that did not require users (such as pub, build_runner, etc) to

    To the best of my knowledge there was no such guarantee for the past several years: During the transition from Dart 1 to Dart 2 the VM was changed to use kernel for script snapshots (4b1180a was seemingly the first commit to introduce this behavior, bc7220a made it probably the default).

    Before this time, the VM has embedded a hash (based on a subset of VM source files) into script snapshots, assuming the hash will change if snapshots become incompatible (see 3dfb90f) – whether this list covered everything (see #41802 (comment)) is another question.

    With the introduction of kernel, a kernel format version number was added to kernel files. The VM would, as usual, issue an API error if it cannot read such a kernel file. But as outlined in #41802 (comment), that version number does cover only the kernel format itself and not various other compatibilities.

    => I agree, we could consider this a breaking change, back in 2018.
    => Since then it was effectively not really safe to re-use kernel files across SDK versions.
    => The reason this was not really noticed is because the VM only very infrequently updates kernel transformers, adds recognized methods, … and only in certain situations would a user actually notice it.

    We have been relying on the VM reporting for us the incompatibility with exit code 253 since 2014.

    Originally this seemingly came from c859d25. Though notice that the exit code 253 is not reserved for incompatible snapshots, we issue API errors also in other situations.

    This was useful behavior that made sense to rely upon because anything else would require additional
    tracking of the previous sdk versions etc by all of these tools.

    I understand that this seems very useful, though one could argue that it would be much better to add a flag to the VM to ask it whether it can run a snapshot instead of trying to run it and see what happens (e.g. imagine a user program is actually exiting with this exit code, would it trigger an infinite re-snapshoting attempts?)

    In any case, because of all these problems I have advocated (a lot, despite strong opposition) for embedding the SDK hash into the kernel, which we have now – the CL re-landed in 0ce8398.

    We will leave this bug open until it has rolled through flutter and then close it.

Comments are closed.