Proposal: Dll-like Plugin for an external context – external manifest, dynamic module-map injection and loading

Currently, the DllPlugin and DllReferencePlugin allows loading external modules from separately generated bundles. The problem is that we need to provide a manifest with all the modules being bundled at the time of bundling. This is limiting in certain scenarios, where we would like to dynamically load modules which are not known at the time of build (user input, external files).

Example use case:

Imagine an docs-browser app, that dynamically loads the documentation text and executable examples, based on a JSON file that lists where they are contained. The documentation and examples are stored and built separately. They both can grow (more files are added) and we shouldn’t need to republish the entire documentation website after every new addition to the documentation.

What I propose is that we delegate the manifest to the Dll bundle itself. This way the following could be possible:

require('https://domain.com/dll-bundle.js#jquery');

Assuming the dll-bundle.js file is a bundle that contains the module jquery, the above require would download the file (proper CORS required), load the manifest from it (listing all the mappings of modules and merging them with the preloaded ones) and finally load the code from the bundled module jquery.

This would require the module IDs to be long and random (e.g. UUIDs) so that collisions do not occur, but they could technically also be generated based on prefixed some seeded value.

This would solve questions like #1421 or #150.

EDIT:
A helper method for requiring from an external bundle might make more sense, something like:

require.ensureExternal('https://domain.com/dll-bundle.js', function(require) {
  var $ = require('jquery');
})

// or even:
require.ensureExternal(dynamicallyGeneratedUrl, function(require) {
  var someModule = require(totallyDynamicName);
})

Author: Fantashit

6 thoughts on “Proposal: Dll-like Plugin for an external context – external manifest, dynamic module-map injection and loading

  1. webpack doesn’t take care of the module loading. You need to use another library for the loading part. But you can use webpack to bundle the main file and the loaded files. For your use case you don’t need complex constructs like DLLs. Just define a interface for the loaded scripts, expose it with the output.library function and load them with a script loader like scriptjs.

    Example:

    function loadDocumentation(name) {
      return new Promise(function(resolve, reject) {
        scriptjs("https://domain.com/documentation-" + name + ".js", function() {
          var doc = window["documentation-" + name];
          resolve(doc);
        });
      });
    }

    documentation part build with output.library = "documenation-" + name

  2. // dll-bundle-entry.js
    module.exports = function(request) {
      switch(request) {
        case "jquery": return require("jquery");
        // ...
      }
    };
    // or: module.exports = require.context(...)
    var bundleContentsPromise = fetch('https://domain.com/dll-bundle.js');
    var requireInBundlePromise = bundleContentsPromise.then(evalContent);
    
    requireInBundlePromise.then(function(require) {
      require("jquery");
    });

    You don’t need any special support for the runtime. You dll-bundle is just a library which exposes a require-like API, but you could use any API.

    If writing the entry point manually is too much work, you could generate it with a loader.

  3. But require.ensure automatically loads bundles, and it would be really nice to have that functionality for DllReferencePlugin as well. Something like this:

    require.ensure([], () => {
      require('somelibrary');
    });

    And in webpack config:

    plugins: [
      new DllReferencePlugin({
        context: '.',
        manifest: require('../dll-manifest.json')
      })
    ]

    Where the dll-manifest.json file contains a manifest generated by DllPlugin for a build that contains the somelibrary module.

    If the DllReferencePlugin can cause this to autoload the DLL bundle, that would be awesome.

  4. Actually, require.ensure is unrelated. It would be nice if DllReferencePlugin, or perhaps a wrapper around it, can auto-load DLL bundles when modules are require()ed from them.

  5. I need to share very large amount of code across several apps, and it’s impossible to build them together in one project(they will be developed by different people and different time). In order to use the browser cache, reduce the file size in server and client cache,DllReferencePlugin is a very good idea.

    But since shared code are in big size, load them with script tag in html will make first load very slow. supporting async loading DllReferencePlugin will resolve this.

    I know use scriptjs or other module loader can do this, even use requirejs with ‘requirejs([‘module’], a => { //todo })’. but those loaders cannot read dependency,alias and path settings from webpack,a duplicate setting is need. it is so ugly and inconvenient

  6. @CameronPlimsoll It’s my pleasure.

    1、In you dll generator webpack config,use assets-webpack-plugin to outp a JSON object to descript you output libaray, like

    module.exports = merge(baseWebpackConfig, {
         entry: utils.getLibaryEntry(),
         output: {
             path: config.libraryPath,
             jsonpFunction: `webpackJsonp${Date.now()}`,  //!! to avoid webpackJsonp conflict with reference webpack 
             filename: 'bundle.[id].js',
             chunkFilename: 'chunk.[id].[chunkhash].js',
             publicPath: '../../',   //you public Path
             libraryTarget: 'commonjs2'  //output  cmd 
         },
        plugins: [
             new AssetsPlugin({
                 filename: 'assets.json',
                 fullPath: false,
                 path: config.libraryPath,
                 metadata: {
                     version: (new Date()).getTime()
                 }
             }),
    1. In my project , I use SystemJS to load js from dll project, I wrote a simple plugin to insert SystemJS code to webpack bootstrap, but insert SystemJS to html template maybe easier. After that, you need to set map for Systemjs, like
    return fetch(mapJsonLocation)
            .then(response => response.json())
            .then((asset) => {
                const map = {}
                if (asset.metadata) {
                    delete asset.metadata
                }
                Object.keys(asset).forEach((key) => {
                    const aliasKey = `${packageName}/${key}`
                    const aliasPath = `${prefixPath}/${asset[key].js}`
                    map[aliasKey] = aliasPath
                })
                SystemJS.config({
                    map
                })
            })
    1. Now you can write in your file, Notice that async load is the only option, sync load nope, it is a little verbose
    SystemJS.import('packageName/moduleName')
      .then((module) => {
         //todo 
      })
    1. If you use requirejs, Output AMD format libary and use
     requirejs(['module'], (module) => {})
    

    UPDATE: If you use commnChunkPlugin to get a vendor output,you MUST load it before load any dll js file ! or you will get Error: (SystemJS) webpackJsonp1490526215268 is not defined LIKE @CameronPlimsoll

Comments are closed.