make devtool: “source-map” fast by producing concatenated sourcemaps

I’m submitting a bug report (this could be seen as a bug or a feature)
Webpack version:
1.10.x/2.x
Please tell us about your environment:
OSX 10.x / Linux / Windows 10
Current behavior:
Lots of people are currently being forced to use devtool: “source-map”, the most expensive sourcemap option, because of a bug in chrome. For example, see: #2145.
This makes the incremental build quite slow :(.
According to the source maps revision 3 spec, there is a “concatenated” version of sourcemaps:

To support concatenating generated code and other common post processing, an alternate representation of a map is supported:
{
version : 3,
file: “app.js”,
sections: [
{ offset: {line:0, column:0}, url: “url_for_part1.map” }
{ offset: {line:100, column:10}, map:
{
version : 3,
file: “section.js”,
sources: [“foo.js”, “bar.js”],
names: [“src”, “maps”, “are”, “fun”],
mappings: “AAAA,E;;ABCDE;”
}
}
],
}

The index map follow the form of the standard map
Line 1: The entire file is an JSON object.
Line 2: The version field. See the description of the standard map.
Line 3: The name field. See the description of the standard map.
Line 4: The sections field.

The “sections” field is an array of JSON objects that itself has two fields “offset” and a source map reference. “offset” is an object with two fields, “line” and “column”, that represent the offset into generated code that the referenced source map represents.
The other field must be either “url” or “map”. A “url” entry must be a URL where a source map can be found for this section and the url is resolved in the same way as the “sources” fields in the standard map. A “map” entry must be an embedded complete source map object. An embedded map does not inherit any values from the containing index map.

The sections must be sorted by starting position and the represented sections may not overlap.

Unfortunatelly, when webpack generates the sourcemap for a chunk, it just treats it as one big file. So when a chunk gets invalidated on the incremental build, it needs to rebuild the sourcemap for every module in the chunk.

Expected/desired behavior:
I think there could be a big win here if we can just produce and cache the sourcemaps for each individual module (similar to the devtool:’eval’) situation. Then, on incremental build, you just need to:

  1. regenerate the sourcemap for the modified module
  2. compute the offsets and create the “concatenated” sourcemap for the whole chunk

I think (1) can easily be achieved by adding some code in the SourceMapDevToolPlugin similar to EvalSourceMapDevToolModuleTemplatePlugin.js. Something like:

compilation.moduleTemplate.plugin("module", function(source, module) {
        if(source.__SourceMapDevToolData)
            return source;
        var sourceMap;
        var content;
        if(source.sourceAndMap) {
            var sourceAndMap = source.sourceAndMap(options);
            sourceMap = sourceAndMap.map;
            content = sourceAndMap.source;
        } else {
            sourceMap = source.map(options);
            content = source.source();
        }
        if(!sourceMap) {
            return source;
        }

        // Clone (flat) the sourcemap to ensure that the mutations below do not persist.
        sourceMap = Object.keys(sourceMap).reduce(function(obj, key) {
            obj[key] = sourceMap[key];
            return obj;
        }, {});
        var modules = sourceMap.sources.map(function(source) {
            var module = self.compilation.findModule(source);
            return module || source;
        });
        var moduleFilenames = modules.map(function(module) {
            return ModuleFilenameHelpers.createFilename(module, self.moduleFilenameTemplate, this.requestShortener);
        }, this);
        moduleFilenames = ModuleFilenameHelpers.replaceDuplicates(moduleFilenames, function(filename, i, n) {
            for(var j = 0; j < n; j++)
                filename += "*";
            return filename;
        });
        sourceMap.sources = moduleFilenames;
        if(sourceMap.sourcesContent) {
            sourceMap.sourcesContent = sourceMap.sourcesContent.map(function(content, i) {
                return content + "\n\n\n" + ModuleFilenameHelpers.createFooter(modules[i], this.requestShortener);
            }, this);
        }
        sourceMap.sourceRoot = options.sourceRoot || "";
        sourceMap.file = module.id + ".js";
        source.__SourceMapDevToolData = sourceMap;
        return source;
    });

So now each module source.__SourceMapDevToolData is a cache of the sourcemap for that module.
Part (2) is a bit trickier as you need to somehow calculate the offsets, optimally without putting all those strings together.

compilation.plugin("after-optimize-chunk-assets"
   ... compute each module source offset (I believe the "assets" are CachedSource holding a ref to a ConcatSource, so some recursive logic is probably needed here)...
  ... put together the concat sourcemap ...
...
  • What is the motivation / use case for changing the behavior?
    fast sourcemaps that work in chrome (and other browsers) 😛
  • Browser: [all]

Author: Fantashit

2 thoughts on “make devtool: “source-map” fast by producing concatenated sourcemaps

  1. allright, so I finally got some time and put together a partially hacked plugin to produce indexed sourcemaps. It seems to work pretty well.

    You will need the following dependency:
    $ npm i extend

    and you can use it like so:

    devtool: false,
    plugins: [
       ...
       new SourceMapPlugin({
            enabled: true,
            // only produce sourcemaps for files matched by this regex
            moduleAssetIncludeFilter: /\.(js|css|less)$/,
            moduleAssetExcludeFilter: /((\\|\/)node_modules(\\|\/)|\(webpack\))/,
            sourceMapStyle: {
                columns: false
            }
        })
    ]

    Note: I don’t mind anybody taking bits of this code and doing a proper implementation in webpack.

    SourceMapPlugin.js:

    "use strict";
    
    const extend = require('extend');
    const ModuleFilenameHelpers = require("webpack/lib/ModuleFilenameHelpers");
    const ConcatSource = require("webpack-core/lib/ConcatSource");
    const RawSource = require("webpack-core/lib/RawSource");
    const path = require("path");
    
    const PrefixSource = require("webpack-core/lib/PrefixSource");
    const util = require('util');
    
    /**
     * This plugin is used to generate faster (indexed) sourcemaps
     * @param  {object} options
     *     - enabled {boolean} whether plugin is enabled
     *     - moduleFilenameTemplate {string} the filename that shows up in the chrome dev tools for each module
     *     - jsSourceMappingURLComment {string} the sourcemapping comment to use for js files ("[url]" gets relaced by actual url)
     *     - cssSourceMappingURLComment {string} the sourcemapping comment to use for css files ("[url]" gets relaced by actual url)
     *     - moduleAssetIncludeFilter {regex} controls the module assets to include in sourcemapping (js and css by default)
     *     - moduleAssetExcludeFilter {regex} controls the module assets to exclude in sourcemapping
     *     - sourceMapStyle.columns {boolean} if true, sourcemaps also do column mapping instead of just lines (more expensive)
     */
    var SourceMapPlugin = module.exports = function(options) {
        this.options = extend(true, {
            enabled: true,
            moduleFilenameTemplate: "webpack:///[resource-path]",
            chunkAssetFilter: /\.(js|css)($|\?)/i,
            jsSourceMappingURLComment: "\n//# sourceMappingURL=[url]",
            cssSourceMappingURLComment: "\n/*# sourceMappingURL=[url] */",
            moduleAssetIncludeFilter: /\.(js|css)$/,
            moduleAssetExcludeFilter: undefined,
            sourceMapStyle: {
                columns: false
            }
        }, options);
    };
    
    SourceMapPlugin.prototype.apply = function(compiler) {
        var options = this.options;
        if (options.enabled === false) {
            return;
        }
    
        compiler.plugin("compilation", function(compilation) {
            // when a module is compiled, also generate the associated sourcemap
            // Note: the following plugin code is a slight modification of the one in EvalSourceMapDevToolModuleTemplatePlugin
            compilation.moduleTemplate.plugin("module", function(source, module) {
                if (module.__moduleSourcemap) {
                    source.__moduleSourcemap = module.__moduleSourcemap;
                    return source;
                }
    
                // only produce sourcemaps for the things we care about
                if (options.moduleAssetExcludeFilter && options.moduleAssetExcludeFilter.test(module.resource)) {
                    return source;
                }
                if (!options.moduleAssetIncludeFilter.test(module.resource)) {
                    return source;
                }
    
                var sourceMap;
                var content;
                if(source.sourceAndMap) {
                    var sourceAndMap = source.sourceAndMap(options.sourceMapStyle);
                    sourceMap = sourceAndMap.map;
                    content = sourceAndMap.source;
                } else {
                    sourceMap = source.map(options.sourceMapStyle);
                    content = source.source();
                }
                if(!sourceMap) {
                    return source;
                }
    
                // TODO: not sure why some of the following code is necessary as there is only one module...
    
                // Clone (flat) the sourcemap to ensure that the mutations below do not persist.
                sourceMap = Object.keys(sourceMap).reduce(function(obj, key) {
                    obj[key] = sourceMap[key];
                    return obj;
                }, {});
                var modules = sourceMap.sources.map(function(source) {
                    var module = compilation.findModule(source);
                    return module || source;
                });
    
                var moduleFilenames = modules.map(function(module) {
                    return ModuleFilenameHelpers.createFilename(module, options.moduleFilenameTemplate, this.requestShortener);
                }, this);
                moduleFilenames = ModuleFilenameHelpers.replaceDuplicates(moduleFilenames, function(filename, i, n) {
                    for(var j = 0; j < n; j++) {
                        filename += "*";
                    }
                    return filename;
                });
                sourceMap.sources = moduleFilenames;
    
                if(sourceMap.sourcesContent) {
                    sourceMap.sourcesContent = sourceMap.sourcesContent.map(function(content, i) {
                        return content + "\n\n\n" + ModuleFilenameHelpers.createFooter(modules[i], this.requestShortener);
                    }, this);
                }
                sourceMap.sourceRoot = options.sourceRoot || "";
                sourceMap.file = module.id + ".js";
                module.__moduleSourcemap = source.__moduleSourcemap = sourceMap;
                 // console.log("### producing sourcemap for asset ", moduleFilenames);
                return source;
            });
    
            // after chunks are produced, create an indexed sourcemap for each chunk by putting together the sourcemaps
            // coming from constituting modules
            compilation.plugin("after-optimize-chunk-assets", function(chunks) {
                var shouldProduceSourcemapsForAsset = ModuleFilenameHelpers.matchObject.bind(undefined, { test: options.chunkAssetFilter });
                for (let chunk of chunks) {
                    for (let file of chunk.files) {
                        if (!shouldProduceSourcemapsForAsset(file)) {
                            continue;
                        }
    
                        var asset = compilation.assets[file];
    
                        // check if we have the file/sourcemap in the cache and use them if yes
                        if (asset.__sourceMapDevToolData) {
                            var data = asset.__sourceMapDevToolData;
                            for (var cachedFile in data) {
                                compilation.assets[cachedFile] = data[cachedFile];
                                if (cachedFile !== file) {
                                    chunk.files.push(cachedFile);
                                }
                            }
                            continue;
                        }
    
                        // console.log("!!! producing sourcemaps for chunk: ", file);
    
                        var assetSourceMap = {
                            version: 3,
                            file: file,
                            sections: []
                        };
    
                        visitAssetSources(asset, {line: 0, column: 0}, function(asset, offset) {
                            if (asset.__moduleSourcemap) {
                                assetSourceMap.sections.push({
                                    offset: Object.assign({}, offset), // clone offset as it gets modified
                                    map: asset.__moduleSourcemap
                                });
                            }
                        });
    
                        asset.__sourceMapDevToolData = {};
                        if (assetSourceMap.sections.length === 0) {
                            continue; // sourcemap not necessary
                        }
    
                        // compute sourcemap filename
                        var filename = file,
                            query = "";
                        var idx = filename.indexOf("?");
                        if(idx >= 0) {
                            query = filename.substr(idx);
                            filename = filename.substr(0, idx);
                        }
                        var sourceMapFile = compilation.getPath(compiler.options.output.sourceMapFilename, {
                            chunk: chunk,
                            filename: filename,
                            query: query,
                            basename: path.basename(filename)
                        });
                        var sourceMapUrl = path.relative(path.dirname(file), sourceMapFile).replace(/\\/g, "/");
    
                        // update source with sourceMappingURL comment and add asset for sourcemap
                        var currentSourceMappingURLComment = /\.css($|\?)/i.test(file) ? options.cssSourceMappingURLComment
                                                                                       : options.jsSourceMappingURLComment;
                        asset.__sourceMapDevToolData[file] = compilation.assets[file] = new RawSource(
                            new ConcatSource(asset, currentSourceMappingURLComment.replace(/\[url\]/g, sourceMapUrl)).source()
                        );
                        asset.__sourceMapDevToolData[sourceMapFile] = compilation.assets[sourceMapFile] = new RawSource(JSON.stringify(assetSourceMap));
                        chunk.files.push(sourceMapFile);
                    }
    
                }
            }.bind(this));
        }.bind(this));
    };
    
    /**
     * visits a nested asset source structure recursively, computing the offset of the current sub-source within the
     * overall asset and invoking the callback.
     * @param  {webpack Source} asset
     * @param  {offset obj} offset {line:<int>, column:<int>}
     * @param  {function(<offset obj> offset, <webpack Source> asset)} fnCallback - visitor function
     */
    function visitAssetSources(asset, offset, fnCallback) {
        // Note: instanceOf was not used everywhere here because there are multiple versions of those files.
        // One version is pulled in by extract-text-plugin dependency webpack-sources, the other is from webpack-core
        if (typeof asset === "string") {
            incrementOffset(offset, asset);
        } else if (asset.constructor.name === "CachedSource") {
            fnCallback(asset, offset);
            visitAssetSources(asset._source, offset, fnCallback);
        } else if (asset.constructor.name === "ConcatSource") {
            fnCallback(asset, offset);
            for (let childAsset of asset.children) {
                visitAssetSources(childAsset, offset, fnCallback);
            }
        } else if (asset.constructor.name === "PrefixSource") {
            fnCallback(asset, offset);
            if (typeof asset._source === "string") {
                incrementOffset(offset, asset.source());
            } else {
                visitAssetSources(asset._source, offset, fnCallback);
            }
        } else if (typeof asset.source === "function") {
            fnCallback(asset, offset);
            incrementOffset(offset, asset.source());
        } else {
            throw new Error("SourceMapPlugin::visitAssetSources() - unrecognized asset type encountered: " + asset.constructor.name);
        }
    }
    
    /**
     * increments given offset (line/column) according to given string as if the string was being appended
     * Note that the offset argument is modified.
     * @param  {offset obj} offset {line:<int>, column:<int>}
     * @param  {string} str
     * @return {offset obj}
     */
    function incrementOffset(offset, str) {
        for (let i = 0; i >= 0; ) {
            var j = str.indexOf("\n", i);
            if (j >= 0) {
                offset.column = 0;
                offset.line += 1;
                i = j + 1;
            } else {
                offset.column += str.length - i;
                i = j;
            }
        }
    
        return offset;
    }
    
    // we need to patch PrefixSource for 2 reasons:
    // 1. all the unnecessary string replacement are performance intensive
    // 2. ability to easily visit the wrapped source
    // Note that this patching will basically disable the ability to add prefixes before each line (which by defaults adds a '\t' char)
    // ... not really as important as speed in our case
    PrefixSource.prototype.source = function() {
        return (typeof this._source === "string") ? this._source : this._source.source();
    };
    
    PrefixSource.prototype.node = function(options) {
        return this._source.node(options);
    };
    
    PrefixSource.prototype.listMap = function(options) {
        return this._source.listMap(options);
    };
    
    PrefixSource.prototype.updateHash = function(hash) {
        if (typeof this._source === "string") {
            hash.update(this._source);
        } else {
            this._source.updateHash(hash);
        }
    };

Comments are closed.