Problems with loading the Web Assembly module when using code bundlers (Vite, Webpack)

This issue has been tracked since 2021-10-16.

I'm playing around in a react project created with create-react-app (which uses webpack for bundling) and between how Vizzu is trying to locate its WASM and how webpack bundles it, I can't figure out an elegant way to get it to work.

I realize this is not an issue with Vizzu per se, but wanted to highlight it, as it will be important for the adoption.

Details as I understand them right now.
The vizzu js module tries to import its own wasm module based on its own import path.
image

In my case this resolves to http://localhost:3000/node_modules/vizzu/dist/cvizzu.wasm
image

This file path is not recognized by the local dev server (as it only bundles .js/.ts files by default)
Since the path is not recognized it serves the main webpage I'm trying to develop to test the module.
image

The wasm parser naturally fails to parse this and I get an empty page.
image

I know that there is a way to update the webpack config in order to serve the wasm file on this exact path, which is a half-decent solution but I don't (yet) know how to do it. I'll add details here once I have them

korompaiistvan wrote this answer on 2021-10-16

There seem to be two problems:

  1. By default the webpack config of CRA does not load .wasm files
  2. Even if it did (or one updated the config so it did), it wouldn't load cvizzu.wasm file, as it's not the target of any require or import statements

Potential solutions are:

  1. change the import of the wasm in the js file, so webpack (or any other bundler) can pick up the reference to wasm. they cannot do this because of the reliance on import.meta.url to construct the require path of the wasm. I assume this is needed for non-npm uses, e.g. when served from a cdn, so this is not straightforward
  2. referencing the wasm from the application itself without any real use just so webpack reaches it. In this scenario the output path would need to be fixed to the one the .js file looks for
christopher-caldwell wrote this answer on 2021-10-17

I'm having a similar issue using Vite. The request to wasm returns 404 as it makes the request to http://localhost/cvizzu.wasm.

This is also not something wrong with the lib, more so needing some experimentation and a "how to use with x". I am still trying to figure it out and will write up a guide when I do.

korompaiistvan wrote this answer on 2021-10-17

This has left me quite frustrated. Apps created with create-react-app cannot seem to elegantly solve this problem, so the most straightforward solution is adding a step to one's build process which moves the .wasm file into the public folder (which is copied and served as-is after bundling).

Create-react-app now uses Webpack 5 which might solve this, as it has a built-in WASM loader (I've been trying to solve it using webpack 4). At some point I'll give that a try to see if it works, but for now I just admitted defeat and copied the wasm file to my static folder.

In general, I think it would be really valuable to set up a working example with at least one bundler and document it

korompaiistvan wrote this answer on 2021-10-18

I gave it a go on NextJS which uses webpack 5.
It gives me the following error:
image

Updating vizzu's package.json (adding the type: "module" line to it) solves this error but introduces another
image

In my understanding the core issue is the usage of BOTH import.meta and require(), which makes the npm package a weird mix of sorta-an-ES6-module-but-not-really.

I believe the issue can be solved in two ways:

  1. Avoiding import.meta -> Adding the USE_ES6_IMPORT_META=0 flag to the emscripten build, which would avoid the use of import.meta. This should be the safer solution as it would work with older toolchains, e.g. webpack 4.
  2. Avoiding 'require()' -> Adding the WASM_ASYNC_COMPILATION=0 MODULARIZE=1 EXPORT_ES6=1 flags to the build instead, which would export a "proper" ES6 module where import.meta can be used. (Caveat: I'm not sure that this is the right combination of flags, and what the potential side effects may be for other environments)
christopher-caldwell wrote this answer on 2021-10-18

I've got a Vite example to go off. Not a bad starting place.

Have to copy the wasm to the public/, but eh.

korompaiistvan wrote this answer on 2021-10-18

yea for now, that's the best course of action. I'm 100% that we'll figure this out so future users won't have a problem

simzer wrote this answer on 2021-10-18

I've got a Vite example to go off. Not a bad starting place.

Thanks, this is great! I have added a markdown file to the project (and a link to it into the readme) to list 3rd party projects:
https://github.com/vizzuhq/vizzu-lib/blob/main/PROJECTS.md
If you could put a link to your repo along with a description line there, that would be great!

simzer wrote this answer on 2021-10-18

Never mind, I have put in the link (got the description from the repo).

simzer wrote this answer on 2021-10-23

Unfortunately, USE_ES6_IMPORT_META=0 introduces another problem, this way it tries to load the wasm relative to the HTML document, not to the cvizzu.js.

simzer wrote this answer on 2021-10-23

Another solution could have been to create a single file (using SINGLE_FILE=1 emscripten flag), but that would increase the package size by ~25% due to the base64 encoding of the wasm file.

korompaiistvan wrote this answer on 2021-10-25

that's a shame. how about the modularization option?

Avoiding 'require()' -> Adding the WASM_ASYNC_COMPILATION=0 MODULARIZE=1 EXPORT_ES6=1 flags to the build instead, which would export a "proper" ES6 module where import.meta can be used. (Caveat: I'm not sure that this is the right combination of flags, and what the potential side effects may be for other environments)

I did some more digging and everyone on webpack 4 is doing the same thing Christopher and I did (explicitly copying the .wasm file to the right place) just with different methods.

simzer wrote this answer on 2021-10-25

We use the MODULARIZE=1 EXPORT_ES6=1 flags currently, I'm not sure that the WASM_ASYNC_COMPILATION=0 flag would make any difference, but we can give it a try.

simzer wrote this answer on 2021-10-25

WASM_ASYNC_COMPILATION=0 seems to have no effect on the problematic part of the generated code.
The file begins with the same code:

    var VizzuModule = (function() {
        var _scriptDir = import.meta.url;

There are two other problems with this flag. It won't work with the ASSERTIONS=0 flag which I would like to introduce in the next release, shaving off ~80kb of the generated JS code with it.
There is also a wasm size limit in Chrome which won't allow compilation of bigger wasm modules on the main thread.

simzer wrote this answer on 2021-10-25

Maybe we could modify the generated code in a post-build step? Would a conditional use of the import.meta object solve the problem? If so, could you create a patch to the generated file which works with CRA/Webpack?

korompaiistvan wrote this answer on 2022-02-26

I have some renewed motivation in trying to resolve this. is there way to get non-minified js out of the generation process?
I think one of the main problems webpack has is that there is no outright require or import statement in the js so "it doesn't know" that the .wasm file is used at all

korompaiistvan wrote this answer on 2022-02-26

I just found this repo, between that and Christopher's solution, I'll come up with something fairly easy to set up. the more I read about this the more I think it can be solved on the Emscripten side

simzer wrote this answer on 2022-02-26

@korompaiistvan
On this url you can find the build of the most recent development version:
https://vizzu-lib-main.storage.googleapis.com/lib/vizzu.js

simzer wrote this answer on 2022-02-26

But I'm afraid the emscripten wrapper is still not readable in this version:
https://vizzu-lib-main.storage.googleapis.com/lib/cvizzu.js

If you need a debug build (emscripten generates non-minified code for debug build), then I can produce that for you.

simzer wrote this answer on 2022-02-26

If you would like to experiment with the emscripten configuration, and build the project, then
https://github.com/vizzuhq/vizzu-lib/blob/main/project/build.md
This is the way! :)

simzer wrote this answer on 2022-02-27

@korompaiistvan
Thank you for providing the example and the tutorial on how to use Vizzu with React!
https://vizzuhq.github.io/vizzu-react-example/
https://github.com/vizzuhq/vizzu-react-example

simzer wrote this answer on 2022-03-10

@korompaiistvan
I have made the url of the wasm file configurable in Vizzu:

Vizzu.options({ wasmUrl: './whatever-folder/cvizzu.wasm' });

How much would this help with this bundler issue?

simzer wrote this answer on 2022-03-10

I have managed to reduce the problem. Now integrating Vizzu with Webpack needs only the steps listed here:
https://github.com/simzer/vizzu-webpack-demo/blob/main/README.md#adding-vizzu-to-a-project-which-using-webpack

korompaiistvan wrote this answer on 2022-03-10

i think that solution in the readme is a decent compromise. not as elegant as it would be if it worked automatically, but basically the next best thing

simzer wrote this answer on 2022-03-10

@korompaiistvan
Thanks, we will put this into our tutorial with some explanation then.
I will also try to update https://github.com/vizzuhq/vizzu-react-example and send a PR for you if you'd be willing to do a review.

simzer wrote this answer on 2022-03-11

I was unable to update the react example.
If I use the solution from the webpack-demo, it won't work with npm start.
However, I was able to build a correctly working version with npm run build.

korompaiistvan wrote this answer on 2022-03-11

that's probably got to do something with how CRA deals with webpack config (badly).
did you modify the config overrides js file or create a separate webpack config file?

also as a note: webpack devserver (npm start) and build works slightly differently so there might be a difference there as well.

I'm also wondering about postprocessing your npm package instead so it includes this explicit import but im not sure how exactly

simzer wrote this answer on 2022-03-11

I have modified the config-override.js file.

simzer wrote this answer on 2022-03-11

I have tried devserver (webpack serve) in the webpack-demo repo, and it works there.

simzer wrote this answer on 2022-03-11

Well, maybe the problem was that the cvizzu.wasm and vizzu.min.js were overwritten by a local build. Now that the necessary changes are released in 0.4.5, npm start is working.

simzer wrote this answer on 2022-03-12

PR sent to https://github.com/vizzuhq/vizzu-react-example. Closing this issue.

More Details About Repo
Owner Name vizzuhq
Repo Name vizzu-lib
Full Name vizzuhq/vizzu-lib
Language JavaScript
Created Date 2021-06-07
Updated Date 2022-11-22
Star Count 1630
Watcher Count 19
Fork Count 65
Issue Count 17

YOU MAY BE INTERESTED

Issue Title Created Date Updated Date