Zeromq usage with webpack fails

[devzzzero]

test-ui.zip
Describe the bug
I actually did manage to get zeromq.js + electron to work, sort of. At least running a single test file.

The problem began when I tried to start up zmq.Subscriber sockets in the main thread.

I then got this error:

Error: Failed to load zeromq.js addon.node: TypeError: The "path" argument must be of type string. Received undefined
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at Object.join (node:path:1244:7)

I saw #675, but I'm not convinced that my fail is related: First off, I am importing the zmq code in index.ts (the main thread), NOT in the renderer. Plus, a bunch of other bug reports (such as #673) seems to hint that zeromq and electron SHOULD work well together.
Furthermore, in a prior life, I did work on an electron app that used the built in net from node.js which worked fine.

Help!

This is in the index.ts (i.e. the main thread)

Reproducing

I started this test project by doing

1. npm init electron-app@latest test-ui -- --template=webpack-typescript
   I then added preact , zeromq, sass, and started to play with it using a single .ts file.
   Problems occurred when I imported that working zmq code to the main thread (index.ts)
2. (just unpack test-ui.zip into test-ui to get my project configs)
3. copy paste any zeromq test code into index.ts and watch it fail like above

The same class (which is just a simple Subscriber socket read -> console.log) when run separately
using npx ts-node test/krapola.ts runs just fine.
And here it is: LogRelay.ts

import * as zmq from "zeromq"

interface WatcherDesc {
  topic:string;
  sock:zmq.Subscriber;
};
class SocketMap extends Map<string,WatcherDesc> {};
class LogWatcher {
  static readonly  CATEGORIES = [ 'BUSY', 'LOGM', 'SUBQ', 'RESP' ];
  url:string;
  sockets:SocketMap;
  constructor(url:string="tcp://localhost:35501") {
    this.url = url;
    this.sockets = new SocketMap();
    for (const topic of LogWatcher.CATEGORIES) {
      const sock = new zmq.Subscriber();
      sock.connect(url);
      sock.subscribe(topic);
      console.log(`connected to ${url} for ${topic}`);
      this.sockets[topic] = { topic, sock  };
    }
    // console.log(this.sockets);
    for (const topic of LogWatcher.CATEGORIES) {
      console.log(`for topic ${topic}`, this.sockets[topic]);
    }
  };
  // ... other code cut
};

Expected behavior
I expect it to work!

Tested on

• OS: OpenSUSE 15.5
• ZeroMQ.js version: [email protected]

[aminya]

Webpack is likely replacing __dirname with undefined.

zeromq.js/src/load-addon.ts

Line 21 in 9f1172d

__dirname,

We have to provide a more robust solution that works without __dirname being needed. I plan to add ES Modules support soon, but there might be a Webpack workaround as well.

https://stackoverflow.com/questions/43527016/dirname-is-not-working-in-node-js-with-webpack-bundling

[devzzzero]

Hi, thankyou @aminya for your response.
After putting in some console.log, I found that process.arch and process.platform are both undefined. __dirname seems to be resolved. __dirname is undefined in renderer, but is defined in main. I tried commenting out the NodePolyFillPlugin stuff from webpack config, but it changed nothing.

This was after I put in `npm install --save-dev process (https://github.com/defunctzombie/node-process#readme) but it seems like that it does not provide the crucial arch/platform ....

As you saw, I had the polyfills turned on webpack.plugins.ts, (with process in the list), so perhaps the polyfill code isn't doing the right thing (or has bitrotted)

On a standalone testcode outside of webpack, they resolve to x64 and linux

What do you think should be done in this case?

Thank you.

[devzzzero]

issue webpack/18987 webpack/webpack#18987

issue Richienb/node-polyfill-webpack-plugin#58

[devzzzero]

and this: electron/forge#3763

[devzzzero]

I suppose I can kludge up something awful on linux side to get past this particular issue, but the interesting thing is what to do on Windows (?!!)

I don't yet understand how webpack and electron-forge work in Windows, and how that should interact with the polyfill stuff, and how that impacts zeromq.

Uhhh help!
Thank you.

[devzzzero]

bloody awful kludge. DON'T USE!

function findAddon() {
  let addon = undefined;
  let [theDirname, platform, arch]  = [ undefined, 'linux', 'x64'];
  try {
    theDirname = __dirname;
    platform = process.platform;
    arch = process.arch;
  } finally {
    if ((theDirname === undefined) || (platform === undefined) || (arch === undefined)) {
      theDirname = '/';
      platform = 'linux';
      arch = 'x64';
    }
  }

      const addonParentDir = path_1.default.resolve(path_1.default.join(theDirname, "..", "..",  "node_modules", "zeromq", "build", platform, arch, "node"));

I think maybe instead of trying to depend on fragile stuff like polyfill, maybe zeromq can get some magic variables from somewhere?

[devzzzero]

Just rambling now:

electron/forge#3763 and webpack/webpack#18987 and Richienb/node-polyfill-webpack-plugin#58

__dirname gets set to '/' when the app finally runs on someone's machine, apparently.
when using electron-forge+webpack, it still unclear to me exactly when/how __dirname (and other missing defaults) are supposed to be set, so that it functions during development time, and after the app is installed.

zeromq has its own non-trivial setup for compiling the .cc files and such, and so how to best navigate this complexity?

[aminya]

Can you avoid not bundling zeromq by adding it to the externals? Just copy the files over as is.

[devzzzero]

Can you avoid not bundling zeromq by adding it to the externals? Just copy the files over as is.

Almost, but not quite. renderer side hangs when I try to send stuff over from the main process using ipcMain. Still working on this avenue.

Injecting zeromq on on externals on the renderer side seems to be a no-go,
even with

      sandbox: false,
      contextIsolation:false,
      nodeIntegration: true, // <--- flag
      nodeIntegrationInWorker: true, // <---  for web workers

l am stuck with 'fs' and 'path' not being available during findAddon.
I can't seem to inject fs nor path.. The polyfill plugins are pretty much a no-op, and putting fs, process, path, etc.. on externals or on preload just errors out on the renderer.

How the heck do people get electron-forge+ webpack and zeromq to work on Windows???

[aminya]

Here's an example of Zeromq with Electron from another issue.
https://github.com/diefbell-grabcad/ZeroMQ-Electron-Refresh-Crash-Repro/tree/696892db983885d79a767f43e897d3d0d8ff7ea3