Best way to handle a bad connection?

[ianforsyth]

Hi there!

I'm trying to make sure I properly handle the case where BullMQ can't establish a connection to redis either because it's down or misconfigured. I'd ideally like to catch the error and execute some other code, not have it error on the server.

If I just try to create a queue on a bad connection like this:
const queue = new Queue(queueName, { connection });

I get this:

Error: Promise {
  <rejected> Error: getaddrinfo ENOTFOUND bad-host-name
      at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'bad-host-name'
  }
}
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
Error: Promise {
  <rejected> Error: getaddrinfo ENOTFOUND bad-host-name
      at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'bad-host-name'
  }
}
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)
[ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)

And it seems like ioredis will continuously retry from there.

I looked in the source and queues appear to emit an error event when the connection fails so I updated to this:

  const queue = new Queue(queueName, CONNECTION);
  queue.on('error', (err) => {
    console.log('Error occured');
    queue.close();
  });

And that gets me to:

Error occured
Error: Promise {
  <rejected> Error: getaddrinfo ENOTFOUND bad-host-name
      at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'bad-host-name'
  }
}
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: getaddrinfo ENOTFOUND bad-host-name
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)

And there are no more retries because I'm closing the queue. But is there a recommended way to catch that other error and handle it gracefully?

[nguernse]

Hello! I wanted to revive this discussion, as I had a similar question regarding bad connections, and what the best practice should be.

Prereqs

I have [email protected] installed on MacOS.

Scenario

I have the following running:

• API / Queue to add jobs
• Workers to process jobs
• Redis queue

On the Worker service I am following the advice for gracefully shutting down workers:

const gracefulShutdown = async (signal: string) => {
  console.log(`Received ${signal}, closing server...`);

  try {
    console.log("Gracefully closing worker...");
    await worker.close();
    console.log("Worker closed...");
  } catch (error: any) {
    console.error("Failed to close worker....", error.message);
    process.exit(1);
  }

  // Other asynchronous closings
  process.exit(0);
};

process.on("SIGINT", () => gracefulShutdown("SIGINT"));
process.on("SIGTERM", () => gracefulShutdown("SIGTERM"));

Situation A

If I have all services up and running, and I provide a valid Redis host to the worker in the config: { connection: { host: <valid-host>, port: 6379}}, then in the terminal where I am running the Worker service I do ctrl+c.

The results are as expected and it gracefully shuts down.

Likewise, if I spin up everything, then I shut down the Redis server. The worker service will begin logging ECONNREFUSED errors. And if I do the same ctrl+c, it gracefully shuts down and the connection error stops.

Situation B

If I have all three running, but I provide an initially invalid Redis host to the worker in the config: { connection: { host: "bad", port: 6379}}, then I do the same ctrl+c, it does not gracefully shut down.

It will try to call the worker.close() but the worker keeps trying to connect to a bad connection infinitely, with ECONNREFUSED. In this situation it'll log Gracefully closing worker, but it will not reach Worker closed or any of the code below.

Eventually in the terminal it'll just keep logging the ECONNREFUSED errors until I stop the node process in the Activity Monitor.

Furthermore, if I update to a valid host and re-run the Worker service, it will keep running the errors in the background while the new service is up and running as expected.

Question

How would you handle Situation B? It seems like even after trying to do worker.close() it will continue trying to connect to the bad connection, unless doing a complete reboot. Maybe I am overthinking this?

[manast]

I think situation B should work as A. The fact that the host is invalid when the worker is instantiated should not change the mechanics of error handling, retrying and closing from when the worker did start from a working connection and then the connection for some reason stopped working. If you could create a minimal case that reflects the B situation we can look more into it to see if there is an issue with BullMQ.

[nguernse]

Sure, I've made a minimal case here: https://github.com/nguernse/bullmq-connection-bug. I've put steps to repeat in the Readme. Let me know if there's anything I can do to help.