nsqlookupd: fast push notification of topic availability

[buguang01]

When using lookup to determine the receiving connection of a new topic, if there is no topic, there will be an error. Then the listening is done by HTTP. Is there a B solution?

[buguang01]

My current solution is to reduce the time required for this HTTP request, but it feels terrible.

[buguang01]

@ploxiln hello?

[ploxiln]

It sounds like you want a way to be notified of new topics appearing on various nsqd immediately, instead of polling for them. We do not currently have such a feature. This sounds like it could be a reasonable feature, but it would be a change in the api between the consumer and nsqlookupd. Maybe an optional "http-long-polling" mode would be useful and not too hard. Or a custom tcp protocol thing.

For many common uses of nsq, the topics availability do not change very often. The same list of topics might exist for months without changing, and it is often multiple days between a new nsqd instance appearing with an existing topic. Having each make a brief request every 30 seconds or so to poll for the availability of a topic is an efficient way to handle many thousands of consumers with just a few redundant nsqlookupd instances without too much resources. If these connections have longer-lived state, that may use more resources, which may be problematic for some existing setups. For many it may not be a problem at all. But it would need to be implemented by nsq consumer libraries to be helpful.

[buguang01]

Hope to have this function as soon as possible.

There's also a question, I have N * nsqd, 3 * lookup, and each nsqd will connect to each lookup, which seems to be the configuration you want, right?

And my customers will connect to each lookup at the same time.

[stair-go]

Sorry my English is not very good. I have a similar problem

Polling query can be tolerated, but this can happen:

A, B and C all subscribe to a D topic, but the D topic has not been created yet, which will continue to query from nsqlookupd,

At this time, another service sends 100 messages to D topic. At this time, service a may connect to the nsqd where the topic is located and consume all the messages. As a result, although B and C have subscribed, they have not received the data

This is not only the case where the topic is not created, but also the case when the topic is sent to the new nsqd, resulting in message loss. Is there any solution or my usage error?

[buguang01]

Sorry my English is not very good. I have a similar problem

Polling query can be tolerated, but this can happen:

A, B and C all subscribe to a D topic, but the D topic has not been created yet, which will continue to query from nsqlookupd,

At this time, another service sends 100 messages to D topic. At this time, service a may connect to the nsqd where the topic is located and consume all the messages. As a result, although B and C have subscribed, they have not received the data

This is not only the case where the topic is not created, but also the case when the topic is sent to the new nsqd, resulting in message loss. Is there any solution or my usage error?

我现在的做法就是如果我这个服务监听某个主题，那么在发现还没有这个主题的时候，自己发一个这个主题的空消息（也就是类似心跳啊什么的，不处理的消息），这样就可以在nsq上面创建好主题了。

[ploxiln]

If a topic already exists on an older nsqd (and has consumers and channels there), when it is newly published on a new nsqd, the new nsqd will query nsqlookupd to get the existing channels present on older nsqd, and will pre-create all existing channels, so no channels miss any messages.

If a topic does not exist on any older nsqd but there are multiple consumers waiting to consume it on different channels, you should use the nsqd http api to pre-create the topic on the new nsqd a couple minutes before publishing starts. Or, you can use the nsqlookupd http api to pre-create the topic and all channels just a couple seconds before publishing starts.

[stair-go]

Thank you for the answer. I think I've found a solution.
Thank you very much ~