WakuV2 filter debugging and refactor

[vitvly]

Some of the message deliverability issues listed here (status-im/status-mobile#16873) are related to filter operation.

Investigations have been previously carried out in this issue: status-im/status-mobile#17091 (comment). Seems that in that particular case peer connection between filter server and filter client was dropped and not resurrected for some reason.

This issue exists for the purpose of tracking ongoing investigations.

[vitvly]

Related PR: #4020

Extra debug output has been added.

Code analysis has uncovered potential problems that are being addressed in that PR:

• access to filter-to-subscription data map was not properly synchronised
• not-so-nice timed loop in Subscribe() that might result in many repeated subscribe calls
• there existed a possibility of multiple subscribes with similar ContentTopics to the same peer. PR to address these issues here: feat: topic refactors for filter client waku-org/go-waku#750.

[vitvly]

Update: fixed WakuV2Filter test issues in #4020, added some extra logs.
There is new build tag added with recent go-waku changes - gowaku_no_rln, need to fix status-go build.

[vitvly]

status-go PR build fixed
waku-org/go-waku#750 merged

[vitvly]

Changes in status-go PR #4020 - replace weird callbacks with channels passing Filter objects, thus all Subscribes occur from within the same goroutine.

[vitvly]

Created a status-mobile PR to run some client-side tests on: status-im/status-mobile#17344

Fixing status-go mobile build issues that surfaced there.

[vitvly]

After meeting with @richard-ramos and @cammellos it was decided to

• allow multiple subscriptions to the same [peer, contentFilter]
• fix filter.client.UnsubscribeWithSubscription so that it allows fine-grained unsubscribes for a particular subscription

[vitvly]

Added a go-waku PR implementing changes discussed on yesterday's meeting: waku-org/go-waku#762

[vitvly]

Tiny PR: waku-org/go-waku#766, rebased waku-org/go-waku#762. Need to fix some race condition in TestWakuFilter.

Relevant changes for multiple subs and sub-specific unsubscribe have been added to https://github.com/status-im/status-go/pulls/4020.

[vitvly]

Fixed race condition in tests in waku-org/go-waku#762, merged. Updated status-go PR https://github.com/status-im/status-go/pulls/4020 to point to updated go-waku.

[vitvly]

Testing mobile PR status-im/status-mobile#17344, fixing issues discovered there.

[vitvly]

Progress update: status-im/status-mobile#16873 (comment)

[vitvly]

Refactored health checks again, now these a single event loop.

[vitvly]

Merged in #4082 and status-im/status-mobile#17472.

There are still issues with community sending, as there is no retry. To be investigated.

[vitvly]

Recent updates:

• some issues were found with new event loop code, fixed here: Test ping peers fix for filter status-mobile#17604
• after looking through logs it turned out that LightPush was not enabled on nwaku fleets, fixed here: store: set logrotate frequency to hourly infra-status#32

[vitvly]

I think it's safe to close for now, will be replaced by more specific issues later on.