No centrality matches

[deepy420]

I created a fresh describegraph.json file and had the script generate its own improvecentrality.conf file, yet I still don't get any matches. Very odd.

I do see matches when I check a different tool:
https://www.moneni.com/nodematch?node=03cd502dee15bb9859a6d9c56a5608c1f76894736ec4cfec906eb025b7623293dc

Loaded graph. Number of nodes: 19004 Edges: 82124
Latest update in graph: Mon Oct 11 08:21:45 2021
Simplified graph. Number of nodes: 11871 Edges: 4423
Performing analysis for BitcoinRevolution
First filtering pass found 0 candidates for new channels
Running modified farness score calculations
Completed modified farness score calculations in 1.7s
Checking 1ML availability statistics
1ML availability filter selected 0 candidates for new channels in 0.0s
No candidates found, is your graph stale?
If issue persists, delete describegraph.json and improvecentrality.conf
Traceback (most recent call last):
File "scripts/lndpytools/improvecentrality.py", line 248, in
main()
File "scripts/lndpytools/improvecentrality.py", line 239, in main
raise ValueError('No valid candidates')
ValueError: No valid candidates

[Gridflare]

Are you using the default configuration?
If not please share the contents of improvecentrality.conf

[deepy420]

Yes, it's the default configuration.

[Gridflare]

I cannot replicate your issue when reading the graph directly from lnd

Fetching graph data from lnd
Loaded graph. Number of nodes: 12089 Edges: 61995
Latest update in graph: Mon Oct 11 10:25:40 2021
Simplified graph. Number of nodes: 11236 Edges: 27852
Performing analysis for BitcoinRevolution
First filtering pass found 613 candidates for new channels
Running modified farness score calculations
Completed modified farness score calculations in 6.5s
Checking 1ML availability statistics
Failed to fetch availability for 03e3fc2b4f0aeb2e17866b4134e3fb0800ab11ca54f30ccc8b6d927209d2f57602
1ML availability filter selected 35 candidates for new channels in 0.9s
Starting baseline centrality computation
Queuing computations for new centralities
Waiting for baseline centrality calculation to complete
Our current centrality is approximately 14395
Collecting centrality results, this may take a while
Completed centrality difference calculations in 16.8s
delta MFscor Avail Relbty Alias Pubkey
|    | delta   |   MFscor |   Avail | Relbty   | Alias                 | Pubkey                                                             |
|---:|:--------|---------:|--------:|:---------|:----------------------|:-------------------------------------------------------------------|
|  0 | 13.1%   |  18.9401 |     366 | 100.0%   | SoulexBoy_BXXI        | 024e0d88d5cf3ac6df50697ac95be7fde0ac3017bc0a918edbfdb70ba70b00c17f |
|  1 | 8.0%    |  18.0633 |     562 | 100.0%   | PocketBitcoin.com     | 02765a281bd188e80a89e6ea5092dcb8ebaaa5c5da341e64327e3fadbadcbc686c |
|  2 | 7.7%    |  18.386  |     670 | 100.0%   | ln1.lightning.co.uk   | 021bc2416752fb15c3b740187669afc2514426e41caf2828978ad613f7e61370a0 |
|  3 | 7.4%    |  18.8287 |     630 | 100.0%   | CommonLawIsTheAnswer  | 02067f33c1b0f0cd79ca9bd2df85af0c5655d19726bd16dcaf3b60c39b59d3067a |
|  4 | 6.9%    |  31.5345 |       5 | 97.7%    | ACINQ                 | 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f |
|  5 | 6.3%    |  18.6665 |    1753 | 100.0%   | Strato                | 038b9c4b9a0d6f1381ac0e44f6f83da059d0c9db1faf7395961e54d254e3571f11 |
|  6 | 5.9%    |  18.3921 |    1531 | 100.0%   | CloudNalu             | 03e47facaeb2504a8c196ae34314cd05b9bc480278f649aeaeaed8b165b03f28ea |
|  7 | 5.7%    |  20.1541 |      45 | 99.0%    | DiamondHands💎🙌      | 035b1ff29e8db1ba8f2a4f4f95db239b54069cb949b8cde329418e2a83da4f1b30 |
|  8 | 5.4%    |  18.774  |     649 | 98.5%    | ThunderHub⚡          | 02f63f49339c8b438c3291ab21e35d1b5642ac2360240068b5ecd3fd5183f2c042 |
|  9 | 5.2%    |  18.9486 |     305 | 97.1%    | Einundzwanzig         | 03a09f56bba3d2c200cc55eda2f1f069564a97c1fb74345e1560e2868a8ab3d7d0 |
| 10 | 5.1%    |  18.7679 |     145 | 100.0%   | LightningNetwork.Plus | 0302536817c335e45bcd0b10b78fd06bc246609405d98eb64672410dce04f49e5f |
| 11 | 4.9%    |  18.4425 |     300 | 97.8%    | FatBear.io            | 020593994806de7eb9a8ae63d1687bd43abebdd158ac8fcb426ece6571980e09d8 |
| 12 | 4.6%    |  18.1578 |     886 | 100.0%   | BTC-LND               | 032b96613d99075560051f6312e34ca6da5b445bee338613df85389e97aed516f4 |
| 13 | 4.4%    |  18.3995 |     761 | 100.0%   | Orange🍊Vibe          | 0215986537a6ca91ee696f1431ec5c09aeb03494a99104407793bbf1a26c324298 |
| 14 | 4.3%    |  18.0064 |    1896 | 100.0%   | jayber                | 0253c0bcd2105d41efa6925488ec428255ae469d2e90277cc105a401ec8e343b7b |
| 15 | 4.0%    |  19.0106 |     136 | 100.0%   | LNBIG.com [lnd-33]    | 028a8e53d70bc0eb7b5660943582f10b7fd6c727a78ad819ba8d45d6a638432c49 |
| 16 | 3.8%    |  18.3917 |     418 | 97.9%    | WizardofAus           | 0283de1355ab232080de2904edc77388c6651e2d81a0336bfb739f893b35aa76ce |
| 17 | 3.6%    |  19.2679 |      34 | 99.4%    | lnmarkets.com         | 03271338633d2d37b285dae4df40b413d8c6c791fbee7797bc5dc70812196d7d5c |
| 18 | 3.6%    |  18.6892 |     150 | 98.9%    | LNBIG.com [lnd-27]    | 03fce165537aea120bffe8505876b44d5119354f825b3eac329b761fc5636bf334 |
| 19 | 3.4%    |  18.453  |     212 | 100.0%   | LNBIG.com [lnd-31]    | 03e5ea100e6b1ef3959f79627cb575606b19071235c48b3e7f9808ebcd6d12e87d |
| 20 | 3.2%    |  20.0778 |      66 | 99.4%    | LNBIG.com [lnd-11]    | 033e9ce4e8f0e68f7db49ffb6b9eecc10605f3f3fcb3c630545887749ab515b9c7 |
| 21 | 2.9%    |  18.705  |      91 | 100.0%   | LNBIG.com [lnd-39]    | 02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490 |
| 22 | 2.9%    |  19.1096 |     115 | 99.1%    | LNBIG.com [lnd-10]    | 02bb24da3d0fb0793f4918c7599f973cc402f0912ec3fb530470f1fc08bdd6ecb5 |
| 23 | 2.8%    |  18.9364 |      53 | 98.8%    | LNBIG.com [lnd-12]    | 034ea80f8b148c750463546bd999bf7321a0e6dfc60aaf84bd0400a2e8d376c0d5 |
| 24 | 2.7%    |  19.552  |      92 | 100.0%   | LNBIG.com [lnd-09]    | 02de11c748f5b25cfd2ce801176d3926bfde4de23b1ff43e692a5b76cf06805e4a |
| 25 | 2.5%    |  18.1712 |     272 | 100.0%   | Lightning.Store       | 02bfe975eb18b29455ee6d5c3a2dc6c8097cb4d9a38fb440ded1e464b33ccc9f60 |
| 26 | 2.3%    |  18.697  |     613 | 98.7%    | LPU                   | 0331f80652fb840239df8dc99205792bba2e559a05469915804c08420230e23c7c |
| 27 | 2.3%    |  18.7416 |     449 | 98.5%    | midoric               | 020783a7022993d2bbd1de05aa070d92bc58759452247b4586d542fb565e3dcf59 |
| 28 | 2.2%    |  23.279  |      31 | 99.7%    | BCash_Is_Trash        | 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f |
| 29 | 1.7%    |  17.9998 |     106 | 100.0%   | Fold                  | 02816caed43171d3c9854e3b0ab2cf0c42be086ff1bd4005acc2a5f7db70d83774 |
| 30 | 1.6%    |  18.3502 |     741 | 100.0%   | VincentVega           | 0371507087e9e9d0aea4a0405b3be4cc8b9e7d3582211bb26864d0797448d39ef3 |
| 31 | 1.5%    |  18.0746 |      44 | 100.0%   | Bitrefill             | 03d607f3e69fd032524a867b288216bfab263b6eaee4e07783799a6fe69bb84fac |
| 32 | 1.5%    |  18.3866 |    1459 | 100.0%   | ln-beast              | 033719864ecae30f766a05f84dc74a88af6a268c5b8289e6b31bad4d9e635f20de |
| 33 | 0.4%    |  18.8248 |      94 | 100.0%   | SimpleCoin            | 03bec0f5799c4ae2d0fa8943f089324bddd6cbbbb6178f7ac2f2588696280e6587 |
| 34 | 0.2%    |  18.0141 |    1178 | 100.0%   | TheBolt               | 03bec3989098df0675033530169ef105b1bd3be8fd989cb7955cb4b59616ab3530 |

I notice your filtered edge count is very small, try removing this line (66) from GraphFilter
t - edge['last_update'] <= 1.5 * 24 * 60 * 60,
If there's something weird with how LND is handling the last_update field the above should work around it.

[deepy420]

I'm using lncli version 0.13.3-beta commit=v0.13.3-beta.

I commented out that line in GraphFilter and I still get the same result:
Loaded graph. Number of nodes: 19004 Edges: 82124
Latest update in graph: Mon Oct 11 08:21:45 2021
Simplified graph. Number of nodes: 11871 Edges: 4560
Performing analysis for BitcoinRevolution
First filtering pass found 0 candidates for new channels
Running modified farness score calculations
Completed modified farness score calculations in 1.9s
Checking 1ML availability statistics
1ML availability filter selected 0 candidates for new channels in 0.0s
No candidates found, is your graph stale?
If issue persists, delete describegraph.json and improvecentrality.conf
Traceback (most recent call last):
File "scripts/lndpytools/improvecentrality.py", line 248, in
main()
File "scripts/lndpytools/improvecentrality.py", line 239, in main
raise ValueError('No valid candidates')
ValueError: No valid candidates

[deepy420]

Also, why is it recommending ACINQ if I already have a channel with them? Maybe because the liquidity isn't currently balanced enough in that channel?

[Gridflare]

I cannot replicate your error, my best guess now is that the minimum relevant channel size was set to something large.
But if you're using the default config that shouldn't be an issue.

It really should not recommend channels you've already connected to, but it appears that my node does not know about your channel with ACINQ, it is only 2 days old.

[deepy420]

Should I upload my describegraph.json file here to help you reproduce the issue?

Here's the conf file:
[Node]
pub_key = 03cd502dee15bb9859a6d9c56a5608c1f76894736ec4cfec906eb025b7623293dc

[GraphFilters]
minrelevantchan = 500000

[CandidateFilters]
minchancount = 8
maxchancount = 10000
mincapacitybtc = 0.3
maxcapacitybtc = 1000
minavgchan = 1000000
minmedchan = 1000000
minavgchanageblks = 4000
minchannels = 1500k4 3M2
minreliability = 0.97
max1mlavailability = 2000
finalcandidatecount = 11

[Other]
csvexportname = newchannels.csv

thanks!

[Gridflare]

I suggest waiting a couple of days for the graph data to readjust.
I think your node is simply very new and not up to date with all the gossip yet.

[deepy420]

That makes sense. Ok, thanks!

[deepy420]

9 days later and now only 1 match.

Loaded graph. Number of nodes: 20094 Edges: 84252
Latest update in graph: Wed Oct 20 03:49:32 2021
Simplified graph. Number of nodes: 13003 Edges: 7465
Performing analysis for BitcoinRevolution
First filtering pass found 1 candidates for new channels
Running modified farness score calculations
Completed modified farness score calculations in 2.8s
Checking 1ML availability statistics
Failed to fetch availability for 0316a0f72a6db1a4d6ac041778179f142f1e788a680cb417bbfc9871b40628bc95
1ML availability filter selected 0 candidates for new channels in 0.7s
No candidates found, is your graph stale?
If issue persists, delete describegraph.json and improvecentrality.conf
Traceback (most recent call last):
File "scripts/lndpytools/improvecentrality.py", line 248, in
main()
File "scripts/lndpytools/improvecentrality.py", line 239, in main
raise ValueError('No valid candidates')
ValueError: No valid candidates

[Gridflare]

Are you running on Neutrino by chance? My node is also running into issues right now, and I highly suspect Neutrino at this point.

[deepy420]

I don't think so. How can I check? It is running umbrel. The blockchain is fully synced.

[Gridflare]

As far as I know, Umbrel is a full node, not Neutrino.
The config line is bitcoin.node=neutrino
If you're running a full or pruned node, it will say bitcoin.node=bitcoind

My best guess for a graph issue is channel sizes not being recorded correctly (I'm currently witnessing this on my node)
If that's the case, setting minrelevantchan=0 and maybe the other capacity min filters to 0 should fix it.

If that's not the problem, I'll have to take a look at your copy of the graph to get a better understanding of your node's network view.

[deepy420]

Yes, it says bitcoin.node=bitcoind

I set minrelevantchan=0 and it gave 3 matches:
"First filtering pass found 3 candidates for new channels"

Is that normal? How many matches does it usually return?

[Gridflare]

It should find 400-800 depending on the size of your node and your settings.
I really suspect there's something odd with your graph, it might be worth inspecting the channel data in it.

[deepy420]

I really have no idea how to inspect that data. I've uploaded the file here in case you have a moment to check it:
https://pubsharefiles.s3.amazonaws.com/describegraph.json

[Gridflare]

I don't see anything wrong from a first inspection. I'll take a more detailed look later.

[deepy420]

Thanks!

[Gridflare]

I've replicated your issue with your copy of the graph.
You are missing the capacity and channel point information for a number of your peers.
For example, search for Boltz's pubkey in your graph, lots of zeros that shouldn't be there.
I don't know why this would be happening to you, but you might want to pass it along to the Umbrel folks.

[deepy420]

I searched a bit and found the cause of the issue you described:
lightningnetwork/lnd#3024

They say that there's no way to rebuild the graph without closing all the channels and doing everything all over again.

Even if my LND is running with --routing.assumechanvalid, will your script work if I connect to LND remotely instead of using the describegraph.json file? I'm guessing that it wouldn't work since it would just use the same internal graph data.

So it seems that I won't be able to use your script locally and will only be able to rely on your website!

I think more people will face this issue because I'm using a standard Umbrel configuration.

[Gridflare]

Thanks, that helps.
Remote connection is the same graph as the json file.
Currently, my website is encountering the same issue, but to a lesser extent, it sounds like I'll have to create a new node just to backend it.

Does Umbrel really run with assumechanvalid? I'm not convinced that the issue is fully explained yet.

[deepy420]

Yea, I installed on AWS 2 weeks ago straight from the Umbrel repo. Maybe older versions didn't have that flag.

Your website gives me a few suggestions and almost all of them are channels that I have already opened.

[Gridflare]

That flag is deprecated and should not be enabled for Umbrel.

Based on the issue you linked, it sounds like I will have to create a new node just for tracking the graph for the website.

[deepy420]

I searched the umbrel repo for assumechanvalid and I don't see any mention of it. It seems like it might not be enabled:
https://github.com/getumbrel/umbrel/search?q=assumechanvalid

[kroese]

I researched this issue because Im running into the same problem. The reason why you didnt find assumechanvalid in the Umbrel code is because this setting has been renamed to "neutrino.validatechannels" and defaults to false.

; Validate every channel in the graph during sync by downloading the containing 
; block. This is the inverse of routing.assumechanvalid, meaning that for 
; Neutrino the validation is turned off by default for massively increased graph 
; sync performance. This speedup comes at the risk of using an unvalidated view 
; of the network for routing. Overwrites the value of routing.assumechanvalid if 
; Neutrino is used. (default: false) 
; neutrino.validatechannels=false

I guess the only option I have is to re-install LND on another machine without Neutrino, and re-sync the whole graph :s

[Gridflare]

If you have used neutrino on your node at any point some channels will have unknown capacity values.
The workaround is to drop the graph with chantools and fetch the graph with include_unannounced.
[GraphFilters]includeunannouced has been added to improvecentrality.conf.
Make sure Neutrino is fully disabled and replaced with a full or pruned node first.

IMPORTANT:
Using chantools dropchannelgraph has side effects, all of your own channels will disappear from describgraph unless fetched with include_unannounced, which will also include your private channels, which may not be desirable.
lightninglabs/chantools#48