Rebalancing rollout

[mliu-xlabs]

Task
We implemented the feature of automatic rebalancing for the wallets but its currently disabled. This will be the rollout of the feature

• requires testing
• checking metrics
• building dashboards
• stress testing with volume and unbalanced wallets

This tracks with our OKRs

• We can say uptime of the wallets
• amount of time we have to manually add funds or manually rebalance
• the number of alerts that we are getting today vs. what we are getting in the future

[mliu-xlabs]

There are dependencies right now

[abhidtu2014]

As discussed with @solanoepalacio, we will be working on below metrics (some of which are already in place):

• expenditure by chain (exists)
• expenditure by chain (usd)
• rebalancing count (exists)
• rebalancing errors
• wallet-pool available wallets at any point in time

Derived metrics

• For uptime of the wallets, we will try to derive it from number of active wallets per chain metrics
• For number of alerts comparison: now vs then, we can use grafana to query alerts and create a pannel showing the alerts count for a time range and we should see how we get less balance alerts with time

Also, we will be adding these metrics in a new dashboard called "Relayer Accounts and Rebalancing".

cc: @mliu-xlabs

[abhidtu2014]

Tested below metrics on testnet:

• expenditure by chain
• rebalancing count (exists)
• rebalancing errors
• wallet-pool available wallets at any point in time
• wallet lock period per chain