node 259 can't provision deployments

[Omarabdul3ziz]

Describe the bug

deployment with zdb workload failed with timeout after about 10m

To Reproduce

try deploy zdb on node 259 on devnet

Screenshots

TimeoutError: TimeoutError: Deployment with contract_id: 172829 failed to be ready after 10 minutes.

[Omarabdul3ziz]

trying to reproduce on a virtual node and here is what happened

• the deployment looks succeeded, i got the myc IP and i manage to connect to it
• but the node logs shows this
  
  but the file already exists
  

but when i tried to deploy on node 259, looks the workload doesn't get ready even after about 15 min.
also looks the sock file is exists and the zdb instance is running

but couldn't get an idea what is happening there, it misses debug logs

i will try to change the log level there to figure out what is happening

[Omarabdul3ziz]

Update

• it is not a zdb issue, it is a node issue
  not only zdb deployments fails, deploying network/vms also fails after waiting about 10m, contract is created on the chain but the node didn't start the deployment.
• it is not a global zos4 issue
  other devnet zos4 nodes (249, and my local node) works fine. only this node doesn't
• also it is not specific to a client, it happens in go/ts client

i see nothing more we could debug in the current state

zos/pkg/kernel/kernel.go

Line 15 in 895849a

Debug = "zos-debug"

this is a kernel flag that will help viewing debug logs, but will require a reboot for the node

[Omarabdul3ziz]

just noticed this in the contracts list

same code made a deployment type

• vm on node 259 which is failing
• vm-light on node 249 which succeeded

so it is either

• the node report as zos4 and it is not
• the (both) client agreed to send zos3 to the node and it is zos4

the node looks good running as zos4

and also repot with the light features https://gridproxy.dev.grid.tf/nodes/259

more debugging next week :D

[Eslam-Nawara]

tried to deploy on the same node and had a problem deploying on it, it couldn't complete any deployment it always times out with no log showing the problem

tried to restart provisiond and deployed a vm, I was able to successfully deploy a vm of type vm-light and network-light network on the node and is accessible through mycelium

I think this means the client sent a valid deployment with valid metadata but still not sure what happened that resulted of a deployment of type vm instead of vm-light before,

maybe an old version of the client was used, or other client than the go client

I agree with Omar we need to reboot the node with the debug flag to get more info about the problems causing the node to not be able to provision