Kernel Died when loading Custom model using pynq-dpu

[prawin-kumar]

Setup:
Using Vitis AI 2.5 docker image
Pynq-DPU v2.5
Compiled a custom CNN model and quantized to create the xmodel file.
Trying to upload the xmodel on the Kria KV 260 SOM via the pynq-dpu overlay. However, directly loading the model via Overlay causes the python kernel to crash. So I used the xir and vart methods directly. But when using the create_runner() with the obtained subgraph the kernel crashes.

[skalade]

Hi @prawin-kumar,

There could be a number of things that can cause the kernel to crash. Do the example models run OK for you, e.g. mnist, resnet50?

For debug I find it useful to convert the notebook into a script and run it in your jupyter terminal. When scripts fail can get more verbose outputs of the errors causing it than in jupyter.

Additionally you can try checking the jupyter log in /var/log/jupyter.log if there's anything useful there.

[prawin-kumar]

Hey @skalade ,
The example models run perfectly fine on the board(Tested the resnet50,yolov3 and inceptionv1).
Tried to run the notebook as a script and found this:

Kernel is stuck for quite sometime and automatically restarts and attached a relevant section of the log file:

[skalade]

If I were you, as a sanity test, I would try compiling something simple, like the mnist model https://github.com/Xilinx/DPU-PYNQ/blob/master/host/train_mnist_model.ipynb and creating a runner with that.

Given that that works then you know that your host->target flow is OK. This would only leave the model architecture as a culprit (you've ruled out runtime because the shipped models work, and you've ruled out compiler because re-compiled mnist works). Perhaps you're using an exotic layer that's not supported by the compiler? See if there's anything funny with your compilation output.

Nothing out of the ordinary in your jupyter log, that's quite common, I'd be looking for vart error messages or something like that.