About use mmdetection with Zed

[kankanzheli]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues.
• This issue is not a question, bug report, or anything other than a feature request directly related to this project.

Proposal

I'm sorry to ask this question. I am now trying to use mmdetection to directly detect the image obtained by zed mini or the data in the SVO file obtained by zed mini camera.

To tell you the truth, it's a difficult job for me. But I noticed that in this work, you have successfully run mmdetection on zed cameras:

https://github.com/stereolabs/zed-pytorch/blob/master/other/MODEL_ZOO.md

So can you share this part of the code? Or provide a tutorial on how to make mmdetection run successfully on zed?

Use-Case

No response

Anything else?

No response

[kankanzheli]

Can anyone answer this question?

[kankanzheli]

Those Job had been mentioned in zed-pytorch.
In https://github.com/stereolabs/zed-pytorch/blob/master/other/MODEL_ZOO.md
You said:
Comparison with Detectron and mmdetection
In the following section, we compare our implementation with Detectron and mmdetection. The same remarks from mmdetection about different hardware applies here.

[P-yver]

Hi,
you should be able to use our sample to get the current ZED image
as shown here
and then use the mmdetection sample to do the actual detection,
as shown here.

[kankanzheli]

Thank you very much for your answer. But I don't understand what you mean. What I want to achieve is real-time detection. Output the results while the camera acquires the image. (we also hope that mmdetection can run directly on the SVO data obtained by zed, similar to the one shown on zed pytorch.)

[kankanzheli]

I roughly studied the code of "zed_object_detection. Py". In zed pytorch, it mainly relies on the code of importing maskrcnn benchmark. So I may need to make changes in this code to get the desired results. But the specific operation is more complex.

I saw you mention mmdetection in this code introduction So I thought you could already use mmdetection on zed cameras But now it seems that you haven't achieved that, right?

[kankanzheli]

Of course, if you have relevant work or suggestions, please provide them to me.

[adujardin]

Hi @kankanzheli
The zed-pytorch repository is originally a fork from https://github.com/facebookresearch/maskrcnn-benchmark, the mmdetection benchmark is actually from this repo and not from a custom mmdetection ZED integration unfortunetly.

So you would need to start from the zed-pytorch code for instance and integrate it yourself into a mmdetection sample.

[kankanzheli]

Oh thanks. How about the pyzed.sl? I try to do some work with the zed_object_detection.py. But I don't understand how to use the pyzed.sl. Is there a tutorial on how to use "pyzed. SL"?

[kankanzheli]

And what I need is to make mmdetection run on the data output by zed. So I don't need to transfer zed-pytorch to the mmdetection architecture?

[adujardin]

pyzed is the python wrapper of the C++ ZED SDK and will provide all the features of the SDK. It's required to use the camera and is used on all python related integration. The examples have been unified with the other languages and can be found here https://github.com/stereolabs/zed-examples (there's a subfolder for python in each). There's also the API documentation https://www.stereolabs.com/docs/api/python/

You need to follow the code structure and main functions used in zed-pytorch on mmdetection. The tutorials will provide you with the basics of pyzed usage to better understand the workflow https://github.com/stereolabs/zed-examples/tree/master/tutorials

[kankanzheli]

One more question. In the process of learning pyzed, I found that the object recognition and body tracking module of zed can only run on zed2. Can zed pytorch run on zed Mini now?

[adujardin]

Yes, the Object detection and body tracking module are now working with ZED2, and ZED Mini since ZED SDK 3.6

[kankanzheli]

So how can I ungrade the SDK of ZED camera?

[kankanzheli]

At the same time, I have installed pyzed-3.5 in the anconda enviroment. So, What should I do?

[adujardin]

The ZED SDK is running on the host PC and is independent of the camera firmware. You just need to install it on your PC (including the corresponding python API), then the Object detection feature could be run with either camera using the samples.

[kankanzheli]

I already download the ZED-SDK-3.5.5. Now I want to Use ZED-SDK-3.6.2. So I Just need to install ZED-SDK-3.6.2. Should I need to take care of ZED-SDK-3.5.5?

[adujardin]

Yes, it will overwrite the previous SDK versions (on Linux it's installed in /usr/local/zed and doesn't support multiple versions at once)

[kankanzheli]

I face a problem when I work mmdetection with zed.
In the mmdetection, they output the result in this way:

camera = cv2.VideoCapture(args.camera_id)

print('Press "Esc", "q" or "Q" to exit.')
while True:
    ret_val, img = camera.read()
    result = inference_detector(model, img)

    ch = cv2.waitKey(1)
    if ch == 27 or ch == ord('q') or ch == ord('Q'):
        break

    model.show_result(
        img, result, score_thr=args.score_thr, wait_time=1, show=True)

Then, I changed the codes into:

while running:
    err_code = zed.grab(runtime)
    ....
    zed.retrieve_image(left, sl.VIEW.LEFT, resolution=res)
    img = cv2.cvtColor(left.get_data(), cv2.COLOR_RGBA2RGB)
    result = inference_detector(model, img)     
    ch = cv2.waitKey(10)
    if ch == 27 or ch == ord('q') or ch == ord('Q'):
        break
    model.show_result(img, result, score_thr=args.score_thr, wait_time=1, show=True)

But my code just output the result of the first frame. Then the terminal will report error:

RuntimeError: CUDA error: invalid resource handle

What is the problem?

[adujardin]

Yes, this is an annoying problem and is caused by a conflict of CUDA context, both Pytorch and the ZED SDK have their own and it can't be mixed implicitly. Related issues:

• Could I share my CUDA context with a PyZEDCamera? zed-python-api#35
• cudnn error when running zed api alongside a pytorch model  zed-python-api#154

The solution to this is to put the ZED functions in a different thread than pytorch, like the zed-tensorflow code https://github.com/stereolabs/zed-tensorflow/blob/master/object_detection_zed.py

[kankanzheli]

I see.
The most import part in https://github.com/stereolabs/zed-tensorflow/blob/master/object_detection_zed.py is:

# Start the capture thread with the ZED input
print("Starting the ZED")
capture_thread = Thread(target=capture_thread_func, kwargs={'svo_filepath': svo_filepath})
capture_thread.start()
# Shared resources
global image_np_global, depth_np_global, new_data, exit_signal

Right? For this part, the global is very import. At begining of object_detection_zed.py:
width = 704
height = 416
confidence = 0.35

image_np_global = np.zeros([width, height, 3], dtype=np.uint8)
depth_np_global = np.zeros([width, height, 4], dtype=np.float)

For it, I have one question. As you can see, in my codes, I want to get rgb image from zed. Then, in the mmdetection part, I can work out the result with the RGB image. So in the two threads, I need to shared the resource -- RGB-img
So, I should change the image_np_global = np.zeros([width, height, 3], dtype=np.uint8) into what?

[kankanzheli]

In fact, I haven't been exposed to multithreading before

[kankanzheli]

Oh， that problem had been solved!

[kankanzheli]

There is a new problem. After running zed mini camera on ROS, can mmdetection be used for processing? Is there a similar code to use the instance segmentation method to process the image obtained by zed mini camera?