[ByteTrack] - redesign update_with_detections to use IoU to match input Detections with tracker_id

[SkalskiP]

Description

• Currently, update_with_detections returns the predicted position of boxes, not their actual coordinates received from the detector. Many users have complained about the deterioration of box quality when using ByteTrack. (#743)
• ByteTrack does not work with segmentation models because masks are not transferred to the update_with_detections output.
• The Detections.data field is lost after passing through update_with_detections.

All these issues can be resolved by changing the logic in update_with_detections. Instead of mapping values obtained from update_with_tensors to new Detections objects, we should use IoU to map the results of update_with_tensors to input Detections objects. This way, the input xyxy coordinates and the input state of the mask and data fields will be preserved.

For this purpose, we can utilize the already existing function box_iou_batch. The matching procedure has been demonstrated in one of our videos on YouTube.

Additional

• Note: Please share a Google Colab with minimal code to test the new feature. We know it's additional work, but it will definitely speed up the review process. Each change must be tested by the reviewer. Setting up a local environment to do this is time-consuming. Please ensure that Google Colab can be accessed without any issues (make it public). Thank you! 🙏🏻

[rolson24]

Hi @SkalskiP!
Can I work on this issue? I think I can implement this in the way you want without modifying update_with_tensors and in a pretty simple way.

[SkalskiP]

Hi @rolson24 👋🏻! Feel free to submit PR. You will make me very happy.

[SkalskiP]

The solution for this issue is implemented via #1035. I am closing the issue. Thanks @rolson24 ! 🔥