Wrong size of mask in RTMDet-Ins post-processing

[flytocc]

Thanks for your error report and we appreciate it a lot.

Checklist

1. I have searched related issues but cannot get the expected help.
2. I have read the FAQ documentation but cannot get the expected help.
3. The bug has not been fixed in the latest version.

Describe the bug
In _bbox_mask_post_process function, when the argument rescale is True, the mask_logits will be interpolated to the original image size, but the width and height in scale_factor are reversed when calculating the original image size. If keep_ratio in Resize tranform is False, the width and height scale factor in scale_factor is difference, and it will make mask size wrong.

mmdetection/mmdet/models/dense_heads/rtmdet_ins_head.py

Lines 498 to 507 in ee2e542

	if rescale:
	ori_h, ori_w = img_meta['ori_shape'][:2]
	mask_logits = F.interpolate(
	mask_logits,
	size=[
	math.ceil(mask_logits.shape[-2] * scale_factor[0]),
	math.ceil(mask_logits.shape[-1] * scale_factor[1])
	],
	mode='bilinear',
	align_corners=False)[..., :ori_h, :ori_w]

Reproduction

1. What command or script did you run?

python tools/test.py configs/rtmdet/rtmdet-ins_tiny_8xb32-300e_coco.py \
    https://download.openmmlab.com/mmdetection/v3.0/rtmdet/rtmdet-ins_tiny_8xb32-300e_coco/rtmdet-ins_tiny_8xb32-300e_coco_20221130_151727-ec670f7e.pth \
    --cfg-options test_dataloader.dataset.pipeline.1.keep_ratio=False

2. Did you make any modifications on the code or config? Did you understand what you have modified?

There are no modifications on the code or config.

3. What dataset did you use?

coco

Environment

not important

Error traceback
If applicable, paste the error trackback here.

none

Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

#11231

[flytocc]

@RangiLyu Hi, can you take a look at this issue, the same error is fixed in mmdeploy open-mmlab/mmdeploy#2581