Wrong depth warrpping

[phongnhhn92]

Hi,
I am testing the depth_estimation.ipynb examples that you guys have provided to warp the source image to the target image using the depth map of the target image. However, when I look at the wrapped image at the cell 7 it looks weird to me. Why does it look like there are two overlapping image in the wrapped image ?

[edgarriba]

this might be the effect produced by occlusions. Are we talking about this example ?
https://github.com/kornia/kornia-examples/blob/master/depth_estimation.ipynb

[chenusc11]

I also encountered the same problem.
To reproduce it, you just need to change the line:
image_dst_to_src = kornia.warp_frame_depth(image_dst, depth_src, dst_trans_src, intrinsics_src)
to:
image_dst_to_src = kornia.warp_frame_depth(image_i, depth_src, i_trans_j, intrinsics_src)

[edgarriba]

@phongnhhn92 @chenusc11 could you send a quick fix to that code?

[phongnhhn92]

@phongnhhn92 @chenusc11 could you send a quick fix to that code?

I am sorry for slow reply. I think @chenusc11 found the solution. I didnt notice that you combine 2 transformation matrices at cell 6
dst_trans_src = torch.cat([i_trans_j, k_trans_j], dim=0)

This leads to the overlapping between i->j and k-> j warped images. I think you should be a bit more clarify in the ipnb file so there is no more confusion :D

[phongnhhn92]

Hello guys, I hate to do this but I have to reopen the issue again. This is the code that I copy and adapt from the example:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision.utils import save_image
import kornia

import cv2
import numpy as np
import matplotlib.pyplot as plt


use_cuda = torch.cuda.is_available()
device = torch.device('cuda' if use_cuda else 'cpu')
print('Using ', device)

def load_data(filename):
    data_np = np.load(filename)
    data = torch.from_numpy(data_np)
    return data

def data_to_vis(data):
    return kornia.tensor_to_image(data)

# load the camera matrix
intrinsics_i = load_data('data/depth/camera_intrinsics_0.npy')[..., :3, :3]  # 1x3x3
intrinsics_j = load_data('data/depth/camera_intrinsics_1.npy')[..., :3, :3]  # 1x3x3
intrinsics_k = load_data('data/depth/camera_intrinsics_2.npy')[..., :3, :3]  # 1x3x3

# load the camera pose
i_pose_w = load_data('data/depth/camera_pose_0.npy')  # 1x4x4
j_pose_w = load_data('data/depth/camera_pose_1.npy')  # 1x4x4
k_pose_w = load_data('data/depth/camera_pose_2.npy')  # 1x4x4

# load images
image_i = load_data('data/depth/image_0.npy')  # 1x3xHxW
image_j = load_data('data/depth/image_1.npy')  # 1x3xHxW
image_k = load_data('data/depth/image_2.npy')  # 1x3xHxW
save_image(image_i,'image_i.png')
save_image(image_j,'image_j.png')
save_image(image_k,'image_k.png')
# warp by depth
i_trans_j = i_pose_w @ torch.inverse(j_pose_w)
k_trans_j = k_pose_w @ torch.inverse(j_pose_w)
dst_trans_src = torch.cat([i_trans_j, k_trans_j], dim=0)

# load the depth map (in meters)
depth_i = load_data('data/depth/depth_0.npy')  # 1x1xHxW
depth_j = load_data('data/depth/depth_1.npy')  # 1x1xHxW
depth_k = load_data('data/depth/depth_2.npy')  # 1x1xHxW

depth_src = depth_j
intrinsics_src = intrinsics_j

image_src = image_j
image_dst = torch.cat([image_i, image_k], dim=0)
image_dst_to_src = kornia.warp_frame_depth(image_i, depth_src, i_trans_j, intrinsics_src)
print(image_dst_to_src.shape)
save_image(image_dst_to_src,'warped.png')

However, I still have that weird problem in the warp.png.

This warped image is far from being consistent with the J image since we are warping i->j:

@edgarriba can you double check this ? I can using kornia '0.5.8'. It seems like you guys have made some changes to the kornia.warp_frame_depth() so the original example code doesnt work anymore. But somehow I still get the overlapping patches here. I think that the overlapping area in the warped image should be zeros-pixels due to occlusion, is that correct ?

[edgarriba]

@phongnhhn92 are you aware which is the latest version that works well ?

[phongnhhn92]

I am afraid to tell that I made a mistake thinking the problem is solved. I thought just simply change the line as @chenusc11 would help but it turns out doesn't. Sorry for the confusion!