Runtime error in I3D model

[ThisisBillhe]

Hi! Thanks for your work. I have this error when running demo.py: RuntimeError: forward() is missing value for argument 'x'. Declaration: forward(torch.src.i3dpt_jit.I3D self, Tensor x, bool rescale=False, bool resize=False, bool return_features=False) -> Tensor. It comes from line 27 in fvd/styleganv/fvd.py. It seems that the input is taken as self' and no value for argument x'.

[JunyaoHu]

Hello! Which version of torch do you use? The version in my env is 1.13.1.

Let me know your env then I will try to repeat your issue.

ubuntu@ubuntu:~/hjy/code/common_metrics_on_video_quality$ conda list | grep torch
pytorch 1.13.1 py3.8_cuda11.7_cudnn8.5.0_0 pytorch
pytorch-cuda 11.7 h778d358_5 pytorch
pytorch-mutex 1.0 cuda pytorch
torchaudio 0.13.1 py38_cu117 pytorch
torchvision 0.14.1 py38_cu117 pytorch

[ThisisBillhe]

pytorch 2.1.1 py3.8_cuda12.1_cudnn8.9.2_0 pytorch
torchaudio 2.1.1 py38_cu121 pytorch
torchtriton 2.1.0 py38 pytorch
torchvision 0.16.1 py38_cu121 pytorch

[JunyaoHu]

I haven't tried version > 2 yet, please wait. This may take a long time.

[ThisisBillhe]

That's all right. Thanks for your reply.

[JunyaoHu]

Hello, according to the torch version you provided, I re-run the demo program, and it can calculate the results and the results are correct.

I directly git clone the repository without modifying any code.

[ThisisBillhe]

sorry, it seems has something wrong with my video tensor. My shape is

videos1.shape
torch.Size([1, 14, 3, 576, 1024]).

It works fine with

NUMBER_OF_VIDEOS = 8
VIDEO_LENGTH = 30
CHANNEL = 3
SIZE = 64
videos1 = torch.zeros(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)
videos2 = torch.ones(NUMBER_OF_VIDEOS, VIDEO_LENGTH, CHANNEL, SIZE, SIZE, requires_grad=False)

[JunyaoHu]

[1, 14, 3, 576, 1024]. It is also fine for me in pytorch 1.13 and 2.1.

import torch
from calculate_fvd import calculate_fvd
from calculate_psnr import calculate_psnr
from calculate_ssim import calculate_ssim
from calculate_lpips import calculate_lpips

videos1 = torch.zeros(1, 14, 3, 576, 1024)
videos2 = torch.ones(1, 14, 3, 576, 1024)
device = torch.device("cuda")

import json
result = {}
result['fvd'] = calculate_fvd(videos1, videos2, device, method='styleganv')
result['ssim'] = calculate_ssim(videos1, videos2)
result['psnr'] = calculate_psnr(videos1, videos2)
result['lpips'] = calculate_lpips(videos1, videos2, device)
print(json.dumps(result, indent=4))

[ThisisBillhe]

It is not working for me with

videos1 = torch.zeros(1, 14, 3, 576, 1024)
videos2 = torch.ones(1, 14, 3, 576, 1024)

For videogpt method, there will be error in line 119 of common_metrics_on_video_quality-main/fvd/videogpt/fvd.py, where the shape of x1 is [1,400] and sigma is a scalar, which will lead to an error with torch.svd(mat)

linalg.svd: input should have at least 2 dimensions, but has 0 dimensions instead

For styleganv method, I have the same error with the initial post.

[JunyaoHu]

Hello, there are two issues.

In fact, the original calculation code of the two methods does not support the calculation of one pair of videos, at least two pairs of videos are required (covariance calculation is required).

I improved stylegan's method so that I could compute a pair of videos, but I didn't modify the videogpt code.

Now I have updated the code so that videogpt also supports fvd computation for a pair of videos.

Updated for ./common_metrics_on_video_quality/fvd/videogpt/fvd.py

def frechet_distance(x1, x2):
    x1 = x1.flatten(start_dim=1)
    x2 = x2.flatten(start_dim=1)
    m, m_w = x1.mean(dim=0), x2.mean(dim=0)
    sigma, sigma_w = cov(x1, rowvar=False), cov(x2, rowvar=False)
    mean = torch.sum((m - m_w) ** 2)
    if x1.shape[0]>1:
        sqrt_trace_component = trace_sqrt_product(sigma, sigma_w)
        trace = torch.trace(sigma + sigma_w) - 2.0 * sqrt_trace_component
        fd = trace + mean
    else:
        fd = np.real(mean)
    return float(fd)

Now, the two methods are both good for me to calculate [1, 14, 3, 576, 1024]

I'm not sure why it's not working on your side, I'm sorry, maybe you can try using another server or recreate a new virtual environment.

I wish I could help you.

[JunyaoHu]

I'm going to keep this issue open until someone else has a similar problem and solves it.

[ThisisBillhe]

Hello, there are two issues.

In fact, the original calculation code of the two methods does not support the calculation of one pair of videos, at least two pairs of videos are required (covariance calculation is required).

I improved stylegan's method so that I could compute a pair of videos, but I didn't modify the videogpt code.

Now I have updated the code so that videogpt also supports fvd computation for a pair of videos.

Updated for ./common_metrics_on_video_quality/fvd/videogpt/fvd.py

def frechet_distance(x1, x2):
    x1 = x1.flatten(start_dim=1)
    x2 = x2.flatten(start_dim=1)
    m, m_w = x1.mean(dim=0), x2.mean(dim=0)
    sigma, sigma_w = cov(x1, rowvar=False), cov(x2, rowvar=False)
    mean = torch.sum((m - m_w) ** 2)
    if x1.shape[0]>1:
        sqrt_trace_component = trace_sqrt_product(sigma, sigma_w)
        trace = torch.trace(sigma + sigma_w) - 2.0 * sqrt_trace_component
        fd = trace + mean
    else:
        fd = np.real(mean)
    return float(fd)

Now, the two methods are both good for me to calculate [1, 14, 3, 576, 1024]

I'm not sure why it's not working on your side, I'm sorry, maybe you can try using another server or recreate a new virtual environment.

I wish I could help you.

Hi Junyao,
I really appreciate your work and your response. Wish you all the best.

[ningmenghongcha]

当且仅当NUMBER_OF_VIDEOS = 8时，才能运行demo.py。换成其他数值都跑不了
RuntimeError: forward() is missing value for argument 'x'. Declaration: forward(torch.src.i3dpt_jit.I3D self, Tensor x, bool rescale=False, bool resize=False, bool return_features=False) -> Tensor

[JunyaoHu]

当且仅当NUMBER_OF_VIDEOS = 8时，才能运行demo.py。换成其他数值都跑不了 RuntimeError: forward() is missing value for argument 'x'. Declaration: forward(torch.src.i3dpt_jit.I3D self, Tensor x, bool rescale=False, bool resize=False, bool return_features=False) -> Tensor

你好，你的运行环境是？我这里不同视频数量都可以计算的。

[ningmenghongcha]

当且仅当NUMBER_OF_VIDEOS = 8时，才能运行demo.py。换成其他数值都跑不了 RuntimeError: forward() is missing value for argument 'x'. Declaration: forward(torch.src.i3dpt_jit.I3D self, Tensor x, bool rescale=False, bool resize=False, bool return_features=False) -> Tensor

你好，你的运行环境是？我这里不同视频数量都可以计算的。

torch 2.1.0
torchvision0.16.0
py3.10.0

[zhjj666]

当且仅当NUMBER_OF_VIDEOS = 8时，才能运行demo.py。换成其他数值都跑不了 RuntimeError: forward() is missing value for argument 'x'. Declaration: forward(torch.src.i3dpt_jit.I3D self, Tensor x, bool rescale=False, bool resize=False, bool return_features=False) -> Tensor

你好，你的运行环境是？我这里不同视频数量都可以计算的。

你好，想问一下视频的格式应该是哪一种

[JunyaoHu]

当且仅当NUMBER_OF_VIDEOS = 8时，才能运行demo.py。换成其他数值都跑不了 RuntimeError: forward() is missing value for argument 'x'. Declaration: forward(torch.src.i3dpt_jit.I3D self, Tensor x, bool rescale=False, bool resize=False, bool return_features=False) -> Tensor

你好，你的运行环境是？我这里不同视频数量都可以计算的。

你好，想问一下视频的格式应该是哪一种

视频格式应该都支持，不管是avi、mp4还是啥，用opencv之类的读取视频，然后转换为pytorch的tensor就可以了。

Video formats should be supported, whether avi, mp4 or whatever, read the video with opencv and convert it to pytorch tensors.

[Bravo5542]

Hi! Thanks for your work. I have this error when running demo.py: RuntimeError: forward() is missing value for argument 'x'. Declaration: forward(torch.src.i3dpt_jit.I3D self, Tensor x, bool rescale=False, bool resize=False, bool return_features=False) -> Tensor. It comes from line 27 in fvd/styleganv/fvd.py. It seems that the input is taken as self' and no value for argument x'.

In line 27 of the common_metrics_on_video_quality/fvd/styleganv/fvd.py file, Add an x argument as below:
feats = np.vstack([feats, detector(x=torch.stack([preprocess_single(video) for video in videos[i*bs:(i+1)*bs]]).to(device), **detector_kwargs).detach().cpu().numpy()])
it'll work