mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)

[Justobe]

I get an exception when I load model using MXNET as keras's backend. I can load the model correctly with Tensorflow , Theano and cntk as the backend, but I get the following error with MXNET:

Traceback (most recent call last):
File "/data/code/lemon-git/scripts/patch_mxnet/crash_checker.py", line 31, in
model = keras.models.load_model(file_path,custom_objects=ModelUtils.custom_objects())
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 496, in load_model
model = deserialize_model(f, custom_objects, compile)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 302, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 535, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 301, in from_config
model.add(layer)
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
output_tensor = layer(self.outputs[0])
File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 470, in call
output = self.call(inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 175, in call
dilation_rate=self.dilation_rate)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 3705, in conv2d
padding_mode=padding, data_format=data_format)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 94, in func_wrapper
train_symbol = func(*args, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5045, in _convnd
padding, is_slice, out_size = _preprocess_padding_mode(padding_mode, x.shape,
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4395, in shape
return self.get_shape()
File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4404, in get_shape
_, out_shape, _ = self.symbol.infer_shape_partial()
File "/root/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1152, in infer_shape_partial
return self.infer_shape_impl(True, *args, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1210, in infer_shape_impl
ctypes.byref(complete)))
File "/root/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
raise MXNetError(py_str(LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4) :
Stack trace:
[bt] (0) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4b09db) [0x7fd6920089db]
[bt] (1) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x235e45c) [0x7fd693eb645c]
[bt] (2) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2620d12) [0x7fd694178d12]
[bt] (3) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x26235fb) [0x7fd69417b5fb]
[bt] (4) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShapeEx+0x103e) [0x7fd6940dff7e]
[bt] (5) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShapePartialEx+0x82) [0x7fd6940e0672]
[bt] (6) /root/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7fd6c688bec0]
[bt] (7) /root/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7fd6c688b87d]
[bt] (8) /root/anaconda3/lib/python3.6/lib-dynload/ctypes.cpython-36m-x86_64-linux-gnu.so(ctypes_callproc+0x2ce) [0x7fd6c6d33ede]

The code is as follows:
script.py:

import os
import sys
import argparse

"""Parser of command args"""
parse = argparse.ArgumentParser()
parse.add_argument("--backend", type=str, help="name of backends")
parse.add_argument("--crash_dir", type=str, help="path")
flags, unparsed = parse.parse_known_args(sys.argv[1:])

bk = flags.backend
os.environ['KERAS_BACKEND'] = bk

from keras import backend as K
import keras
import traceback


def custom_objects():

    def no_activation(x):
        return x

    def leakyrelu(x):
        import keras.backend as K
        return K.relu(x, alpha=0.01)

    objects = {}
    objects['no_activation'] = no_activation
    objects['leakyrelu'] = leakyrelu
    return objects


print("INFO:Using {} as backend for states extraction| {} is wanted".format(K.backend(), bk))
files = os.listdir(flags.crash_dir)
pass_cnt = 0
for f in files:
    print(f)
    file_path = os.path.join(flags.crash_dir,f)
    try:
        print(f"Loading model. {pass_cnt+1} of {len(files)}")
        model = keras.models.load_model(file_path,custom_objects=custom_objects())
        pass_cnt += 1
    except:
        traceback.print_exc()

print(f"Backend:{bk} Total: {len(files)} Pass:{pass_cnt}")

This is a very simple code. Put the model in a folder, and then run the code to load the model with different backend.

You can run like this:(change mxnet to other backend like tensorflow )

python -u script.py --backend mxnet --crash_dir your/path/

My related library version is:

numpy 1.16.1
tensorboard 1.14.0 py36hf484d3e_0
tensorflow-estimator 1.14.0
tensorflow-gpu 1.14.0
Theano 1.0.4
cntk-gpu 2.7
mxnet-cu101 1.5.1.post0
mkl 2018.0.3 1
mkl-service 1.1.2 py36h17a0993_4
mkl_fft 1.0.1 py36h3010b51_0
mkl_random 1.0.1 py36h629b387_0
Keras 2.2.4
keras-applications 1.0.8 py_0
keras-mxnet 2.2.4.2
keras-preprocessing 1.1.0 py_1

You can download the model from this link:
https://1drv.ms/u/s!Aj6dGBsJFcs0jWiaHhB85gLYKQIQ?e=xdgTSX

thanks in advance!