About Model Conversion！

[sunmooncode]

感谢您的开源！

您的项目非常有趣，我想问能使用mgeconvert 能将该模型转换成onnx嘛，或着有其他方法嘛！

期待您的回复！

[sunmooncode]

我尝试用一下代码转换mge模型出现错误： Segmentation fault (core dumped)

data1 = mge.Tensor(np.random.random([1, 3, 384, 512]).astype(np.float32)), 
data2 = mge.Tensor(np.random.random([1, 3, 384, 512]).astype(np.float32))

pretrained_dict = mge.load("model/crestereo_eth3d.mge",map_location="cpu")
model = CREStereo(max_disp=256, mixed_precision=False, test_mode=True)

model.load_state_dict(pretrained_dict["state_dict"], strict=True)

model.eval()
@jit.trace(symbolic=True, capture_as_const=True)
def infer_func(a,b, *, model):
    pred = model(a,b)
    return pred
infer_func(data1,data2,model=model)
infer_func.dump("./test.mge", arg_names=["left_image","right_image"],output_name=["disp"])

[ibaiGorordo]

@sunmooncode So far I was able to convert to traced module using the following code:

import numpy as np
import megengine.functional as F
from megengine import jit
import megengine.traced_module as tm
import megengine as mge
from nets import Model

print(mge.__version__)

data1 = mge.tensor(np.random.random([1, 3, 480, 640]).astype(np.float32))
data2 = mge.tensor(np.random.random([1, 3, 480, 640]).astype(np.float32))

pretrained_dict = mge.load("crestereo_eth3d.mge")
model = Model(max_disp=256, mixed_precision=False, test_mode=True)
model.load_state_dict(pretrained_dict["state_dict"], strict=True)
model.freeze_bn()

output = model(data1,data2)
print(output)

traced_model = tm.trace_module(model, data1, data2)
traced_model.eval()

mge.save(traced_model, "traced_model.tm")

Running it will give you an error, but you can fix it by commenting this line:

CREStereo/nets/corr.py

Line 45 in 924d9f2

assert right_crop.shape == left_feature.shape

My idea is to see if the tracedmodule_to_onnx function in the mgeconverter will work. However, when I try to convert the model, I get the following error:

tm_to_onnx.py", line 51, in tracedmodule_to_onnx
    traced_module = mge.load(traced_module)
  File "/usr/local/lib/python3.7/dist-packages/megengine/serialization.py", line 107, in load
    return load(fin, map_location=map_location, pickle_module=pickle_module)
  File "/usr/local/lib/python3.7/dist-packages/megengine/serialization.py", line 112, in load
    return pickle_module.load(f)
ValueError: unsupported pickle protocol: 5

For some reason in that machine, loading the model does not work, but in another machine I have, I am able to do traced_module = mge.load(traced_module) without any issue. However in that machine I am not able to install mgeconverter . So I will keep trying tomorrow.

[sunmooncode]

@ibaiGorordo thanks for your help！
When I use mgeconvert's convert tracedmodule_to_onnx -i tracedmodule.tm -o out.onnx I get the following error:

Traceback (most recent call last):
  File "/home/zt/.local/bin/convert", line 525, in <module>
    main()
  File "/home/zt/.local/bin/convert", line 518, in main
    args.func(args)
  File "/home/zt/.local/bin/convert", line 280, in convert_func
    opset=args.opset,
  File "/home/zt/.local/lib/python3.6/site-packages/mgeconvert/converters/tm_to_onnx.py", line 61, in tracedmodule_to_onnx
    irgraph = tm_resolver.resolve()
  File "/home/zt/.local/lib/python3.6/site-packages/mgeconvert/frontend/tm_to_ir/tm_frontend.py", line 71, in resolve
    self.get_all_oprs()
  File "/home/zt/.local/lib/python3.6/site-packages/mgeconvert/frontend/tm_to_ir/tm_frontend.py", line 103, in get_all_oprs
    assert op_gen_cls, "METHOD {} is not supported.".format(expr.method)
AssertionError: METHOD __rmul__ is not supported.

The reason is that many OPs of CREstereo do not have corresponding operator support for mgeconverter.

In addition:
When I use the script to evaluate the real-time performance of CREstereo, my computer is an RTX1050ti, and the result is less than 1 Fps. However, under the same conditions, RAFTstereo has nearly 3 Fps, and the realtime model has more than 6 Fps. So CRestereo is too big for me.

[PINTO0309]

I get segmentation fault even if I build megengine and megconvert for my environment.
Installing MgeConvert CREStereo - Zenn - my article

By the way, I have already confirmed that CREStereo works in the CPU environment on which it was built. I have given up on exporting to ONNX because MegEngine will segfault no matter what workaround I try.

${HOME}/.local/bin/convert mge_to_onnx \
-i crestereo_eth3d.mge \
-o crestereo_eth3d.onnx \
--opset 11

/home/user/.local/lib/python3.8/site-packages/megengine/core/tensor/megbrain_graph.py:508: ResourceWarning: unclosed file <_io.BufferedReader name='crestereo_eth3d.mge'>
  buf = open(fpath, "rb").read()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "/home/user/.local/bin/convert", line 525, in <module>
    main()
  File "/home/user/.local/bin/convert", line 518, in main
    args.func(args)
  File "/home/user/.local/bin/convert", line 283, in convert_func
    converter_map[target](
  File "/home/user/.local/lib/python3.8/site-packages/mgeconvert/converters/mge_to_onnx.py", line 50, in mge_to_onnx
    irgraph = MGE_FrontEnd(mge_fpath, outspec=outspec).resolve()
  File "/home/user/.local/lib/python3.8/site-packages/mgeconvert/frontend/mge_to_ir/mge_frontend.py", line 21, in __init__
    _, outputs = load_comp_graph_from_file(model_path)
  File "/home/user/.local/lib/python3.8/site-packages/mgeconvert/frontend/mge_to_ir/mge_utils.py", line 106, in load_comp_graph_from_file
    ret = G.load_graph(path)
  File "/home/user/.local/lib/python3.8/site-packages/megengine/core/tensor/megbrain_graph.py", line 511, in load_graph
    cg, metadata = _imperative_rt.load_graph(buf, output_vars_map, output_vars_list)
RuntimeError: access invalid Maybe value

backtrace:
/home/user/.local/lib/python3.8/site-packages/megengine/core/lib/libmegengine_shared.so(_ZN3mgb13MegBrainErrorC1ERKSs+0x4a) [0x7f3b39dfe1fa]
/home/user/.local/lib/python3.8/site-packages/megengine/core/lib/libmegengine_shared.so(_ZN3mgb17metahelper_detail27on_maybe_invalid_val_accessEv+0x34) [0x7f3b39f060f4]
/home/user/.local/lib/python3.8/site-packages/megengine/core/_imperative_rt.cpython-38-x86_64-linux-gnu.so(+0x14c605) [0x7f3b94873605]
/home/user/.local/lib/python3.8/site-packages/megengine/core/_imperative_rt.cpython-38-x86_64-linux-gnu.so(+0x14c823) [0x7f3b94873823]
/home/user/.local/lib/python3.8/site-packages/megengine/core/_imperative_rt.cpython-38-x86_64-linux-gnu.so(+0x11d62e) [0x7f3b9484462e]
/usr/bin/python3(PyCFunction_Call+0x59) [0x5f5e79]
/usr/bin/python3(_PyObject_MakeTpCall+0x296) [0x5f6a46]
/usr/bin/python3(_PyEval_EvalFrameDefault+0x5d3f) [0x570a1f]
/usr/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f6226]
/usr/bin/python3(_PyEval_EvalFrameDefault+0x5706) [0x5703e6]

[PINTO0309]

@ibaiGorordo

$ pip install pickle5

• /usr/local/lib/python3.7/dist-packages/megengine/serialization.py - line 112

import pickle5

return pickle5.load(f)

[ibaiGorordo]

Hi @PINTO0309 thanks for the tip. @sunmooncode I got the same error after fixing my issue.

Also I have create a Google Colab notebook to reproduce the error:
https://colab.research.google.com/drive/1IMibaByKwiAIam8UAvyI-2U_Z9O7rBWS?usp=sharing

Because of the CUDA version, it crashed if I load the model with a runtime with GPU. So, run it without GPU.

[ibaiGorordo]

Commenting these two lines fixes the rmul error:

CREStereo/nets/crestereo.py

Lines 98 to 99 in 924d9f2

	image1 = 2 * (image1 / 255.0) - 1.0
	image2 = 2 * (image2 / 255.0) - 1.0

However, next I get the following error:
AssertionError: Module <class 'megengine.module.normalization.InstanceNorm'> is not supported.

[sunmooncode]

That is because megconvert does not support this operation. I have seen that many in the network do not support it, so it should not be converted.

[ibaiGorordo]

Yeah... I tried converting the model to .mge (using @trace(symbolic=False, capture_as_const=True), but using mge_to_onnx outputs the following error: AssertionError: OP PowC is not supported (I updated the Google Colab notebook).

I do not understand the framework enough, but seems to be hard to fix the issues

[ibaiGorordo]

Since it seems to be hard to convert, I have tried to implement the model in Pytorch. The model seems to run normally, but since I don't have the weights I cannot fully test it.

My hope is that somehow we can translate the weights from this model there.

https://github.com/ibaiGorordo/CREStereo-Pytorch

[sunmooncode]

@ibaiGorordo Good job
You can train a model with a small step count and run through that model to see if it converts. Secondly, pytorch to onnx cannot contain dynamic structures. The structure needs to be instantiated.

[ibaiGorordo]

@sunmooncode I was able to convert the weights directly, however, it seems that there is some parameter in my Pytorch implementation that is probably not correct. But overall the conversion seems to work.

Haven't tried converting it to other frameworks yet.

[ibaiGorordo]

I was able to fix the implementation issues and convert the model to ONNX: https://github.com/ibaiGorordo/ONNX-CREStereo-Depth-Estimation. From there it should be easier to convert to other platforms. Here is a video with the output in ONNX: https://youtu.be/ciX7ILgpJtw

@sunmooncode regarding the low speed, if you use a low resolution (320x240) and only do one pass without flow_init, you can get decent speed with good quality.

[PINTO0309]

I have committed a large number of ONNX models of various resolution and ITER combinations. I imagine the ITER10 version is twice as fast.

https://github.com/PINTO0309/PINTO_model_zoo/tree/main/284_CREStereo

• init - ITER 2,5,10,20
  This model generates flow_init.

• next - ITER 2,5,10,20
  This model takes as input a total of three images: flow_init generated by the init model and two LEFT/RIGHT images.

• combined - ITER 2,5,10,20
  Two levels of inference, init and next, are merged into a single onnx.

[sunmooncode]

@PINTO0309 @ibaiGorordo Thanks for your help!
I would like to know how you converted these models, can you provide their conversion scripts, it would be very helpful to me.