[Linux] onnxruntime not detected

[SheeperGit]

Bug Description

onnxruntime is not being detected by the app.
Tried through Github release v2.8.1 as well as building it from source with the same result.
Any help would be greatly appreciated. Thanks in advance, fellas!

Steps to Reproduce

Outline the steps to reproduce the behavior:

Downloaded dotnet-runtime and dotnet-sdk via pacman.
(i.e., sudo pacman -S dotnet-runtime dotnet-sdk)

Download v2.8.1 release from github.

cd
chmod +x DatasetProcessorDesktop
./DatasetProcessorDesktop

App launches.
Go to Generate Tags.
Set input/output folders.
WDv3 model selected.

Hit Generate Tags button.
Log messages say "Downloading WDV3Model.onnx file..." for ~18secs.
"Something went wrong! Error log will be saved in the logs folder."

Tags not generated.

Expected Behavior

The application should detect onnxruntime and generate the tags.

Environment Details

Complete the following information:

• Operating System: Arch Linux x86_64
• Application Version: 2.8.1

Additional Context

Here is the output of the error log:

Exception Details

Date and Time: 2024-07-27 15:39:26
Source: Microsoft.ML.OnnxRuntime
Message: The type initializer for 'Microsoft.ML.OnnxRuntime.NativeMethods' threw an exception.
Help Link:
HResult: -2146233036

Inner Exception Details

Source: Microsoft.ML.OnnxRuntime
Message: Unable to load shared library 'onnxruntime' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable:
/home/sheep/Programs/linux-x64/onnxruntime.so: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.6/onnxruntime.so: cannot open shared object file: No such file or directory
/home/sheep/Programs/linux-x64/libonnxruntime.so: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.6/libonnxruntime.so: cannot open shared object file: No such file or directory
/home/sheep/Programs/linux-x64/onnxruntime: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.6/onnxruntime: cannot open shared object file: No such file or directory
/home/sheep/Programs/linux-x64/libonnxruntime: cannot open shared object file: No such file or directory
/usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.6/libonnxruntime: cannot open shared object file: No such file or directory

Help Link:
HResult: -2146233052

Stack Trace

at Microsoft.ML.OnnxRuntime.SessionOptions..ctor()
at SmartData.Lib.Services.Base.BaseAIConsumer2.LoadModel() in E:\dev\DatasetHelpers\SmartData.Lib\Services\Base\BaseAIConsumer.cs:line 60 at SmartData.Lib.Services.Base.BaseAutoTaggerService2.LoadModel() in E:\dev\DatasetHelpers\SmartData.Lib\Services\Base\BaseAutoTaggerService.cs:line 64
at SmartData.Lib.Services.Base.BaseAutoTaggerService`2.GenerateTags(String inputPath, String outputPath, Boolean weightedCaptions) in E:\dev\DatasetHelpers\SmartData.Lib\Services\Base\BaseAutoTaggerService.cs:line 99
at DatasetProcessor.ViewModels.GenerateTagsViewModel.CallautoTaggerService(IAutoTaggerService autoTagger) in E:\dev\DatasetHelpers\DatasetProcessor\ViewModels\GenerateTagsViewModel.cs:line 214
at DatasetProcessor.ViewModels.GenerateTagsViewModel.MakePredictionsAsync() in E:\dev\DatasetHelpers\DatasetProcessor\ViewModels\GenerateTagsViewModel.cs:line 160

Target Site

Declaring Type: Microsoft.ML.OnnxRuntime.SessionOptions
Method Name: .ctor

Additional Information

[Particle1904]

I believe I've found the problem. The Linux build didn't have the correct NuGet package with the OnnxRuntime files.
I've configured the project to use conditional NuGet packages so Linux builds from this point forward will be built with the correct NuGet package for the OnnxRuntime.

Heres the release: https://github.com/Particle1904/DatasetHelpers/releases/tag/v2.8.2
Let me know if it works, I currently don't have WSL installed and configured to test it out.

[SheeperGit]

Hey, thanks for the fast reply!

This new release did indeed fix the issue with the onnxruntime.
The log message says it was able to successfully download the WDv3 onnx file and tagged the images shortly after.

Thank you very much for your help, my friend! You guys built an awesome application.

Since the issue has been resolved, I'm marking it as resolved.

[SheeperGit]

It seems I've closed the issue too soon :)
When going over to the Tag/Caption Editor and selecting the input folder, I get the following error in my terminal:

Unhandled exception. Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:NoSuchFile] The ONNX Runtime extensions library was not found. The Microsoft.ML.OnnxRuntime.Extensions NuGet package must be referenced by the project to use 'OrtExtensions.RegisterCustomOps.
at Microsoft.ML.OnnxRuntime.SessionOptions.RegisterOrtExtensions()
at SmartData.Lib.Services.MachineLearning.CLIPTokenizerService.LoadModel() in E:\dev\DatasetHelpers\SmartData.Lib\Services\MachineLearning\CLIPTokenizerService.cs:line 67
at SmartData.Lib.Services.MachineLearning.CLIPTokenizerService.CountTokens(String inputText) in E:\dev\DatasetHelpers\SmartData.Lib\Services\MachineLearning\CLIPTokenizerService.cs:line 38
at DatasetProcessor.ViewModels.TagEditorViewModel.b__33_0() in E:\dev\DatasetHelpers\DatasetProcessor\ViewModels\TagEditorViewModel.cs:line 302
at System.Threading.Tasks.Task1.InnerInvoke() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at DatasetProcessor.ViewModels.TagEditorViewModel.CountTokensForCurrentImage() in E:\dev\DatasetHelpers\DatasetProcessor\ViewModels\TagEditorViewModel.cs:line 302 at DatasetProcessor.Views.TagEditorView.OnEditorTextChangedAsync(Object sender, EventArgs args) in E:\dev\DatasetHelpers\DatasetProcessor\Views\TagEditorView.axaml.cs:line 74 at DatasetProcessor.Views.TagEditorView.<.ctor>b__5_1(Object sender, EventArgs args) in E:\dev\DatasetHelpers\DatasetProcessor\Views\TagEditorView.axaml.cs:line 41 at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_0(Object state) at Avalonia.Threading.SendOrPostCallbackDispatcherOperation.InvokeCore() at Avalonia.Threading.DispatcherOperation.Execute() at Avalonia.Threading.Dispatcher.ExecuteJob(DispatcherOperation job) at Avalonia.Threading.Dispatcher.ExecuteJobsCore(Boolean fromExplicitBackgroundProcessingCallback) at Avalonia.Threading.Dispatcher.Signaled() at Avalonia.X11.X11PlatformThreading.CheckSignaled() at Avalonia.X11.X11PlatformThreading.RunLoop(CancellationToken cancellationToken) at Avalonia.Threading.DispatcherFrame.Run(IControlledDispatcherImpl impl) at Avalonia.Threading.Dispatcher.PushFrame(DispatcherFrame frame) at Avalonia.Threading.Dispatcher.MainLoop(CancellationToken cancellationToken) at Avalonia.Controls.ApplicationLifetimes.ClassicDesktopStyleApplicationLifetime.Start(String[] args) at Avalonia.ClassicDesktopStyleApplicationLifetimeExtensions.StartWithClassicDesktopLifetime(AppBuilder builder, String[] args, Action1 lifetimeBuilder)
at DatasetProcessor.Desktop.Program.Main(String[] args) in E:\dev\DatasetHelpers\DatasetProcessorDesktop\Program.cs:line 17
Aborted (core dumped)

Outlining my steps:

Download new v2.8.2 release from Github.

cd linux-x64
chmod +x DatasetProcessorDesktop
./DatasetProcessorDesktop

Go to Tag/Caption Editor
Select input folder
App crashes
Terminal displays error mentioned above

[Particle1904]

Its failing to load the Clip Tokenizer onnx extension that is used to count number of tokens, but I don't really understand why. It works on Windows. Maybe the onnx extension package isn't supported in Linux.

I'll add a hotfix so it doesn't crash if it can't load it.

Edit: Seems like its a common issue with Linux

• ONNX Runtime Extensions fails to load on Linux due to missing libssl.so.1.1 microsoft/onnxruntime-extensions#761
• https://stackoverflow.com/questions/72133316/libssl-so-1-1-cannot-open-shared-object-file-no-such-file-or-directory/72633324#72633324
  If you want, you can try downloading the package in your machine. I'll make it so this feature is disabled in Linux if it fails to load the library for the onnxruntime extensions.

[Particle1904]

I've released a new hotfix for this issue: https://github.com/Particle1904/DatasetHelpers/releases/tag/v2.8.3
I tried to simulate the issue on Windows and it should no longer crash, instead it will inform the user that it failed to load the CLIP Tokenizer but the Editor Page should remain usable.

Let me know if it works.

[SheeperGit]

Hey, good news!

v2.8.3 does prevent the crash.
I've also tested all the previous steps I've done before (e.g., downloading the WDv3.onnx file, processing tags, etc.) to test for regressions, and there aren't any errors. It does disable use of the CLIP Tokenizer, but I don't particularly mind this minor inconvenience.

As for the possible cause of this being a dependency on a missing libssl, I've checked that I have openssl-3.3.1-1 which is what my package manager tells me to install when typing sudo pacman -S libssl.so (sudo pacman -S libssl errors out, saying there is not package with that name).
I've tried to reproduce the steps done in the first link you've sent microsoft/onnxruntime-extensions#761 with somewhat similar results. I've done export LD_DEBUG=all and redirected stdout and stderr to error.txt (see attached google drive link below) on the release version that started this issue (v2.8.1) and a similar message does crop up, along with other messages that claim a package is needed by another. You can see this by searching for needed by in error.txt

The fix mentioned in the second link you've sent (https://stackoverflow.com/questions/72133316/libssl-so-1-1-cannot-open-shared-object-file-no-such-file-or-directory/72633324#72633324) contains a fix likely to be specific to Debian-based Linux distribution, of which my operating system is not. This fix may work for those Debian-based distros, but these files on Arch Linux may brick the OS, so I won't be trying them. Perhaps another Linux user will run into the same problem and may try to see if this resolves their error.

That being said, the application is now perfectly usable on my Arch Linux machine!
Thanks a bunch for your help, my friend.
I don't know whether you plan to investigate the issue further, but if you are, and have more fixes to suggest for me to try, I'd be glad to test the application further for you. If not, then that's alright too, I'm perfectly happy with how things are now and we can mark this issue as closed.

error.txt:
https://drive.google.com/file/d/1mzwKYH0vW5KKlyILfDdKFzVZTgAu6vS7/view?usp=sharing

[Particle1904]

I won't investigate any further for now, I honestly have no idea what the issue really is. Thanks for the help with testing.