Support native multi-platform non-blocking texture loading

[robertlong]

Texture2D.LoadImage is slow, can only be ran on the main thread, and blocks the renderer while loading. This is a dealbreaker for a lot of projects, especially those involving VR as dropping frames causes motion sickness.

[stevenvergenz]

The only alternative I can find is to load and decompress texture files into color arrays in a separate thread, then pass that back to the main thread for LoadImage to consume. Depending on where the bottleneck is, this could be helpful.

[robertlong]

Yeah, I think moving the decoding to a separate thread will remove most of the latency. However, I don't personally have the benchmarks to prove it yet. Then there's three methods to try when loading the image data on the main thread:

Texture2D.SetPixels which requires decoding the image and creating the Color array using Unity's Color class. I'm pretty certain we wont see a boost in performance from a PNG or JPEG decoder written in C# though.

Texture2D.LoadRawTextureData which takes a formatted byte array. Here's an example of someone decoding an image in C# and loading the raw data. Their numbers seem to indicate that most of the time spent is in decoding the texture data which means that there may be significant performance gains to be had if you did the decoding in an optimized native plugin.

Texture2D.CreateExternalTexture This method would require using a native plugin and marshaling a single pointer to the texture data. Here's an example of someone using this technique on iOS. I would assume this is the most efficient method.

It'd be great if Unity offered an async texture loading function themselves and there's a feature request open here. For now I think we'll need to settle for one of these options.

[CWolfs]

There is one more option, which you've touched on.

Internally Unity uses FreeImage library for its texture processing. You can jump into another thread, do all the image work in FreeImage then feed back the results into the main thread for the last mile. We do this with our Collada Importer for Unity for our game's mod tools.

You see definite speed increases and non-blocking behaviour.

[robertlong]

@CWolfs that's good to know! Can you explain your process more? I'm about to start a native plugin for texture loading.

[mhm90]

@robertlong Did you found a plugin for async texture loading? I tried @CWolfs method but it requires building the whole FreeImage lib for Android & iOS which is a huge binary, The FreeImage lib wrapper is only supported in unity editor.

[robertlong]

I'm working on the WebGL/WebVR side of glTF tooling now so unfortunately I no longer have the time to look into async texture loading.

I wish Unity would include this in their engine and have requested the feature multiple times. They must have image compression in their Android/iOS runtime builds already, we just need access to it. There's also probably much smaller libraries to use for this.

I think I remember @bghgary saying he had done some sort of async texture loading for Unity. Perhaps he can lead you in the right direction?

[bghgary]

some sort of async texture loading for Unity

I have, but it's really old now. It is a native plugin that I wrote for x86, so there is some work to do to make it cross-platform.

[amenzies]

I'm working on a Unity implementation of 3D Tiles and texture loading is by far and away the biggest performance offender. We have identified two potential performance concerns:

1. Texture loads are synchronous (decompression and transfer to GPU)
2. Decompression of jpg and png files is slow

Focusing on number 2 for the moment.

Our approach is to encode a GPU encoded DDS texture in the glTF file (as per this extension) and load them in unity using LoadRawTextureData (as per this example).

The downside of this approach is that we still need to encode both textures in the glTF so that viewers without the DDS extension still work. So we do end up with larger glTF files but they load much more smoothly. Theoretically, the image resources could be external references in the glTF and the loader could then only download the encoding it prefers.

Long term it looks like glTF might adopt a universal GPU texture encoding such as Basis KHR_texture_transmission Extension
but it looks like that is still a way off.

In the mean time, would folks be interested in me adding support to this project for the MSFT_texture_dds extension?

[amenzies]

Related to issue #59

[nsmith1024]

Hello, is anyone working on this, because i need it bad, my Unity App sucks because it uses WWW to load a lot of images and ht keeps freezing because of the above problem. Did anyone create a threaded version of it?

[blgrossMS]

@JohnCopicMS has been looking into it. Do you have any status?

[nsmith1024]

One thing that helped, is that when using WWW, for example WWW xxx= new WWW("www.example.com/my.jpg") to load a texture, is never use xxx.texture anywhere in your code, not even to check for null, just the act of accessing it will cause it to create the texture on the main thread causing the freeze. Use it exactly like this only

https://docs.unity3d.com/ScriptReference/WWW.LoadImageIntoTexture.html

I removed all reference to xxx.texture from my code and its WAY faster now, still has a tiny freeze but i dont know if its the texture causing it or something else.

[StephenHodgson]

www is depreciated

[StephenHodgson]

This was an interesting read: https://docs.unity3d.com/Manual/NativePluginInterface.html

[philiptolk]

In the mean time, would folks be interested in me adding support to this project for the MSFT_texture_dds extension?

I and my team here at the Institute for Creative Technology are very interested in your adding the MSFT_texture_dds extension, thanks so much for your insight into the texture loading issue

[MephestoKhaan]

Dont use WWW but UnityWebRequest for loading textures, it does all decompression in a background thread for you.

[blgrossMS]

@pmtolk We will be implementing a new extension mechanism for the library #259 and will be adding dds support then.

[adamkvd]

Is there any update on this? Not actually using GLTF, but facing the same blocking texture load issue in a project and would be very interested in seeing other solutions.

I've explored the manual texture decode in a worker thread, combined with a GetRawTextureData<byte>(), but so far this has failed to provide a good solution across platforms, as custom decode turned out to be quite inefficient compared to whatever LoadImage does. The fact that the decode is on a worker thread doesn't hide the newly introduced inefficiency on weaker platforms, and it's still not eliminating hiccups fully, because allocating texture memory and texture.Apply on the main thread can still take time for large (>=1024) textures. This seems to reflect what @amenzies mentioned earlier.

As far as alternatives go, it seems UnityWebRequestTexture does support background loading of textures as @MephestoKhaan mentioned, but according to the docs only for sRGB color data, not linear colors and data textures, so that's not really a viable solution with PBR in mind.

@amenzies Do you have any data on the efficiency of DDS texture loading for larger texture sizes? Also, did you end up finding an intermediate workaround for platforms without GPU support for DDS textures?

@CWolfs Is FreeImage built into Unity across all platforms, and is there a "safe" way to access it with the right DllImport keywords?

[amenzies]

@adamkvd, I did run a few quick profiling tests with DDS and CRN compressed textures. My primary use case is loading lots of small textures as part of a 3D Tiles implementation and I have found that performance hitches can mostly be avoided even when using jpg/png compression if the texture size is kept at 256x256 and only one or two textures are loaded at a time. I am using UnityWebRequest as noted above. I've found I can actually load a lot more textures per frame on desktop systems but by limiting to 1 or 2 I can hit 60 FPS on a first generation hololens which is pretty good.

I did see some improvements when testing with DDS and CRN but it was quite some time ago and I can't remember how significant the boost was. I think things were leaning slightly in favor of CRN. Could depend on what DDS settings you are using though since there are a lot of options. I'm sure image size makes a big difference here.

I have this branch where I added prototype support for loading DDS and CRN if you want to experiment
https://github.com/amenzies/UnityGLTF/tree/feature/dds-crn-images

I think I was using this utility to create my CRN files but my understanding is that there is a newer version but that it is a commercial solution.
https://github.com/BinomialLLC/crunch

[DaZombieKiller]

As of Unity 2019.3 there's a new SetPixelData API that may come in handy for this, figured it would be useful to point out: https://docs.unity3d.com/2019.3/Documentation/ScriptReference/Texture2D.SetPixelData.html

[nsmith1024]

Do you still have to do "Apply" after you do the SetPixelData? If you do it will still freeze, "Apply" is the big problem because its done on the main thread, if you have a large image it will freeze the engine for a while.

[LeandroExhumed]

There is one more option, which you've touched on.

Internally Unity uses FreeImage library for its texture processing. You can jump into another thread, do all the image work in FreeImage then feed back the results into the main thread for the last mile. We do this with our Collada Importer for Unity for our game's mod tools.

You see definite speed increases and non-blocking behaviour.

Does FreeImage work fine with Unity? Here at the company that I work we are trying to implement it on a Unity project but we are having some problems when the image is loaded.

[CWolfs]

It works but not entirely out of the box. We noticed there were problems with the red and blue channels being reversed. It seemed it was loading as BGR instead of RGB on some operating systems.

We took the source and changed it so it always loads as RGB regardless of operating system. Then this worked great for us on another thread to have non-blocking texture loading.

We load the texture in memory using FreeImage then use Unity's LoadRawTextureData so that it doesn't block as Unity doesn't need to do anything but set the data instead of any pre-processing on the data.

You also need to take into account Normal maps yourself. When you know you're trying to load a normal map then you need to ensure the channels are set up right for a normal map on that texture.

[StephenHodgson]

You also need to take into account Normal maps yourself. When you know you're trying to load a normal map then you need to ensure the channels are set up right for a normal map on that texture.

Thanks for the tidbit. That's important.

[michaellu88]

hi, i am new to unity and i found this loadimage freezing problem when i upload vrm at runtime. After reading the above comment, using FreeImage look like a good option but i can't find a good source to load FreeImage into my Unity. got error on DLLnotfound. i am running on webgl. i need help pls.

[mhm90]

i am running on webgl

FreeImage is only used in the UnityEditor. Beyond the editor, you can't use it. All you can do is load the texture using WebGL calls and then reference it in unity using Texture2D.CreateExternalTexture(). I have implemented a basic native plugin for android long ago, but it lacks the support of multiple texture types and managing the lifecycle of textures since they are created beyond unity.

[pfcDorn]

With the latest updates, I would recommend to use KTX in GLBs. With this format it don't block the mainthread too much :)

[DaZombieKiller]

To my knowledge, the current best way to implement this (as of Unity 2022.1) is as follows:

• Create the Texture2D instances in bulk, passing createUninitialized: true to the constructor.
• For each texture, store the NativeArray<byte> you get from texture.GetRawTextureData<byte>()
  • This will give you access to the CPU-side buffer for all mipmap levels in the texture.
• On background threads, read all of the texture data into the NativeArray<byte> for each texture.
• Call Apply in bulk (possibly using a time-sliced coroutine) on all of the textures, passing updateMipmaps: false.

There is no way to avoid the main thread delay caused by Apply currently. Unity has the capability to do this asynchronously on the C++ side, but none of the functionality necessary to replicate it is exposed to C#. Unity 2023 now exposes Texture.graphicsTexture which is part of what the C++ side uses to perform async texture uploads, but the full functionality is not yet exposed to C# so it can't really be used for manual async upload right now.

It's worth noting that Unity doesn't expose GetRawTextureData<T> right now for the following texture types:

• Cubemap
• CubemapArray
• Texture2DArray
• Texture3D

But it does expose GetPixelData<T>, so it is trivial to manually implement GetRawTextureData<T> as shown here.