Implement support for glTF as an interchange format (import/export, Khronos/OMI extensions)

[aaronfranke]

Is your feature request related to a problem? Please describe.

The status quo of 3D content development (Unity, Unreal, Godot, VRChat, Resonite, etc) is to use an engine-specific or even game-specific format. This means that, beyond the basics (mesh and skeleton data), a lot of content cannot be easily ported from one platform to another, as it requires a lot of work to set up game objects.

For example, if you import a chair into VRChat/Resonite/etc, you typically would get only the mesh but can't interact with the chair, and you would need to do additional work to inform the game of how players should sit in the chair.

The same problem applies to many other situations. Let's say you want to import a boom box into Resonite. You'd bring in a 3D model, but all it will do is sit there and look like a boom box. To actually use it, you'd need to attach an audio emitter to it, add physics shapes, make it a rigid body, define its mass, and so on.

Describe the solution you'd like

My proposed solution is to use glTF plus extensions from Khronos, OMI, and more as a a general-purpose engine-agnostic 3D asset interchange format.

OMI has been developing glTF extensions for this purpose. We have extensions for things like physics shapes (boxes, spheres, etc), physics bodies (dynamic, kinematic, mass, velocity, etc), audio emitters (playing MP3 or WAV files globally or at a specific point in the world), gravity, seats, spawn points, and more. You can find the repo with OMI's extensions here: https://github.com/omigroup/gltf-extensions/tree/main/extensions/2.0

The glTF extensions are built upon the glTF format as a base, which can be thought of as an open standard replacement for FBX files, while glTF with extensions may one day replace Unity packages, or at least reduce the amount of engine-specific setup needed after importing a 3D model file.

Khronos, the creators of glTF, are also working on extending glTF. The lights extension allows for lights to be exported from Blender. Some other interesting extensions worth mentioning are KHR_animation_pointer, allowing for any property to be animated (light color, camera FOV, and more), and KHR_interactivity, allowing for engine-agnostic scripting using behavior graphs (like visual scripting).

Most of the extensions I mentioned above have been implemented in Godot Engine (mostly by me). Here's a list of extensions and their implementation status in Godot Engine.

Officially in Godot Engine:

• Lights: Implemented in Godot 4.0 and later (and also in Blender).
• WebP/KTX textures: Implemented in Godot 4.2 and later (and WebP is in Blender).
• Root node handling: Implemented in Godot 4.2 and later. GLTF: Add root node export options and GODOT_single_root extension godotengine/godot#81851
• Physics: Implemented in Godot 4.3 and later (several PRs, since this is such a big feature).
• Animation pointer: Not yet merged, Godot 4.4+ GLTF: Implement KHR_animation_pointer for animating custom properties godotengine/godot#94165
• Audio: Not yet merged, Godot 4.4+ Add audio support to the GLTF module godotengine/godot#88204
• Visibility: Not yet merged, Godot 4.4+ Implement KHR_node_visibility in the GLTF module godotengine/godot#93722

Godot Plugins:

• Copyright: Not in the engine to allow easy extensibility https://github.com/aaronfranke/godot-gltf-khr-xmp-copyright
• Seat: Probably too game-specific to be included in the engine, but might be added later https://github.com/omigroup/omi-godot/tree/main/addons/omi_extensions/seat
• Spawn point: Also probably too game-specific to be included in the engine https://github.com/omigroup/omi-godot/blob/main/addons/omi_extensions/misc/omi_spawn_point.gd
• Gravity: Implemented here Add OMI_physics_gravity implementation and test files omigroup/omi-godot#5 but requires an unmerged engine PR Area2D/3D Gravity: Replace point bool with a type enum, add Target gravity type godotengine/godot#82878
• Scene composition: glTFX, not yet finalized by Khronos, but may be added to the engine officially in a few years https://github.com/aaronfranke/godot-gltfx-reference-format
• Scripting: We may implement KHR_interactivity support in a Godot plugin in the future, but it probably won't be within the next year, as this feature would be huge.
• Vehicles: Implemented here Add OMI_vehicle_* implementation and test files omigroup/omi-godot#11
• Avatars: VRM is a format for 3D characters built on top of glTF. It has a Godot implementation here https://github.com/V-Sekai/godot-vrm and is also implemented in apps like VRoid Studio, VSeeFace, Pawperty Damage, and more. VRM includes glTF extensions for things like bone constraints (ex: digitigrade legs, taurs, etc) and spring bones (ex: floppy ears on a furry character).

This is an ongoing effort, the glTF format itself is very new (the 2.0 version that everyone uses is only about 7 years old), OMI is only 3 years old, and several of these extensions are not yet finalized.

Also, if there is anything important missing that would be helpful for a Blender/etc -> Resonite pipeline, feel free to propose new extensions. One extension that I have in my head that haven't been developed yet is one for skybox/environment settings.

Describe alternatives you've considered

The alternative to glTF would be to use another open format (not recommended because that would cause fragmentation) or to use an engine-specific or game-specific format (not recommended for the reasons listed at the top).

If using a game-specific format, this would mean that Resonite needs to implement support for importing from many platforms, instead of just implementing import of glTF+extensions and having the other platforms export to this format. Ultimately I believe the effort will be better spent if we have a pipeline of Resonite, Blender, Godot, etc being able to share content via glTF + OMI/etc extensions, instead of separately implementing pipelines for Blender -> Resonite, Godot -> Resonite, Unity -> Resonite, VRChat -> Resonite, etc (and potentially, the reverse of all of those, and all other possible combinations...).

Additional Context

See the discussion in #2335

Khronos glTF core spec (posting for completeness but you probably don't need to look at this) https://github.com/KhronosGroup/glTF/tree/main/specification/2.0

Khronos glTF extensions https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Khronos

Khronos-reviewed vendor extensions https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Vendor

Khronos WIP interactivity/scripting extension KhronosGroup/glTF#2293

More vendor extensions, unreviewed and work-in-progress https://github.com/KhronosGroup/glTF/pulls

VRM specification https://github.com/vrm-c/vrm-specification (note: original documents are written in Japanese, the English version can be hard to read) and see issue #7

OMI extension plugins for Godot (anything not in the engine) https://github.com/omigroup/omi-godot

And I'll post again for redundancy the link to OMI's extensions https://github.com/omigroup/gltf-extensions/tree/main/extensions/2.0

Requesters

@aaronfranke @madjin and everyone else at @omigroup