Scripts to bake and play tens of thousands of GPU vertex animations

[KontosTwo]

Describe the project you are working on

I am developing a real time tactics game, currently capable of supporting up to 30,000 animated 3D soldiers fighting in 1,200 units. However, this was not possible using out-of-the box Godot functionality which leads to the next section

Describe the problem or limitation you are having in your project

The out-of-the-box Godot functionality to tackle this task of animating tens of thousands of 3D soldiers would initially be a MeshInstance3D and AnimationPlayer pair for every single soldier. However, this would result in massive CPU and GPU time spent both submitting draw calls and computing skeletal mesh animation. The next attempt would be to use MultiMeshInstance3D, but it lacks direct AnimationPlayer integration and can only render one mesh anyway. Thus, it's not possible to easily animate a large number of 3D meshes unless significant work is done.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The feature is based off of the GPU vertex animation tooling I've already implemented. It consists of three modules:

1. Defining: To define an animated 3D mesh, a VertexAnimation Resource stores three other Resources: a mesh (slime), an AnimationPlayer (slime crawl, slime squish, slime dies) compatible with that mesh, and all possible texture variations (blue slime, red slime, evil slime) for that mesh.
2. Baking: To convert these human-readable Resources into a GPU-friendly format, a baker script will run through every single VertexAnimation Resource and convert the three child Resources into vertex and normal textures, which will be stored in a specified folder, and their paths stored in their originating VertexAnimation resource. In addition, all possible texture variations will be combined into one Texture2DArray
3. Playing: A script will accept a VertexAnimation Resource and upload its vertex and normals textures, as well as the texture variations array, into a shader that is implemented to replace the vertex and normals with those stored in the array

This suite of tools and scripts solve the following problems:

• It makes animating tens of thousands or even more 3D meshes possible. Vertex animations are a common optimization technique and adding this feature to Godot would enable its developers to have this many animations as an option.
• It reduces time needed to implement their own GPU vertex animation solution, which is already a common pattern. I can testify that developing these tools took a stressful 3 weeks which others won't have to experience if this proposal were accepted

Here is a demonstration of the results of using these modules

https://www.youtube.com/watch?v=OTbQH3k0q6Q

For 5,000 critters, CPU Time is 2ms, and GPU Time is 10ms for a crowded screen

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The above section gives an introduction to how the three modules work. Here are some code snippets or images to illustrate them in action

1. Defining:
   Here's an example of a resource definition

3. Baking Here's the baker script

using Godot;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using WaywardSoul.Battle;
using WaywardSoul.Helpers;

namespace WaywardSoul.Graphics
{
    public partial class CritterBaker : Node
    {
        [ExportGroup("Actions")]

        [Export]
        private bool Meshes { get; set; } = true;

        [Export]
        private bool Textures { get; set; } = true;

        [ExportGroup("Baking")]

        [Export]
        private float LODBias { get; set; } = 10f;

        [Export(PropertyHint.Dir)]
        private string VertexDirectory { get; set; }

        [Export(PropertyHint.Dir)]
        private string NormalDirectory { get; set; }

        [Export(PropertyHint.Dir)]
        private string TextureArrayDirectory { get; set; }

        [ExportGroup("Dependencies")]

        [Export]
        private AnimationPlayer AnimationPlayer { get; set; }

        [Export]
        private Skeleton3D Skeleton { get; set; }

        public int NumFrames => AnimationPlayer.GetTotalFrameCount(FPS);

        private HashSet<string> BakedMeshes { get; set; }

        private static readonly string NONE_CRITTER = "none.tres";
        public static readonly float FPS = 8;
        public static float SPF => 1 / FPS;

        public override void _Ready()
        {
            BakedMeshes = [];

            if (string.IsNullOrEmpty(VertexDirectory) 
                || string.IsNullOrEmpty(NormalDirectory) 
                || string.IsNullOrEmpty(TextureArrayDirectory))
            {
                GD.PrintErr("Vertex or Normals directory not set");
                return;
            }
            BakeCritterEquipmentProcess();

            GetTree().Quit();
        }

        private void BakeCritterEquipmentProcess()
        {
            var equipmentAndNames = new List<(CritterEquipment, string)>();

            var files = GetCritterEquipmentFiles();
            foreach(var file in files)
            {
                var filePath = file.Item1;
                var resource = ResourceLoader.Load(filePath);
                if (resource is CritterEquipment)
                {
                    equipmentAndNames.Add((resource as CritterEquipment, file.Item2));
                }
            }

            var meshToEquipments = new Dictionary<Mesh, List<(string, CritterEquipment, Texture2D)>>();
            foreach (var equipmentAndName in equipmentAndNames)
            {
                var equipment = equipmentAndName.Item1;
                var name = equipmentAndName.Item2;

                var mesh = equipment.Mesh;
                var texture = equipment.Texture;

                if (!meshToEquipments.ContainsKey(mesh))
                {
                    meshToEquipments.Add(mesh, []);
                }
                meshToEquipments[mesh].Add((name, equipment, texture));
            }

            foreach(var meshPathToEquipment in meshToEquipments)
            {
                var textureArray = new Texture2DArray();
                var equipments = meshPathToEquipment.Value;
                var textures = new Godot.Collections.Array<Image>(equipments.Select(e =>
                {
                    var texture = e.Item3;
                    var image = texture.GetImage();
                    return image;
                }));

                var mesh = meshPathToEquipment.Key;
                var meshPath = mesh.ResourcePath;
                var meshFileName = Path.GetFileNameWithoutExtension(meshPath);

                var vertexImagePath = VertexDirectory + "/" + meshFileName + ".png";
                var normalImagePath = NormalDirectory + "/" + meshFileName + ".png";

                if (Meshes)
                {
                    BakeCritterMesh(mesh, vertexImagePath, normalImagePath);
                }

                if (Textures)
                {
                    var textureArrayErrorCode = textureArray.CreateFromImages(textures);
                    if (textureArrayErrorCode != Error.Ok)
                    {
                        GD.PrintErr("Texture2DArray " + meshFileName + " could not be created: " + textureArrayErrorCode);
                    }

                    var textureArrayPath = TextureArrayDirectory + "/" + meshFileName + ".res";
                    var saveErrorCode = ResourceSaver.Save(textureArray, textureArrayPath);
                    if (saveErrorCode != Error.Ok)
                    {
                        GD.PrintErr("Texture2DArray " + meshFileName + " could not be saved: " + saveErrorCode);
                    }

                    var textureArrayResource = ResourceLoader.Load<Texture2DArray>(textureArrayPath);

                    var numMeshPathToEquipment = equipments.Count;
                    for (var i = 0; i < numMeshPathToEquipment; i++)
                    {
                        var equipment = equipments[i];
                        AssignCritterEquipmentMesh(
                            equipment.Item2,
                            vertexImagePath,
                            normalImagePath,
                            textureArrayResource,
                            i
                        );
                    }
                }
            }
        }

        private void BakeCritterMesh(
            Mesh mesh,
            string vertexImagePath,
            string normalImagePath
        )
        {
            var instance = new MeshInstance3D
            {
                Mesh = mesh,
                LodBias = LODBias,
                Skeleton = Skeleton.GetPath()
            };
            Skeleton.AddChild(instance);

            var meshDataTool = mesh.MeshDataTool();
            var numVertices = meshDataTool.GetVertexCount();

            var numTotalFrames = NumFrames;

            var vertexImage = Image.Create(numTotalFrames, numVertices, false, Image.Format.Rgbaf);
            var normalImage = Image.Create(numTotalFrames, numVertices, false, Image.Format.Rgbaf);

            var frameCounter = 0;
            foreach (var animation in AnimationPlayer.GetAnimationList())
            {
                AnimationPlayer.Play(animation);

                var animationDuration = AnimationPlayer.GetAnimation(animation).Length;
                var numAnimationFrames = Mathf.CeilToInt(animationDuration * FPS);
                for (var i = 0; i < numAnimationFrames; i++)
                {
                    AnimationPlayer.Advance(SPF);

                    var bakedFrame = BakeCritterEquipmentMeshForAnimation(meshDataTool);
                    var bakedFrameEnumerator = bakedFrame.GetEnumerator();

                    for (var j = 0; j < numVertices; j++)
                    {
                        bakedFrameEnumerator.MoveNext();
                        var (vertex, normal) = bakedFrameEnumerator.Current;

                        vertexImage.SetPixel(frameCounter, j, vertex);
                        normalImage.SetPixel(frameCounter, j, normal);
                    }

                    frameCounter++;
                }
            }

            vertexImage.SavePng(vertexImagePath);

            normalImage.SavePng(normalImagePath);

            Skeleton.RemoveChild(instance);
            instance.QueueFree();
        }

        private void AssignCritterEquipmentMesh(
            CritterEquipment equipment, 
            string vertexImagePath,
            string normalImagePath,
            Texture2DArray textureArray, 
            int textureIndex
        )
        {
            equipment.Vertex = ResourceLoader.Load<CompressedTexture2D>(vertexImagePath, cacheMode: ResourceLoader.CacheMode.Ignore);

            equipment.Normal = ResourceLoader.Load<CompressedTexture2D>(normalImagePath, cacheMode: ResourceLoader.CacheMode.Ignore);

            equipment.TextureArray = textureArray;
            equipment.TextureIndex = textureIndex;

            ResourceSaver.Save(equipment);
        }

        private IEnumerable<(Color, Color)> BakeCritterEquipmentMeshForAnimation(MeshDataTool meshDataTool)
        {
            var boneTransformRests = new Dictionary<int,Transform3D>();
            var boneTransformCurrents = new Dictionary<int, Transform3D>();
            for (var i = 0; i < Skeleton.GetBoneCount(); i++)
            {
                boneTransformRests.Add(i, Skeleton.GetBoneGlobalRest(i));
                boneTransformCurrents.Add(i, Skeleton.GetBoneGlobalPose(i));
            }

            for (var i = 0; i < meshDataTool.GetVertexCount(); i++)
            {
                var bones = meshDataTool.GetVertexBones(i);
                var weights = meshDataTool.GetVertexWeights(i);
                var vertex = meshDataTool.GetVertex(i);
                var normal = meshDataTool.GetVertexNormal(i);

                var globalVertex = Vector3.Zero;
                var globalNormal = Vector3.Zero;
                var numBones = bones.Length;
                for (var j = 0; j < numBones; j++)
                {
                    var bone = bones[j];
                    var boneTransformRest = boneTransformRests[bone];
                    var boneTransformCurrent = boneTransformCurrents[bone];
                    var boneWeight = weights[j];

                    var localVertex = boneTransformRest.Inverse() * vertex;
                    var localVertexTransformed = boneTransformCurrent * localVertex;
                    globalVertex += localVertexTransformed * boneWeight;

                    var localNormal = boneTransformRest.Inverse() * normal;
                    var localNormalTransformed = boneTransformCurrent * localNormal;
                    globalNormal += localNormalTransformed * boneWeight;
                }

                globalVertex += Vector3.One;
                globalVertex /= Vector3.One * 2;

                globalNormal = globalNormal.Normalized();
                globalNormal += Vector3.One;
                globalNormal /= Vector3.One * 2;


                var globalVertexColor = new Color(globalVertex.X, globalVertex.Y, globalVertex.Z);
                var globalNormalColor = new Color(globalNormal.X, globalNormal.Y, globalNormal.Z);

                yield return (globalVertexColor, globalNormalColor);
            }
        }

        private List<(string, string)> GetCritterEquipmentFiles()
        {
            return GetCritterEquipmentFiles("res://wayward_soul");
        }

        private List<(string, string)> GetCritterEquipmentFiles(string path)
        {
            var files = new List<(string, string)>();
            var dir = DirAccess.Open(path);
            dir.ListDirBegin();
            var fileName = dir.GetNext();
            while(fileName != "")
            {
                var filePath = path + "/" + fileName;
                if (dir.CurrentIsDir())
                {
                    files.AddRange(GetCritterEquipmentFiles(filePath));
                }
                else
                {
                    if (fileName.EndsWith(".tres") && !fileName.Equals(NONE_CRITTER))
                    {
                        files.Add((filePath, fileName.Replace(".tres","")));
                    }
                }
                fileName = dir.GetNext();
            }
            return files;
        }
    }
}

4. Playing
   Arranging the transform and frame number for the CUSTOM_DATA

[MethodImpl(MethodImplOptions.AggressiveInlining)]
private void UpdateGPUData()
{
    CritterGPUPayloadInstanceData.Transform = Transform;

    var globalAnimationFrame = Access.CritterAnimationCumulativeFrameIndex(CurrentAnimation) + AnimationFrame;

    CritterGPUPayloadInstanceData.FrameIndex = Mathf.FloorToInt(globalAnimationFrame);
}

Uploading transform and custom data to multimesh, and limiting the number of visible instances

for (var j = 0; j < NumThreads; j++)
{
    var model = CritterGPUPayloadCollection.AcquireForThread(j).Models[meshIndex];
    var data = model.Data;
    var start = modelInstanceCount;
    var numInstancesForModel = model.InstancesCount;
    for (var k = 0; k < numInstancesForModel; k++)
    {
        RenderingServer.MultimeshInstanceSetTransform(critterMeshRid, start + k, data[k].Transform);
        RenderingServer.MultimeshInstanceSetCustomData(critterMeshRid, start + k, data[k].CustomData);
    }

    modelInstanceCount += numInstancesForModel;
}
RenderingServer.MultimeshSetVisibleInstances(critterMeshRid, modelInstanceCount);

Shader for MultiMeshInstance3D

shader_type spatial;

#define USE_ALPHA 0
#define USE_ALPHA_CUTOFF 0
#define USE_EMISSION 0
#define USE_REFLECTIONS 0
#define USE_NORMAL_MAP 0
#define USE_OCCLUSION 0
#define USE_ANISOTROPY 0
#define USE_BACKLIGHT 0
#define USE_REFRACTION 0

#if USE_ALPHA
render_mode depth_draw_always;
#endif

//#include "includes/base-cel-shader.gdshaderinc"

#if USE_EMISSION
#include "includes/emission.gdshaderinc"
#endif

#if USE_REFLECTIONS
#include "includes/reflections.gdshaderinc"
#endif

#if USE_NORMAL_MAP
#include "includes/normal-map.gdshaderinc"
#endif

#if USE_OCCLUSION
#include "includes/occlusion.gdshaderinc"
#endif

#if USE_ANISOTROPY
#include "includes/anisotropy.gdshaderinc"
#endif

#if USE_BACKLIGHT
#include "includes/backlight.gdshaderinc"
#endif

#if USE_REFRACTION
#include "includes/refraction.gdshaderinc"
#elif !USE_REFRACTION && USE_ALPHA
#include "includes/transparency.gdshaderinc"
#endif

group_uniforms BaseProperties;
#if USE_ALPHA_CUTOFF
uniform float alpha_cutoff: hint_range(0.0, 1.0) = 0.5;
#endif

uniform float vertex_count;
uniform float frame_count;
uniform sampler2D gpu_animation_vertex: hint_default_white;
uniform sampler2D gpu_animation_normal: hint_default_white;
uniform sampler2DArray gpu_animation_texture;

void vertex() {
	
	float pixel = 1.0 / vertex_count;
	float half_pixel = pixel * 0.5;
	float frame = 1.0 / frame_count;
	float half_frame = frame * 0.5;

	int span_r = floatBitsToInt(INSTANCE_CUSTOM.r);
	float frame_number = float(span_r & 0xFFFF);

	float x = frame_number / frame_count;
	float y = float(VERTEX_ID) / vertex_count;
	vec2 offset = vec2(half_frame, half_pixel);
	vec2 coord = (vec2(x,y) + offset);

	vec4 vertex_color = texture(gpu_animation_vertex, coord);
	VERTEX = (vertex_color.xyz - 0.5) * 2.0;

	vec4 normal_color = texture(gpu_animation_normal, coord);
	NORMAL = (normal_color.xyz - 0.5) * 2.0;

	float tintR = float((span_r >> 24) & 0xFF);

	int span_g = floatBitsToInt(INSTANCE_CUSTOM.g);
	float tintG = float(span_g & 0xFF);
	float tintB = float((span_g >> 8) & 0xFF);

	vec3 tint = vec3(tintR / float(255), tintG / float(255), tintB / float(255));

	float texture_index = float((span_r >> 16) & 0xFF);

	COLOR.rgba = vec4(tint, texture_index);
}
void fragment() {
	float texture_index = COLOR.a;

	ALBEDO = COLOR.rgb * texture(gpu_animation_texture, vec3(UV, texture_index)).rgb;
#if USE_ALPHA
	float alpha = color.a * texture(base_texture, UV).a;
	ALBEDO *= alpha;
#elif USE_ALPHA_CUTOFF
	ALPHA = color.a * texture(base_texture, UV).a;
	ALPHA_SCISSOR_THRESHOLD = color.a * texture(base_texture, UV).a;
#endif
}
}

If this enhancement will not be used often, can it be worked around with a few lines of script?

This feature requires some trial and error to implement the first time, costing developers many weeks for a feature that is common in game requiring many animated 3D meshes

Is there a reason why this should be core and not an add-on in the asset library?

This provides a high requested feature as seen from these posts:

https://forum.godotengine.org/t/how-to-instance-animations/46857
https://godotforums.org/d/19323-anyone-have-luck-with-implementing-gpu-instancing
https://www.reddit.com/r/godot/comments/8d54yy/anyone_have_luck_with_implementing_gpu_instancing/
https://www.reddit.com/r/godot/comments/11d0iot/15000_zombies_rendered_in_godot_on_my_macbook/

The author of the last one actually managed to implement it, but hasn't shared implementation details yet, leaving curious developers in the dark. So this commonly requested feature, which is tooling to easily manage GPU vertex animations to animate tens of thousands of 3D meshes, doesn't exist at the moment. However, if it does, then developers can eliminate weeks of development time to leverage the freedom that comes with being able to animate huge numbers of 3D meshes.

This feature could also be a core feature because it relies entirely on existing Godot public API (basically, no internal calls to the engine's C++ code). This makes maintenance quite easy since nothing fancy is being done.

[KontosTwo]

Here's a better illustration of how the public API could work once the vertex and normal arrays, and the texture variations have been baked