Back-face collisions and ray-cast surface normals

[mihe]

This is sort of a two-part question.

Godot allows for enabling back-face collision for its ConcavePolygonShape3D (JPH::MeshShape) on a per-shape basis. I implemented this in a somewhat quick-and-dirty manner by simply duplicating the triangles, flipping the winding order and relying on your Indexify to deduplicate the vertices, although I'll admit I haven't confirmed that this last step actually happens.

So I guess my first question is: Is this the way I should be doing it, or is there a better way to achieve it?

If this is indeed how it should be done, then this approach seems to run up against a problem with regards to retrieving the surface normal from a ray-cast. As per your documentation I'm using Body::GetWorldSpaceSurfaceNormal to fetch the surface normal, using the position retrieved from RayCast::GetPointOnRay. This seems to become a problem when you have two perfectly overlapping triangles like I do for these duplicated back-face triangles, where it seems to somewhat randomly pick either of the two triangle normals, meaning you sometimes end up with the wrong one.

One way of resolving this could be for me to just check if the normal is facing in the same direction as the ray itself, in which case I just flip it, but I figured I should check to see what ideas you might have.

I'd be happy to throw together a Samples repro if something here sounds off.

[jrouwe]

The first thing that comes to mind is to create a DecoratedShape for this. This DecoratedShape would directly forward all collision queries to its child and only override the CastRay and CollisionDispatch::sRegisterCastShape methods. The CastRay function can copy the RayCastSettings object but set mBackFaceMode to EBackFaceMode::CollideWithBackFaces to make the mesh double sided. You would indeed need to flip the normal yourself based on the ray cast direction or you could add an extra bit to the SubShapeID to identify a back facing hit by wrapping CastRayCollector, adding the bit and then forwarding it to the user provided CastRayCollector. Override Shape::GetSurfaceNormal to read this bit, call the inner shape Shape::GetSurfaceNormal and flip the normal if the bit is set.

For CastShape you can use a similar trick on the ShapeCastSettings.

[mihe]

That's an interesting approach! It sounds like this might also allow me to conditionally ignore back-face hits based on Godot's hit_back_faces ray-cast parameter, even when the obstructing shape uses back-face collision, which is another problem I've been noodling over for a while now.

I'll give that all a try. Thank you!

[mihe]

The DecoratedShape coupled with the custom dispatch functions worked like a charm.

[AndreaCatania]

This question helped me to solve the same issue, thanks a lot!