Getting accurate tangent space normal calculations

[ted.mcclung]

Lately I've been trying to workout a problem with tangent space normal maps in Unigine. Basically, there's no real standardization for tangent space coordinate systems, and each engine presents it's own set of variables that can be drastically different from the next. Thus, baking normal maps becomes a problem, since in some basic cases it becomes very obvious that there's a mismatch. 

There's a piece of software called Handplane 3D that attempts to bridge that gap by reading in object space normal maps and the mesh, and spitting out tangent space maps that are calibrated to various engines. My current problem is that they don't have a preset for Unigine, and all of the calibrations I can bake display some degree of warping.

 

These are the meshes I'm using. The right one is the low poly, and the left is the high poly I'm baking from.

 

 

 

These are the various tangent space normal maps applied onto the low poly model in Unigine. Notice how almost all of them display some kind of inverted shadow on the top corner, and similar effects on the surface facing the camera. These surfaces are supposed to be flat. The Source and Unreal 3 engine calibrations are arguably the best, but even those display obvious problems in different areas. 

 

 

 

For reference, this is the high poly model, next to the Source normal map. Notice how the lighting on both sides of the high poly is completely smooth.

 

 

 

Has anybody noticed this problem? There are times when objects are moving that it becomes painfully obvious. What can be done about this?

[ted.mcclung]

Incidentally, I also tried baking from Unigine's resource editor, but the results were even worse. 

 

[silent]

Hi Ted,

 

Did you have a chance to check SDK Browser -> Demos -> Samples 2 -> materials/normal_maps sample? There are couple examples of tangent space normals usage.

 

Thanks!

[ted.mcclung]

Actually, that demo displays exactly the same problem I'm talking about. It also has a mismatch in tangent space calculations.

 

This is the soft normal tangent space object:

 

However, when the object's normal is read in object space in the normal_maps demo it looks great, I haven't seen any of these problems, and the seams look better too. So I guess the solution for me is to start baking things in object space instead. 

 

Interestingly, I also discovered that importing through different means yields different shading on objects when using tangent space normal maps. For example, I imported my box test object through the FBX importer, and also exported using unigine's own 3ds max .mesh exporter plugin, and applied the same material to both objects, and the shading was wildly different. The screenshot I posted on September 10th is of a box exported with the 3ds max .mesh exporter plugin, and it has terrible shading, but when I looked at the same material on the same box imported through the fbx importer, it looked practically perfect. It was a huge change. It seems really weird to me that these two importing methods treat the object in such contrasting ways, when the objects brought in should be identical.

[ted.mcclung]

The one instance where I got a decent result with tangent space normals was importing through fbx and baking with unigine's resource editor, which to be honest, leaves a lot to be desired in the pipeline department. I had problems with different formats scaling incorrectly, it doesn't handle cages nearly as elegantly as xnormal, and most importantly I can't automate the process at all, or bake all the maps that I need, meaning I'd have to set up the bakes in xnormal as well anyway. Considering I often have complex projects requiring me to individually bake 300+ pieces in only a few hours, baking through the resource editor sadly just isn't a viable option.

 

Unfortunately, now that I think about it, using object space normal maps severely limits me in things like reusing uv space, so that's not really a solution either. 

[cryunreal]

Hi! Yes, a tangent space is a real problem for all game engines yet. The best solution is baking normal map in Maya and export .fbx from there. After that on an import step into Unigine you need to turn on Import Tangent Space. With 3ds max this solution doesn't work, because 3ds max writes a wrong tangent space to .fbx

[ted.mcclung]

What version of fbx are you using?

[ted.mcclung]

I've read about quite a few other companies writing scripts to convert world space maps into tangent space maps using their engine's unique tangent basis. Unigine already supports both world space and tangent space normal maps, if there was a feature to do that conversion, either by script or in the editor itself, it would eliminate these problems. 

[danni.coy]

does this mean that if we use handplane and set the output to maya we will get the right results?