Claudia Herbst-Tait
Professor
Department of Digital Arts
Pratt Institute



Extracting Displacement Maps in Mudbox for Use in Maya

To demonstrate how to convert detail of high res meshes in Mudbox into displacement maps for Mental Ray renderings in Maya, I'll use the rose stencil I previously used in class and apply it to a simple sphere. You can use other stencils or sculpting techniques on any object you wish. Please note that later on you will need the low and high-res version of your objects -- thus, make sure not to save over your low-res model.

Import your low-res model of choice; up the level of resolution (shift+D) and apply the rose stencil (or any other stencil or sculpt tool...). In my case, it looks like this (figure 1). If you look closely, you'll notice I added a lot of detail to my sphere for the rose stencil to work.

Figure 1

Select your object and go to File, Export Selection. Name it "high-res". Now import the low-res mesh. Both objects should be sitting on top of each other. You can also look at the Object List (the tab next to Layers) and confirm both your objects are there. You can also easily select objects in the Object List and rename them. Right-click on the low-res model to rename it "low-res". Right-click on the high-res model to rename it "high-res".

In the main menu, got to Maps, Extract Texture Maps, New Operation. Check Displacement Map and give your operation a name (I named mine "Class Demo"). Also specify your Target Model (low-res mesh) and your Source Model (high-res mesh). Lastly, on the bottom, next to Base File Name, click the little folder icon and specify a path where to save the file and file name. I named mine "detail". (FYI, notice that this will generate a .tif 32-Bit FP image.) See figure 2.

Figure 2

Click Extract. After a moment, you should get a message that the extraction was completed without a problem. Now open the image in Photoshop. You will be prompted the following window -- use the same setting as indicated in figure 3.

Figure 3

Now save the image again as a ".tif" (give it a new name). Use the settings as indicated in figure 4. Why do this? Because otherwise Mental Ray won't be able to recognize your ".tif" image and report an error message.

Figure 4

Now let's open Maya. Before we get started assigning our map, go to main menu, Windows, Setting/Preferences, Plug-in Manager. Make sure the tiffFloatReader.bundle is loaded (see figure 5).

Figure 5

Next, set the renderer to Mental Ray. Create a Blinn material (you may also use different materials I just want to have a specular attribute). Assign the map you just saved to the Blinn's Displacement map attribute (just in case you forgot, to get to the Displacement attribute input, in the material's Attribute Editor, click the small icon as indicated in figure 6).

Figure 6

Import your low-res model and assign the Blinn material. Using Mental Ray, do a test render. In my case, it looks like this (figure 7).

Figure 7 (eeks!)

Ok, so this needs work. Typically, when adjusting a displacement map, I would start with the geometry itself (select the object, open the Attribute Editor, open the Displacement Map tab -- adjusting the Initial Sample Rate can give you a better representation of your displacement map. See this link for an example. While you want to be aware of the Initial Sample Rate attribute, it won't solve our problem... Moving right along... smooth your object and do another test render... when smoothed, the sphere now looks like this (figure 8).

Figure 8

What's up with the hole? We'll talk about this in classs... First thing we want to do is delete the smooth node (we're trying to save time rendering). Now let's fix the hole...

Back to Mudbox for a moment. In Mudbox, perform another map extraction but this time set the Choose Samples option to "Furthest Outside" and increase the search distance as indicated in figure 9 (in my case, I changed the value from 7 to 12 units -- depending on your model, you may have to try a larger value). Chose a new Base File Name and click Extract.

Figure 9

In Maya, replace the old displacement map with the new one you just created. Do another test render -- the hole should now be gone. Here's the rose without the hole... (figure 10)

Figure 10

Now, to make this look nicer without smoothing the object... Select your object, go to Window, Rendering Editors, Mental Ray, Approximation Editor. Click the Create button next to the Subdivision section as indicated in figure 11.

Figure 11

A new node has been created. You should see it in the Attribute Editor -- see the following options (figure 12) -- increase the N Subdivision one unit at a time, test render and compare the results. In my case, I increase the N Subdivision to a value of 4 and get the result shown in figure 13. Note: The higher the values used, the longer the render time... You may also experiment withincreasing the Length value (in small steps, try 0.001 or 0.01, this value depends on the level of detail in your map) and the Approx Method option (try Spatial).

Figure 12

(Lucky) Figure 13

Looks pretty good...

Instead of a Displacement Map, you can also save out a Normal Map -- here just a few notes, more on this in class...

In Mudbox, load your low and high res objects as you did above. In the Extract Texture Maps options window, select Normal Map. Set image size to 2048 by 2048, Coordinate Space to Tangent, Compatibility to Maya, set a Base File Name (as you did above...) -- make sure to select 8 bit and .tif image format when specifying a file name. Click Extract.

Next, open the Normal Map in Photoshop and save under a new file name -- for the options that come up, use the same settings you used above.

In Maya, import your low-res object, create a Phong or Blinn material and assign it to your object. Apply a Bump map to your material of choice, and -- in the bump2d node, set the Use As option to Tangent Space Normals (default is Bump). Go into shaded mode with textures (hit 6) -- now, in your perspective window panel, go to Renderer, change this setting from Default Quality Rendering to High Quality Rendering. You should see your map show up in the perspective window while in shaded mode.

The object still looks rough -- with the object selected, in the Polygon module, go to Normals, Soften Edge... you should now see a smooth object that includes the detail of your high-res model and renders quickly.... Do a test render using Mental Ray.... Notice that, unlike when using a Displacement map, a Normal Map does not change the appearance of your object in profile. See figure 14 for a straight on rendering with the Normal Map (it looks like a three-dimensional relief on the object) and figure 15 for the same map in profile (no relief)....

Figure 14

Figure 15