1. 3D & Motion Graphics
  2. Maya

Occlusion Shading Opacity Mapped Geometry In Maya

Scroll to top
Read Time: 12 min

Today author Dimitar Katerov will be showing us how to create a custom shading network inside Maya, which will allow you to get Occlusion information from opacity mapped geometry, something that is always a challenging task. Dimitar will also cover creating a tree using PaintFX, as well as the lighting and rendering setup using Mentalray.

Republished Tutorial

Every few weeks, we revisit some of our reader's favorite posts from throughout the history of the site. This tutorial was first published in April of 2011.

Paint FX is a very powerful way to create vegetation in your 3D scene. Getting occlusion from Opacity mapped geometry is a little bit tricky, but it isn't hard at all. It's very useful to be able to do Occlusion shading on opacity mapped planes, in order to get detailed tree leaves, without the need of modeling them completely. Just mapping the planes with leaf opacity texture saves a lot of time and polycount. Onyx tree and similar software also generate planes instead of leaves so this method is applicable as well.

I will show a shading network for a 2 sided material, because Maya doesn't have one built in,
This is very handy when making leaves as well, because the top and bottom sides have different Color, Bump and Reflectivity values.

Step 1

A little bit of theory and explanation, before we start with the actual tutorial. Most artists use Occlusion to add an extra bit of realism to their renders. The problem with the conventional Occlusion Shading methods
is that you can only occlude true geometry. You can't directly apply an opacity map and even if you apply a mib_transparency node (which is used for creating the custom shading network) you can see that the whole plane is generating shadows.

Step 1 ImageStep 1 ImageStep 1 Image

Because we are going to use Mental ray nodes, make sure Mental ray is loaded in Maya by choosing Window>Settings/Preferences>Plug-in manager and in there check both Loaded and Auto load check boxes for Mayatomr.mll.

So lets start with the actual tutorial.

Step 2

For creating the custom occlusion network, first open Hypershade (Window>Rendering Editors>Hypershade.) Make two Lambert materials by choosing Maya>Surface>Lambert from the Create panel. The first is for all geometry that is not Opacity mapped and the other is for the Opacity mapped geometry.

Step 2 ImageStep 2 ImageStep 2 Image

Step 3

Create mib_transparency (mental ray>Sample Compositing>mib_transparency) and mib_fg_occlusion (mental ray>MentalRay Lights>mib_fg_occlusion.) The final node for this network is an actual texture for the Opacity mapping. Create file node (Maya >2D Textures > File.)

Step 3 ImageStep 3 ImageStep 3 Image

Step 4

Double click on file2 node. This will open the Attribute Editor for the file2 node. Choose your texture by clicking the Folder icon next to Image Name field. Make sure the part that you want to be transparent is white. Choose Filter Type Off and Alpha Is Luminance . That way your texture isn't going to be blurred by the filter.

Step 4 ImageStep 4 ImageStep 4 Image

Step 5

Select all the nodes that you've created in the Hypershade editor and click the Input and output connections shown in the screenshot. This way you reveal the Shading Groups for Lambert2 and Lambert3 shaders.

(You can find them in Shading Groups tab in Hypershade and MMB-drag them to your Work Area below)

Step 5 ImageStep 5 ImageStep 5 Image

Step 6

Middle mouse button-drag (MMB-drag) your file node to mib_transparency node and choose transp. This way you link the texture to the Transparency of the mib_transparency node.

Step 6 ImageStep 6 ImageStep 6 Image

Step 7

MMB-drag your file node to mib_transparency node to lambert3SG (lambert3 Shading group) and choose Other.

Step 7 ImageStep 7 ImageStep 7 Image

Step 8

In the following window choose to connect the outValue to miMaterialShader. This is connecting the resulting value of color plus transparency of the mib_transparency to the lambert 3 shader (when using Mental ray).

Step 8 ImageStep 8 ImageStep 8 Image

Step 9

Do the same for the connection between mib_fg_occlusion node and lambert2SG. Your connections should look like this when you hover over them.

Step 9 ImageStep 9 ImageStep 9 Image

Step 10

Now the Lambert2 material is getting Occlusion information and can be assigned to all objects in the scene which don't need a Opacity map. Lambert3 is casting Occlusion shadows but its color is solid and doesn't get any shading info. To fix this you must connect mib_fg_occlusion to the color of non transparent parts of the mib_transparency node.

To do it just MMB-drag mib_fg_occlusion to mib_transparency and choose input.

Step 10 ImageStep 10 ImageStep 10 Image

Step 11

That is all for the Occlusion Shader. Now lets make the two sided material for the leaves. As you probably know Maya does not have a built in two sided material that allows one face to have different shaders on the top and bottom sides. To make good looking leaves you need to use different materials, because the top and bottom sides of a leaf are usually different. To avoid Extruding and UV-mapping all of your planar geometry, or Extruding and assigning different materials to the top and bottom faces, you can achieve this by making a simple two sided material. This material will automatically assign different shaders to the top and bottom faces based on the face normals.

For the two sided material make two mia_material_x_passes by (mental ray>Materials>mia_material_x_passes) in the create panel of the Hypershade editor. One for the top and one for the bottom side. (You can make any two kinds of shaders, I just prefer mia_material_x_passes, you can use for example Blinn and Lambert)

Step 11 ImageStep 11 ImageStep 11 Image

Step 12

Create a condition node, (Maya>Utilities>Condition) and a samplerInfo node (Maya>Utilities>Sampler Info.) If the old shaders (the occlusion ones) are still in the work area, you can Select the two mia_material_x_passes nodes and those which you just created, and select the Input connections icon ( left from the Input and output connections ). This way you will be left only with these nodes in the screenshot. They are the only ones that you need for the two sided material network.

Step 12 ImageStep 12 ImageStep 12 Image

Step 13

Because mia_material_x_passes1 and mia_material_x_passes2 are too long of names and you cannot distinguish them at all, you can rename them for ease of use. Rename them by holding the right mouse button (RMB) on the material and choosing Rename.

Rename one of them to topMat and the other to botMat.

Step 13 ImageStep 13 ImageStep 13 Image

Step 14

MMB-drag topMat to the condition node and choose colorIfTrue.

Step 14 ImageStep 14 ImageStep 14 Image

Step 15

In the following window, in the left panel scroll down and find result, click it and on the right side choose colorIfTrue. This way the result from the topMat shader (color, bump, spec, refl ... etc.) will be shown if the condition of the condition node is met.

Step 15 ImageStep 15 ImageStep 15 Image

Step 16

Do the same for the connection between botMat and the condition node, except you connect it's result to colorIfFalse. Your connections should look like this when you hover over them.

Step 16 ImageStep 16 ImageStep 16 Image

Step 17

Now you must connect the samplerInfo node to the condition node. Again MMB-drag the samplerInfo node to the condition node, select Other and from the Connection Editor window that appears, choose flippedNormal in the left panel and firstTerm in the right panel. This sets the condition node to use the normal information on all faces to decide which material is to be used ( topMat or botMat ).

Step 17 ImageStep 17 ImageStep 17 Image

Step 18

Create one more mia_material_x_passes and rename it to leafMat. This will be the material that you are going to assign to the leaves. The condition will take care of assigning the topMat or botMat. MMB-drag the condition node to the leafMat and choose Other.

Step 18 ImageStep 18 ImageStep 18 Image

Step 19

In the Connection Editor window that appears, choose outColor in the left panel and diffuse in the right panel. This sets all the material connections. You will have to make one more adjustment and assign textures for color, bump and opacity.

Step 19 ImageStep 19 ImageStep 19 Image

Step 20

Create a file node (Maya>2D Textures>File) then select the file node itself and duplicate it (Ctrl+D) four times so you end up with five file nodes, then MMB-drag the place2dTexture node to the four new file nodes and choose Default. This allows you to control all of them by adjusting one placement node and also makes sure they all are placed in exactly the same way. If you for example want to make your leaves more narrow, you can simply adjust the placement node and it will do it automatically for the color bump and opacity textures. Rename the first two - topColor and topBump, next two botColor and botBump and the last one opacity.

Step 20 ImageStep 20 ImageStep 20 Image

Step 21

Double click on each one and assign the according texture in the Attribute editor. Make the Filter = 0.1 so it doesn't blur your textures too much. For the opacity texture choose filter type Off.

Step 21 ImageStep 21 ImageStep 21 Image

Step 22

Double click on the topMat and it will open in the Attribute Editor. MMB-drag topColor to topMat's Color and topBump to Standard Bump of the topMat. When you drag the topBump to Standard Bump it will create a new bump node and open it in the Attribute editor. For bump depth enter 0.2. Do the same for botMat with botColor and botBump.

Step 22 ImageStep 22 ImageStep 22 Image

Step 23

Then double click on the leafMat and MMB-drag the opacity file node to CutoutOpacity in the leafMat attributes.

Step 23 ImageStep 23 ImageStep 23 Image

Step 24

Double click on the topMat and adjust the Reflection values (Reflectivity. Glossiness, Glossy Samples). Do the same for the botMat, but make it less reflective (Reflectivity 0.1).

Step 24 ImageStep 24 ImageStep 24 Image

Step 25

That's it for the materials. For creating the tree on which we will test the created materials, first make the backdrop by making a plane and extruding one of the edges up and beveling the edge between the two faces.

Step 25 ImageStep 25 ImageStep 25 Image

Step 26

To make the actual PaintFX tree, you first switch your menu setting to Rendering and then choosing - Paint Effects>Paint Effects Tool and then Paint Effects>Get Brush

Step 26 ImageStep 26 ImageStep 26 Image

Step 27

The Get Brush command will open the Visor. In the Visor window choose trees and from the trees that appear in the right panel, choose the one that best resembles the final look you are after. I chose the keyakiPark tree. Click somewhere on the Grid and PaintFX will create that tree for you.

Step 27 ImageStep 27 ImageStep 27 Image

Step 28

Close the Visor and select the tree you just created. You should be able to find the attributes of that tree in the Attributes Editor . If you don't see the Attribute editor press Ctrl+A. Choose the tab with the name of your tree brush, in my case keyakiPark. In the Tubes/Creation part of the Attributes, crank up the Segments to make it more detailed, and the Length Min and Max to make it taller and set Tube Width 1 and 2 to make it thicker.

Step 28 ImageStep 28 ImageStep 28 Image

Step 29

Turn on Branches, Twigs and Leaves and Branch After Twigs check boxes. Adjust the settings for Branches and Twigs. I tend not to use more then two for the Num Branches, because trees rarely split into more then two branches at the same point. This is also true for the Twigs In Cluster.

Step 29 ImageStep 29 ImageStep 29 Image

Step 30

Adjust the Twig Length Scale to achieve a good looking tree crown. For the leaves Choose Leaf Location On Secondary Branches Only, Leaf Start to near zero, so there are leaves from the beginning of the branches. Turn off Leaf Forward Twist. Which if on, makes all the leaves face the camera. Crank the Num Leaf Clusters until you have enough leaves.

Step 30 ImageStep 30 ImageStep 30 Image

Step 31

This i's what my tree looks like.

Step 31 ImageStep 31 ImageStep 31 Image

Step 32

Create an area light, select it and turn on - Use Ray Trace Shadows, Use Light Shape and set High Samples to 64.

Step 32 ImageStep 32 ImageStep 32 Image

Step 33

Make another Area Light, then go ahead and adjust the same attributes like the first Area Light, just give it a little less Intensity. Position the lights roughly at the positions shown below (the left one is the first Area light).

Step 33 ImageStep 33 ImageStep 33 Image

Step 34

Because you are going to use Mental ray as the rendering engine, you must convert PaintFX trees to polygons because Mental ray can't render PaintFX. Just select the tree and choose Modify>Convert>Paint Effects to Polygons.

Step 34 ImageStep 34 ImageStep 34 Image

Step 35

Select everything in your scene, go to Channel Box / Layer Editor choose Render layers and hit Create new layer and assign selected objects (the outlined icon in the right side of the Render layers Panel). Double click on the newly created layer and rename it Color.

Step 35 ImageStep 35 ImageStep 35 Image

Step 36

Choose the Color layer, then choose the leaves, hold RMB (Right Mouse Button) and choose Assign Existing Material and then select leafMat. Leave the trunk with the shader that the PaintFX to Polygons conversion made for it, or make a new one with color, bump textures and glossy reflections.

Step 36 ImageStep 36 ImageStep 36 Image

Step 37

Open the Render Settings window and choose mental ray for the rendering engine. In the Quality tab make the following adjustments: Adaptive Sampling with Max Sample Level of 2, Filter - Mitchell (Filter Size 4,4). For Raytracing set the Reflections and Refractions to 10, and Max Trace Depth to 20. This is very important for the Occlusion to work well. In the Indirect Lighting tab everything should be turned off (GI off, FG off etc.)

Step 37 ImageStep 37 ImageStep 37 Image

Step 38

Select the tree and the backdrop (don't select the lights) and create a new Render layer and name it Occ (for occlusion). Assign lambert3 to the leaves and lamber2 to the trunk and backdrop. Click the Render settings button for this layer (the button outlined with green in the screenshot).

Step 38 ImageStep 38 ImageStep 38 Image

Step 39

In the Render Settings window, in the Indirect Lighting tab, hold RMB over Final Gathering and choose Create layer override.

Step 39 ImageStep 39 ImageStep 39 Image

Step 40

This will make Final Gathering orange which means that if you turn it on or off it will affect only this render layer. Turn Final Gathering on by clicking on the checkbox and enter the following settings: Accuracy 100, Point Density 2, Point Interpolation 10.

Step 40 ImageStep 40 ImageStep 40 Image

Step 41

Close the Render Settings window and from the Camera menu choose View>Camera Attribute Editor...

Step 41 ImageStep 41 ImageStep 41 Image

Step 42

In the Attribute Editor, Environment tab hold RMB and make layer Override for the Background Color, and make it white.

Step 42 ImageStep 42 ImageStep 42 Image

Step 43

Choose Options and turn on Render All Layers in the Render layers panel. Then choose Render All Layers Options by clicking the square icon next to Render All Layers.

Step 43 ImageStep 43 ImageStep 43 Image

Step 44

Set Keep image mode to Keep layers and close this window. Hit render and you will get one Color and one Occlusion image saved in your render view. Just save them to your hardrive and you can combine them later in Photoshop.

Step 44 ImageStep 44 ImageStep 44 Image

Step 45

You can see the final result with Overlay Blending mode in Photoshop.

That's all for this tutorial I really hope you liked it.

final Imagefinal Imagefinal Image

Did you find this post useful?
Want a weekly email summary?
Subscribe below and we’ll send you a weekly email summary of all new 3D & Motion Graphics tutorials. Never miss out on learning about the next big thing.
Looking for something to help kick start your next project?
Envato Market has a range of items for sale to help get you started.