Today author Dimitar Katerov will be showing us how to create a custom shading network inside Maya, which will allow you to get Occlusion information from opacity mapped geometry, something that is always a challenging task. Dimitar will also cover creating a tree using PaintFX, as well as the lighting and rendering setup using Mentalray.
Every few weeks, we revisit some of our reader's favorite posts from throughout the history of the site. This tutorial was first published in April of 2011.
Paint FX is a very powerful way to create vegetation in your 3D scene. Getting occlusion from Opacity mapped geometry is a little bit tricky, but it isn't hard at all. It's very useful to be able to do Occlusion shading on opacity mapped planes, in order to get detailed tree leaves, without the need of modeling them completely. Just mapping the planes with leaf opacity texture saves a lot of time and polycount. Onyx tree and similar software also generate planes instead of leaves so this method is applicable as well.
I will show a shading network for a 2 sided material, because Maya doesn't have one built in,
This is very handy when making leaves as well, because the top and bottom sides have different Color, Bump and Reflectivity values.
A little bit of theory and explanation, before we start with the actual tutorial. Most artists use Occlusion to add an extra bit of realism to their renders. The problem with the conventional Occlusion Shading methods
is that you can only occlude true geometry. You can't directly apply an opacity map and even if you apply a mib_transparency node (which is used for creating the custom shading network) you can see that the whole plane is generating shadows.
Because we are going to use Mental ray nodes, make sure Mental ray is loaded in Maya by choosing Window>Settings/Preferences>Plug-in manager and in there check both Loaded and Auto load check boxes for Mayatomr.mll.
So lets start with the actual tutorial.
For creating the custom occlusion network, first open Hypershade (Window>Rendering Editors>Hypershade.) Make two Lambert materials by choosing Maya>Surface>Lambert from the Create panel. The first is for all geometry that is not Opacity mapped and the other is for the Opacity mapped geometry.
Create mib_transparency (mental ray>Sample Compositing>mib_transparency) and mib_fg_occlusion (mental ray>MentalRay Lights>mib_fg_occlusion.) The final node for this network is an actual texture for the Opacity mapping. Create file node (Maya >2D Textures > File.)
Double click on file2 node. This will open the Attribute Editor for the file2 node. Choose your texture by clicking the Folder icon next to Image Name field. Make sure the part that you want to be transparent is white. Choose Filter Type Off and Alpha Is Luminance . That way your texture isn't going to be blurred by the filter.
Select all the nodes that you've created in the Hypershade editor and click the Input and output connections shown in the screenshot. This way you reveal the Shading Groups for Lambert2 and Lambert3 shaders.
(You can find them in Shading Groups tab in Hypershade and MMB-drag them to your Work Area below)
Middle mouse button-drag (MMB-drag) your file node to mib_transparency node and choose transp. This way you link the texture to the Transparency of the mib_transparency node.
MMB-drag your file node to mib_transparency node to lambert3SG (lambert3 Shading group) and choose Other.
In the following window choose to connect the outValue to miMaterialShader. This is connecting the resulting value of color plus transparency of the mib_transparency to the lambert 3 shader (when using Mental ray).
Do the same for the connection between mib_fg_occlusion node and lambert2SG. Your connections should look like this when you hover over them.
Now the Lambert2 material is getting Occlusion information and can be assigned to all objects in the scene which don't need a Opacity map. Lambert3 is casting Occlusion shadows but its color is solid and doesn't get any shading info. To fix this you must connect mib_fg_occlusion to the color of non transparent parts of the mib_transparency node.
To do it just MMB-drag mib_fg_occlusion to mib_transparency and choose input.
That is all for the Occlusion Shader. Now lets make the two sided material for the leaves. As you probably know Maya does not have a built in two sided material that allows one face to have different shaders on the top and bottom sides. To make good looking leaves you need to use different materials, because the top and bottom sides of a leaf are usually different. To avoid Extruding and UV-mapping all of your planar geometry, or Extruding and assigning different materials to the top and bottom faces, you can achieve this by making a simple two sided material. This material will automatically assign different shaders to the top and bottom faces based on the face normals.
For the two sided material make two mia_material_x_passes by (mental ray>Materials>mia_material_x_passes) in the create panel of the Hypershade editor. One for the top and one for the bottom side. (You can make any two kinds of shaders, I just prefer mia_material_x_passes, you can use for example Blinn and Lambert)
Create a condition node, (Maya>Utilities>Condition) and a samplerInfo node (Maya>Utilities>Sampler Info.) If the old shaders (the occlusion ones) are still in the work area, you can Select the two mia_material_x_passes nodes and those which you just created, and select the Input connections icon ( left from the Input and output connections ). This way you will be left only with these nodes in the screenshot. They are the only ones that you need for the two sided material network.
Because mia_material_x_passes1 and mia_material_x_passes2 are too long of names and you cannot distinguish them at all, you can rename them for ease of use. Rename them by holding the right mouse button (RMB) on the material and choosing Rename.
Rename one of them to topMat and the other to botMat.
MMB-drag topMat to the condition node and choose colorIfTrue.
In the following window, in the left panel scroll down and find result, click it and on the right side choose colorIfTrue. This way the result from the topMat shader (color, bump, spec, refl ... etc.) will be shown if the condition of the condition node is met.
Do the same for the connection between botMat and the condition node, except you connect it's result to colorIfFalse. Your connections should look like this when you hover over them.
Now you must connect the samplerInfo node to the condition node. Again MMB-drag the samplerInfo node to the condition node, select Other and from the Connection Editor window that appears, choose flippedNormal in the left panel and firstTerm in the right panel. This sets the condition node to use the normal information on all faces to decide which material is to be used ( topMat or botMat ).
Create one more mia_material_x_passes and rename it to leafMat. This will be the material that you are going to assign to the leaves. The condition will take care of assigning the topMat or botMat. MMB-drag the condition node to the leafMat and choose Other.
In the Connection Editor window that appears, choose outColor in the left panel and diffuse in the right panel. This sets all the material connections. You will have to make one more adjustment and assign textures for color, bump and opacity.
Create a file node (Maya>2D Textures>File) then select the file node itself and duplicate it (Ctrl+D) four times so you end up with five file nodes, then MMB-drag the place2dTexture node to the four new file nodes and choose Default. This allows you to control all of them by adjusting one placement node and also makes sure they all are placed in exactly the same way. If you for example want to make your leaves more narrow, you can simply adjust the placement node and it will do it automatically for the color bump and opacity textures. Rename the first two - topColor and topBump, next two botColor and botBump and the last one opacity.
Double click on each one and assign the according texture in the Attribute editor. Make the Filter = 0.1 so it doesn't blur your textures too much. For the opacity texture choose filter type Off.
Double click on the topMat and it will open in the Attribute Editor. MMB-drag topColor to topMat's Color and topBump to Standard Bump of the topMat. When you drag the topBump to Standard Bump it will create a new bump node and open it in the Attribute editor. For bump depth enter 0.2. Do the same for botMat with botColor and botBump.
Then double click on the leafMat and MMB-drag the opacity file node to CutoutOpacity in the leafMat attributes.
Double click on the topMat and adjust the Reflection values (Reflectivity. Glossiness, Glossy Samples). Do the same for the botMat, but make it less reflective (Reflectivity 0.1).
That's it for the materials. For creating the tree on which we will test the created materials, first make the backdrop by making a plane and extruding one of the edges up and beveling the edge between the two faces.
To make the actual PaintFX tree, you first switch your menu setting to Rendering and then choosing - Paint Effects>Paint Effects Tool and then Paint Effects>Get Brush
The Get Brush command will open the Visor. In the Visor window choose trees and from the trees that appear in the right panel, choose the one that best resembles the final look you are after. I chose the keyakiPark tree. Click somewhere on the Grid and PaintFX will create that tree for you.
Close the Visor and select the tree you just created. You should be able to find the attributes of that tree in the Attributes Editor . If you don't see the Attribute editor press Ctrl+A. Choose the tab with the name of your tree brush, in my case keyakiPark. In the Tubes/Creation part of the Attributes, crank up the Segments to make it more detailed, and the Length Min and Max to make it taller and set Tube Width 1 and 2 to make it thicker.
Turn on Branches, Twigs and Leaves and Branch After Twigs check boxes. Adjust the settings for Branches and Twigs. I tend not to use more then two for the Num Branches, because trees rarely split into more then two branches at the same point. This is also true for the Twigs In Cluster.
Adjust the Twig Length Scale to achieve a good looking tree crown. For the leaves Choose Leaf Location On Secondary Branches Only, Leaf Start to near zero, so there are leaves from the beginning of the branches. Turn off Leaf Forward Twist. Which if on, makes all the leaves face the camera. Crank the Num Leaf Clusters until you have enough leaves.
This i's what my tree looks like.
Create an area light, select it and turn on - Use Ray Trace Shadows, Use Light Shape and set High Samples to 64.
Make another Area Light, then go ahead and adjust the same attributes like the first Area Light, just give it a little less Intensity. Position the lights roughly at the positions shown below (the left one is the first Area light).
Because you are going to use Mental ray as the rendering engine, you must convert PaintFX trees to polygons because Mental ray can't render PaintFX. Just select the tree and choose Modify>Convert>Paint Effects to Polygons.
Select everything in your scene, go to Channel Box / Layer Editor choose Render layers and hit Create new layer and assign selected objects (the outlined icon in the right side of the Render layers Panel). Double click on the newly created layer and rename it Color.
Choose the Color layer, then choose the leaves, hold RMB (Right Mouse Button) and choose Assign Existing Material and then select leafMat. Leave the trunk with the shader that the PaintFX to Polygons conversion made for it, or make a new one with color, bump textures and glossy reflections.
Open the Render Settings window and choose mental ray for the rendering engine. In the Quality tab make the following adjustments: Adaptive Sampling with Max Sample Level of 2, Filter - Mitchell (Filter Size 4,4). For Raytracing set the Reflections and Refractions to 10, and Max Trace Depth to 20. This is very important for the Occlusion to work well. In the Indirect Lighting tab everything should be turned off (GI off, FG off etc.)
Select the tree and the backdrop (don't select the lights) and create a new Render layer and name it Occ (for occlusion). Assign lambert3 to the leaves and lamber2 to the trunk and backdrop. Click the Render settings button for this layer (the button outlined with green in the screenshot).
In the Render Settings window, in the Indirect Lighting tab, hold RMB over Final Gathering and choose Create layer override.
This will make Final Gathering orange which means that if you turn it on or off it will affect only this render layer. Turn Final Gathering on by clicking on the checkbox and enter the following settings: Accuracy 100, Point Density 2, Point Interpolation 10.
Close the Render Settings window and from the Camera menu choose View>Camera Attribute Editor...
In the Attribute Editor, Environment tab hold RMB and make layer Override for the Background Color, and make it white.
Choose Options and turn on Render All Layers in the Render layers panel. Then choose Render All Layers Options by clicking the square icon next to Render All Layers.
Set Keep image mode to Keep layers and close this window. Hit render and you will get one Color and one Occlusion image saved in your render view. Just save them to your hardrive and you can combine them later in Photoshop.
You can see the final result with Overlay Blending mode in Photoshop.
That's all for this tutorial I really hope you liked it.
Subscribe below and we’ll send you a weekly email summary of all new 3D & Motion Graphics tutorials. Never miss out on learning about the next big thing.Update me weekly
Envato Tuts+ tutorials are translated into other languages by our community members—you can be involved too!Translate this post