Throughout the course of this two part rendering tutorial you will learn how to use the powerful render layers inside Maya 2010, and also how to output them and prepare them for compositing. In the second part, you will see how you can use After Effects to composite all of those rendered layers to achieve more realistic look with custom depth of field.
Open your scene file, or use the one supplied with the project files (found at the top of the post). Be sure that you have Mental Ray turned on in Maya (Windows -> Settings/Preferences -> Plug-in Manager’)
In the Plug-in Manager window, scroll and find ‘Mayatomr.mll’ and tick on the ‘Loaded’ option.
Then click on the ‘Channel Box / Layer Editor’ (in the top right corner) to start adding the render passes.
At the bottom part of the panel you’ll see the ‘Layer Editor’.
Click on the ‘Render’ tab, next to the ‘Display’ tab.
You’ll see one pre-made layer. Now, the first layer that you’re going to make is called the Ambient Occlusion layer. To make it, first select all of your objects and cameras (except the lights). If you’re using the Project Files, open the ‘Outliner’ and select the objects that are in the ‘objects_group’ and also the ‘Camera’.
In the ‘Layer Editor’ click on the button show in the image below. This will create a new render layer and assign the select objects to it.
Double click on the new layer and rename it ‘Ambient_Occlusion’.
Before you continue you need to set up the basic render settings for the scene, so click on the ‘Render Settings’ button for the ‘masterLayer’.
First, make sure that your 'Default Renderer' is set to 'Mental Ray'.
In the ‘Common’ tab, set the ‘Image format’ to be ‘TIFF’, the ‘Frame/Animation ext’ to ‘name.#.ext’, the ‘Frame padding’ to 3, the ‘Start frame’ to '1', and the ‘End frame’ to '150'.
Scroll down further and set the ‘Renderable Camera’ to ‘Camera’ and the ‘Width’ and ‘Height’ to '700' by '405'.
Click on the ‘Quality’ tab, and turn the ‘Max Sample Level’ to '2'. This will smooth out the result of the final render.
Scroll some more and turn the ‘Filter’ to ‘Gauss’ then expand the ‘Raytracing’ sub menu.
Then under ‘Raytracing’ set the ‘Reflections’ and ‘Refractions’ to '4', and the ‘Max Trace Depth’ to '8', to allow the light to fully pass through any transparent objects.
Under ‘Indirect Lighting’, turn on ‘Global Illumination’ and set the ‘Accuracy’ to '1000' (to prevent any artifacts in the render). Also set the ‘Radius’ to '5' and the ‘Merge Distance’ to '0.2' (to smooth it out further). The ‘Radius’ and the ‘Merge Distance’ will depend on the scale of your scene in Maya units, so if your scene is a different size, then change the properties accordingly.
If you’re using a custom scene for the tutorial, you can scale your scene so it will match the grid size from the scene in the project files, and then you can use the same numbers for the properties as I have used in the tutorial.
Go back to the ‘Layer Editor’, and select the ‘Ambient_Occlusion’ layer.
Select all of the objects.
Alternately, you can right click on the ‘Ambient_Occlusion’ layer, and from the menu select ‘Select Objects in Layer’.
Then right click on the ‘Ambient_Occlusion’ layer again, and go to ‘Overrides -> Create New Materials Override -> Surface Shader’ (see next step for a picture).
...which will create an override material for the objects on the current render layer.
All of the objects assigned to this layer should not be visible now because of the surface shader. If you press the wireframe button, however, you will be able to see them again in wireframe mode.
Go to ‘Windows -> Rendering Editors -> Hypershade’.
The ‘Hypershade’ window will open with all the available materials.
Scroll down and you should be able to see the surface shader you just created.
Right click on the surface shader, and select its name in the menu that shows up. Alternately, you can double click it for the same result.
The attributes of the surface shader should now appear. Click on the name of the shader and rename it to ‘AmbientOcclusion_shader’.
Click on the checkered square next to the ‘Out Color’.
In the ‘Create Render Node’ window that appears, click on the ‘mental ray’ tab.
Scroll to the ‘Texture’ sub menu, and select the ‘Mib_amb_occlusion’ node.
In the attributes window (for the ambient occlusion node), set the ‘Samples’ to '64' and the ‘Max Distance’ to '2.5' to smooth the result. The ‘Max Distance’ is also relative to the scale of your scene.
If you’re not using the scene from the Project Files you will need to tweak this attribute to match your scene size.
To make sure that your viewport is using the 'renderable camera' from the ‘Render Settings’, go to ‘Panels -> Perspective ->Camera’ or your main 'renderable camera' if you’re using a custom scene.
Select ‘Camera attributes’ in the viewport, or click the camera in the ‘Outliner’.
In the camera attributes, scroll to the ‘Environment’ sub menu, expand it, and right click on the ‘Background Color’. Then choose ‘Create Layer Override’, so when you change the attribute they will only change for the current render layer.
Double click the color picker, choose a pure white color, and then click ‘Accept’. This is needed in order to see the result properly. Otherwise you would get a completely black image.
Now go to the ‘Layer Editor’ again, and click on the ‘Render Settings’ button for the ‘Ambient_Occlusion’ layer.
Under the ‘Indirect Lighting’ tab, right click on ‘Global Illumination’ and choose ‘Create Layer Override’.
Turn ‘Global Illumination’ off, because ambient occlusion doesn’t use the lights from the scene.
To do a test render. Click on the ‘Open Render View’ button.
Now in the ‘Render View’ window, select ‘Render -> Render -> Camera’, or your main renderable camera if you’re using a custom scene.
Your ambient occlusion layer is now finished. You should have something like this:
Next up is the Zdepth layer. Again, select only the 'renderable camera' and all of the objects.
Click the button in the ‘Layer Editor’ to assign the selection to a new render layer.
Double click on the new layer and rename it to ‘zdepth’.
Go to the the command line (bottom left) with the 'zdepth' layer still selected in the ‘Layer Editor’.
Now paste this code into the command line: "renderLayerBuiltinPreset linearDepth zdepth". This will attach nodes from the zdepth preset to your layer. Note: The word 'zdepth' from the code refers to the 'zdepth layer'. If your layer is called something else, then you will need to change the code accordingly.
If everything is right, when you input the command there should be no warnings in the command response field.
The ‘zdepth’ layer should be still selected.
Click on the ‘Render Settings’ button for the 'zdepth layer'.
Select ‘mental ray’ as the 'default renderer'
In the ‘Render Settings’, under the ‘Common’ tab, right click on the ‘Image format’ and then select ‘Create Layer Override’.
Set the ‘Image format’ to be ‘TIFF uncompressed’, because you need the maximum image information from the render
Scroll down and right click on the ‘Depth channel’, then select ‘Create Layer Override’.
Enable the ‘Depth channel’ so it can output the zdepth information.
Click on the ‘Indirect Lighting’ tab, and then right click on ‘Global Illumination’ and choose ‘Create Layer Override’.
Un-check ‘Global Illumination’.
Select all of the objects, lights, and the 'renderable camera' in the ‘Outliner’.
Press the button to create a new layer with the selected objects.
Double click the new layer and rename it 'Beauty_pass'. This will be the base for your other passes that will be composited together later.
Open the ‘Render Settings’ for the new layer.
In the ‘File name prefix’ field write ‘scene’, and set the ‘Frame padding’ to '3'.
Click the ‘Smooth shade all’ button, so all of your objects are visible again.
In the ‘Layer Editor’, make sure that for all of the layers that you have created, the enable button is not crossed, except for the one for the ‘masterLayer’.
Choose the ‘Rendering’ option from the shelf selection drop-down menu.
Go to ‘File -> Project -> Edit Current…’.
In the window that pops up, the ‘Location’ is where your project is situated, and the ‘Images’ folder is where your render passes will be rendered into separate folders. So in this example the scene is going to be rendered at: ‘C\Users\User\My Projects\images’.
To render the layers, go to ‘Render -> Batch Render’. It will depend on the power of your computer as to how long it will take to render the layers. The 'ambient occlusion' and the 'zdepth' layer will render much faster than the 'beauty' layer because they don’t use 'global illumination'.
When the renders are complete, your render folder should look something like this. If you followed the tutorial exactly, you should have 150 frames in both the 'ambient occlusion' and 'beauty layer' folders, and 300 frames in the 'zdepth' folder