1. 3D & Motion Graphics
  2. Maya

Achieving Realism and Depth using Render Layers in Maya - Day 1

Read Time:8 minsLanguages:
This post is part of a series called Achieving Realism and Depth using Render Layers in Maya.
Achieving Realism and Depth using Render Layers in Maya - Day 2

Throughout the course of this two part rendering tutorial you will learn how to use the powerful render layers inside Maya 2010, and also how to output them and prepare them for compositing. In the second part, you will see how you can use After Effects to composite all of those rendered layers to achieve more realistic look with custom depth of field.

Step 1

Open your scene file, or use the one supplied with the project files (found at the top of the post). Be sure that you have Mental Ray turned on in Maya (Windows -> Settings/Preferences -> Plug-in Manager’)

Step 2

In the Plug-in Manager window, scroll and find ‘Mayatomr.mll’ and tick on the ‘Loaded’ option.

Step 3

Then click on the ‘Channel Box / Layer Editor’ (in the top right corner) to start adding the render passes.

Step 4

At the bottom part of the panel you’ll see the ‘Layer Editor’.

Step 5

Click on the ‘Render’ tab, next to the ‘Display’ tab.

Step 6

You’ll see one pre-made layer. Now, the first layer that you’re going to make is called the Ambient Occlusion layer. To make it, first select all of your objects and cameras (except the lights). If you’re using the Project Files, open the ‘Outliner’ and select the objects that are in the ‘objects_group’ and also the ‘Camera’.

Step 7

In the ‘Layer Editor’ click on the button show in the image below. This will create a new render layer and assign the select objects to it.

Step 8

Double click on the new layer and rename it ‘Ambient_Occlusion’.

Step 9

Before you continue you need to set up the basic render settings for the scene, so click on the ‘Render Settings’ button for the ‘masterLayer’.

Step 10

First, make sure that your 'Default Renderer' is set to 'Mental Ray'.

Step 11

In the ‘Common’ tab, set the ‘Image format’ to be ‘TIFF’, the ‘Frame/Animation ext’ to ‘name.#.ext’, the ‘Frame padding’ to 3, the ‘Start frame’ to '1', and the ‘End frame’ to '150'.

Step 12

Scroll down further and set the ‘Renderable Camera’ to ‘Camera’ and the ‘Width’ and ‘Height’ to '700' by '405'.

Step 13

Click on the ‘Quality’ tab, and turn the ‘Max Sample Level’ to '2'. This will smooth out the result of the final render.

Step 14

Scroll some more and turn the ‘Filter’ to ‘Gauss’ then expand the ‘Raytracing’ sub menu.

Step 15

Then under ‘Raytracing’ set the ‘Reflections’ and ‘Refractions’ to '4', and the ‘Max Trace Depth’ to '8', to allow the light to fully pass through any transparent objects.

Step 16

Under ‘Indirect Lighting’, turn on ‘Global Illumination’ and set the ‘Accuracy’ to '1000' (to prevent any artifacts in the render). Also set the ‘Radius’ to '5' and the ‘Merge Distance’ to '0.2' (to smooth it out further). The ‘Radius’ and the ‘Merge Distance’ will depend on the scale of your scene in Maya units, so if your scene is a different size, then change the properties accordingly.

Step 17

If you’re using a custom scene for the tutorial, you can scale your scene so it will match the grid size from the scene in the project files, and then you can use the same numbers for the properties as I have used in the tutorial.

Step 18

Go back to the ‘Layer Editor’, and select the ‘Ambient_Occlusion’ layer.

Step 19

Select all of the objects.

Step 20

Alternately, you can right click on the ‘Ambient_Occlusion’ layer, and from the menu select ‘Select Objects in Layer’.

Step 21

Then right click on the ‘Ambient_Occlusion’ layer again, and go to ‘Overrides -> Create New Materials Override -> Surface Shader’ (see next step for a picture).

Step 22

...which will create an override material for the objects on the current render layer.

Step 23

All of the objects assigned to this layer should not be visible now because of the surface shader. If you press the wireframe button, however, you will be able to see them again in wireframe mode.

Step 24

Go to ‘Windows -> Rendering Editors -> Hypershade’.

Step 25

The ‘Hypershade’ window will open with all the available materials.

Step 26

Scroll down and you should be able to see the surface shader you just created.

Step 27

Right click on the surface shader, and select its name in the menu that shows up. Alternately, you can double click it for the same result.

Step 28

The attributes of the surface shader should now appear. Click on the name of the shader and rename it to ‘AmbientOcclusion_shader’.

Step 29

Click on the checkered square next to the ‘Out Color’.

Step 30

In the ‘Create Render Node’ window that appears, click on the ‘mental ray’ tab.

Step 31

Scroll to the ‘Texture’ sub menu, and select the ‘Mib_amb_occlusion’ node.

Step 32

In the attributes window (for the ambient occlusion node), set the ‘Samples’ to '64' and the ‘Max Distance’ to '2.5' to smooth the result. The ‘Max Distance’ is also relative to the scale of your scene.

Step 33

If you’re not using the scene from the Project Files you will need to tweak this attribute to match your scene size.

Step 34

To make sure that your viewport is using the 'renderable camera' from the ‘Render Settings’, go to ‘Panels -> Perspective ->Camera’ or your main 'renderable camera' if you’re using a custom scene.

Step 35

Select ‘Camera attributes’ in the viewport, or click the camera in the ‘Outliner’.

Step 36

In the camera attributes, scroll to the ‘Environment’ sub menu, expand it, and right click on the ‘Background Color’. Then choose ‘Create Layer Override’, so when you change the attribute they will only change for the current render layer.

Step 37

Double click the color picker, choose a pure white color, and then click ‘Accept’. This is needed in order to see the result properly. Otherwise you would get a completely black image.

Step 38

Now go to the ‘Layer Editor’ again, and click on the ‘Render Settings’ button for the ‘Ambient_Occlusion’ layer.

Step 39

Under the ‘Indirect Lighting’ tab, right click on ‘Global Illumination’ and choose ‘Create Layer Override’.

Step 40

Turn ‘Global Illumination’ off, because ambient occlusion doesn’t use the lights from the scene.

Step 41

To do a test render. Click on the ‘Open Render View’ button.

Step 42

Now in the ‘Render View’ window, select ‘Render -> Render -> Camera’, or your main renderable camera if you’re using a custom scene.

Step 43

Your ambient occlusion layer is now finished. You should have something like this:

Step 44

Next up is the Zdepth layer. Again, select only the 'renderable camera' and all of the objects.

Step 45

Click the button in the ‘Layer Editor’ to assign the selection to a new render layer.

Step 46

Double click on the new layer and rename it to ‘zdepth’.

Step 47

Go to the the command line (bottom left) with the 'zdepth' layer still selected in the ‘Layer Editor’.

Step 48

Now paste this code into the command line: "renderLayerBuiltinPreset linearDepth zdepth". This will attach nodes from the zdepth preset to your layer. Note: The word 'zdepth' from the code refers to the 'zdepth layer'. If your layer is called something else, then you will need to change the code accordingly.

Step 49

If everything is right, when you input the command there should be no warnings in the command response field.

Step 50

The ‘zdepth’ layer should be still selected.

Step 51

Click on the ‘Render Settings’ button for the 'zdepth layer'.

Step 52

Select ‘mental ray’ as the 'default renderer'

Step 53

In the ‘Render Settings’, under the ‘Common’ tab, right click on the ‘Image format’ and then select ‘Create Layer Override’.

Step 54

Set the ‘Image format’ to be ‘TIFF uncompressed’, because you need the maximum image information from the render

Step 55

Scroll down and right click on the ‘Depth channel’, then select ‘Create Layer Override’.

Step 56

Enable the ‘Depth channel’ so it can output the zdepth information.

Step 57

Click on the ‘Indirect Lighting’ tab, and then right click on ‘Global Illumination’ and choose ‘Create Layer Override’.

Step 58

Un-check ‘Global Illumination’.

Step 59

Select all of the objects, lights, and the 'renderable camera' in the ‘Outliner’.

Step 60

Press the button to create a new layer with the selected objects.

Step 61

Double click the new layer and rename it 'Beauty_pass'. This will be the base for your other passes that will be composited together later.

Step 62

Open the ‘Render Settings’ for the new layer.

Step 63

In the ‘File name prefix’ field write ‘scene’, and set the ‘Frame padding’ to '3'.

Step 64

Click the ‘Smooth shade all’ button, so all of your objects are visible again.

Step 65

In the ‘Layer Editor’, make sure that for all of the layers that you have created, the enable button is not crossed, except for the one for the ‘masterLayer’.

Step 66

Choose the ‘Rendering’ option from the shelf selection drop-down menu.

Step 67

Go to ‘File -> Project -> Edit Current…’.

Step 68

In the window that pops up, the ‘Location’ is where your project is situated, and the ‘Images’ folder is where your render passes will be rendered into separate folders. So in this example the scene is going to be rendered at: ‘C\Users\User\My Projects\images’.

Step 69

To render the layers, go to ‘Render -> Batch Render’. It will depend on the power of your computer as to how long it will take to render the layers. The 'ambient occlusion' and the 'zdepth' layer will render much faster than the 'beauty' layer because they don’t use 'global illumination'.

Step 70

When the renders are complete, your render folder should look something like this. If you followed the tutorial exactly, you should have 150 frames in both the 'ambient occlusion' and 'beauty layer' folders, and 300 frames in the 'zdepth' folder

Looking for something to help kick start your next project?
Envato Market has a range of items for sale to help get you started.