Extra Ordinary, the Series

The “Omniverse Basics, Create for Production Part 1: USD and Layers” video describes how things work technically, but I found the “why” hard to understand at times. (Looking at the comments left on the video, I don’t think I am alone!) Here is my understanding from the video (combined with other reading), expanded into the “why” rather than just the “what”. This does not follow the video exactly, but I suggest reading this blog then watching the video again. I don’t repeat the video here, I provide background that will hopefully make the video make more sense.

The problem

The example project in the video is creating a short animated video. The video shows a small number of camera “shots” in from a few “scenes”. To render the video, a USD (Universal Scene Description) file is created per shot, the shot USD files are rendered to video files, and the resultant video files are finally combined to make the complete animated video. (The video does not show the entire process, but that is what ultimately the project is doing.)

So how best to lay out all your USD files on disk? There is the question of what is technically possible, but also what makes logical sense.

For example, USD files supports “layering”, like in Adobe Premiere Pro and other similar apps. You can have a series of USD files that are layered in a stack where each file contributes new content or modifies the contributions of lower layers in the stack. For example, you can add more content using layers (start with a background scene, then add a layer per character and prop in the scene, with another layer for music, and so on), but you can also modify previous layers (one layer adds a chair, another layer moves the chair to the desired position). USD often refers to this process of modifying (overriding) underlying layers as “non-destructive changes”. The original model is not moved, you just apply a “delta” (a change) to make to the underlying layer.

A naïve solution

The most naïve solution is to identify all the models needed for each shot and make a complete copy of everything per shot file. Combine all of the models of all the characters into a single mega file.

The problem with this approach (apart from being fundamentally stupid) is if you change something reused across shots, you will have to make another copy per shot. It is also not very storage efficient. While this may not be that important for a very short video, for a full movie this would become painful to manage.

Sidebar: If you use multiple layers, there are tools to creates such a single mega file from the separate layer files. This is referred to as “flattening the stack”.

Sharing content across shots

Sharing content across shots is where the importance of having multiple USD files comes in. Don’t make a copy of all the models per shot, instead refer to shared files so each shot can reuse them. Start a new file each time you have something that is worth reusing.

In reality however, you typically want to make some adjustments. The most common and simplest to understand adjustment is to change the position of a model. A model’s position is a part of the original model, but you don’t want to change that since the model is shared. So instead, you create a new USD layer to override the position of the model.

Taken to the extreme, it is common to have tens to hundreds of small files that are combined to make a final shot. In order to keep this understandable to mere mortals, conventions are put in place. That is the focus of this blog – understanding what conventions you should put in place. (The video gets more into the technical details of how to use the UI of their tool to put it into practice.)

Scopes

In the video there are effectively 4 levels of scoping:

• Assets shared across projects
• Assets within a project
• Assets within a scene
• Assets within a shot

For example, Toy_Jansen (the character model of the NDIVIA CEO) is used in multiple projects, not just this particular video. Within a project, locations (“sets” if you like movie terms, the video uses “environments”) and props may exist. A set may be reused within a longer movie, with some changes (such as the time of day affecting the position of the sun). A scene is a section of a movie that occurs at a location. Finally, you also need to make changes per shot, such as the positioning and lens settings of the camera.

The sample project in the video above use different file names and different directory structures for files related to the different scopes. Each scope may also leverage multiple USD files for layering of changes. So it is useful to first understand the scope for a group of files, then understand the purpose of a layer file within that group.

The file system namespace

There are two namespaces that need to be taken into account: for USD files and the prim path in stages. Let’s start with files.

The directory structure for USD files is normally organized to help make the large number of assets in and across projects manageable.

• You may have a library of assets like furniture and plants that can be reused across projects (as well as characters like Toy_Jensen from the video). You may use directory names like “characters”, “props”, “foliage”, and “sets”.
• Each asset may be broken into multiple layer files to make them easier to repurpose. For example, you may have a model layer with the mesh, but a separate materials layer to apply materials to the mesh. You can then change the clothes of the character by changing the materials but reusing the same mesh.

There is no fixed directory structure that has to be used, but it is worth planning a logical grouping of files. Moving them later can be painful as USD files will contain path names to these files. (Inside USDA files, which are a text version of files, path names are surrounded with “@” characters, like @C:\Omniverse\Library\Characters\Toy_Jensen.usd@.)

Path names in files raises a challenge however. How to share a project between different team members on different computers? One strategy is to use relative path names as much as possible. NVIDIA Omniverse offers another solution however. They support NVIDIA Nucleus, which is like a shared cloud drive. You can then use a “omniverse://host/path” syntax (like a URL to a web page) to reference a file, and the path will work for everyone else who has access to the same Nucleus server. (There are lots of other goodies you get with Nucleus as well, like collaborative editing.) You still have a hierarchical directory structure of files, but it means the path to the file will work for everyone without change.

The world is your stage (the stage namespace)

To render a shot, the ultimate goal is to build up the “stage” to contain all the sets, props, characters, and so on needed. To reference objects (“prims”) in a stage, USD uses paths. Paths in USD files appear inside “<” and “>”. So, if you are used to HTML or XML, when you see “</World>” it is NOT a close tag for a World element! Rather it is the stage path “/World”.

You can add everything (characters, sets, props, cameras, animation clips, etc) to one big directory, or you can place them in subdirectories. For example, you might decide to put the character in /World/Characters/Toy_Jensen. The names are up to you – they don’t have to match the filenames on disk. Consistency however can make things less confusing! The prim path names are very important for overrides – they are the unique identifier for assets. As such, it is painful to change the path of a prim after a project has been created as you have to find all the references to the path and modify them. It is therefore worth spending a bit of time thinking about the path names you want to use in advance, such as creating subdirectories per type of asset, and setting up descriptive naming conventions. (Use “Toy_Jensen” and not “Character 1”.)

Having said that, you also don’t need to get as carried away planning out prim paths compared to files on disk. Ultimately you are trying to create a single shot and only bring in the assets needed for that shot. So there are fewer objects to worry about colliding that the full file system. You can also rename things when you bring them into a stage if you really have to. The NVIDIA video put most things directly under /World and did not worry much about nesting.

Sidebar: Each USD file can nominate a prim in the file as the “default” prim. Referencing the file from another USD file will then include only that prim in the layer stack. You can put other things in the file that are not included. For example, you can create a character USD file and add lights to it. If only the character is nominated as the default prim for the file, the lights will not be included when the file is referenced.

External assets (libraries)

So, your first step is to work out how you are going to organize files that you may want to reuse across projects into libraries. This includes setting up standards for your own use of file names and prim paths to avoid collisions or confusion. Also investigate whether going to a Nucleus server makes sense or not.

If you have multiple developers working on such assets, you may wish to split one asset into multiple files so different people can work on the asset at the same time, without fear of overwriting the changes of the other person. For example, one person may adjust the model mesh, another the textures for materials. Obviously some mesh changes may need texture changes (e.g. if you add a hat, you will need artwork for the hat), but minor adjustments to the mesh to round off a rough point will not require texture changes. While you may separate the character into multiple layer files, you probably should still create a single USD file that includes (as sublayers) all the other layers, so projects can refer to that single file, even though the asset internally has multiple layers. (Note: Toy_Jensen did not seem to do this – the project referenced the model and material layers separately from the project files, which surprised me.)

Project assets

You may have some assets that are specific to a project. This is not that different to a library in a way, but these files would all reside under a project directory. Examples of project level assets might be the render pipeline settings to use for the project, so all shots are rendered consistently. Other settings might be the frame rate of the videos you want to create. You can of course create characters for a specific project if you desire, but I am a little wary of this as you may decide to start a new project say for a promotional video.

In order to bring assets into your project, you may decide to create “bridge” files. These files refer to the external asset and give you a single handle to the external file. If the external file moves to a new directory (or you want to replace it with a new version), you can update a single file in your project. You may also make some customizations to the external asset, such as changing a material or adjusting its scale factor.

Sets for use within scenes might also be defined at the project level. This allows a movie where a set (a location/environment) is visited multiple times during the movie. Create a separate directory for each such type of asset you identify to help organize your files.

Scene assets

Examples of scene level assets are building up the final set for the scene to use. This might include a common project level set (using a sublayer), but then add or move props in the set, adjust lighting based on the time of day, and so on. All of the shots in a scene typically share the same set, so create a USD file for the scene that shots can reuse. For example, /myfiles/project1/shots/sc10/sc10.usd for scene 10.

Shot assets

Examples of shot level assets are cameras and the final positioning of characters and props. You may also add additional lights to make objects of particular focus pop a bit more. A USD file is created per shot which would normally include as a base layer the scene USD file that the shot is a part of. It then adds additional layers to make other changes. For example, you may decide as a convention to add all the lights per shot in a separate light layer file (sc10_sh100_lights.usd), or create a subdirectory for it.

A final example directory structure

Putting all the above together, you may end up with a directory structure such as follows. Please note, I made many of these paths up. I tried to be similar to the video, but I have made some changes along the way. The video also did not show the contents of all the subdirectories. So this is a bit of “what I might do”, not just what the video shows.

First, let’s look at some possible paths for library files. (I am using Window path names here just to make it clearer they are not prim paths.)

C:\Omniverse\Library\assets\props\foliage\lucious\fern1\fern1.usd
C:\Omniverse\Library\assets\props\foliage\lucious\fern1\fern1_model_v1.usd
C:\Omniverse\Library\assets\props\foliage\lucious\fern1\fern1_material_v1.usd
C:\Omniverse\Library\assets\props\foliage\desert\palm1\parlm1.usd
C:\Omniverse\Library\assets\props\foliage\desert\palm1\palm1_model_v1.usd
C:\Omniverse\Library\assets\props\foliage\desert\palm1\palm1_material_v1.usd

Names like “fern1” and “palm1” might not be ideal (not good global descriptors), but by having a parent “asset pack” directory (like “luscious” or “desert”) you can have multiple ferns without the filenames colliding. In this example, the model and materials have been separated out, but this is not required. Having a base USD file that is always referenced however is a good ideal. It then does not matter if you split out the layers or not for the model.

Should “foliage” or “plants” be put under “props”? That is up to you. Think about how it makes sense to group and classify the assets you have. Remember it can be annoying to change path names later, so it’s generally better to add too many subdirectories than too few. Have a look around the NVIDIA provided assets and sample USD projects people have published for inspiration. Strange conventions used may be the result of painful lessons from real projects.

Another library example for a character that is shared across projects, with different versions being created over time.

C:\Omniverse\Library\assets\characters\Toy_Jensen\toy_jensen.usd
C:\Omniverse\Library\assets\characters\Toy_Jensen\materials\toy_jensen_materials_V1.usd
C:\Omniverse\Library\assets\characters\Toy_Jensen\model\toy_jensen_model_V1.usd
C:\Omniverse\Library\assets\characters\Toy_Jensen\model\toy_jensen_model_V2.usd
C:\Omniverse\Library\assets\characters\Toy_Jensen\textures\black_jacket.png

Having separate layers for the model and materials allows them to be versioned separately. Is it important to create subdirectories for models and materials? They do this in the video, but it feels overkill to me. I like the practice of the final filenames being fairly descriptive (toy_jensen_model_V1.usd not model_V1.usd). Descriptive filenames can be useful when doing operations like searches that might not show the full path name. My concern is when you expect a directory only to have one or two files in it, it can end up more confusing than beneficial. However, creating separate directories might be helpful if you have different people working on models vs meshes. They can “own” their respective directory without fear of their files colliding with other users. So for bigger teams, subdirectories might make more sense. (With a Nucleus server, you can make live changes that others will see, so fear of bumping into others can be a real issue.)

Sidebar: One important thing to understand about “references” in USD is when in Omniverse Create (now Composer) is if you “reference” another file, the editor won’t let you change that file contents. That can save you from accidental modifications. You can lock layers in Create, but references is safer. So it is a useful practice to create references as a form of isolation to protect shared assets from accidental change.

How then to bring an external asset into a project? The video introduces the concept of a bridge file.

C:\Projects\EduDemo\assets\characters\Toy_Jensen\bridge_toy_jensen.usd
C:\Projects\EduDemo\assets\characters\Toy_Jensen\deltas\delta_toy_jensen.usd
C:\Projects\EduDemo\assets\characters\Toy_Jensen\ref_holders\ref_material_Toy_Jensen.usd
C:\Projects\EduDemo\assets\characters\Toy_Jensen\ref_holders\ref_model_Toy_Jensen.usd

Within the project the bridge location is the only place the external files are referenced, and local adjustments can be made (e.g. to change the scale factor to a consistent value with the rest of the project).

All external assets would have a bridge file that is then used by the rest of the project.

C:\Projects\EduDemo\assets\environments\kitchen\bridge_kitchen.usd
C:\Projects\EduDemo\assets\props\boxes\bridge_box1.usd

You may however create locations within the project using props (furniture etc) from your library. This would then go within your project directory structure. You don’t want it in a library as someone may decide to “improve” it later, breaking your project. Decide if an asset is to be shared or not. Library assets should be stable (not changed often).

C:\Projects\EduDemo\sets\kitchen.usd

What about shots? Shots are grouped under scenes for scoping reasons. A scene contains settings that are shared by all shots in the scene.

C:\Projects\EduDemo\shots\sc10\shared\sc10_base.usd
C:\Projects\EduDemo\shots\sc10\shared\delta_starting_placement.usd
C:\Projects\EduDemo\shots\sc20\shared\sc20_base.usd

Each scene gets is own directory with a “sc” prefix and the scene number. You can increment numbers by 1, but I like incrementing by 10 as it’s easier to add other scenes in later without renumbering. An example base file from the video has the following sublayers:

• delta_starting_placement.usd
• bridge_spoon.usd
• bridge_Toy_Jensen.usd
• bridge_salt_box.usd
• bridge_coffee_grinder.usd
• bridge_Kitchen_Closeup.usd
• bridge_HDR.usd

That is, it includes all the bridge files that are needed for the scene, then has a single file for holding all the new positions for the brought in assets. You don’t have to have a separate file for holding the positions for models (it could go in the base scene file directly), but the most important thing is to have a convention you follow.

Note that the above (from the NVIDIA video) references a delta_starting_placement.usd file which does not include the scene number. If this is positioning for a specific scene, I would include it in the filename. But if you are scared of being inconsistent, take hope! Even the NVIDIA example I don’t think is consistent in their rules, and it all still works.

Let’s look at some actual shots.

C:\Projects\EduDemo\shots\sc10\sh100\sc10_sh100_V1_01.usd
C:\Projects\EduDemo\shots\sc10\sh100\delta\delta_positions_sc10_sh100.usd
C:\Projects\EduDemo\shots\sc10\sh100\camera\Camera_sc10_sh100.00_1001-1200.usd
C:\Projects\EduDemo\shots\sc10\sh100\clip\Anim_Toy_Jensen_sc10_sh100.usd
C:\Projects\EduDemo\shots\sc10\sh100\clip\Anim_P_Spoon_sc10_sh100.usd

Each shot includes the parent scene base file (../shared/sc10_base.usd) then adds additional layers. In the example given, the characters are added in the base scene file and then moved. (In my personal projects in Unity, I usually add the characters in each shot as characters can enter or leave the field of view. Do whatever makes more sense for your project.)

The video included “1001-1200” in the filename which is the frame range within the shot timeline the camera is for, but they leave the frame range out of the prim path. They probably have a reason why. I would have left it out myself, but I have not done much with camera movements in Omniverse yet. For example, for a “shot” in Unity, I sometimes create multiple video files from a single “shot” to save effort.

The shot file they also appeared to include version number information, maybe so you can compare variations of the shot until you pick a final version. If you do this, I assume there would be some master list to keep track of the full list of shot files for the project.

Wrapping up

The above is my version of understanding the NVIDIA video contents. I have not described exactly what is in the video, but I tried to draw out concepts such as scoping that they demonstrate how to implement in the video. For example, you have external assets in libraries, you bring them in via bridge files, you have a hierarchy of scenes and shots, etc. Do you have to use this exact structure? No! The goal however is to give a feel of the planning you should be doing on your own projects and then set in place your own set of conventions. I plan to have multiple episodes in a series, so I would include an episode layer to the shot hierarchy.

Appendix: Further reading (Added 4/25/23)

In a NVIDIA study session Mati from NVIDIA and others shared some additional reading that could be relevant to animation in particular (most of these assets are worthy of independent blog posts):

• A great list of useful tutorials
• Guidelines for Structuring USD Assets
• https://www.youtube.com/watch?v=jcLaeVd9qH0 – a more recent video on USD from NVIDIA
• How Animal Logic organizes files with concepts such as Entities and Fragments (and more). Entities are things like characters and props, whereas fragments are one aspect of an entity and don’t really makes sense being used on their own (e.g. a mesh without textures, or lighting for a camera). (A nice diagram.)
• Dreamworks first adoption of USD – includes how they broke down various asset USD files. E.g. a shot USD file includes a manifest USD file that loads up separate USD files for characters, props, sets, cameras, and sequences. They then layer an animation file over the top of the manifest file to change the poses etc of characters. Then add a instancing USD file that turns on instancing for selected assets for performance. Then each department gets another file on top (lighting, layout, visual effects, etc). It also showed their pipeline stages: modeling, surfacing, rigging, layout, animation, character fx, crowds, visual fx, matte painting, lighting and composition, and finally editing the video clips together.

Also in the session I saw a syntax I had not seen before combining a file path with a prim path:

def "CubeTwo" (references = [@./cube-model.usda@</World/CubeA>]) { 
    ...
}

I assume this means the prim path of /World/CubeA inside the file ./cube-model.usda.

The other question is how to render all the shots. I have come across NVIDIA Farm for queuing up multiple render jobs (designed for a farm of servers to delegate work to), Omniverse Machinima for easy game cinematics, and Omniverse RTX Renderer which talks about the capabilities of the renderer itself. I need to read more to work out how these all fit together