Document ContentLayout.json and publish its schema

Analyzer/Analyzer.csproj

@@ -25,6 +25,7 @@
   <ItemGroup>
     <ProjectReference Include="..\UnityBinaryFormat\UnityBinaryFormat.csproj" />
     <ProjectReference Include="..\UnityFileSystem\UnityFileSystem.csproj" />
+    <ProjectReference Include="..\UnityDataModels\UnityDataModels.csproj" />
   </ItemGroup>
 
   <ItemGroup>

Analyzer/AnalyzerTool.cs

@@ -4,8 +4,8 @@
 using System.IO;
 using UnityDataTools.Analyzer.SQLite.Handlers;
 using UnityDataTools.Analyzer.SQLite.Parsers;
-using UnityDataTools.Analyzer.SQLite.Parsers.Models;
 using UnityDataTools.Analyzer.SQLite.Writers;
+using UnityDataTools.Models;
 using UnityDataTools.BinaryFormat;
 using UnityDataTools.FileSystem;
 

Analyzer/SQLite/Parsers/AddressablesBuildLayoutParser.cs

@@ -4,7 +4,7 @@
 using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
 using UnityDataTools.Analyzer.SQLite.Handlers;
-using UnityDataTools.Analyzer.SQLite.Parsers.Models;
+using UnityDataTools.Models;
 using UnityDataTools.Analyzer.SQLite.Writers;
 
 namespace UnityDataTools.Analyzer.SQLite.Parsers

Analyzer/SQLite/Writers/AddressablesBuildLayoutSQLWriter.cs

@@ -4,7 +4,7 @@
 using Microsoft.Data.Sqlite;
 using Newtonsoft.Json;
 using UnityDataTools.Analyzer.SQLite.Commands.AddressablesBuildReport;
-using UnityDataTools.Analyzer.SQLite.Parsers.Models;
+using UnityDataTools.Models;
 
 namespace UnityDataTools.Analyzer.SQLite.Writers
 {

Documentation/contentlayout.md

@@ -0,0 +1,73 @@
+# ContentLayout.json
+
+`ContentLayout.json` describes the content that a content directory build produced. It is written by [`BuildPipeline.BuildContentDirectory`](https://docs.unity3d.com/6000.6/Documentation/ScriptReference/BuildPipeline.BuildContentDirectory.html) into the build report directory, alongside the other build report files. For an overview of the build report directory and the other files it contains, see [Build report and build history](https://docs.unity3d.com/6000.6/Documentation/Manual/build-reporting.html) in the Unity Manual.
+
+This page explains what the file contains conceptually to aid in creation of build-analysis tooling or inspection of content directory build output.  The C# types that define the schema are published alongside this documentation in [`ContentLayout.cs`](../UnityDataModels/ContentLayout.cs), which is the authoritative reference for the individual fields.
+
+## What it is for
+
+Use `ContentLayout.json` to:
+
+* Find which source assets a build included, and which serialized file each one ended up in.
+* See the dependencies between the files the build produced.
+* Understand what contributes to the size or loading footprint of the content in the runtime.
+
+## Relationship to the build manifest
+
+`ContentLayout.json` is a superset of the build manifest that Unity ships with the content. The build manifest is a minimal summary that contains only what is required to load the content at runtime. `ContentLayout.json` is not shipped, and adds a more complete picture, including a mapping from the built content back to the source assets in the project.
+
+## What it does not contain
+
+* **Packaging information.** It does not record how the content is stored, for example whether the artifacts are packed inside Unity Archive files. It describes the logical content, not its on-disk packaging.
+* **Object-level detail.** It does not describe individual Unity objects inside the files, and contains no type information. This is intentional, to keep the file size reasonable. For object-level analysis, run [`UnityDataTool analyze`](command-analyze.md) on the build output. For per-type and per-asset size statistics, use the `ContentSummary` of the [BuildReport](buildreport.md).
+* **Non-deterministic data.** To avoid non-deterministic data this file contains no timestamps, and no information about the build process that created it. Two builds that produce identical content produce an identical `ContentLayout.json`.
+
+## Terminology
+
+The file uses a few terms consistently:
+
+* **Artifact** — a unit of build output. The term is used instead of "file" because the build output is not necessarily written as individual files (for example, it could be packed inside Unity Archives, or served from key/value storage).
+* **SerializedFile** — a Unity binary file containing serialized objects (the `.cf` content files of the build). See [Unity Content Format](unity-content-format.md).
+* **Loadable** — an object that can be loaded on demand, identified independently of the serialized file that happens to contain it.  Loadable will reference a specific object, for example the root GameObject of a prefab, but loading that object will load the entire SerializedFile.
+* **LoadableSceneId** — Similar to a Loadable, but referencing a Unity Scene.
+* **Source asset** — an asset in the source project (identified by GUID and asset path) that contributed to the build output.
+
+## Top-level structure
+
+`ContentLayout.json` is a single JSON object with the following members. Entries in several of the arrays cross-reference each other by array index or by hash, so the file describes a graph rather than a flat list.
+
+| Member | Description |
+|--------|-------------|
+| `Version` | Schema version of the file. See [Schema versioning](#schema-versioning). |
+| `BuildManifestHash` | Hash of the build manifest this layout corresponds to. |
+| `SerializedFiles` | One entry per serialized file in the build. Each entry records the source assets it contains, its content hash, and its dependencies on other serialized files, loadables, and loadable scenes. The same source asset can appear in more than one serialized file (for example, a single FBX file can be split into multiple output files). |
+| `RootAssets` | The `ObjectIdHash` of each root asset the build was made from. Each one has a corresponding entry in `LoadableObjectIds`. |
+| `LoadableObjectIds` | The objects that can be loaded on demand. Each entry records its `ObjectIdHash`, where the object lives in the built content (which serialized file) and where it came from in the source project (GUID, asset path, local file ID, and identifier type). |
+| `LoadableSceneIds` | The scenes in the build, each with its source project path and GUID, and the serialized file that contains it. |
+| `BinaryArtifacts` | The artifacts that make up the build output. See [Binary artifacts](#binary-artifacts). |
+
+### Binary artifacts
+
+`BinaryArtifacts` is essentially the list of files in the build output. Each entry has a `Category` and a `Size`, and lists its direct dependencies in `ArtifactReferences`:
+
+* The entry with category `manifest` is the root of the graph.
+* Entries with category `contentfile` each have a matching entry in `SerializedFiles` (matched by content hash).
+* `BinaryArtifacts` also reports the additional data files that hold audio, video, texture, and mesh data (the `.resource` and `.resS` files).
+* BinaryArtifacts are identified by the hash of their content.  When saved as a file, the filename is the hash and the file extension is based on the category.
+
+`ArtifactReferences` lists only direct dependencies. Dependencies that go through a loadable or loadable scene are not included, and the dependency graph is never cyclical. Together, this makes it possible to see every artifact required to load a particular serialized file, excluding data that is loaded on demand through a loadable or loadable scene.
+
+## Schema versioning
+
+The schema is subject to change. The `Version` field records the schema version of the file, independently of the Unity version that produced it. When the schema changes, the version number increments.
+
+[`ContentLayout.cs`](../UnityDataModels/ContentLayout.cs) always represents the latest schema version (currently version 1).
+
+## Related documentation
+
+| Topic | Description |
+|-------|-------------|
+| [Build report and build history](https://docs.unity3d.com/6000.6/Documentation/Manual/build-reporting.html) | The build report directory and the files in it (Unity Manual). |
+| [BuildReport Support](buildreport.md) | Analyzing Unity build report files with UnityDataTool. |
+| [Unity Content Format](unity-content-format.md) | SerializedFiles, Unity Archives, and how build output maps back to source assets. |
+| [`analyze` command](command-analyze.md) | Object-level analysis of build output. |

Documentation/unity-content-format.md

@@ -124,3 +124,5 @@ You can include a Unity BuildReport file when running `UnityDataTools analyze`.
 `UnityDataTools analyze` can also import Addressables build layout files, which include source asset information.  See [Addressable Build Reports](./addressables-build-reports.md).
 
 For AssetBundles built by [BuildPipeline.BuildAssetBundles()](https://docs.unity3d.com/ScriptReference/BuildPipeline.BuildAssetBundles.html), Unity creates a .manifest file for each AssetBundle that has source information.  This is a text-based format.
+
+For content directory builds, the [ContentLayout.json](contentlayout.md) file maps the build output back to the source assets in the project.

Documentation/unitydatatool.md

@@ -99,3 +99,4 @@ If you see a warning about `UnityFileSystemApi.dylib` not being verified, go to
 | [Analyze Examples](analyze-examples.md) | Practical database queries |
 | [Comparing Builds](comparing-builds.md) | Strategies for build comparison |
 | [Unity Content Format](unity-content-format.md) | TypeTrees and file formats |
+| [ContentLayout.json](contentlayout.md) | The content layout file produced by content directory builds |

Analyzer/SQLite/Parsers/Models/BuildLayout.cs → UnityDataModels/BuildLayout.cs

@@ -2,7 +2,7 @@
 
 // this file comes from Addressables and is used to serialize and deserialize
 // build layout information to JSON
-namespace UnityDataTools.Analyzer.SQLite.Parsers.Models
+namespace UnityDataTools.Models
 {
     public class BuildLayout
     {

UnityDataModels/ContentLayout.cs

@@ -0,0 +1,194 @@
+// This file defines the structure of the ContentLayout.json file produced by BuildPipeline.BuildContentDirectory
+// (available starting in Unity 6.6).
+//
+// See Documentation/contentlayout.md for further details.
+//
+// ContentLayout always represents the latest schema version (currently 1). If the schema changes
+// in future, older versions can be preserved under a version-specific namespace while this type
+// continues to track the latest version.
+namespace UnityDataTools.Models
+{
+    /// <summary>
+    /// Category strings used by <see cref="BinaryArtifact.Category"/>.
+    /// </summary>
+    public static class BuildArtifactCategory
+    {
+        /// <summary>Texture data, written to a streaming resource (.resS) file.</summary>
+        public const string Texture = "texture";
+
+        /// <summary>Mesh data, written to a streaming resource (.resS) file.</summary>
+        public const string Mesh = "mesh";
+
+        /// <summary>Audio data, written to a resource (.resource) file.</summary>
+        public const string Audio = "audio";
+
+        /// <summary>Video data, written to a resource (.resource) file.</summary>
+        public const string Video = "video";
+
+        /// <summary>A SerializedFile / Content File (.cf).</summary>
+        public const string ContentFile = "contentfile";
+
+        /// <summary>A manifest file (.json).</summary>
+        public const string Manifest = "manifest";
+    }
+
+    /// <summary>
+    /// Describes a SerializedFile in the build output.
+    /// </summary>
+    public struct SerializedFileLayout
+    {
+        /// <summary>Index of this entry inside <see cref="ContentLayout.SerializedFiles"/>.</summary>
+        public int Index;
+
+        /// <summary>Stable identifier used to reference this SerializedFile from other SerializedFiles
+        /// in a way that doesn't break when the content changes. Currently based on the cluster or guid of the source.</summary>
+        public string ID;
+
+        /// <summary>True for synthetic entries representing built-in Unity resources that are not produced
+        /// by the build (currently only "Library/unity default resources"). Such entries have no ContentHash.</summary>
+        public bool IsBuiltIn;
+
+        /// <summary>The source assets included in this SerializedFile.</summary>
+        public string[] SourceAssets;
+
+        /// <summary>Indices into the containing <see cref="ContentLayout.SerializedFiles"/> array, identifying
+        /// other SerializedFiles that need to be loaded prior to loading this SerializedFile.</summary>
+        public int[] SerializedFileDependencies;
+
+        /// <summary>ObjectIdHash values for loadable objects referenced from this SerializedFile.</summary>
+        public string[] LoadableDependencies;
+
+        /// <summary>Scene paths for scenes referenced from this SerializedFile.</summary>
+        public string[] LoadableSceneDependencies;
+
+        /// <summary>xxhash3 hash of the content, used for the filename (+".cf") and for lookup into UDS.
+        /// Matches the <see cref="BinaryArtifact.ContentHash"/> of the corresponding entry in
+        /// <see cref="ContentLayout.BinaryArtifacts"/>.</summary>
+        public string ContentHash;
+    }
+
+    /// <summary>
+    /// Records a loadable object in the build. Listed at the top level of the ContentLayout so that a
+    /// loadable's identity is described independently of the SerializedFile that happens to contain it.
+    /// </summary>
+    public class LoadableObjectIdLayout
+    {
+        /// <summary>Hash of the GUID, LFID and IdentifierType.</summary>
+        public string ObjectIdHash;
+
+        /// <summary>AssetDatabase GUID of the source asset.</summary>
+        public string GUID;
+
+        /// <summary>Path of the source asset.</summary>
+        public string AssetPath;
+
+        /// <summary>Local file id of the source object.</summary>
+        public long LFID;
+
+        /// <summary>Identifier type of the source object.</summary>
+        public int IdentifierType;
+
+        /// <summary>Index into <see cref="ContentLayout.SerializedFiles"/> for the file that contains this
+        /// loadable, or -1 if it was dropped (e.g. server build shader references).</summary>
+        public int SerializedFile = -1;
+    }
+
+    /// <summary>
+    /// Records a scene exposed as loadable in the build, and the SerializedFile that contains it.
+    /// </summary>
+    public class LoadableSceneIdLayout
+    {
+        /// <summary>AssetDatabase GUID of the scene.</summary>
+        public string GUID;
+
+        /// <summary>Scene path.</summary>
+        public string Path;
+
+        /// <summary>Index into <see cref="ContentLayout.SerializedFiles"/>, or -1 if not mapped.</summary>
+        public int SerializedFile = -1;
+    }
+
+    /// <summary>
+    /// Describes a binary artifact produced by the build (SerializedFile, streaming resource, manifest, etc.).
+    /// </summary>
+    public class BinaryArtifact
+    {
+        /// <summary>Index of this entry inside <see cref="ContentLayout.BinaryArtifacts"/>.</summary>
+        public int Index;
+
+        /// <summary>Content addressable hash. For ContentFile artifacts, the matching
+        /// <see cref="SerializedFileLayout"/> can be found by ContentHash.</summary>
+        public string ContentHash;
+
+        /// <summary>One of the strings in <see cref="BuildArtifactCategory"/>.</summary>
+        public string Category;
+
+        /// <summary>Size in bytes.</summary>
+        public ulong Size;
+
+        /// <summary>Indices into <see cref="ContentLayout.BinaryArtifacts"/> identifying additional artifacts
+        /// referenced from this one (e.g. mesh, audio).
+        /// Note: this does not track references to other ContentFiles — those are recorded in
+        /// <see cref="SerializedFileLayout.SerializedFileDependencies"/>.</summary>
+        public int[] ArtifactReferences;
+
+        /// <summary>The on-disk file extension for this artifact, derived from <see cref="Category"/>.
+        /// For unrecognized categories, returns the category name.</summary>
+        public string FileExtension
+        {
+            get
+            {
+                switch (Category)
+                {
+                    case BuildArtifactCategory.Texture:
+                    case BuildArtifactCategory.Mesh:
+                        return ".resS";
+                    case BuildArtifactCategory.Audio:
+                    case BuildArtifactCategory.Video:
+                        return ".resource";
+                    case BuildArtifactCategory.ContentFile:
+                        return ".cf";
+                    case BuildArtifactCategory.Manifest:
+                        return ".json";
+                    default:
+                        return string.IsNullOrEmpty(Category) ? "" : "." + Category;
+                }
+            }
+        }
+    }
+
+    /// <summary>
+    /// In-memory representation of the ContentLayout.json file written by the build.
+    ///
+    /// The Layout is a companion to the BuildManifest, recording additional details about the build
+    /// (including source assets and information about which object an ObjectId hash refers to).
+    /// It is not shipped with the build; it exists for tools and tests that analyze build output.
+    /// The schema is subject to change and there is currently no backward compatibility.
+    /// </summary>
+    public class ContentLayout
+    {
+        /// <summary>The schema version this type represents.</summary>
+        public const int CurrentVersion = 1;
+
+        /// <summary>Schema version of the ContentLayout.json file.</summary>
+        public int Version;
+
+        /// <summary>Hash of the BuildManifest this layout corresponds to.</summary>
+        public string BuildManifestHash;
+
+        /// <summary>The SerializedFiles in the build output.</summary>
+        public SerializedFileLayout[] SerializedFiles;
+
+        /// <summary>ObjectIdHash values of the root assets; resolve via <see cref="LoadableObjectIds"/>.</summary>
+        public string[] RootAssets;
+
+        /// <summary>Loadable objects in the build.</summary>
+        public LoadableObjectIdLayout[] LoadableObjectIds;
+
+        /// <summary>Loadable scenes in the build.</summary>
+        public LoadableSceneIdLayout[] LoadableSceneIds;
+
+        /// <summary>Binary artifacts produced by the build.</summary>
+        public BinaryArtifact[] BinaryArtifacts;
+    }
+}

UnityDataModels/UnityDataModels.csproj

@@ -0,0 +1,17 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <OutputType>Library</OutputType>
+    <TargetFramework>net9.0</TargetFramework>
+    <LangVersion>latest</LangVersion>
+  </PropertyGroup>
+
+  <PropertyGroup Condition=" '$(Configuration)' == 'Debug' ">
+    <PlatformTarget>AnyCPU</PlatformTarget>
+  </PropertyGroup>
+
+  <PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
+    <PlatformTarget>AnyCPU</PlatformTarget>
+  </PropertyGroup>
+
+</Project>