Add LoadFromDDSStream{,Ex}()

[solbjorn]

Many games support own archive formats and implement own sort of a virtual file system. In that case, LoadFromDDSFile() doesn't fit, while LoadFromDDSMemory() requires 2x filesize: one buffer to read the file and another one to load the texture, since ScratchImage is always owning.

Add a simple InputStream class and LoadDDSFromStream() to avoid this memory overhead and allow games and apps to use their own functionality to operate with files or memory.
More formats could support this in future, as well as writing to an OutputStream if/when needed.

[solbjorn]

@microsoft-github-policy-service agree

[walbourn]

For loading DDS content at runtime in a game, I'd suggest not using DirectXTex or at least not the library. It's designed to load and convert all kinds of legacy content to modern formats, so it brings a lot of code to bear.

The DDSTextureLoader module is intended to be more light-weight and focused on the needs of runtime. That may be a good place to look at adding the Stream version instead.

[solbjorn]

We considered using DDSTextureLoader, but unfortunately it's a poor fit for our engine, probably not only ours. We don't want our texture loader to require a Dx context or to create SRVs/UAVs/whatever for us. Once we have a list of textures to load, we load them and create ID3DBaseTexture (using CreateTextureEx()) on as many threads as possible and that's it. Later, we create the associated resources before the first usage. Using DDSTextureLoader would limit this to 8 threads or even to 1 if a deferred context can't be used to do this.
When using DXT1/5 or BC6H/7 (99% of our textures), LoadFromDDS*() is pretty straightforward: it parses the header and then reads the texture directly to the buffer that ScratchImage allocated and that's it.

[walbourn]

For scenarios where you are just wanting to load content, DirectXTex is going to be overkill unless your dataset includes lots of legacy formats (24bpp, etc.) you have to fix-up at load-time.

For most titles/engines, DirectXTex is an excellent library to use for a 'content cooking/packing' step that produces a 'WAD-style' file-system-in-a-file' data file that your application can efficiently load with ASYNC I/O. Loading them as individual DDS files generally gets dominated by the overhead of Open/Close and AV scans.

Is there a particular design requirement here that your engine consume 'loose' files?

[solbjorn]

The textures are stored in game archives (either legacy custom format or squashfs) and are read using the corresponding APIs. There are no individual opens/closes; the archives are indexed, opened, and reading a file from them is very cheap.
Since the archives are compressed, there are two ways of reading: 1) allocate a buffer for the whole file and read it; 2) use a stream for decompressing and reading on the fly. If we take the first approach, then a 8k BC7 texture would require allocating 88 Mb to read it into the memory and then ScratchImage would allocate additional 88 Mb. If I use streams from this PR, only ScratchImage actually allocates the memory and then the texture is read directly into its buffer.

[walbourn]

And to verify for your scenario, you DO NOT expect the loader at runtime to do legacy conversions/expansions (24bpp, 3:2:2, etc.)?

[solbjorn]

Nope, we only use DXT1 (it's not under the 24bpp category I guess?), DXT5, and BC7 (+ BC6H in some future probably). But we only expect a ID3DBaseTexture from our texture loader it expects only a ID3D11Device; the rest is created later separately, so DDSTextureLoader is not an option for us in its current implementation. If it had a function only to do this first step, I'd reconsider switching to it.

Here's all what we need, load + create:

https://github.com/solbjorn/reaper-engine/blob/master/ogsr_engine/Layers/xrRenderDX10/dx10Texture.cpp#L165

(that fallback part after CreateTextureEx() could be removed easily, I think it's a leftover which nobody hits)

(I mean, we'd still need DxTex, but only for screenshot capturing)

[walbourn]

Did you look at ScreenGrab for screenshots?

[solbjorn]

We only use DirectX::CaptureTexture() from DirectXTex to get our main render target and later we process and save the image manually, as we have to handle different types of screenshots (regular screenshots are in PNG or JPG and for this, ScreenGrab would be enough; we also have thumbnails for game saves, which are saved in DDS, but resized and compressed in BC7). So it's not a big deal which API to use for that (and screenshots is not something perf-critical), although we usually write files using our engine API for consistency.