Training the diffusion model on my own data

[ritvikpasham]

I have my own frames and actions from training my own RL agent to play some of the Atari games that your model is trained on. I wanted to run your diffusion model on my data to see how well it trains on my data and whether I need to collect more frames and actions. Is there a good way of going about this?

[tryfonaskam]

this sound very intresting. i am going to try to get some models to work with diamond if i do it i will post here

[jovog002]

I would also like to train diamond on the frames of another game. It would be ideal if i had to put the videos for training in a folder so that I could then simply triage them. Is that easily possible? Either I'm missing something in the code or this is not yet possible without further ado?

[Dere-Wah]

To do so you need to process your dataset to fit a specific format. As shown in here:

https://github.com/eloialonso/diamond/tree/csgo?tab=readme-ov-file#training

You need to follow the format of the following dataset: https://github.com/TeaPearce/Counter-Strike_Behavioural_Cloning

If you take a look at what the dataset looks like, it is basically a list hdf5 files, each one containing information about 1000 frames or so called "Episode". A single episode, from what I can understand from the code of this project, is a combination of Image data & key inputs.

If you actually download a .hdf5 file and inspect it with an online inspector, you will find a data structure documented here:
https://github.com/TeaPearce/Counter-Strike_Behavioural_Cloning?tab=readme-ov-file#structure-of-hdf5-files-image-and-action-labels

So if you want to create your hdf5 files for this model, you can just create the frame_i_x & frame_i_y structures, the first one containing image data and the second one an array of bits that indicate if in that frame a specific key (mapped to the index) is being pressed or not. Again, I recommend downloading a sample from the dataset and inspecting it to get a clear view of what the final data should look like.

Once you have your list of hdf5 files, each one containing 1000 episodes, you need to preprocess them. Take a look at this:
https://github.com/eloialonso/diamond/blob/csgo/src/process_csgo_tar_files.py

What it does is decompressing the .tar which contains the hdf5 files, and then converts in the Dataset-Episodes format that this project uses. But here we directly created hdf5 files, with no .tar compression! So you'll need to adapt the code to directly look at your hdf5 files instead of expecting a .tar to decompress.

Each episode has an "obs" (observation) , which is basically the image, and an act (action) which is the array of buttons. (Look at /data/Episode.py)

diamond/src/process_csgo_tar_files.py

Line 128 in 3ab441c

episode.obs, (30, 56), interpolation=T.InterpolationMode.BICUBIC

If you're training at a different resolution you may want to change these values. (height, width)

I figured all of this stuff out while looking through the code, as I am working on training a model to simulate a SM64 stage. I'll try my best to document my progress on the discord of this project.

[tryfonaskam]

@Dere-Wah do you have a python script for recording data? if so could you share it?
if not do you know of any that would be easy to modify to be compatable with other games?