File Format

Format, Codec and Library Decision

The first part of this development was to determine what format was best suited to recording the streams from the Kinect in terms of space, write performance, read performance and complexity. The initial ideas were:

• Use a very simple method of writing out the pure binary data to the file
• Use FFMPEG or similar to write to an AVI file, or similar
• Use OpenCV to write to an AVI file or similar

The initial route followed was to try and use FFMPEG libraries to write a simple, uncompressed AVI file. This, unfortunately proved to be quite difficult to research and implement and the decision was made to move on to another format before running down a dead end street.

The next step was to try OpenCV to write an AVI file. Although on paper this looked promising with the simple enough interface of OpenCV, the lack of certain documentation made this a prolonged implementation, with less than adequate results; the process used by OpenCV was a blocking type I/O, meaning that the program had to halt the execution temporarily in order to finish writing data to disk. This eventually lead to dropping frames when dealing with both depth and RGB streams of a high and moderate quality respectively. As an attempt to tackle this problem, a threaded approach was taken, to separate the part of the application to receive frames from the Kinect and the part of the program to write the output file. Unfortunately, whatever way OpenCV was structured, this lead to the same problem of the entire application being paused for I/O.

The final fallback was to standard binary output libraries or similar to write the pure binary data. This posed a few challenges. Whereas using AVI implicitly separated frames from each other in libraries like OpenCV and is a common format to store streams while containing info about the stream such as width and height, this needed to be explicitly defined in our format as this defines the amount of data to be read in per frame. On top of this, the format information has to define what streams are present, and should also give the frame time in ms from the start of recording, in case a ToF algorithm or likewise is used after the recording is done.

Note that the RGB stream from the Kinect is in BGRA format. Although the Alpha channel is removed, the resulting format is still BGR. To get RGB, the channels have to be reversed.

File Structure

Stream Info

The start of the file defines the streams present, and the resolution of each stream. The first 5 bytes are chars, defining a stream, followed by 8 bytes, again chars, defining the resolution. In the case of a second stream, the same 5 + 8 byte structure is repeated. The valid types and resolutions are shown in the table below

Stream Speficier (5 Bytes)	Resolution Specifier (8 Bytes)
DEPTH	80X60
	320X240
	640X480
RGB	640X480
	1280X960

An example of the total 5 + 8 bytes stream and resolution specifier for an 80 x 60 depth stream is shown below.

| D | E | P | T | H | 8 | 0 | X | 6 | 0 | | | |

Frames

The sequence of frames is predetermined. It is always depth first if any, followed by RGB. Each frame is raw, uncompressed data, lead by frame info. The first 5 bytes are chars defining the stream represented in the frame. This is useful in the event a frame from one stream is dropped while the frame from the other stream is received correctly. The next 4 bytes form a 32-bit unsigned long representing the frame time in ms since the start of recording. The rest of the frame data limits is determined by the frame dimensions and bytes per pixel. For example a depth image uses 2 bytes per pixel (1 * unsigned short) whereas the RGB color image uses 3 bytes per pixel (3 * unsigned char). At a resolution of 80 x 60, the depth frame data would be 80 x 60 x 2 = 9600 bytes.

File Sequence

The sequence of info and data is shown below.

Stream info	Frames
SS RS (SS RS)	[SS TC FD]

Where:

• SS is the 5 byte stream specifier
• RS is the 8 byte resolution specifier
• TC is 4 byte frame time code
• FD is the actual frame data