Skip to content

readme2.md

Latest commit

 

History

History
64 lines (55 loc) · 4.5 KB

readme2.md

File metadata and controls

64 lines (55 loc) · 4.5 KB

Video creation based on PRNet + Synthesizing Obama

Terms

Common terminology for this doc and the rest of the project:

Position map
A 256x256x3 array whose [v, u] index stores the position of the vertex with uv-space coordinates (u, v) as an (x, y, z) triple
Texture
A 256x256x3 array whose [v, u] index stores the color of the vertex with uv-space coordinates (u, v) as a (b, g, r) triple
Canonical face/position-map/texture
Canonical face is any face that is used to align other faces, so that the center, size and orientation of the other faces are similar to the canonical face. Usually, there is one canonical face evaluated for every video chunk.
This face itself can have any size and orientation, but usually tends to be aligned so that the x, y, z values lie between ±2.5, with mean 0; and the face is looking directly in the front.
Frontalized position map
A position map that has been aligned with a specific canonical position map

Directory structure

Each video/video-chunk should have a corresponding directory, usually located in the videos folder, which should have the following files/subdirectories:

  • input: [Directory] Contains each video frame as an image (named 00001.jpg, 00002.jpg etc.)

  • posmap: [Directory] [Optional, not used after data extraction] Contains position map detected by PRNet for each video frame (named 00001.exr, 00002.exr etc.)

  • canonical: [Directory] Contains data for the canonical position-map and texture. Has the following files:

    • frontalized.exr: The canonical position map
    • texture.webp: The canonical texture
    • keypoints.npy: A 68x3 array of keypoint positions corresponding to frontalized.exr
    • normals.exr: A 256x256x3 array whose [v, u] index stores the area-vector of the quadrilateral formed by all vertices adjacent to the vertex with uv-space coordinates (u, v)
    • selected.txt: [Optional, not used for anything] A list of all the frames that were used to create this canonical face.

  • output: [Directory] Contains a directory for each face-containing video frame (named 00001, 00002 etc.) containing information of the face. Each such directory has the following files within it:

    • frontalized.exr: The frontalized position map of the face
    • texture.webp: The texture of the face
    • keypoints.npy: A 68x3 array of frontalized keypoint positions corresponding to frontalized.exr
    • params.npy: A vector (usually 20-dimensional) containing PCA parameters for mouth shape and inner-mouth colors
    • lighting.npy: A 4-dimensional vector representing the lighting in the frame
    • texWithoutL.exr: The texture with the lighting separated
    • transform.txt: A 4x3 matrix which, when when multiplied with a homogenized frontalized position, gives the actual non-frontalized position. (i.e. The affine transform required to take the positions in frontalized.exr to the actual positions in the video)

  • pca.npz: Contains the matrices and coefficients required to go from actual frontalized positions and colors of inner mouth to the PCA parameters, and vice-versa.

  • albedo.npy: Contains the matrix required to go from actual texture to the lighting parameters, and vice-versa.

  • video.mp4: [Optional, not used for anything] The actual video.

Instructions for Extracting data

For instructions on how to run a particular python script within this repo, use python <script> -h

  1. Create base directory and the input directory and video.mp4 within it
  2. Extract images using ffmpeg -i <path/to/base>/video.mp4 -q:v 2 <path/to/base>/input/%05d.jpg
  3. Run pipeline/demo.py to get posmap folder
  4. Run pipeline/make_canonical.py to get the canonical face
  5. Run pipeline/get_output.py to get the output folder
  6. Run TMFR to get better results for the keypoints
  7. Run pipeline/convert_tmfr_output.py to get a new_output folder using TMFR results. Replace the output folder with this folder
  8. Run pipeline/get_pca_params.py to get pca.npz in the base directory, and params.npy files in each directory in output
  9. Run pipeline/save_light_albedo.py to get albedo.npy in base, and lighting.npy and texWithoutL.exr in each directory in output

Alternatively, just use the extract-data repository (which contains this one and TMFR as submodules).