Unexpected Artifacts in Dynamic Renders for Aria Dataset

[Kennethii2i]

Hi!

I've been testing the model with the Aria Digital Twin (ADT) dataset, following the data preparation steps in the README using arguments/aria_data.py.

I'm getting some results in the dynamic_final renders that I didn't quite expect, and I was hoping you might be able to offer some guidance. I'm not sure if I've misunderstood the expected output or perhaps missed a step in my setup.

Here are the two cases I'm referring to:

1. Background elements in "Apartment" scene render
   • Sequence: Apartment_release_golden_skeleton_seq100_10s_sample_M1292
   • Observation: When I view the dynamic_final render, it seems to contain a significant amount of what looks like the static background.
   • Question: I was expecting the dynamic_final render to mostly isolate the dynamic object. I was wondering if this inclusion of the background is expected behavior for this kind of complex scene.

2. Bright spots in "WoodenFork" scene render

• Sequence: Lite_release_recognition_WoodenFork_seq032_61283
• Observation: In this sequence, the dynamic_final render seems to include some bright static light fixtures on the ceiling.
• Question: I was curious if the model might sometimes classify very bright static spots as dynamic?

I apologize if these are basic questions.
Thank you for your time and consideration!

[BatFaceWayne]

Hi this is very good observation and may be very helpful for other users! Currently to balance the scene learning rate for various input scene scale and pose estimation method, the deformation MLP and grid learning rate is scaled by "spatial_lr_scale", obtained by the camera trajectory range:

And it works nicely with camera moving with human. For example:

However for your sequences where the camera wearer is standing still(leading to very small camera range), "spatial_lr_scale" is smaller than ideal case.

Therefore for these squences you could carefully scale_up the "spatial_lr_scale", e.g. add in front of the line 295 of the gaussian_model.py.

self.spatial_lr_scale = self.spatial_lr_scale * 2

Moreover, specifically for the first sequnce as it is very short(~10s) with limited scene coverage, the point cloud init is also very sparse(~9k). In this case you could set the foreground point cloud initialization to be 1k instead of 10k random points, in dataset_reader.py.

As the result you would be able to obtain correct decomposition results as

And

I hope this helps!

Hint: in aria_data.py it is recommended to set the last term of grid resolution to be half of the sequence length.

[Kennethii2i]

Thank you for your kind reply.
I have two follow-up questions regarding the P0001_10a27bf7 in Aria Hot3D dataset.

First, regarding the hint you mentioned at the end: "set the last term of grid resolution to be half of the sequence length."
How should I interpret "sequence length" in this context?

• Case 1: The number of frames after preprocessing? (e.g., I have 350 frames after the nerfstudio preprocess, so should I set the value to 175?)
• Case 2: The total number of frames from the original sequence? (e.g., The original is 124.39s at 30fps. Should I use (30 * 124.39) / 2, which is approximately 1866?)

Second, I observed a discrepancy. Although I run DeGauss train.py in 350 frames with aria_data config, the final result of static_raw and mask_comp only contain 300 image frames, starting from the 50th frame (frames 51-350).
Is this expected behavior, or did I misconfigure something?

[BatFaceWayne]

Hi, I would recommend to set the sequence to be the half actual sequence length. For this specific sequence(results could be found on our project website: https://batfacewayne.github.io/DeGauss.io/static/new_videos/aria_quali/hot3d_vis.mp4), we simply set the last resolution parametes to be 2000. But we also got roughly the same number after the nerfstudio preprocess(~3.7k frames). Could you please double check that?

And for the second point, as stated in our paper, we drop the first 50 frames allowing the recording to be stablized (camera exposure, ISO etc.). You may modify this line in dataset_reader to disable this cropping behavior.

https://github.com/BatFaceWayne/DeGauss/blob/e8abdc02ea977545b105afcef9c9bb7516bc8807/scene/dataset_readers.py#L574C1-L574C28

[Kennethii2i]

Thank you very much for your reply and clarification.

You mentioned that for the P0001_10a27bf7 sequence, you "got roughly the same number after the nerfstudio preprocess(~3.7k frames)."

When I followed the data preparation steps, I only got 350 frames.
I suspect I may be downloading the wrong data or using an incorrect method. Here are the exact steps I took:

1. Went to the P0001_10a27bf7 Hot3D page.

2. Clicked URLs for downloader CLI on the bottom right to download the .json file.

3. Ran the command:

aria_dataset_downloader --cdn_file {your_json_file} --output_folder {output_folder} --data_types 0 1 2 3 4 5 6 7

4. Ran the nerfstudio preprocess command as per the nerfstudio Aria custom data documentation:

ns-process-data aria --vrs-file {your_vrsfile_in_output_folder} --mps-data-dir {your_mps/slam_directory_in_output_folder} --output-dir {path_to_your_output_directory}

Could you please confirm if these are the correct steps, or if there is a different procedure I should be following to obtain the ~3.7k frames you mentioned?

Thank you again for your guidance.

[BatFaceWayne]

Hi, we noticed that in a recent nerfstudio update --max-frames is set to 350 by default. You could override this by manually setting max frames number:

ns-process-data aria --vrs-file {your_vrsfile_in_output_folder} --mps-data-dir {your_mps/slam_directory_in_output_folder} --output-dir {path_to_your_output_directory} --max-frames 4000

[Kennethii2i]

That worked perfectly! I got ~3.7k images. Thank you so much for pointing that out!

[Kennethii2i]

Question: Regarding #10 (comment), first sequence

Following your suggestions, I made the following changes:

• gaussian_model.py: Added self.spatial_lr_scale = self.spatial_lr_scale * 2 before line 295.
• dataset_readers.py: Changed the number of random points from 10k to 1k in the use_1_user block within the readColmapSceneInfo function for aria dataset.
• aria_data.py: Set the last term of the grid_resolution to 150 (half of the 300-frame sequence).

However, even with these changes, I was still unable to reproduce the clean decomposition results shown in your screenshots. The result is still not ideal. I have attached the images of my results for your reference.
Do you have any idea what might be causing this? In this case, should I try increasing the spatial_lr_scale even further (e.g., x3 or x4)?
Just to be safe, I have pasted my aria_data.py configuration below. Could you please check if it matches the configuration you are using?

Thank you again for your help.

aria_data.py

_base_="default.py"
ModelParams=dict(
    kplanes_config = {
     'grid_dimensions': 2,
     'input_coordinate_dim': 4,
     'output_coordinate_dim': 16,
     'resolution': [256, 256, 256, 150]
##### 1400 as half the sequence length of aria data
#### lower resolution and multi res [1, 2] for faster training
    },
    multires=[1, 2, 4],
    defor_depth=1,
    net_width=128,
    no_do=False,
    no_dshs=False,
    no_ds=False,
)

OptimizationParams = dict(
    iterations = 120_000,
    coarse_iterations = 1000,
##### coarse_iterations as half the sequence length of aria training data(assuming batchsize 2)

    densify_until_iter = 100_000,
    position_lr_max_steps = 120_000,
    ########### grid lr is recommended to be larger for aria data, but consider lower lr in case training is unstable
    grid_lr_init= 0.0004,
    grid_lr_final=0.000016,
    densify_grad_threshold_after=0.0002,
    densify_grad_threshold_coarse=0.0002,
    densify_grad_threshold_fine_init=0.0002,
    deformation_lr_init=0.00016,
    deformation_lr_final=0.000016,
    prune_small_foreground_visbility = False,
    downscale_mask_deform_lr = 0.1,
    accumulation_steps=1,
    reset_SH = False,
    lambda_main_loss =4,
    SH_lr_downscaling_start=8,
    SH_lr_downscaling_end=8,
    lambda_depth_smoothness=0.1,
    opacity_reset_interval = 4000,
    pruning_interval = 300,
    weight_penal_light_end=0.1,
    ###### slower background foreground assignment
    downscale_ulti_loss = 0.5,
    ############  push background floaters further away from the camera and pruned later
    separation_high_prob = True,
    detach_background_separation = False,

    eval_include_train_cams = True,
    vignette_mask = './assets/vignette_imx577.png',
    camera_mask = './assets/aria_camera_mask.png',
    foreground_oneupshinterval = 1000,
    background_oneupshinterval = 10000,
    lambda_loss_depth_back = 1,
    max_gaussian_foreground = 100000,
    max_gaussian_background = 360000,
    make_background_thresh_larger = False,
    timenet_width = 512,
    timenet_output = 128,
    coarse_mean_override = False,
    ####### use white bg for aria data
    force_white_background = True
)

image reference

[BatFaceWayne]

Hi, this is interesting — your modification is actually exactly the same as what I implemented. Our method should generally work robustly as long as the learning rate stays within a reasonable range.

However, I noticed that the depth renderings you obtained look very different from mine. Could you try setting up a fresh environment and installing everything exactly as described in the README (especially installing depth-diff-gaussian-rasterization via
pip install -e submodules/depth-diff-gaussian-rasterization)?

Please note that newer versions of the original depth-diff-gaussian-rasterization use a different depth scale, and other diff-gaussian-rasterization variants may not support depth rasterization in backward passes. It’s important to use the version of depth-diff-gaussian-rasterization bundled with 4DGaussians or the one included in our repository. I hope this helps!

[Kennethii2i]

Hi, thank you so much for your quick and helpful response!

Regarding depth-diff-gaussian-rasterization: I have confirmed that it is installed correctly from the submodules in your repository and is functioning without any errors.

Now that the rasterization is working properly, however, I have noticed a different behavior in the results that I would like to ask about:

Regarding to the same dataset (Apartment_release_golden_skeleton_seq100_10s_sample_M1292), Why is the empty space (where the cup used to be) being classified as Foreground?

• In the Dynamic Final render, the model appears to identify the empty table surface as Foreground after the cup is lifted.
  I suspect there might be some underlying issue with the training process causing the model to converge to this state. Do you have any insights into what might be causing this behavior?

Thank you again for your time.

[BatFaceWayne]

Hi glad to see the rasterizer works properly!

For the foreground/background confusion, it is expected to have these artifacts for long-time semi-static objects especcially for short recordings, as our method does not take any semantic information as input. :)

[Kennethii2i]

Thanks for the clarification!