Demo for the navigation task

[TimeEscaper]

Dear authors,

Thank you very much for releasing the code of the great work! We are now trying to deploy your model for the mobile robot navigation task. We are using inference_pretrained.ipynb as a reference and trying to adapt it for the navigation demo, but facing some troubles with that. Can you please outline the steps that should be done for running your model in navigation mode?

Thank you in advance!

[HomerW]

To adapt the inference example from the notebook for navigation:

• Use the navigation head by passing head_name='nav' into sample_actions
• Use the omnimimic_gnm statistics to unnormalize the action by passing these into sample_actions
• The observation dictionary should contain a single key image_nav containing the image observation

However, note that the full trajectory inference example in the notebook doesn't actually handle the timestep pad mask correctly. To actually roll out the policy on a robot you can use the evaluation server here. This is how we evaluated on mobile navigation robots. This server has a function stack_and_pad that will correctly stack the history of observations and create a correct timestep padding mask. Hope this helps!

[jisa]

Cool project!

Using head_name='nav' and unnormalization_statistics=model.dataset_statistics["omnimimic_gnm_dataset"]["action"] leads to an error:
ValueError: Incompatible shapes for broadcasting: shapes=[(6, 4, 2), (100, 2)]

Where 6 is my input batch size. 4 is the number of steps predicted by the model. 100 is number of steps in the statistics. 2 is the dimensionality of a single action.

I.e. my understanding is that this is because the model predicts 4 steps, while the statistics is for 100 steps.

• Is there a simple way to either make the model predict 100 steps or to trim the statistics to 4 steps? For performance reasons, I would prefer the later. Looking at the statistics, I understand that worst case I can trim it myself.
• What is the interpretation of the outputs of the nav head? Some kind of waypoints? Are they all relative to the robot, or somehow relative to the previous waypoint?

[jisa]

After trimming the stats myself with unnormalization_statistics=dict((stat_name, stat_value[:4,...]) for (stat_name, stat_value) in model.dataset_statistics["omnimimic_gnm_dataset"]["action"].items()), combined with knowledge from the paper and about previous related projects, I am guessing I am looking at metric waypoints relative to the robot, covering a short distance. The robot collecting the nav data is small and not very fast.

[molnarszilard]

Could you provide a minimal working example of what the client should look like if I want to use the server?
I would be interested in either single-arm manipulation or ground robot navigation.