Train HugWBC for other robots

[minolta001]

Hello! Thanks for sharing your great work! I am working on using your method to train the wbc policy for Unitree H1-2 and G1 robot. However, the training for H1-2 is not successful. The robot can learn to stand after around 200 iterations. However, as the training time increases, the robot tends to perform more unstable behaviors and never learns how to walk.

I implemented the environment for H1-2 following your code with some modifications due to the difference between H1 and H1-2. Could you please share any insight for the issue I met? Is the symmetry matrix might cause the issue?

[XiMianTLX]

Hi! Have you successfully migrated to G1? We had issues keeping G1 stable at all during our migration.

[XiMianTLX]

We noticed that the “stand still” penalty seems to make G1's feet tend to stay in place.

[litleWY]

Hi! Have you successfully migrated to G1? We had issues keeping G1 stable at all during our migration.

Hello! I also tried to migrate this to G1. Now, I have some issues where the disturb_curriculum is always zero because the rew_tracking_lin_vel is too low to reach the thresholds. Have you encountered this?

[minolta001]

@XiMianTLX Hi! I am currently working on H1-2, however, I met the same issue mentioned by @litleWY. The disturb_curriculum is always zero. Though I didn't realize the issue is cause by the low rew_tracking_lin_vel reward.

@XiMianTLX A suggestion for you. Please try to increase the penalize_curriculum_sigma, which I believe can somehow have effect on the learning process.

[litleWY]

@XiMianTLX Hi! I am currently working on H1-2, however, I met the same issue mentioned by @litleWY. The disturb_curriculum is always zero. Though I didn't realize the issue is cause by the low rew_tracking_lin_vel reward.

@XiMianTLX A suggestion for you. Please try to increase the penalize_curriculum_sigma, which I believe can somehow have effect on the learning process.

    def update_disturb_curriculum_grid(self, env_ids, noise_env_ids):
        if len(env_ids)==0: return
        timesteps = int(self.cfg.commands.resampling_time / self.dt)
        ep_len = min(self.max_episode_length, timesteps)

        # only for disturb masks
        curr_is_pass = torch.ones(len(noise_env_ids), dtype=bool, device=self.device)
        curr_is_down = torch.zeros(len(noise_env_ids), dtype=bool, device=self.device)

        for key, value in self.curriculum_thresholds['disturb'].items():
            all_rew = self.command_sums[key][noise_env_ids] / ep_len
            success_threshold = value * self.reward_scales[key]
            if key in self.curriculum_reward_list:
                success_threshold *= self.curriculum_scale

            curr_is_pass *= (all_rew > success_threshold)
            curr_is_down += (all_rew < success_threshold / 2)
        
        self.disturb_rad_curriculum[noise_env_ids] = torch.where(
            curr_is_down,
            (self.disturb_rad_curriculum[noise_env_ids] - 0.05).clip(min=0),
            torch.where(
                curr_is_pass,
                (self.disturb_rad_curriculum[noise_env_ids] + 0.05).clip(max=self.cfg.disturb.max_curriculum),
                self.disturb_rad_curriculum[noise_env_ids]
            )
        ) 

        # resample all noise envs disturb
        self.disturb_masks[env_ids] = (torch.rand(len(env_ids))<=0.5).to(self.device) # Reset with half with disturb.
        is_noise = torch.rand(len(env_ids)) <= self.disturb_noise_ratio
        self.disturb_isnoise[env_ids] = is_noise.to(self.device)
        self.disturb_actions[env_ids] = self.dof_pos[env_ids, -self.disturb_dim:] - self.default_dof_pos[:, -self.disturb_dim:]
        if self.disturb_replace_action:
            self.interrupt_mask[env_ids] = self.disturb_masks[env_ids]
        else:
            self.interrupt_mask[env_ids] = self.disturb_masks[env_ids] * (~self.disturb_isnoise[env_ids])

    class curriculum_thresholds:
        class upper_motion:
            tracking_lin_vel = 0.8  
        class terrains_level:
            tracking_lin_vel = 0.8
            dof_vel_limits = 0.1 
        class terrains_type:  
            tracking_lin_vel = 0.8  
        class commands:
            tracking_lin_vel = 0.8  
            tracking_ang_vel = 0.4  
            tracking_contacts_shaped_force = 0.8
            tracking_contacts_shaped_vel = 0.8

Level Up Conditions
To enter a more difficult terrain (terrain_levels += 1), the robot must meet all three of the following conditions at the end of an Episode:
Physical displacement meets the standard (Distance Task):
• Judgment criterion: real_distance > self.terrain.env_length / 2
• Meaning: The straight-line distance the robot actually moves from the spawn point must exceed half the length of the current terrain block. If your G1 has too small a stride due to "crouching" or "leg clamping", even if it walks steadily, it cannot level up as long as it does not cross this line.
Orientation Stability:
• Judgment criterion: ~self.large_ori_buf[env_ids]
• Meaning: The robot does not experience violent deflection or spinning in place during movement. If the robot travels far but keeps "spinning in circles", the system will determine its posture is uncontrollable.
Movement quality meets the standard (Reward Success):
• Judgment criterion: is_success_level is True
• Detailed calculation: The system will iterate through each item in curriculum_thresholds['terrains_level'] (usually including tracking_lin_vel). It is required that the average reward (task_reward) for this item must be greater than the preset success threshold:
task_reward > value × reward_scales[key]
• Veto: As long as one assessment indicator (such as speed tracking) fails to meet the standard, even if the robot reaches the end of the terrain block, it cannot be promoted.

This is the condition of updating the curriculum, which is answered by gemini, you can refer to that!
Part of the reason on my end is tracking_lin_vel

[XiMianTLX]

Thank your very much! We'll try it soon!!

[litleWY]

Thank your very much! We'll try it soon!!

I've tried to decrease the tracking_lin_vel thresholds, but it still doesn't work. I think G1 can't meet the other two factors either.

[minolta001]

Thank your very much! We'll try it soon!!

I've tried to decrease the tracking_lin_vel thresholds, but it still doesn't work. I think G1 can't meet the other two factors either.

@litleWY I did the same. I reduced the tracking_lin_vel threshold, and it actually ruined the training. Have you tried modified other factors?

[minolta001]

@litleWY BTW, do you might share where you track the conditions? I know there are multiple factors will be checked during training, I just want to see if I could track the variables changes in order to see how far the robot is away from upgrading the terrain difficulty and disturbance.