RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot

|-- RH20T
    |-- RH20T_cfg1
    |   |-- calib/                            # Calibration folder, including calibration-time Gripper Cartesian pose, intrinsic and extrinsic matrices etc. Extrinsic matrices are the Aruco marker's Translations with respect to the camera frame.
    |   |-- task_0001_user_0001_scene_0001_cfg_0001/        # Robotic manipulation data
    |   |    |-- metadata.json                # Robot manipulation scene metadata, including scene finishing timestamp, task completion rating (0 denotes robot failure, 1 denotes task failure, 2-9 denotes completion quality, higher is better), calibration timestamp and calibration quality (0 means some cameras are not calibrated, 1-5 means calibration accuracy, lower is better), etc.
    |   |    |-- cam_[serial_number]/         # Multiple cameras
    |   |    |    |-- color.mp4               # Color images, encode as video. The extraction code is available in our API code.
    |   |    |    |-- timestamps.npy          # Timestamp for each image, our extraction code will use it to decode images.
    |   |    |    `-- depth.mp4 (optional)    # Depth images, encode as video. The extraction code is available in our API code.
    |   |    |-- transformed/
    |   |    |    |-- tcp.npy                 # Gripper Cartesian pose in each cam's coord, {serial number: [{"timestamp": ..., "tcp": ..., "robot_ft": ...}]}, where "tcp" values are xyz+quat (7D) Gripper Cartesian poses
    |   |    |    |-- tcp_base.npy            # Gripper Cartesian pose in base coord, {serial number: [{"timestamp": ..., "tcp": ..., "robot_ft": ...}]}, where "tcp" values are xyz+quat (7D) Gripper Cartesian poses
    |   |    |    |-- joint.npy               # Joint angles, {serial number: {timestamp: joint angle array}}
    |   |    |    |-- gripper.npy             # Gripper commands and information, {serial number: {timestamp: {"gripper_command": 3D array, "gripper_info": 3D array}}}, where the 1st element in the 3D array is the actual gripper width in millimeters(0-110)
    |   |    |    |-- force_torque.npy        # 6-DoF force/torque in cam's coord, {serial number: [{"timestamp": ..., "zeroed": ..., "raw": ...}]}, where "zeroed" values are pre-processed
    |   |    |    |-- force_torque_base.npy   # 6-DoF force/torque in base coord, {serial number: [{"timestamp": ..., "zeroed": ..., "raw": ...}]}, where "zeroed" values are pre-processed
    |   |    |    `-- high_freq_data.npy      # High frequency data, {serial number: [{"timestamp": ..., "zeroed": ..., "raw": ..., "tcp": ...}]}
    |   |    `-- audio_mixed/
    |   |
    |   |-- task_0001_user_0001_scene_0001_cfg_0001_human/  # Human demonstration data corresponds to the above robotic manipulation
    |   |    |-- metadata.json                # Human demonstration metadata, including scene starting and finishing timestamps, calibration timestamp and quality
    |   |    |-- cam_[serial_number]/
    |   |    |    |-- color.mp4
    |   |    |    |-- timestamps.npy
    |   |    |    `-- depth.mp4 (optional)
    |   |    `-- audio_mixed/
    |   |
    |   |
    |   `-- ... ...
    |
    |
    |-- RH20T_cfg2/
    |   `-- same as above
    |
    |
    |-- ...
    |
    |
    `-- RH20T_cfg7/

@inproceedings{
    fang2023rh20t,
    title = {RH20T: A Robotic Dataset for Learning Diverse Skills in One-Shot},
    author = {Fang, Hao-Shu and Fang, Hongjie and Tang, Zhenyu and Liu, Jirong and Wang, Junbo and Zhu, Haoyi and Lu, Cewu},
    booktitle = {RSS 2023 Workshop on Learning for Task and Motion Planning},
    year = {2023}
}

The dataset is licensed under a mixture of licenses as it is partly funded by a company. It is divided into two subsets: RH20T-C (commercial) and RH20T-NC (non-commercial).

The RH20T-C subset contains episodes with names containing 'scene_0001' to 'scene_0005'. It is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0).

The RH20T-NC subset contains episodes with names containing 'scene_0006' to 'scene_0010'. It is licensed under a Creative Commons Attribution 4.0 Non-Commercial License (CC BY-NC 4.0), which is freely available for free non-commercial use, and may be redistributed under these conditions. Commercial use of the RH20T-NC subset or models trained on it is not allowed.

If you have any further questions, please contact fhaoshu@gmail.com.