Forked from PointsCoder/OpenReal2Sim — extended for 3DGS-based scene reconstruction with ground-truth depth calibration.

OpenReal2Sim is a toolbox for real-to-sim reconstruction and robotic simulation. This fork adds key modifications for the HumanoidEverywhere pipeline, where we reconstruct objects from 3DGS-rendered scenes rather than from RGBD sensors.

Problem: The original pipeline calibrates predicted depth using RGBD sensor data (.r3d files). In our pipeline, scenes are captured as 3D Gaussian Splatting (3DGS) models — there is no RGBD sensor.

Solution: We use GT depth rendered from 3DGS (depth_capture.npy) to calibrate the monocular depth predictor. The depth calibration module (openreal2sim/preprocess/depth_calibration.py) was modified to:

• Accept .npy depth maps from 3DGS rendering (instead of only RGBD .r3d files)
• Read camera intrinsics and depth scale from per-key metadata JSONs (e.g., real_test1_metadata.json)
• Use robust Huber-weighted least squares to fit scale and shift parameters: depth_gt = scale × depth_pred + shift
• Optionally refine depth prediction with MoGe-2 before calibration

Config (config/config.yaml):

local:
  real_test1:
    preprocess:
      metadata_file: real_test1_metadata.json  # Camera intrinsics + pose
      depth_scale: 1.0                         # GT depth is in meters
      depth_min: 0
      depth_max: 10.0

Problem: Object placement on surfaces (tables, counters) needs accurate collision-free positioning. The original uses plane-fitting which can be inaccurate for complex surface geometry.

Solution: Added gt_depth collision optimization mode (openreal2sim/reconstruction/modules/scenario_collision_optimization.py):

• gt_depth mode: Uses the actual GT depth map from 3DGS to compute the surface height at each object's XY position. Objects are placed so their bottom surface sits exactly on the depth-derived surface, then lifted by a configurable clearance.
• sdf mode: Uses Kaolin to build a volumetric SDF of the background mesh, then optimizes object Z position to minimize penetration while maintaining contact.
• Gaussian PLY translation: When objects are repositioned during collision optimization, their corresponding 3DGS PLY files are also translated to match (gaussian_transform.py).

Config:

local:
  real_test1:
    reconstruction:
      collision_mesh_source: "fdpose"      # Use FoundationPose mesh for collision
      collision_optimization_mode: "gt_depth"  # "plane", "sdf", or "gt_depth"
      collision_clearance: 0.05            # 5cm above surface

Added silhouette-based rotation search for better object pose estimation:

reconstruction:
  use_azimuth_sweep: true        # Enable rotation search
  azimuth_n_candidates: 36       # Test 36 angles (every 10°)

This renders the object silhouette at each candidate rotation and picks the angle that best matches the masked region in the input image.

Added openreal2sim/simulation/prepare_simulation.py — prepares reconstructed meshes for downstream simulation:

• Selects the best mesh variant per object (priority: optimized > registered > fdpose)
• Copies GLB meshes to the simulation output directory
• Generates scene.json with Docker-compatible paths
• Optional VLM-based scaling for real-world dimensions

Clone this repository recursively:

git clone --recurse-submodules https://github.com/Junyingw/OpenReal2Sim.git

For Docker setup, see docker installation.

See the HumanoidEverywhere pipeline docs for the full workflow.

cd OpenReal2Sim

# 1. Depth Calibration (uses GT depth from 3DGS)
PYTHONPATH=$(pwd):$PYTHONPATH python openreal2sim/preprocess/preprocess_manager.py --key_name <key>

# 2. Object Segmentation (interactive)
PYTHONPATH=$(pwd):$PYTHONPATH python openreal2sim/reconstruction/tools/segmentation_annotator.py

# 3. Object Mesh Generation
PYTHONPATH=$(pwd):$PYTHONPATH python openreal2sim/reconstruction/recon_agent.py \
    --use-sam --stage object_mesh_generation --key <key>

# 4. Prepare Simulation Assets
python openreal2sim/simulation/prepare_simulation.py --key <key>

Sample data is included in the data/ directory:

Each key includes: RGB image (.png), depth map (_depth.npy), and camera metadata (_metadata.json).

If you find this repository useful in your research, please consider citing:

@misc{openreal2sim,
  title={OpenReal2Sim: A Toolbox for Real-to-Sim Reconstruction and Robotic Simulation},
  author={OpenReal2Sim Development Team},
  year={2025}
}

@inproceedings{rola,
  title={Robot learning from any images},
  author={Zhao, Siheng and Mao, Jiageng and Chow, Wei and Shangguan, Zeyu and Shi, Tianheng and Xue, Rong and Zheng, Yuxi and Weng, Yijia and You, Yang and Seita, Daniel and others},
  booktitle={Conference on Robot Learning},
  pages={4226--4245},
  year={2025},
  organization={PMLR}
}