Apple Unveils HALO: A Robotic "Brain" Scoring 91% Success in Unseen Real-World Spaces

Apple's research team has detailed a significant advance in robotic control with the release of a paper on the arXiv preprint server. The work introduces HALO, a control system for humanoid robots, and goes a step further by open-sourcing the full pipeline code for data generation, model training, and deployment.

Real-World Testing Shows High Proficiency in Single Tasks

The core of the research involved rigorous real-world evaluation. The team conducted 360 tests across six distinct, unmapped indoor environments, asking the robot to perform individual tasks like object pickup and door opening. The system achieved an impressive average success rate of 91%.

To test sequential reasoning, a more complex combined task was introduced: "duck under an obstacle, walk to a target, then push a box." In this scenario, the average success rate was 70%, with the full three-step sequence succeeding 50% of the time. This highlights a remaining challenge in multi-step planning and physical interaction.

Computational Muscle: Training on NVIDIA Hardware

Building HALO required substantial computational resources. The team synthesized the primary training dataset using approximately 250 NVIDIA H100 GPUs over 48 hours. Training was split into two core components:

  • The Motion Generator: Trained via imitation learning on 64 NVIDIA B200 GPUs, this module learns to create plausible action sequences.
  • The Motion Tracker: Trained using reinforcement learning and policy distillation on 4 H100 GPUs, this module ensures the robot faithfully executes those sequences in the real world.

The process wasn't fully automated. For new physical tasks, researchers still needed to manually craft text prompts, design key motion frames, and sometimes use human video or motion-capture data as reference.

Vision-Only Limits and the Path Forward

The paper candidly outlines the system's current limitations. HALO primarily relies on visual perception, lacking direct sensing of contact forces or textures. Performance degrades on contact-heavy tasks when objects leave the camera's view or are occluded.

Consequently, a major future direction identified by the team is the integration of tactile sensing. Equipping robots with a sense of "touch" alongside sight is seen as crucial for operating in unpredictable, real-world environments.