Apple’s research team has stepped into the spotlight of humanoid robotics with the unveiling of its advanced autonomous control system, dubbed HALO. Detailed in a pre-print paper titled ‘Perceptive Humanoid Motion from Simulation’ submitted to arXiv on November 4th, the HALO system represents a significant leap in enabling robots to perform complex tasks in varied environments.
HALO: A Modular Approach to Robot Autonomy
The HALO system is designed with a multi-layered architecture, integrating cutting-edge AI models for perception, decision-making, and motion control. The research team has made the system’s data generation, model training, and deployment codes publicly available, covering key areas such as motion generation, motion tracking, and task orchestration.
System Architecture and Hardware
For its real-world demonstrations, Apple utilized the Unitree G1 humanoid robot. A custom 3D-printed shell houses a Mac mini equipped with an M4 Pro chip and 64GB of unified memory. This onboard Mac mini handles the computationally intensive tasks of the Orchestrator (Vision-Language Model or VLM) and the Motion Generator (DiT), enabling real-time inference. The G1’s native onboard chips are dedicated to high-frequency operations of the Tracker, which employs a Reinforcement Learning (RL) policy network.
Hierarchical Control for Complex Tasks
HALO’s control operates on three collaborative tiers:
- Task Orchestration (Top Layer): This layer leverages a Vision-Language Model (VLM) to interpret tasks and plan sequences. The system defaults to using the Qwen3.6-35B-A3B model but can also access cloud-based models via API.
- Motion Generation (Middle Layer): Another Qwen model, specifically Qwen3.5-2B, is employed in this layer to generate appropriate robot motions based on the orchestrator’s plan.
- Motion Tracking (Bottom Layer): Trained through techniques like reinforcement learning and policy distillation, this layer runs at a high frequency of 50 Hz, ensuring precise execution of generated motions.
Performance Metrics and Testing
The research paper highlights HALO’s performance across various testing scenarios:
- In 360 single-task tests conducted in six previously unmapped environments, the system achieved an average success rate of 91%.
- For long-sequence composite tasks performed in a single real-world environment, three combined tasks were tested 10 times each, resulting in an average success rate of 70%.
- The most complex task, described as “bend down to pass through an obstacle — walk to the target — push a box,” was tested 10 times and succeeded 5 times, yielding a 50% success rate.
Training and Data Generation
The development of HALO involved substantial computational resources and innovative data generation techniques. Approximately 48 hours of computation using 250 Nvidia H100 GPUs were used to synthesize the main training data. The motion generator was trained using imitation learning with 64 B200 GPUs, while the motion tracker utilized 4 H100 GPUs for its reinforcement learning and policy distillation training.
Challenges and Future Directions
Despite its impressive capabilities, the research team acknowledges certain limitations. The current system primarily relies on visual perception, which can hinder performance in tasks requiring intricate contact interactions or when objects move out of the camera’s view or become occluded. Incorporating tactile sensing is identified as a key area for future improvement to enhance the robot’s ability to interact with its physical environment more robustly.
The release of HALO and its associated code signals Apple’s deep engagement in advancing robotics and AI, potentially paving the way for future applications in various fields.
For further details, you can explore the resources provided by the research team:
- Paper: https://arxiv.org/abs/2611.06908
- Project Page: https://apple-aiml-research.github.io/ml-halo/
- GitHub Repository: https://github.com/apple-aiml-research/ml-halo
