Magic Leap 2 controllers have built-in indoor-outdoor tracking

A picture shared on Twitter from the upcoming Magic Leap 2 shows cameras on the controllers, used for upside-down tracking.

Image of Peter Diamandis showing the Magic Leap 2 controller

Back in January, we reported that Magic Leap 2 specs — or at least some of them — were being shared at SPIE Photonics West 2022. But it turns out we missed the company also revealing the controller to it. -even uses backward tracking. It wasn’t clear at the time what exactly that meant, but the picture shared this week by entrepreneur Peter Diamandis shows two front-facing cameras.

The controllers of almost all AR and VR systems available today are headset-tracked or rely on external base stations. Meta’s Quest 2, for example, tracks a pattern of infrared LEDs under the plastic ring of its controllers, while Valve’s Index controllers determine their position relative to SteamVR “Lighthouse” base stations placed in the corner of your room.

Relying on the headset for tracking has a flaw: if the controller moves out of sight of the sensors or any part of your body gets in the way, tracking will temporarily pause. This isn’t a problem for many use cases, but limits complex two-handed interactions and scenarios like looking left while shooting right. Using external base stations can alleviate most of these issues, but it increases setup time and severely limits portability – and the path from controllers to base stations can still be obstructed.

Magic Leap 1 and Pico Neo 2 used magnetic tracking. Unlike visible light, the magnetic field can pass through the human body, so occlusion is not a problem. But magnetic tracking isn’t as accurate as optical tracking systems can be, and adds significant weight and cost to hardware.

Controllers with onboard cameras promise to solve the occlusion problem while maintaining high accuracy by tracking each other in much the same way as upside-down headsets – using a type of algorithm called Simultaneous Localization and Mapping ( SLAM). SLAM basically works by comparing acceleration (from an accelerometer) and rotation (from a gyroscope) to how high-contrast elements in your room move relative to the cameras. The initial SLAM algorithms were handcrafted, but most today use at least some machine learning.

Potential disadvantages of this approach are the cost of a chip powerful enough to run the tracking algorithm, reduced battery life due to the power the chip would consume, and the need to have an environment well-lit with high-contrast features such as posters – although this limitation also applies to upside-down helmets. Some have suggested that tracking quality may be reduced in fast motion due to motion blur, but that shouldn’t be more of an issue than tracking fast-moving LEDs – a global shutter sensor with slower dwell time. low exposure should make this a non-issue.

Cambria Project Controllers
An Apparent Leak of Meta’s Project Cambria Controllers

Meta is apparently also considering using controllers with built-in tracking in its upcoming Project Cambria headset. Images of Quest-like controllers with cameras instead of an LED ring first leaked in September, and the rings aren’t featured in the official trailer either.

Magic Leap 2 and Project Cambria are slated for release this year, though neither has a specific release window. They are very different products – ML2 is a transparent AR headset designed for business while Cambria is an opaque headset for virtual and mixed reality – but whichever launches first will be the first AR or VR system to use this new approach. monitoring controllers.