Apple Has Its Eyes on the Prize

Apple wants to see what you’re seeing. 

The company is seeking to patent tech for “event camera-based gaze tracking” which utilizes neural networks in a “controller,” a.k.a. an AR or VR headset. Here’s how it works: An event camera – or a vision sensor that picks up on “changes in light intensity” – collects data on eye movements, which are then fed to a neural network. 

The neural network then generates a  “gaze characteristic,” which includes the user’s “initial pupil center” and any gaze adjustment from tracking “location-focused input,” or the minuscule movements of just your pupil, not the entire eye.  

Apple notes that this kind of eye tracking works more quickly and efficiently in guiding user interactions, offering “foveated rendering (or better resolution at the point of focus), and cutting down on geometric distortion in rendering 3D objects. It also eats up far less processing power than traditional methods, preventing excessive power consumption and overheating in the user’s headset. 

“Existing gaze tracking systems often include a camera that transmits images of the eyes of the user to a processor that performs the gaze tracking,” Apple said in its filing.  “Transmission of the images at a sufficient frame rate to enable gaze tracking requires a communication link with substantial bandwidth.”

An AR/VR headset has long been rumored to be in the works at Apple. This patent filing may reveal another detail about how this headset will work. 

It also isn’t the first time we’ve seen patents from Apple indicating an interest in eye tracking. The company previously sought to patent tech that tracked eye movement as a means of controlling a “virtual agent” to perform tasks in an augmented or virtual reality environment, which we discussed in Patent Drop a few months ago. Taken together, it seems that eye tracking and control could be a major feature of its buzzy upcoming device. 

Adding advanced features like this could be Apple’s plan to get consumers to shell out despite the steep price tag: The headset will reportedly cost $3,000, Bloomberg’s Mark Gurman reported in late March. The headset also comes with a number of disadvantages, Gurman reported, such as an external battery that needs to be charged every few hours, no companion app and limited media content at launch. It could ship out as soon as the end of this year and is expected to debut at Apple’s Worldwide Developers Conference starting June 5. 

The headset would be Apple’s first step in competing in the artificial reality ring. Microsoft currently offers three versions of the HoloLens headset, ranging from $3,500 to nearly $5,000. Meta, meanwhile, offers several versions of the mixed-reality Meta Quest headset, starting at $1,000. 

With Apple’s core business being consumer electronics, the company’s best shot at competing in the market could be offering state-of-the-art features that appeal to die-hard Apple fans.