All Eyes On Apple Vision Pro and Virtual Reality Headsets

Apple’s newest product addition, Vision Pro, a mixed reality headset that utilizes an eye-tracking system, is making waves in the technology community.

The new device, “seamlessly blends digital content with the physical world,” the tech giant says of its new high-tech goggles that are on track to release to the public next year.

“The breakthrough design of Vision Pro features an ultra-high-resolution display system that packs 23 million pixels across two displays, and custom Apple silicon in a unique dual-chip design to ensure every experience feels like it’s taking place in front of the user’s eyes in real time,” Apple says.

The product may mark the beginning of an era of “spatial computing,” but what does the new frontier of augmented reality mean for users and their vision?

University of Colorado Department of Ophthalmology professor and Slater Family Endowed Chair Malik Kahook, MD, who studies the intersection of artificial intelligence and vision, explains how the goggles use the eyes to execute commands, how users can protect their eyes, and what this technology, paired with other artificial intelligence methods, could mean for the future of medicine, research, and patient care at the Sue Anschutz-Rodgers Eye Center.

MicrosoftTeams-image (6)

This is what Apple Vision Pro will look like to users of the virtual reality headset, which is set to release in 2024. Vision Pro photos courtesy of Apple.

How does this eye-tracking technology work? What components of the eye are at play in a system like this?

Eye-tracking technology works by continuously measuring the position and movement of the eye. This is done using a variety of methods, including:

Pupil tracking: This method uses a camera to track the movement of the pupil. The pupil changes size in response to light as well as changing focus from far to near, so by tracking its movement, the eye tracker can determine where the user is looking.

Cornea reflection tracking: This method uses a camera to track the reflection of a light source off the cornea, which is the clear front part of the eye. The position of the reflection changes depending on where the user is looking, so by tracking the reflection, the eye tracker can determine the user's gaze direction.

Eyelid tracking: This method uses a camera to track the movement of the eyelids. The eyelids close when the user blinks, so by tracking the movement of the eyelids, the eye tracker can determine when the user is looking at something and when they are not.

Mixed reality headsets typically use a combination of these methods to track the user's eye movements. This information can then be used to control the user interface, provide feedback, or simply track the user's gaze for research purposes. Eye-tracking technology is still in its early stages of development, but it has the potential to revolutionize the way we interact with computers and other devices.

Apple promises jaw-dropping, lifelike clarity with this newest product. How healthy do the eyes have to be to make the most out of a product like Apple Vision Pro?

In general, people with healthy eyes can use mixed reality headsets without any problems. However, there are a few conditions that may make it difficult or uncomfortable to use a mixed reality headset. These conditions include:

Refractive errors: Refractive errors, such as nearsightedness, farsightedness, and astigmatism, can make it difficult to see the virtual world clearly. If you have refractive errors, you may need to wear corrective lenses while using a mixed reality headset. Wearing glasses at the same time as a headset is difficult in many cases – similar to wearing glasses under ski goggles. For this reason, contact lenses might make it easier to correct refractive errors and enjoy the proper fit for mixed reality headsets as they are designed today.

Eye strain: Eye strain can be caused by prolonged use of any digital device, including a mixed reality headset. Symptoms of eye strain include headaches, blurred vision, and dry eyes. To prevent eye strain, take breaks every 20 to 30 minutes when using a mixed reality headset. Focusing on any screen also leads to less blinking and thus drying of the eye. Take time to blink even when intensely focused on objects and activities. One way to deal with eyestrain and symptoms of dryness after prolonged use of digital screens is to use artificial tears. These can be used before and/or after using mixed reality headsets. Preservative free options are available and can be obtained over the counter at most pharmacies and other retail stores.

Motion sickness: Motion sickness can be caused by the mismatch between what you see and what your body feels. This can happen when using a mixed reality headset, especially if you are playing a game that involves movement. If you are prone to motion sickness, you may want to avoid using a mixed reality headset or ease into the use by building up time in the headset gradually.

Apple Vision Pro sits on the face, almost like a pair of goggles. As technology gradually moves closer to our eyes, how can users be mindful about their eyes and vision?

There are some tips for using a mixed reality headset in a way that is safe for your eyes. First, make sure that the headset is adjusted properly so that it is comfortable and the lenses are not too close to your eyes. Remember to take breaks every 20 to 30 minutes when using a mixed reality headset. This will help to prevent eye strain.

Additionally, avoid using the headset in bright light, which can make it difficult to see the virtual world clearly. If you are using the headset in a bright environment, try to find a darker space or dim the lights.

Wear corrective lenses if needed. If you have refractive errors, you may need to wear corrective lenses while using a mixed reality headset.

See a doctor if you have any concerns. If you have any concerns about your eye health and using a mixed reality headset, you should talk to your doctor.

Do you see the technology that Apple has developed here, or technology similar to it, being useful in the medical community or translating into use in eye clinics?

Mixed Reality (MR), a combination of Augmented Reality (AR) and Virtual Reality (VR), is an emerging technology that has the potential to revolutionize the medical field and perhaps particularly influence patient care in ophthalmic practices. MR can be used to:

Improve patient education: AR and VR can be used to create interactive educational experiences that help patients learn about their eye health and the treatments available to them. For example, AR and VR can be used to show patients more details about recommended treatments and/or surgery, how to use eye drops, or how to care for contact lenses.

Provide preoperative planning: AR and VR can be used to create 3D models of the eye that can be used to plan surgeries. This can help surgeons better understand the anatomy of the eye and plan the most precise and effective surgery possible.

Provide intraoperative guidance: AR and VR can be used to provide surgeons with real-time feedback during surgery. This can help surgeons make more accurate incisions and avoid damaging delicate structures in the eye.

Provide psychological support: AR and VR can be used to provide psychological support to patients with eye conditions. For example, AR and VR can be used to create virtual environments that can help patients relax and cope with stress.

AR and VR are still in their early stages of development, but they have the potential to revolutionize the field of ophthalmology and ophthalmic clinical care. As the technology continues to develop, we can expect to see even more innovative and effective ways to use AR and VR to improve patient care.