VR Headset See-through Tech to Help in MR Development

• VR headsets’ transparent technology can enable an early MR experience and also help develop the ecosystem needed for MR applications.
• Depth perception is essential in MR applications to demonstrate the occlusion phenomena. The current quality of the VR VST is barely adequate as it is achieved by cameras. However, this could be improved by including an additional ToF sensor.
• Going forward, there will be significantly more VR headsets with VST, promoting an increase in the number of MR applications.

Virtual reality (VR) headsets use real-time six degrees of freedom (DOF) and visual simultaneous localization and mapping (SLAM) to receive real-world information, such as 3D coordinates and object recognition. Initially, the technology was employed for the hand tracking and Guardian system. Nevertheless, some VR headsets include images from monochrome cameras and, unexpectedly, extend the capabilities of video look-through (VST) to mixed reality (MR). Meta launched the Quest Pro VR headset, while ByteDance has the PICO4 with full-color cameras to offer MR applications that are more realistic.

The transparent technology used in current VR headsets is called VST, while the transparent technology used in augmented reality (AR) devices is called optical see-through (OST). The VST technology mixes the digitized three-dimensional view of the real world with a computer-generated simulation of reality, which is then displayed on the opaque screen. By using VST together with the appropriate rendering technique and algorithm, it is possible to enable interaction and occlusion between virtual and real objects, as well as to reconstruct a digital three-dimensional model of the physical world.

Unlike VR headsets, which use cameras to achieve the see-through feature, AR devices are manufactured with the see-through capability. The OST function of AR devices enables virtual images to be directly superimposed over real vision. The resolution of real images in AR devices is significantly higher than in VR headsets. However, current AR devices allow rather limited interaction with the real environment as occlusion cannot be achieved in the absence of depth sensors.

Advantages of VR VST

The VR VST technology allows users to interact with their environment without taking off the VR headset through a physical keyboard, mouse, touchpad or controller. It is even capable of creating expensive gadgets in the virtual world, such as multiple screens. In the future, more experimental AR and MR features are likely to be integrated into VR headsets, greatly enriching the ecosystem and usability of such devices.

Current challenges facing VR VST

• Both stereovision and depth perception are crucial for MR. In the future, VR headsets equipped with a time-of-flight (ToF) sensor will be able to capture more accurate distance information than the camera-equipped ones currently available.
• The VST of a VR headset barely achieves binocular vision. One of the reasons is that there is only one camera for VR VST, which makes it difficult to measure the distance of a nearby object. In addition, there will still be distortion and distortion of field of view objects. Future algorithms must improve this, or designers must avoid users perceiving items that are closer. In addition, more color and white balance sensors will be required to make more realistic approximations of the actual color.
• In a new application of VR, VST technology can be used to bring various exhibits into the virtual world by creating some virtual exhibits. However, the current resolution of virtual screens in VR headsets is less than that in real screens. Nevertheless, this problem can be solved by increasing the resolution of the screen on the VR headset, although it will cost more.
• Latency is a crucial feature of the VST VR headset because processors must encode the image captured by the camera. Therefore, the speed and bandwidth of processors will affect the performance of VR systems, as future mid-range and high-end VR headsets will generally support 4K video streams. Therefore, the VR headset SoC’s requirement should be higher than that of smartphones.

Closure

The VST technology in VR headsets can blur the lines between VR and AR, making it possible for VR to provide MR features. MR can be used for numerous purposes apart from just gaming. It can be used in learning platforms, for athletic purposes, webinars and digital twins in the manufacturing industry. Although VR VST still has some issues that need to be fixed, it can lower the entry barrier for MR applications. Increasing adoption by users and developers can help improve the VR headset VST technology, which will lead to the creation of more MR applications. In turn, these MR applications will drive the development of AR hardware and ecosystems. After all, AR devices have certain advantages that cannot be replicated by VR devices.

Related posts