Currently, creating a lifelike avatar entails capturing large quantities of high-quality audio and video footage of an individual, then adding existing functionality to make the VR environment believable.
However, researchers have developed a system that allows VR users to represent themselves with lifelike avatars, precisely animated in real time, which can interact with other users. With this system (called Codec Avatars), researchers aim to deliver the most engaging experience possible for VR users, building the future of connection within virtual reality, and eventually, augmented reality.
Up to now, photo-realistic avatars rendered in real-time have been achieved and used frequently in computer animation: actors are equipped with optimally placed sensors that capture geometric details of their faces and expressions computationally. However, this technology is not compatible with existing VR headset designs or platforms which typically obstruct different parts of the face making facial capture technology problematic.
By precisely animating photorealistic avatars from cameras mounted on a VR headset, the researchers have configured a headset with minimum sensors for facial capture, and enabled two-way, authentic social interaction in VR – a process at the forefront of computer graphics and interactive techniques.
The system works by tracking users’ facial expressions using a minimum set of headset-mounted cameras (HMCs). The headset prototype has cameras on the regular tracking headset for real-time animation plus cameras positioned for ideal face-tracking. It uses an artificial intelligence technique that translates HMC infrared images to images that look like a rendered avatar but with the same facial expression of the person.
By precisely mapping between the images from tracking headset and the status of the 3-D avatar through differentiable rendering, the neural network is trained to predict face parameters from a minimal set of camera images in real time, finding high-quality mapping for facial expressions
The researchers also aim to create and animate full bodies for expressing more complete social signals – as well as facial expressions – and though this technology still a long way from appearing in consumer headsets, it’s just a matter of time.