Researchers from Microsoft Mixed Reality & AI Lab have introduced a groundbreaking approach called HMD-NeMo (HMD Neural Motion Model) that generates accurate full-body motion in immersive mixed-reality scenarios, even when hands are only partially visible. HMD-NeMo uses a spatiotemporal encoder with novel mask tokens to encourage plausible motion, and it operates in real-time and online. The approach demonstrates superior accuracy and smoothness compared to existing methods, making it a pioneering solution in the field.
A New Approach to Full Body Motion Generation in Mixed-Reality Scenarios
In the world of immersive experiences in mixed-reality scenarios, generating accurate and realistic full-body avatar motion has been a challenge. Existing solutions rely on limited input signals, such as head and hand movements. However, these solutions assume full-hand visibility, which is not always the case in mixed reality experiences.
Researchers from Microsoft Mixed Reality & AI Lab have introduced a groundbreaking approach called HMD-NeMo (HMD Neural Motion Model). This unified neural network generates plausible and accurate full-body motion even when hands are only partially visible. HMD-NeMo operates in real-time and online, making it suitable for dynamic mixed-reality scenarios.
Key Features of HMD-NeMo:
- Utilizes a spatiotemporal encoder with novel temporally adaptable mask tokens (TAMT) to encourage plausible motion in the absence of hand observations.
- Incorporates recurrent neural networks and a transformer to efficiently capture temporal information and model complex relations between input signal components.
- Handles both motion controller and hand tracking scenarios within a unified framework.
- Maintains temporal coherence even when hands are partially or entirely out of the field of view.
The proposed method is trained using a loss function that considers data accuracy, smoothness, and auxiliary tasks for human pose reconstruction. Extensive evaluations on the AMASS dataset, a collection of human motion sequences, demonstrate the superior accuracy and smoothness of HMD-NeMo compared to state-of-the-art approaches.
HMD-NeMo represents a significant advancement in generating full-body avatar motion in mixed-reality scenarios. Its versatility, impressive performance metrics, and ability to handle different scenarios make it a pioneering solution in the field.
If you want to evolve your company with AI and stay competitive, consider using HMD-NeMo. It can redefine your way of work by automating customer interactions and improving sales processes. Connect with us at hello@itinai.com for AI KPI management advice and explore practical AI solutions at itinai.com.
