ReSyncer: Unified Audio-Visually Synced Facial Performer
ReSyncer: Unified framework for audio-visual facial sync. High-fidelity lip-syncing, face-swapping, and style transfer for virtual performers. Innovative 3D facial dynamics.
Frequently Asked Questions
Have questions about ReSyncer? We've got answers. Browse through our FAQs to learn more about ReSyncer
ReSyncer is a unified framework designed to synchronize generalized audio-visual facial information effectively.
The ReSyncer framework consists of two stages that are simply designed and easy to implement.
The key design feature of ReSyncer is revisiting and rewiring the Style-based generator to efficiently adopt 3D facial dynamics predicted by a principled style-injected Transformer.
The contributions of ReSyncer include powerful lip-syncing capabilities, support for various intriguing properties in virtual performer creation, and high-fidelity face-swapping.
ReSyncer goes beyond existing solutions by uniformly addressing a broader range of needs in virtual performer creation and supporting personalized fine-tuning and identity swapping.
ReSyncer allows for simultaneous satisfaction of high-fidelity face-swapping and lip-syncing, with superior temporal consistency and preservation of rich facial details.
ReSyncer predicts 3D facial dynamics from audio input and overlays the predicted mesh onto the target frame to provide strong spatial guidance for accurate synchronization.
The training data for generalized lip-sync in ReSyncer consists of video clips with accompanying audio clip features, aiming to recover the original frame given the audio clip and a reference frame.
ReSyncer outperforms existing methods in lip-sync quality, video generation quality, and video realness, showcasing substantial improvements in visual quality and realism.
A user study with 30 participants showed that ReSyncer outperformed four counterparts in lip-sync quality, video generation quality, and video realness, highlighting the power and superiority of the ReSyncer framework.