88Z AI, 1 June — Xiaomi's large model application team has released the open-source model ControlFoley, which uniformly supports three video dubbing tasks: text-guided, text-controlled, and reference audio-controlled. By leveraging visual encoding and modal robust training, the model addresses issues of weak control and synchronization difficulties, achieving state-of-the-art performance among open-source solutions on multiple benchmarks.
At a glance
- ControlFoley uniformly supports text-guided, text-controlled, and reference audio-controlled video dubbing tasks
- Addresses synchronization challenges via visual encoding and decoupling techniques, achieving open-source SOTA on multiple benchmarks
The update
Xiaomi's large model application team has officially open-sourced ControlFoley, a controllable video sound effect generation model. This model innovatively unifies support for three core tasks: text-guided video dubbing, text-controlled video dubbing, and reference audio-controlled video dubbing. To address pain points in existing solutions, such as weak text control capabilities and difficulties in precisely synchronizing reference audio, ControlFoley employs advanced technical methods including joint visual encoding, time-timbre decoupling, and modal robust training. Experimental data shows that the model has achieved the best performance in the open-source domain on multiple authoritative benchmarks like VGGSound-Test, significantly improving the controllability and synchronization accuracy of video sound effect generation, thereby providing strong tool support for relevant developers and researchers.
Sources & editorial note
This article is an AI-assisted synthesis by 88Z AI based on the public sources below. Product claims and availability are attributed to their publishers. It does not represent independent testing or first-hand reporting.
- 小米技术Source published
Source dates above refer to the original announcements; the date under the headline is this article’s publication on 88Z AI.