Mmaudio
English
Start creating

Multimodal audio synthesis

mmaudio turns video into sound

Generate synchronized audio from video or text with a research-first toolkit built for expressive, frame-aware results.

Try a prompt

Describe the atmosphere your video needs.

video → audio

Free to start · no signup · ~10 s

1 Branch 1 Tag
Code
hkchengrex updated the repository
configlast year docs8 months ago mmaudio8 months ago sets2 years ago traininglast year .gitignore2 years ago LICENSElast year README.md8 months ago batch_eval.py2 years ago demo.py2 years ago eval_onsets.pylast year gradio_demo.pylast year pyproject.toml11 months ago train.pylast year

Mmaudio

Paper · Webpage · Models · Demo · Colab · Replicate

Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji

University of Illinois Urbana-Champaign, Sony AI, and Sony Group Corporation

CVPR 2025

Highlight

mmaudio generates synchronized audio from video and text inputs. Its central idea is multimodal joint training across a broad range of audio-visual and audio-text data, paired with a synchronization module that aligns generated sound to the movement and timing of each frame.

Try a prompt above, or explore the examples below:

Installation

We have tested mmaudio on Ubuntu environments with modern CUDA-capable hardware.

Prerequisites

We recommend using a miniforge environment.

  • Python 3.9+
  • PyTorch 2.5.1+ with the corresponding torchvision and torchaudio packages for your CUDA version.

1. Install the prerequisite packages if needed:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --upgrade

Other CUDA versions may also work when supported by your GPU and driver.

2. Clone the repository:

git clone https://github.com/hkchengrex/MMAudio.git

3. Install mmaudio with pip:

cd MMAudio
pip install -e .

The model files are downloaded automatically when the demo script is run. Checksums and model details are included in the repository documentation.

Demo

By default, the scripts use the large_44k_v2 model. In our experiments, inference uses around 6GB of GPU memory in 16-bit mode, which fits many modern GPUs.

Command-line interface

With demo.py:

python demo.py --duration=8 --video=<path to video> --prompt "your prompt"

Audio is written in .flac format and reconstructed video is written in .mp4 format under ./output. Omit --video for text-to-audio synthesis.

Gradio interface

The interface supports video-to-audio and text-to-audio synthesis. Experimental image-to-audio processing is also available by duplicating an image into a video sequence. The default port is 7860.

python gradio_demo.py

FAQ

  1. Video processing. Higher-resolution videos take longer to encode and decode, but do not necessarily improve output quality. The CLIP encoder resizes frames to 384×384 pixels, while Synchformer resizes the shorter edge to 224 pixels and applies a center crop.
  2. Frame rates. CLIP operates at 8 FPS and Synchformer at 25 FPS. Frame-rate conversion happens through the video reader, with duplication used when the input frame rate is lower than required.
  3. Failure cases. As with most generative models, failures can occur and the reasons are not always obvious. Known failure modes are documented with the project materials.
  4. Performance variations. Hardware, video backends, inference precision, batch sizes, compilation settings, and random seeds can produce subtle differences between runs.

Known limitations

  1. The model can sometimes generate unintelligible human speech-like sounds.
  2. The model can sometimes generate background music without explicit musical training.
  3. The model may struggle with unfamiliar concepts and fine-grained action descriptions.

These limitations can be addressed through higher-quality and more diverse training data.

Training

See TRAINING.md for data preparation, distributed training, and configuration details.

Evaluation

See EVAL.md for benchmark setup and onset evaluation scripts.

Training Datasets

mmaudio was trained on several datasets, including AudioSet, Freesound, VGGSound, AudioCaps, and WavCaps. These datasets have their own licenses and usage terms. The pretrained models may not be suitable for every commercial use case.

Update Logs

  • 2025-03-09: Uploaded corrected TSV files.
  • 2025-02-27: Disabled the GradScaler by default to improve training stability.
  • 2024-12-23: Added training and batch evaluation scripts.
  • 2024-12-14: Replaced the frame reader with PyAV and cached frames during reconstruction.
  • 2024-12-13: Improved CLIP and Sync feature extraction with a batch-size multiplier and removed the unused VAE encoder during inference.
  • 2024-12-11: Replaced the framerate reader for more reliable reconstruction across hosted GPU environments.

Citation

@inproceedings{cheng2025taming,
  title={{MMAudio}: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis},
  author={Cheng, Ho Kei and Ishii, Masato and Hayakawa, Akio and Shibuya, Takashi and Schwing, Alexander and Mitsufuji, Yuki},
  booktitle={CVPR},
  year={2025}
}

Relevant Repositories

License

  • The code is released under the MIT license.
  • The checkpoints are released under the CC-BY-NC 4.0 license.

Disclaimer

We have no affiliation with and have no knowledge of the party behind the domain “mmaudio.net”.

Acknowledgement

Many thanks to:

Build with synchronized sound

Use the repository for experiments, inspect the examples for reference, or hand a prompt to the hosted experience when you want to move from an idea to an audio track.

Start creating
Video-to-audio Text-to-audio Frame alignment Open research Video-to-audio Text-to-audio Frame alignment Open research
Start creating
Start creating