NeurIPS 2026

MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection

Imperceptible audio perturbations that break the audio–visual alignment talking-head models rely on, so a stolen voice can no longer drive a convincing lip-synced deepfake.

Donghyun Kim*¹, Jin Hong*², Seungmin Kim*¹,³, Dain Kim¹, Junseok Kwon†², Daeseon Choi†¹

¹Soongsil University    ²Chung-Ang University    ³AIM Intelligence

* Equal contribution    † Corresponding authors

Overview of MUTE: protected audio produces unsynchronized talking-head output even under black-box models, post-processing, denoising and adaptive attacks
Figure 1. Overview of MUTE. MUTE protects audio before it can be misused for talking-head synthesis. Even when the attacker uses an unseen black-box model or applies post-processing, denoising or adaptive attacks, the generated video stays visibly out of sync with the speech.

Abstract

Talking-head generation models, which synthesize realistic facial animations from audio, are increasingly vulnerable to misuse in multimodal deepfake scenarios. Protecting against them is hard: these models depend on precise audio–visual alignment, yet effective audio protection methods remain largely underexplored.

We propose MUTE, an audio protection framework that uncouples the underlying audio–visual alignment through a multi-level strategy. MUTE combines (1) Alignment-level Disruption, which directly perturbs cross-modal attention to disrupt structural alignment, and (2) Representation-level Degradation, which perturbs temporal audio embeddings to degrade their representations. To improve robustness, perturbations are constrained in the STFT domain to high-energy regions, making them resistant to post-processing, denoising, and adaptive attacks. An optional speaker-level objective further mitigates bypass via TTS-based resynthesis. MUTE consistently degrades lip synchronization across both white-box and black-box talking-head models while preserving perceptual audio quality, remains effective under real-world transformations, and can be combined with image-based approaches for stronger multimodal protection.

Method

Uncoupling audio–visual alignment at two levels

Given clean audio x, MUTE crafts xadv = x + δ with ‖δ‖∞ ≤ ε, optimized with PGD on a surrogate latent-diffusion talking-head model (Hallo). Only the audio is modified; the attacker's portrait is left untouched.

MUTE framework: alignment-level KL loss on cross-attention maps, representation-level frame-wise cosine loss on audio embeddings, and an STFT-domain robustness loss, optimized with PGD
Figure 2. Overall framework of MUTE. The framework uncouples audio–visual alignment through a multi-level strategy, combining alignment-level disruption and representation-level degradation, while an STFT loss keeps the perturbation robust.

Results · Voice protection

Protected audio breaks lip sync

Talking-head videos generated by the surrogate model Hallo from the same portrait, driven either by clean audio (GT) or by audio protected with each method. All videos play in sync; click any video to hear its audio. The perturbations are hard to hear, but MUTE's lip motion visibly drifts away from the speech.

Qualitative comparison: frames and lip crops for GT, C&W, Psychoacoustic, Qin et al., ARO, SAGO, CAF and MUTE
Figure 3. Qualitative results on (a) TalkingHead-1KH and (b) CelebV-HQ. MUTE produces the largest deviation in facial dynamics and lip shape from the ground truth.

Table 1: Comparison with audio adversarial attack baselines (surrogate: Hallo)

TalkingHead-1KHCelebV-HQ
MethodWER↓STOI↑PESQ↑Sync↓M-LMD↑FID↑FVD↑E-FID↑WER↓STOI↑PESQ↑Sync↓M-LMD↑FID↑FVD↑E-FID↑
C&W13.3970.9222.3223.7401.0961.8551.4423.64613.0570.9482.7024.0391.1081.6621.2473.508
Psychoacoustic12.1400.9222.3333.7871.0971.9141.4273.65511.6430.9492.7403.9261.1261.6761.2763.457
Qin et al.15.6490.9252.3014.4680.9831.8251.3293.44313.8700.9422.5444.5431.0541.4791.1823.331
ARO15.6730.9032.5183.5181.1551.9831.5103.66213.8700.9312.6893.8181.1761.5741.2603.457
SAGO12.4630.9302.5153.2641.2002.1761.6674.11511.9700.9562.9123.5971.2261.8941.3993.963
CAF12.1430.9252.2922.1731.3412.9282.2685.43514.6240.9472.6522.5701.3342.7321.7094.734
MUTE11.6810.9362.5221.3321.6273.1832.3635.48910.4670.9592.9402.1521.6932.7561.7905.062
GT0.0001.0004.5005.6560.0000.0000.0000.0000.0001.0004.5005.2940.0000.0000.0000.000

MUTE gives the lowest SyncNet score and the highest M-LMD on both datasets, and at the same time the best audio quality among protection methods (lowest WER, highest STOI and PESQ). Values are means over three seeds; standard deviations are reported in the paper. Bold marks the best protection method; GT is shown for reference.

Results · Black-box transferability

Transfer to unseen talking-head models

Perturbations are crafted once on Hallo and fed directly to four target models the attacker might use instead. Hallo2 shares a similar architecture with the surrogate; Audio2Head, SadTalker and Ditto differ significantly in design.

Table 3: Transferability under black-box settings

Hallo2Audio2HeadSadTalkerDitto
MethodSync↓M-LMD↑Sync↓M-LMD↑Sync↓M-LMD↑Sync↓M-LMD↑
C&W3.7731.1444.2691.2225.5400.6294.0240.697
Psychoacoustic3.7471.1374.2411.1885.5030.6304.0020.697
Qin et al.4.5351.0274.5101.1045.7020.5834.1820.678
ARO3.5461.1924.1821.2035.2970.6583.8030.781
SAGO3.2781.2194.3351.1335.4180.5633.9590.704
CAF2.2801.3734.0051.2465.4250.6423.9940.703
MUTE1.3971.6793.6271.2994.7030.8643.4570.884
GT5.6880.0005.3140.0006.3890.0004.7780.000

MUTE achieves the lowest Sync and highest M-LMD on every target model, showing that disrupting audio–visual alignment transfers across architectures.

Results · Robustness

Post-processing, denoising and adaptive attacks

An attacker may try to clean the protected audio first. We test down/upsampling (16→8→16 kHz) and adaptive spectral filtering, model-based denoising with Demucs and diffusion-based WavePurifier, and gradient-based adaptive attacks (BPDA, EOT) that check for gradient obfuscation.

Table 4: Robustness under post-processing, denoising and adaptive attacks

Down/UpsamplingSpectral FilteringDemucsWavePurifierBPDAEOT
MethodSync↓M-LMD↑Sync↓M-LMD↑Sync↓M-LMD↑Sync↓M-LMD↑Sync↓M-LMD↑Sync↓M-LMD↑
C&W3.7910.9713.8840.9873.9681.0314.0710.9404.8780.5574.8340.542
Psychoacoustic3.8250.9793.8781.0043.9531.0424.0590.9354.8400.5554.8540.564
Qin et al.4.0320.9554.3460.9504.5690.9783.9740.9524.9260.5644.9470.530
ARO3.6071.0693.6471.0743.6121.1324.0590.9854.7780.5554.8290.549
SAGO3.8051.0173.8011.0873.4071.1414.1010.9224.9120.5624.9230.547
CAF3.5331.0893.0631.1782.5661.2593.9400.9914.8100.5634.7940.563
MUTE2.9141.2542.2511.4101.5341.5933.4941.2174.2690.6554.2980.647

MUTE remains the strongest method under every transformation. BPDA and EOT reduce protection for all methods, but MUTE still outperforms every baseline, indicating genuine robustness rather than gradient obfuscation.

Black-box model + purification

The hardest setting combines both: the protected audio is purified and then fed to an unseen talking-head model.

Results · TTS resynthesis

Resisting voice cloning with the speaker-level objective

An attacker aware of the protection could clone the speaker with TTS and synthesize fresh, perturbation-free speech. We use the protected audio as the speaker reference for CosyVoice and XTTS-v2 (white-box) and Chatterbox (black-box), then measure speaker similarity (Sim, ECAPA-TDNN) and the defense success rate (DSR: share of synthesized samples judged a different speaker, cosine similarity ≤ 0.25).

Table 5: Effect of the optional speaker-level objective (λspk)

CosyVoice (white-box)XTTS-v2 (white-box)Chatterbox (black-box)Lip-sync
SettingSim↓DSR↑Sim↓DSR↑Sim↓DSR↑Sync↓M-LMD↑
Unprotected0.4882.530.4692.800.5744.535.6560.000
λspk = 00.36814.670.39214.930.44915.071.3321.627
λspk = 0.010.20067.73−0.10299.330.28043.871.4961.541
λspk = 0.050.18071.47−0.12099.470.24850.401.9181.465
λspk = 0.100.17074.13−0.12099.730.23952.672.2181.415

With λspk = 0.01, MUTE exceeds 43% DSR on all three TTS models while keeping strong lip-sync disruption (Sync 1.496, M-LMD 1.541). Larger λspk raises DSR further at some cost to alignment disruption.

Results · Multimodal protection

Combining with image protection

MUTE only touches the audio, so it can be stacked with image-based protection of the portrait. Below, the portrait is protected with MIS or Silencer and the audio with CAF or MUTE.

Table 2: Image protection combined with voice protection

Image + Voice protectionSync↓M-LMD↑FID↑FVD↑E-FID↑
MIS3.0391.988142.69223.009162.260
Silencer2.1702.168144.55524.029160.298
MIS + CAF1.2122.157145.16624.526163.048
Silencer + CAF0.9412.152146.11125.764165.884
MIS + MUTE0.9022.261147.14125.289162.983
Silencer + MUTE0.7602.233146.54425.843168.829

Pairing MUTE with either image method gives lower Sync and higher M-LMD than pairing it with CAF, reaching a Sync score of 0.760 with Silencer, while keeping comparable visual-quality degradation.

Results · User study

Which video is the most out of sync?

🎯 Try it yourself

Can you spot the most out-of-sync video?

  1. Pick a sample and press ▶ Play all. Clips from the study appear in a shuffled order.
  2. Choose the one whose lips match the speech the least.
  3. Press Reveal methods to see if you picked MUTE.

Study results

Participants watched videos generated from audio protected by MUTE, CAF, ARO and SAGO and picked the one with the most noticeable lip-sync mismatch. With 100 participants and 10 samples (1,000 responses), MUTE was chosen in 78.1% ± 2.56% (95% CI) of cases, against 13.2% for CAF, 5.2% for ARO and 3.5% for SAGO.

User study bar chart: MUTE selected in 78.1% of responses, CAF 13.2%, ARO 5.2%, SAGO 3.5%
Figure 4. Share of responses per question and in total.

Ablation

Ablation study

Effect of the phase ratio ρ (share of iterations spent on the alignment-level objective), the STFT loss weight λ, and the perturbation budget ε. Sync↓ is reported for the alignment columns.

Table 6: Phase ratio ρ

ρTime↓HalloAudio2HeadSadTalker
1.078.43.9203.9585.048
0.220.11.4193.7984.898
0.1 (ours)12.31.3323.6274.703
0.059.21.4703.7994.899
0.04.81.4273.9045.063

ρ = 1 uses only the alignment-level objective, ρ = 0 only the representation-level one. The latter is stronger and cheaper on the surrogate, the former transfers better; ρ = 0.1 balances both.

Table 7: STFT loss weight λ

λCleanDown/UpDemucs
01.4923.1781.816
5 (ours)1.3322.9141.534
101.4292.9331.494
201.4512.9711.564
501.4743.0241.675

Adding ℒstft improves both clean protection and robustness; λ = 5 gives the best balance.

Table 8: Perturbation budget ε

εWER↓STOI↑PESQ↑Sync↓
0.00210.350.9482.9491.898
0.004 (ours)11.680.9362.5221.332
0.00612.250.9072.2981.200
0.00812.890.8912.1081.122
0.01013.300.8851.9451.027

WER in %. A larger budget disrupts lip sync more but lowers audio quality; ε = 0.004 is the operating point.

Citation

BibTeX

@inproceedings{kim2026mute,
  title     = {MUTE: Multi-Level Alignment Uncoupling Against
               Talking-Head Exploitation for Voice Protection},
  author    = {Kim, Donghyun and Hong, Jin and Kim, Seungmin and
               Kim, Dain and Kwon, Junseok and Choi, Daeseon},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}