Audio-Visual Highlight · ICLR 2027 submission · training record

Highlighting one object in sound and picture with a noisy bridge

Given a clip and a mask naming one object, the model must return the same clip with that object’s sound amplified and its pixels highlighted. Source and target latents are paired, so training interpolates between them along a Schrödinger bridge. This page records the objective, the hyper-parameters and the training curves of the run sigma1_sept, whose one change over the previous run is the bridge noise scale.

run avedit_dit_sb_pe_film_cross_attn_new_1452_data0902clean3_sigma1_sept hardware 4×H100 data 34,172 clips training in progress

The data and what is highlighted

Every clip carries exactly one prompted object, named in free text and localised by a mask. The three cleaned splits — train 31,470, val 1,733, test 969 — are pooled below; cleaning removed clips whose target audio stem was silent, whose highlight was a no-op, or that contained more than two instances of the object.

Three panels: bar chart of object count per clip, donut chart of object type, horizontal bar chart of the twenty most frequent prompts. — Left: how many instances of the prompted object a detector finds per clip. Middle: the 2,053 distinct free-text prompts grouped into semantic classes by keyword rules (0.7 % unmapped). Right: the twenty most frequent prompts, coloured by the same classes. Counts come from GroundingDINO, not human labels.

Instances per clip

object_count	clips	share

Per split: train 7 / 80 / 13 %, val 6 / 82 / 12 %, test 6 / 75 / 18 % for counts 0 / 1 / 2.

Object type

class	clips	share

The bridge, and why its noise scale is the whole story

Let x₀ be the source latents and x₁ the target latents of the same clip — the pair is given, so there is nothing to sample from a prior. For each modality m ∈ {a, v} the interpolant is a Brownian bridge pinned at both ends:

with the two times drawn independently and both noise scales set to one:

The network predicts the drift of that bridge. Writing μₜ = (1−t)x₀ + t x₁ for the noise-free midpoint, the regression target is

where c is the per-frame Perception-Encoder embedding of the masked video, entering through adaLN-Zero and cross-attention. Sampling integrates dx/dt = fθ from x(t₀) = x₀ with Euler steps.

Loss weighting

A single constant, unchanged from the previous run. No per-t weighting, no EMA normalisation between the two terms, no extra weight inside the mask region.

Why σ = 1 rather than 0.01

At σ = 0.01 the bridge is essentially the straight line x₀ + t(x₁−x₀), and since x₀ is also an input, the quantity (xₜ−x₀)/t hands the model the answer. Its noise variance is

which for σ = 0.01 is negligible beyond t ≈ 0.05: the model learned that division instead of the conditional edit, and its loss at t ≥ 0.05 was twenty times lower than at t = 0 — which is exactly where sampling starts. At σ = 1 the same variance exceeds the variance of the edit itself for t < 0.4, so the shortcut is closed and the conditioning has to be read. The clipped range [0.02, 0.98] is required because the factor (1−2t) / 2t(1−t) on the noise term is unbounded at the endpoints once σ is not tiny; sampling therefore also starts at t₀ = 0.02, inside the trained range.

Architecture and optimisation

Model — 1.2 B trainable, from scratch

backbone: U-DiT · 6 in / 1 mid / 6 out blocks, skip fusion Linear(2h→h)
width: hidden 1792 · 16 heads · MLP ×4
block: self-attn → cross-attn(c) → MLP, adaLN-Zero (9 modulations)
time: c_adaLN = emb(t_a) + emb(t_v) + PE tokens
tokens: 345 (audio latent length)
audio latent: 64 × 345 (stable-audio VAE, 44.1 kHz × 8 s)
video latent: 16 × 48 × 32 × 32 (CV-VAE SD3, 256² × 192 f × 24 fps)
video path: per frame PatchEmbed(4) → 64 patches → attentive-stat pooling → 1 vector, 48→345 resampled
fusion: channel concat (64+1024) → ConvNeXtV2 → Linear 1792
condition: 192 × 1792 Perception-Encoder tokens
frozen: 245 M — both VAEs; PE features precomputed
mask latent: loaded, not fed to the network

Optimisation

optimiser: AdamW · lr 1e-4 · β (0.9, 0.999) · ε 1e-8
weight decay: 1e-4
warm-up: linear 1e-5 → 1e-4 over 5,000 steps (≈ 80 epochs)
after warm-up: cosine with T_max from epochs=100000 → effectively constant
batch: 4/GPU × 4 GPU × accum 32 = 512 clips / step
pace: ≈ 61 steps per epoch · ≈ 4 min per epoch
precision: bf16 autocast · grad-norm clip 1.0
seed: 2025 (validation re-seeded per batch)
checkpoints: best(val_total), last, every 4,000 steps
sampling: Euler · t₀ = 0.02 · 10 steps

Changed from the previous run (0902): σ 0.01 → 1.0 · one shared t → independent (t_a, t_v) with a second timestep embedder · t ∈ [0,1] → [0.02, 0.98]. Data, architecture, learning rate and λ_v are identical, so this is an objective-only A/B.

Training curves

Pulled from Weights & Biases at epoch 102 (step 6,385). Loss values are not comparable across the two runs: the σ = 1 target carries a noise term the σ = 0.01 target does not.

Validation loss per epoch

Audio still falling monotonically; video flat from epoch 2 on.

val audioval videoval total = audio + 0.5·video

Train versus validation

The two coincide — with the shortcut gone there is nothing to memorise.

train audioval audiotrain videoval video

Learning rate and the evaluated snapshot

Epoch 89 sits fewer than 600 steps past the end of warm-up.

lr · AdamW

Reference: the σ = 0.01 run

Audio loss falls to 0.065 — the shape of a model learning the division, while its FVD degraded from 547 to 666.

val audioval video

val audio 2.69 → 0.98 over 102 epochs and still descending — the main reason to keep training.
val video is flat at 1.35, yet FVD on the held-out subset improved from 547 to 317 over the same period. The video term is dominated by the irreducible σ√(t(1−t)) noise, so it says nothing about edit quality and must not be used for checkpoint selection.
Training was interrupted at epoch 86 when the shared filesystem filled up mid-checkpoint, and resumed from last.ckpt.

Where it stands

All rows below are the same 84 held-out clips, decoded to pixels and waveform. Pass-through means returning the input unchanged — the floor any real edit must clear, and the reason raw PSNR alone is misleading for this task.

checkpoint	steps	audio PSNR ↑	LSD ↓	edit size vs GT	video PSNR ↑	SSIM ↑	LPIPS ↓	FVD ↓
pass-through (input as output)	–	21.20	9.77	0	16.87	0.600	0.302	361.6
σ = 0.01, best-573	36k	19.68	10.19	1.09	17.89	0.616	0.276	547.3
σ = 0.01, earlier data (46k clips)	9.9k	21.61	11.23	0.95	18.31	0.623	0.266	333.9
σ = 1, epoch 89 · t₀=0, 20 steps	5.6k	21.14	10.06	0.97	18.11	0.619	0.271	324.9
σ = 1, epoch 89 · t₀=0.02, 10 steps	5.6k	21.31	9.78	0.93	18.15	0.621	0.270	317.2

“Edit size vs GT” is ‖gen−src‖ / ‖tgt−src‖ on the waveform: 1 means the model moved the audio exactly as far as the ground truth did. The σ = 0.01 run overshot (1.09) while scoring worse; the new run is on scale and, at one sixth of the optimiser steps, is the first to beat the pass-through floor on audio PSNR and LSD while posting the best FVD of the series.

Rendered samples for this checkpoint — input, condition, target, generation and the previous run side by side — are at WitneyWW/av-highlight-sigma1-sept.