Expressive Guitar Synthesis via Controllable Latent Flow Matching

Code Checkpoints

Two-stage architecture: a performance-control predictor (left) turns per-string score curves into predicted pitch/envelope curves via an axial transformer encoder, and a latent flow-matching decoder (right) renders those curves into six-string audio through an axial DiT, an ODE solver, and a neural audio codec decoder.

Abstract

Realistic guitar synthesis requires both audio fidelity and interpretable control over how notes are performed. The input timed MIDI events decide what is played but not how, leaving the continuous within-note decisions of guitar performance—such as pitch inflection and per-string articulation—unspecified. A model that directly maps MIDI to audio must therefore realize the performance implicitly, and users are left with limited control over the synthesized performance. We therefore propose a two-stage guitar synthesis pipeline that predicts editable, per-string f0 and amplitude trajectories from timed note events and note-level dynamics, and renders them into audio using latent flow matching. Experimental results show that the proposed system outperforms a matched single-stage counterpart and DDSP-Guitar in control fidelity and perceived quality, while keeping the controls editable.

1. Full Mix Study

Select a sample and compare the audio of each model output. Click between the model pills to A/B the same sample rendered by each system while the MIDI roll and query stay fixed — listen for overall audio quality and naturalness.

Query FluidSynth render of the score MIDI — a content hint, not a rated system.
Model output
2. Single String Study

Please select a fixed target audio and fixed target string, compare the information between the different model. Pick a model and a string, then play that string's isolated audio while comparing it against the re-extracted pitch/envelope curves below — click a legend entry to isolate one system's line if the overlay gets busy.

Model

String

String audio This string alone, from the selected model.
Audio Feature Re-Exacted from Prediction

Gray background: where a note is actually sounding.

Pitch (Hz)
Envelope (log-RMS)
Predictor input
Synthesizer input
3. Velocity Prediction

A note-gain predictor estimates per-note loudness from the score alone (no real recording needed); this feeds the same pitch/envelope predictor and synth as the 2-stage system above, making a full 3-stage pipeline. Select a sample, compare its mix audio against the 2-stage (ground-truth velocity) and ground-truth arms, and read the predicted vs. ground-truth velocity curve for every string. Use the String pill or a legend entry to focus the plot on one string, then compare the solid (predicted), dotted (re-extracted from the rendered audio), and faint dashed (ground truth) lines to judge whether the synth actually reached the loudness it was asked for.

Query FluidSynth render of the score MIDI — a content hint, not a rated system.
Model output

String

String audio This string alone, from the selected model.

Velocity comparison (all 6 strings)

4. Unseen Long Audio Inference

Four real-length recordings (63-185s), rendered end-to-end by the full 3-stage pipeline from MusicXML scores in the GAPS dataset — unseen by any part of the system during training, played back whole, not cropped to a short window. There is no alternate system or ground truth to A/B here — just listen for coherence over the full length, especially near the chunk boundaries described below.

Long Sample Inference Detail