Automatic Music Transcription (AMT) turns an audio recording into symbolic notes, usually MIDI. Single‑instrument transcription works well, but transcribing a full mix remains hard. MuScriptor closes that gap. It is an open‑weight decoder‑only Transformer trained on real multi‑instrument recordings across many genres.
The model works in three stages. First it pre‑trains on about 1.45 million synthetic MIDI files, augmenting pitch, tempo, velocity and instrument choice with many soundfonts. Second it fine‑tunes on an internal set of 170 000 real recordings (over 11 000 hours) with note alignments obtained by audio‑symbolic synchronization and filtered by warping distance. Third it runs a GRPO‑style reinforcement learning step on 300 manually verified tracks, rewarding onset, frame and offset F‑scores.
On the held‑out test set (372 tracks) the large (~1.3 B) model achieves Onset F1 60.4, Frame F1 73.3, Offset F1 49.0 and Multi F1 48.2, far above the YourMT3+ baseline (Onset 32.5, Frame 45.5, Offset 17.8, Multi 21.9). Each stage adds roughly 20 points, showing that real data and RL post‑training are crucial.
Getting started is simple. Install the package, load the default medium model, and stream note events or write a MIDI file:
pip install muscriptor
from pathlib import Path
from muscriptor import TranscriptionModel
model = TranscriptionModel.load_model()
for event in model.transcribe(‘audio.wav’, instruments=[‘acoustic_piano’,’drums’]):
print(event)
Path(‘out.mid’).write_bytes(model.transcribe_to_midi(‘audio.wav’))
Use cases include extracting a bassline for re‑voicing, converting historic recordings into editable scores, feeding transcriptions to chord or key analysis, building practice tools with live piano rolls, and conditioning on specific instruments such as drums.
Strengths: training on 170 k real recordings, three size variants with MIT‑licensed code, strong Multi F1, instrument conditioning, streaming API and browser UI. Weaknesses: CC BY‑NC 4.0 weights restrict commercial use, tokenizer drops velocity and cannot repeat same‑pitch same‑instrument notes, lower onset/offset on chorals, large model needs a GPU, 5‑second segment limits long‑range context.
#AI #Product #MusicAI #AMT #OpenSource #DeepLearning