Play it once. Keep it forever.

SongScribe is a browser studio backed by a FastAPI processing service. It captures a chosen melody instrument or accompaniment instrument, aligns takes to one musical clock, and turns them into a synchronized piano roll and lightweight notation preview.

Melody choices are Voice, Guitar, or no melody. Accompaniment choices are Guitar, Piano, or no accompaniment. The listener can write either the melody line or accompaniment line. MIDI piano data remains authoritative; microphone pitch and take refinement are processed by the backend.

Requirements: Node.js 22.13 or newer and Python 3.12.

npm install
.venv/bin/pip install -r requirements.txt

Start the processing API in one terminal:

.venv/bin/uvicorn backend.main:app --reload

Then start the frontend in another terminal:

npm run dev

Open the local URL printed by the development server (normally http://localhost:3000). The frontend uses http://localhost:8000 by default. Set NEXT_PUBLIC_PROCESSING_API_URL when the API is hosted elsewhere.

Quality checks:

npm test
npm run typecheck
npm run lint
npm run build
.venv/bin/python -m pytest

1. Open SongScribe in desktop Chrome or Edge and choose Enter the studio. This opens /studio directly.
2. Open the gear button when you need to pick a melody instrument or accompaniment instrument. The default is Voice + Guitar.
3. Toggle Listen for between Melody and Accompaniment.
4. Connect or select a microphone for Voice or Acoustic Guitar and click Enable microphone.
5. Place the microphone close to the intended source. When capturing voice and guitar together, balance them so neither overwhelms the other.
6. Sing or pick a steady note and confirm the correct melody/accompaniment preview responds.
7. If MIDI Piano is selected, connect it over USB MIDI, enable Web MIDI, select the device, and press a key to test.
8. Set tempo, time signature, and count-in, return to the studio, and press Record.

Microphone access is required for Voice and Acoustic Guitar. MIDI permission is requested only when MIDI Piano is selected. If access is denied, use the permission icon in the address bar and retry. SongScribe streams short, filtered PCM analysis frames to the configured processing service; it does not retain or persist the original microphone recording.

• Recommended: current desktop Google Chrome or Microsoft Edge.
• Web MIDI is not consistently available in Safari or Firefox.
• localhost is considered a secure context for development. A remotely hosted build must use HTTPS for microphone access.

SongScribe is a React + TypeScript frontend connected to a Python FastAPI signal-processing backend. It has no account or database dependency.

• lib/music.ts: frontend note types plus display, playback, routing, and musical-clock helpers.
• lib/processing-api.ts: typed HTTP and WebSocket client for backend pitch and performance processing.
• app/page.tsx: browser device orchestration, one recording clock, metronome scheduling, raw capture, playback, and screen flow.
• backend/services/performance_processing.py: YIN pitch estimation, lane de-duplication, MIDI pairing/CC64 sustain, pitch segmentation and quantization.
• backend/api/routes.py: finite-take processing endpoint.
• backend/api/websocket.py: real-time pitch and chord WebSocket endpoints.
• components/PianoRoll.tsx: shared live/review timeline for melody and accompaniment tracks.
• components/SimpleScore.tsx: deliberately lightweight notation preview. The piano roll remains the timing source of truth.

performance.now() is the master monotonic clock. At the end of the count-in, SongScribe stores one recording start timestamp. Every MIDI event and accepted backend pitch estimate is immediately converted to eventTime - recordingStartTime in milliseconds. MIDI timestamps and microphone pitch frames therefore share the same zero point even though they arrive through different APIs.

The audible metronome is scheduled ahead with the Web Audio clock in short windows, while the visual beat and playhead are calculated from the same monotonic timeline. Raw start, end, pitch, frequency, and confidence values are retained; quantization writes separate refined fields instead of destroying the performance.

• MIDI note-on/off events retain pitch, note name, velocity, channel, timing, and sustain state.
• CC64 holds pending note-offs until the pedal is released.
• The browser sends 2048-sample, filtered stereo analysis frames to /ws/pitches about every 42 ms; backend YIN analysis returns accepted pitch candidates.
• Acoustic-guitar accompaniment also streams mono PCM from the same microphone graph to /ws/chords. Stable chord events become timed regions whose complete tones are voiced in guitar register and quantized together.
• The Listen for toggle activates either the high-pass-shaped melody channel or the low-pass-shaped accompaniment channel.
• RMS gating rejects quiet frames; periodicity confidence rejects uncertain frames.
• A short median window smooths pitch jitter.
• Adjacent frames with the same rounded pitch are merged, and fragments under 110 ms are removed.
• The optional MIDI bleed confidence guard lowers microphone confidence near an active MIDI pitch unless acoustic evidence is very strong.
• Every refined note is quantized to the fixed eighth-note grid; raw timing is preserved separately.

1. Before judging, use Chrome/Edge, select Voice + Acoustic Guitar, choose the line you want to capture, and grant microphone permission.
2. Keep the microphone near the singer/guitar and avoid loud room speakers.
3. On Device Check, sing and pick one steady note at a time. Confirm the gold melody and blue accompaniment previews respond.
4. Preview the metronome for two beats, stop preview, and set 112 BPM / 4-4 / one-measure count-in. Note timing uses the fixed eighth-note grid.
5. Open the studio, press Record, wait through the four-beat count-in, then play a clear picked/arpeggiated pattern and sing for 10–15 seconds.
6. Stop, let refinement finish, point out the gold melody and blue accompaniment tracks, then press Play.
7. Raise or lower one detected note and delete an intentionally bad note to show editing.

• Melody microphone detection remains monophonic. Voice works best on sustained, clean singing; guitar accompaniment uses the polyphonic chord stream when selected.
• A single shared microphone cannot reliably separate overlapping voice and guitar fundamentals in the same pitch range. Both mode uses filters and continuity, not source-separation AI.
• Chord onset display compensates for the recognizer's rolling analysis window, but very fast chord changes can still appear late or be suppressed by stability/minimum-duration filtering.
• Microphone audio is analyzed but not recorded. Review playback uses distinct synthesized tones, not the original microphone audio.
• The notation panel is a readable lead-sheet preview rather than production engraving. It does not yet split piano into grand-staff voices or show complete rests, beams, ties, and accidentals.
• Quantization is grid-based and does not infer swing, tuplets, key, or meter changes.
• The MIDI bleed guard is experimental. Headphones remain the reliable solution when using MIDI Piano with a microphone line.

A Python 3.12 FastAPI service that powers SongScribe performance processing and recognizes common guitar chords from uploaded recordings, JSON sample buffers, or a near-real-time microphone stream. The included microphone client records locally with sounddevice, sends 16-bit PCM over a WebSocket, and prints stable detections in the terminal.

The initial recognizer uses signal processing rather than a trained model. It supports all 12 roots for major, minor, dominant 7, major 7, minor 7, sus2, sus4, diminished, and augmented chords. Major and minor triads are intentionally preferred when a seventh is not clearly audible.

Microphone client -- PCM s16le --> /ws/chords -- rolling buffer --+
Uploaded file -------------------> /analyze -----------------------+--> TemplateChordRecognizer
JSON float samples --------------> /analyze-buffer ---------------+         |
                                                                            v
  downmix -> resample -> high-pass/noise gate -> harmonic signal -> CQT chroma
       -> penalized template scoring -> confidence gate -> temporal smoothing

Key modules:

• backend/services/audio_processing.py handles decoding, stereo downmixing, polyphase resampling, safe conditioning, harmonic/percussive separation, CQT chroma, and silence/noise measurements.
• backend/services/chord_templates.py builds 108 weighted pitch-class templates. Internally, sharps are used consistently: C, C#, D, ... B.
• backend/services/chord_detector.py defines the ChordRecognizer protocol and implements TemplateChordRecognizer.
• backend/services/smoothing.py applies rolling confidence-weighted votes, minimum consistent frames, and silence/unknown hysteresis.
• backend/api/ keeps HTTP and per-connection WebSocket state separate from the signal-processing implementation.
• client/ contains the microphone and file command-line clients.

Python 3.12 is required. From this directory:

python3.12 -m venv .venv

macOS/Linux:

source .venv/bin/activate

Windows PowerShell:

.venv\Scripts\Activate.ps1

Install dependencies:

python -m pip install --upgrade pip
pip install -r requirements.txt

The API and file client do not need microphone hardware. The microphone client uses PortAudio through sounddevice.

• macOS: brew install portaudio
• Ubuntu/Debian: sudo apt update && sudo apt install libportaudio2 portaudio19-dev
• Fedora: sudo dnf install portaudio portaudio-devel
• Windows: the sounddevice wheel normally includes what it needs. Check Windows microphone privacy settings if no input devices appear.

soundfile normally handles WAV, FLAC, and many MP3 files. M4A and codecs unsupported by the local libsndfile build use FFmpeg when it is installed:

• macOS: brew install ffmpeg
• Ubuntu/Debian: sudo apt install ffmpeg
• Windows: install FFmpeg and add its bin directory to PATH.

FFmpeg is optional unless the uploaded codec requires it.

uvicorn backend.main:app --reload

Health check:

curl http://localhost:8000/health

Interactive OpenAPI/Swagger documentation is at http://localhost:8000/docs.

Start the server in one terminal, then in a second activated terminal run:

python -m client.microphone_client

List audio devices:

python -m client.microphone_client --list-devices

Select a device and customize streaming:

python -m client.microphone_client \
  --url ws://localhost:8000/ws/chords \
  --device 2 \
  --sample-rate 22050 \
  --chunk-duration 0.25

The client prints only stable chord changes, not every analysis hop:

Listening... Press Ctrl+C to stop.
Chord: E minor      Confidence: 0.84
Chord: C major      Confidence: 0.79
Silence

Place the microphone near the guitar, play cleanly, and let a chord ring for roughly one second. Input peaks around -18 to -6 dBFS work well. Ctrl+C closes the audio stream and client cleanly.

Connect to ws://localhost:8000/ws/chords and first send a JSON text message:

{
  "type": "start",
  "sample_rate": 22050,
  "channels": 1,
  "sample_width": 2,
  "byte_order": "little"
}

Then send binary chunks with this exact format:

• signed 16-bit little-endian PCM (pcm_s16le)
• mono, one interleaved channel
• declared sample rate from 8,000 through 96,000 Hz
• 0.10–0.25 second chunks recommended; chunks over two seconds are rejected

If a client sends binary immediately without metadata, the defaults are 22,050 Hz, mono, signed 16-bit little-endian. The server retains a 1.5-second rolling buffer, begins after 0.75 seconds, and analyzes every 0.25 seconds. These values are configurable.

Example response:

{
  "type": "chord",
  "timestamp": 2.25,
  "chord": "D minor",
  "confidence": 0.82,
  "notes": ["D", "F", "A"],
  "stable": true
}

Silence is {"type":"silence","timestamp":2.5}. Invalid stream metadata or chunks produce a type: "error" message.

Upload and format a local recording:

python -m client.file_client sample_audio/g_major.wav

Or use curl:

curl -F "audio=@sample_audio/g_major.wav" http://localhost:8000/analyze

POST /analyze accepts .wav, .flac, .mp3, and .m4a multipart uploads under the field name audio. Defaults require 0.25–30 seconds and at most 25 MiB. It returns the result plus the top three candidates.

POST /analyze-buffer accepts JSON-compatible floating-point audio. Stereo samples must be frame-interleaved:

{
  "samples": [0.0, 0.05, -0.03, 0.01],
  "sample_rate": 22050,
  "channels": 1
}

Samples should normally be normalized to [-1, 1]; values outside [-4, 4], non-finite values, invalid channel counts, and mismatched stereo lengths are rejected. JSON is convenient for short integration tests; the binary WebSocket is much more efficient for live audio.

1. Audio is downmixed to mono, resampled to 22,050 Hz, DC-corrected, high-pass filtered at 55 Hz, and peak-limited without amplifying quiet noise.
2. The pre-normalization RMS gate identifies silence, while the high-pass stage reduces rumble. Sustained spectral bins are not gated because a ringing guitar string is itself stationary.
3. Harmonic/percussive separation suppresses pick attacks and string taps when it preserves enough signal.
4. A 36-bin-per-octave constant-Q transform is folded into 12 pitch classes. Median-heavy aggregation resists brief attacks while retaining short notes.
5. Each profile is compared with every weighted chord template. Cosine shape similarity and in-chord energy are rewarded; strong out-of-chord pitch classes are explicitly penalized. Roots and chord-defining thirds/sevenths have more weight than fifths, which may be missing.
6. Confidence combines absolute fit, separation from competing roots, and spectral tonalness. Low-confidence tonal ambiguity becomes Unknown; low RMS becomes Silence.
7. Streaming results require repeated confidence-weighted agreement before a chord change is declared.

This design naturally combines repeated notes across octaves and tolerates unequal string volume, missing fifths, mild detuning, open-string overtones, and brief string noise. It does not assume perfect synthesized spectra.

Settings are read once at startup from environment variables:

For example:

CHORD_CONFIDENCE_THRESHOLD=0.52 uvicorn backend.main:app

No microphone is used by automated tests. Synthetic C major, A minor, and G major audio includes harmonics, slight detuning, amplitude imbalance, attacks, decay, and noise.

pytest

The suite covers template generation and rotation, major/minor discrimination, resampling/stereo, silence, white noise, smoothing, Pydantic response validation, upload errors, buffer analysis, and health.

• No microphone devices: run --list-devices; install PortAudio, grant terminal/Python microphone permission, then restart the terminal. On macOS check System Settings → Privacy & Security → Microphone.
• Invalid sample rate/device: use a rate supported by the selected hardware. Most interfaces support 44,100 or 48,000 Hz; the server resamples it.
• Connection refused: start Uvicorn, verify /health, and confirm the WebSocket URL and port. The microphone client checks the connection before opening the audio device.
• M4A/MP3 decode error: install FFmpeg and verify ffmpeg -version works in the activated terminal.
• Mostly Silence: move closer, raise interface gain without clipping, or lower CHORD_SILENCE_RMS_THRESHOLD slightly.
• Mostly Unknown: let chords ring longer, reduce room noise, tune the guitar, or lower CHORD_CONFIDENCE_THRESHOLD carefully.
• Rapid/late changes: reduce/increase CHORD_SMOOTHING_CONSISTENT_FRAMES. Smaller analysis windows respond faster but have less pitch evidence.
• Wrong inversions or extensions: this system estimates pitch classes, not voicing or bass inversion. Muted strings and overtones can make related chords genuinely ambiguous.

Template matching cannot infer inversions, slash chords, exact voicings, capo position, or every altered/extended chord. Closely related chords share pitch classes (for example, C6 and A minor 7), and noisy rooms, distortion, aggressive strumming, or very short/muted chords reduce reliability. Major/minor triads receive the most tuning and testing; seventh, suspended, diminished, and augmented templates are present but need broader real-guitar validation.

The API layer depends only on the ChordRecognizer protocol:

class ChordRecognizer(Protocol):
    def predict(self, audio: np.ndarray, sample_rate: int) -> ChordPrediction: ...

A future TorchChordRecognizer can implement that contract and be injected through create_app(recognizer_factory=...). The routes, Pydantic responses, clients, buffering, and smoothing do not need to change. A practical upgrade would train on varied guitar recordings, preserve the current silence gate and stream buffer, and return calibrated top-k predictions in the existing domain model.