Audio Support for Scope Rework

[BuffMcBigHuge]

Audio Support for Scope (Reworked)

Overall, this approach is simplified and seems to solve audio quality issues. The complex gating mechanics and resampling have been removed, leveraging WebRTC aiortc-style timing.

Summary

Adds end-to-end audio support to Scope's WebRTC streaming pipeline. Pipelines can return audio alongside video in their output dict; the server streams audio over WebRTC. This is a simplified rewrite of the audio path that fixes clipping and audio quality issues reported in PR #480.

What's New

Backend

• Pipeline interface: Pipelines may return {"video": ..., "audio": ..., "audio_sample_rate": ...}. Audio keys are optional; pipelines that don't produce audio are unchanged.
• PipelineProcessor: Uses an audio_callback instead of a queue. Only the last processor in a chain receives the callback.
• FrameProcessor: Simple audio_queue for raw (audio_tensor, sample_rate) tuples. No background drain thread, no video-gated release, no resampling.
• AudioProcessingTrack: Receives raw audio from FrameProcessor, resamples to 48 kHz when needed (per-channel, preserves stereo), buffers samples, and delivers 20ms stereo frames for WebRTC/Opus. Uses aiortc-style monotonic pacing.

Frontend

• VideoOutput: Mute/unmute toggle (speaker icon). Starts muted to satisfy browser autoplay policy; user can unmute once the stream is playing.
• useUnifiedWebRTC: Merges video and audio tracks into a single MediaStream. Adds a recvonly audio transceiver so the SDP offer includes an audio m-line for the backend to attach its track.

WebRTC Handshake

The browser adds addTransceiver("audio", { direction: "recvonly" }) so the offer includes an audio m-line. After setRemoteDescription, the backend finds the audio transceiver, attaches its AudioProcessingTrack, and sets direction to sendonly. The answer then indicates that the server will send audio.

Why This Version (audio-sync-2)

PR #480 received feedback about:

• Audio quality / clipping – video and audio not synchronized, poor experience
• Code complexity – video-gated release, drain thread, MediaClock coupling

This branch is a simplified rewrite that:

1. Keeps stereo – No forced mono mixdown. Pipelines (e.g. LTX-2) output stereo; we pass it through. Forced mono can cause phase cancellation and artifacts.
2. Passthrough when possible – If the pipeline outputs 48 kHz (WebRTC standard), resampling is skipped entirely.
3. Per-channel resampling – When resampling is needed, each channel is resampled separately, preserving stereo.
4. Simpler architecture – Callback → queue → track. No drain thread, no video-gated slicing, no MediaClock for audio.

Architecture

Pipeline.__call__() → {"video": tensor, "audio": tensor, "audio_sample_rate": int}
    │
    ▼
PipelineProcessor.audio_callback(audio_tensor, sample_rate)
    │
    ▼
FrameProcessor.audio_queue  (raw tuples)
    │
    ▼
AudioProcessingTrack.recv()  (resample if needed, buffer, 20ms frames)
    │
    ▼
WebRTC

Trade-offs

Item	Status
NDI audio	Not included in this PR. NDI output sinks do not receive audio. Can be re-added in a follow-up.
A/V sync	Previous video-gated approach removed. Uses independent aiortc-style pacing. Please test sync behavior.

Files Changed

• src/scope/server/frame_processor.py – Simplified audio path (~185 net lines removed)
• src/scope/server/pipeline_processor.py – Callback-based audio delivery
• src/scope/server/tracks.py – Stereo AudioProcessingTrack with per-channel resampling
• src/scope/server/webrtc.py – Audio track wiring (no MediaClock)

Related

• Audio Support for Scope #480 – Original audio support PR (superseded by this approach)
• Architecture doc
• Pipelines that produce audio (e.g. LTX-2, https://github.com/daydreamlive/scope-ltx-2/tree/marco/feat/performance-audio) are wired separately; this PR provides the infrastructure.

[j0sh]

Thanks, this does seem somewhat simpler than the last iteration.

I don't want to block this for the sake of shipping something if it seems to be working alright for now, but there are a couple things I don't quite understand.

• Video and audio are being paced differently. Video effectively uses wall-clock while audio is using sample counts. Is there a reason for this?
• Using wall-clock makes things susceptible to pipeline jitter.
• If audio output is a little delayed, it doesn't get a chance for another 20ms. This accumulates and will lead to desync. Conversely, audio that might be a little bursty can be unnecessarily delayed.
• Is there a reason the audio queue is non-blocking? It seems preferable to block (up to a reasonable duration, then silence can be inserted) instead of sleeping. But maybe I'm missing something about why media pulls are intended to be non-blocking.
• In general, I'd consider using input timestamps or a "reference clock" to timestamp and pace the output, rather depending on wall-clock after the pipeline. Pipelines may produce output at different rates and this architecture generally doesn't account for that. More on this in Discord.

[BuffMcBigHuge]

Thank you @j0sh, let's continue in Discord.

[j0sh]

Do these loop over individual samples? That is probably pretty expensive; best to work on complete 20ms frames if possible. There's probably some numpy wizardry for fast interleaving and effective frame chunking.

[ryanontheinside]

i attempted to address this here #633

[github-actions]

🚀 fal.ai Preview Deployment

App ID	daydream/scope-pr-534
WebSocket	wss://fal.run/daydream/scope-pr-534/ws
Commit	acf091d

Testing

Connect to this preview deployment by setting the fal endpoint in your client:

FAL_WS_URL=wss://fal.run/daydream/scope-pr-534/ws

🧪 E2E tests will run automatically against this deployment.

[github-actions]

✅ E2E Tests passed

Status	passed
fal App	daydream/scope-pr-534
Run	View logs

Test Artifacts

Check the workflow run for screenshots.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 47c7a2c6-bba3-4b02-9005-f61d81e11853

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch marco/feat/audio-sync-2

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}