AI in production and DJing: stem separation, denoise, de-reverb, and intelligent restoration.

AI in production and DJing: stem separation, denoise, de-reverb, and intelligent restoration

Applying Deep Learning algorithms to digital audio has revolutionised deconstruction, filtering, and spectral restoration. Processes traditionally requiring static comb filtering, strict multichannel phase cancellation, or destructive EQ are now addressed with convolutional neural networks (CNN), U-Net architectures, and spectrogram transformers.

This guide examines the algorithmic foundations of Source Separation, intelligent Denoise, De-reverb, and degraded-audio restoration with technical rigour.

---

1. Source and stem separation (SDR, Demucs, Spleeter, U-Net)

AI stem separation isolates individual components (vocals, drums, bass, and other instruments) from a mono or stereo mix without access to individual multitrack recordings.

Algorithmic foundations

1. Spectrogram and Short-Time Fourier Transform (STFT):

   • The time-domain input signal $x(t)$ is transformed into a two-dimensional time/frequency matrix $X(t, f)$ through STFT: $$X(m, \omega) = \sum_{n=-\infty}^{\infty} x[n] w[n-m] e^{-j\omega n}$$
   • Traditional neural networks process magnitude $|X(m, \omega)|$ to estimate a Spectrogram Mask $\hat{M}(m, \omega) \in [0, 1]$, multiplied by the mix’s magnitude. Original mix phase $\angle X(m, \omega)$ is reapplied when reconstructing the waveform with Inverse STFT (ISTFT).

2. Spectral U-Net architectures (Spleeter, Open-Unmix):

   • Convolutional encoders progressively reduce spectral resolution while extracting high-level features, followed by decoders with skip connections preserving time and frequency detail.

3. Demucs v4 (HTDemucs - Hybrid Transformer Demucs):

   • Models the spectrogram domain (2D STFT) and direct time domain (1D Waveform) in parallel.
   • Combining hybrid U-Net with Transformer blocks processes long-range dependencies between track sections, overcoming shared-phase-mask limitations and eliminating cancellation artefacts in fast transients.

       Stereo Mix     (Waveform + STFT 2D)
                        │
         ┌──────────────┴──────────────┐
         ▼                             ▼
   Spectrogram Branch 2D       Time Branch 1D   (Waveform)
   (U-Net + Transformers)     (Temporal Convolutions   )
         │                             │
         └──────────────┬──────────────┘
                        ▼
            Masks & Reconstruction   
                        │
       ┌───────────┬────┴──────┬───────────┐
       ▼           ▼           ▼           ▼
     Vocals       Bass       Drums       Other

Quality evaluation: Signal-to-Distortion Ratio (SDR)

Separation fidelity is measured with the standardised SDR metric (in decibels dB), defined in the BSSEval framework:

$$\text{SDR} = 10 \log_{10} \left( \frac{|s_{\text{target}}|^2}{|e_{\text{interf}} + e_{\text{noise}} + e_{\text{artif}}|^2} \right)$$

Here, $s_{\text{target}}$ is the true isolated source, $e_{\text{interf}}$ represents other instruments’ bleed, $e_{\text{noise}}$ introduced noise, and $e_{\text{artif}}$ distortion or quantisation/phase artefacts generated by the network.

• SDR < 5 dB: Noisy separation with severe bleed and metallic artefacts (“watery effect”).
• SDR 6 - 9 dB: Real-time DJing standard (software such as Rekordbox, Serato DJ Pro, Traktor Pro 4, djay Pro).
• SDR > 10 dB: Studio quality for remixing and resynthesis (HTDemucs models fine-tuned at 44.1/48 kHz).

---

2. Intelligent denoise and de-reverb with neural networks

AI-based denoise and de-reverb replace static gates and profile-based cancellation with dynamic estimates of envelopes and harmonic components.

Dynamic AI denoise

• Traditional filtering limitations: Static profile subtraction assumes a constant noise spectrum. If noise changes (fluctuating electrical hum, wind, preamp hiss), the algorithm cuts vocal harmonic frequencies, producing clicks or hollow timbre.
• DNN/CNN approach (iZotope RX Voice De-noise, Waves Clarity VX):
  • The network is trained with millions of clean/noisy audio pairs covering multiple noise types.
  • It distinguishes primary vocal/instrument formants from ambient noise’s stochastic density in real time, attenuating up to 30 dB of continuous or transient noise without degrading vocal frequency response.

Spectral de-reverb

• Reverberation modifies the audio’s time envelope, creating diffuse energy tails in mid/high frequencies ($500\text{ Hz} - 4\text{ kHz}$) that blur intelligibility.
• Intelligent De-reverb models estimate the room transfer function (RIR - Room Impulse Response) or predict the instrument’s direct anechoic envelope.
• By isolating late and early reflections, the network synthesises a clean signal, removing space resonance without suppressing the instrument’s natural harmonic tail.

---

3. Spectral restoration and de-noising of archival recordings

Restoring old recordings (period vinyl, degraded cassette tapes, or low-fidelity live audio) uses multiple specialised AI modules:

Restoration module	Physical / electroacoustic cause	AI algorithmic solution
De-Click / De-Crackle	Vinyl scratches or static discharge.	Detect ultra-short non-harmonic transient peaks (< 2 ms) and spectrally interpolate through autoregressive networks.
De-Clip / Unclip	Digital or analogue saturation through preamp overmodulation (wave-crest truncation).	Reconstruct waveform peaks by rounding the clipped crest using a neural network trained on harmonic dynamics.
Spectral Recovery / Bandwidth Extension	Compressed files cut at 16 kHz (low-quality MP3) or telephony.	Synthesise lost high frequencies using generative networks predicting upper harmonics from the fundamental.

---

4. Practical restoration and separation workflow

For maximum quality in music production or remixing, follow a strict processing sequence:

1. Cleanup preprocessing (Denoise & De-clip):
   • Clean hiss and saturation before stem separation. Sending a noisy file through Demucs causes the model to confuse background hiss with cymbals or percussion in the drum track.
2. Prior de-reverberation:
   • Reduce room acoustics before separating vocals. Residual reverb is misassigned to the “Other” stem or creates phase cancellation during resynthesis.
3. Run the separation model:
   • Use 32-bit float precision models at the file’s native sample rate (44.1 kHz or 48 kHz).
4. Postprocessing and bleed filtering:
   • Apply a spectral-band noise gate to isolated vocals to mute small drum residues in non-vocal passages.

---

5. Frequently asked questions (FAQ)

Can AI stem-separation models run in real time without perceptible DJ-booth latency?

Yes, but they require hardware-acceleration optimisation (NPU, Apple Silicon Core ML, or CUDA on dedicated GPUs). DJ applications use quantised lightweight models in 16-bit floating point (FP16) or INT8, sacrificing 1.5 to 3 dB of SDR to guarantee separation latency below 10 ms.

Why do “watery” artefacts or phasing appear when isolating acapellas with AI?

They occur when the neural network estimates an unpredictable spectral mask switching rapidly between 0 and 1 in adjacent frequency bins, or when the original complex mix’s phase is reused without correcting isolated formant misalignment.

What is the difference between Spleeter and Demucs v4?

Spleeter (developed by Deezer in 2019) uses a fast spectral U-Net based on TensorFlow, optimised for CPU processing speed. Demucs v4 (developed by Meta AI) uses a hybrid Transformer architecture processing spectrogram and time-domain waveform jointly, achieving significantly higher SDR fidelity at greater computational cost.