Audio-Reactive and Generative Visuals: TouchDesigner, Notch and Resolume.

Real-Time Graphics versus Pre-Rendered Clips

Although playing looping video clips (loops) encoded in DXV3 or HAP remains the backbone of many VJ sessions, the current standard on cutting-edge stages is real-time reactive and generative graphics.

Unlike pre-rendered files with a fixed file format, generative environments have no fixed duration and never repeat in exactly the same way: they calculate every frame live using mathematical algorithms, GLSL shaders and data extracted from music in real time.

---

Audio Analysis: Spectral Extraction through FFT

For a graphics engine to respond organically to music, capture the audio signal and extract its parameters using digital signal processing (DSP) techniques.

1. Fast Fourier Transform (FFT)

The FFT decomposes the input audio signal into a spectrum of discrete frequency bands. This separates the spectrum into three main control ranges:

• Low Pass Filter (Sub-bass / Bass: 20 Hz - 150 Hz): Reserved for extracting kick and bass energy, using them as visual-impact triggers, scale pulses or 3D mesh deformation.
• Band Pass Filter (Mids: 300 Hz - 3000 Hz): Captures vocal and synthetic frequencies to modulate colour intensity or particle-system turbulence.
• High Pass Filter (Highs / Air: 5000 Hz - 20,000 Hz): Isolates hi-hats, snares and cymbals to control rapid rotation, flashes or spectral-noise emission.

2. RMS Envelope Detection and Peak Followers

• RMS (Root Mean Square): Measures the music’s continuous energy level (perceived volume), ideal for controlling ambient brightness or the virtual camera’s Field of View.
• Peak Followers: Detect abrupt rhythmic transients, turning them into pulses (Triggers) that decay according to an adjustable damping filter (Lag / Attack & Release).

---

Generative Development Platforms: TouchDesigner and Notch

                        ┌────────────────────────────────────────┐
                        │   Audio Input      / Control Signal    │
                        └───────────────────┬────────────────────┘
                                            │
               ┌────────────────────────────┴────────────────────────────┐
               ▼                                                         ▼
    [ TouchDesigner (CHOPs) ]                                 [ Notch (VFX Blocks) ]
    ├── TOPs (Textures GPU)                                   ├── Shaders 3D Live   
    ├── SOPs (Geometry  3D)                                   ├── Particle Systems       GPU
    └── MATs (Materials  PBR)                                 └── Raytracing Rendering  
               │                                                         │
               └────────────────────────────┬────────────────────────────┘
                                            │
                       (Spout / Syphon / NDI - Zero Latency )
                                            ▼
                       [ Resolume Arena (Advanced Output LED) ]

1. TouchDesigner (Derivative)

The leading node-based visual programming platform for creating live interactive systems.

• Operator Architecture:
  • CHOPs (Channel Operators): Real-time numerical signal and audio processors.
  • TOPs (Texture Operators): GPU-accelerated 2D image-processing operators.
  • SOPs (Surface Operators): Generators and manipulators of 3D geometry and meshes.
  • MATs (Materials): Assignment of shading materials (PBR, Phong, GLSL Custom Shaders).

2. Notch (Notch VFX)

A real-time 3D graphics engine designed specifically for show visual designers. It compiles into small modules called Notch Blocks, which can be loaded directly as effects within Resolume Arena, Disguise or TouchDesigner.

• Massive GPU Particle Systems: Simulating millions of particles interacting with force fields, gravity and music-reactive volume fields.
• Real-Time Raytracing and Global Illumination: Photorealistic rendering of reflections and caustics directly on stage.

---

Inter-App Video Exchange: Spout vs Syphon vs NDI

When graphics are generated in TouchDesigner or Notch, the final signal usually needs to reach Resolume Arena for final mixing, mapping or layer cropping. This uses ultra-low-latency video streaming protocols:

1. Spout (Windows)

• Mechanism: Uses shared texture memory in VRAM through DirectX 11/12 or OpenGL APIs.
• Latency: 0 Frames (Zero latency).
• Bandwidth: Unlimited by hardware. Uses no network traffic or CPU.

2. Syphon (macOS)

• Mechanism: Uses the macOS GPU server (Metal / IOSurface) to share frames between applications.
• Latency: 0 Frames.

3. NDI (Network Device Interface by Vizrt / NewTek)

• Mechanism: Compresses the video signal with very-low-latency codecs and transmits it as UDP packets over the local Ethernet LAN.
• Latency: 1 to 2 frames (16 - 32 ms).
• Use Case: Ideal for sending the visual signal from a powerful backstage rendering computer to the VJ laptop at FOH.