Audio Visualization

Real-time audio visualization system with waveforms, spectrograms, and FFT analysis

Overview

The audio visualization system in Friday Night Funkin’ provides real-time visual feedback for audio playback. It includes spectral analysis, waveform rendering, and character-specific visualizations like A-Bot’s reactive display.

Location: funkin.audio.visualize

Components

VisShit

Core utility class for audio buffer access and FFT analysis.

Properties

snd · FlxSound

The sound to visualize.

setBuffer · Bool

Whether the audio buffer has been successfully accessed.

audioData · Int16Array

Raw audio sample data as 16-bit integers.

sampleRate · Int · default: "44100"

Sample rate of the audio in Hz.

numSamples · Int

Total number of audio samples.

Constructor

new(snd:FlxSound)

snd · FlxSound · required

The FlxSound instance to analyze.

Methods

checkAndSetBuffer

checkAndSetBuffer():Void

Attempts to access the audio buffer from the playing sound. Must be called while the sound is playing.

getCurAud

static function getCurAud(aud:Int16Array, index:Int):CurAudioInfo

Extracts audio information at a specific sample index.

aud · Int16Array · required

The audio data array.

index · Int · required

Sample index to read.

return · CurAudioInfo

Object containing left, right, and balanced audio channel values (-1.0 to 1.0).

funnyFFT

funnyFFT(samples:Array<Float>, ?skipped:Int = 1):Array<Array<Float>>

Performs Fast Fourier Transform analysis on audio samples.

samples · Array<Float> · required

Array of audio samples to analyze.

skipped · Int · default: "1"

Sample skip rate for optimization.

return · Array<Array<Float>>

2D array of frequency power values over time.

---

PolygonSpectogram

Real-time waveform visualization using polygon mesh rendering.

Properties

vis · VisShit

The VisShit instance providing audio data.

visType · VISTYPE · default: "UPDATED"

Visualization mode: STATIC, UPDATED, or FREQUENCIES.

daHeight · Float · default: "FlxG.height"

Height of the visualization in pixels.

realtimeVisLenght · Float · default: "0.2"

Length of audio to visualize in seconds for real-time mode.

realtimeStartOffset · Float · default: "0"

Time offset for real-time visualization start position.

detail · Float · default: "1"

Visualization detail multiplier. Higher values = more vertices.

thickness · Float · default: "2"

Line thickness for waveform rendering.

waveAmplitude · Int · default: "100"

Horizontal amplitude of the waveform.

Constructor

new(?daSound:FlxSound, ?col:FlxColor = FlxColor.WHITE, ?height:Float = 720, ?detail:Float = 1)

daSound · FlxSound

The sound to visualize.

col · FlxColor · default: "FlxColor.WHITE"

Color of the waveform.

height · Float · default: "720"

Height of the visualization.

detail · Float · default: "1"

Detail level multiplier.

Methods

setSound

setSound(daSound:FlxSound):Void

Sets or changes the sound to visualize.

daSound · FlxSound · required

The new sound to visualize.

generateSection

generateSection(start:Float = 0, seconds:Float = 1):Void

Generates a static waveform visualization for a specific section of audio.

start · Float · default: "0"

Start time in milliseconds.

seconds · Float · default: "1"

Duration to visualize in seconds.

checkAndSetBuffer

checkAndSetBuffer():Void

Initializes the audio buffer for visualization. Automatically called during update.

---

ABotVis

A-Bot character’s reactive audio visualizer with animated bars.

Properties

snd · Null<FlxSound>

The sound being visualized.

analyzer · Null<SpectralAnalyzer>

The spectral analyzer instance for frequency analysis.

Constructor

new(snd:FlxSound, pixel:Bool)

snd · FlxSound · required

The sound to visualize.

pixel · Bool · required

Whether to use pixel art graphics (true) or HD graphics (false).

Methods

initAnalyzer

initAnalyzer():Void

Initializes the spectral analyzer with A-Bot specific tuning. Must be called after the sound starts playing.

Analyzer Configuration:

• 7 frequency bars
• Frequency range: 10 Hz to 22,000 Hz
• dB range: -65 to -25
• Optimized FFT settings per platform

dumpSound

dumpSound():Void

Clears the sound and analyzer references to free memory.

Constants

BAR_COUNT · Int

Number of frequency visualization bars.

---

Type Definitions

CurAudioInfo

typedef CurAudioInfo = {
  var left:Float;
  var right:Float;
  var balanced:Float;
}

Audio channel information at a specific sample:

left · Float

Left channel value (-1.0 to 1.0).

right · Float

Right channel value (-1.0 to 1.0).

balanced · Float

Average of both channels (-1.0 to 1.0).

VISTYPE

enum VISTYPE {
  STATIC;
  UPDATED;
  FREQUENCIES;
}

Visualization rendering modes:

• STATIC - One-time generation
• UPDATED - Real-time following playback
• FREQUENCIES - Frequency-based visualization

Example Usage

Basic Waveform Visualization

// Create a waveform visualizer
var spectogram = new PolygonSpectogram(FlxG.sound.music, FlxColor.CYAN, 720, 1.5);
spectogram.waveAmplitude = 150;
spectogram.thickness = 3;
spectogram.visType = UPDATED;
add(spectogram);

// Will automatically update as music plays

Static Section Visualization

// Generate waveform for a specific section
var spectogram = new PolygonSpectogram();
spectogram.setSound(mySound);
spectogram.visType = STATIC;
spectogram.generateSection(5000, 2); // 2 seconds starting at 5s
add(spectogram);

A-Bot Reactive Visualizer

// Create A-Bot visualizer
var abotVis = new ABotVis(FlxG.sound.music, false); // HD graphics
add(abotVis);

// Initialize after music starts
FlxG.sound.music.play();
abotVis.initAnalyzer();

// Bars will automatically react to music frequencies

Custom FFT Analysis

// Create analyzer
var visShit = new VisShit(FlxG.sound.music);

// Wait for buffer
if (FlxG.sound.music.playing) {
  visShit.checkAndSetBuffer();
  
  if (visShit.setBuffer) {
    // Get sample at current time
    var sampleIndex = Math.floor(visShit.snd.time / 1000 * visShit.sampleRate);
    var audioInfo = VisShit.getCurAud(visShit.audioData, sampleIndex);
    
    trace('Left: ${audioInfo.left}, Right: ${audioInfo.right}');
  }
}

Real-time Waveform with Custom Settings

// High detail, custom colored waveform
var waveform = new PolygonSpectogram(music, FlxColor.LIME, 600, 2.0);
waveform.realtimeVisLenght = 0.5; // Show 500ms of audio
waveform.realtimeStartOffset = 0.1; // Start 100ms ahead
waveform.waveAmplitude = 200;
waveform.thickness = 4;
waveform.visType = UPDATED;
add(waveform);

Multi-track Visualization

// Visualize instrumental and vocals separately
var instVis = new PolygonSpectogram(instrumental, FlxColor.BLUE, 360, 1.0);
instVis.visType = UPDATED;
add(instVis);

var vocalVis = new PolygonSpectogram(vocals, FlxColor.RED, 360, 1.0);
vocalVis.y = 360;
vocalVis.visType = UPDATED;
add(vocalVis);

Performance Considerations

• Buffer Access: checkAndSetBuffer() only works while sound is playing
• Detail Level: Higher detail values create more vertices, impacting performance
• FFT Size: Native platforms use smaller FFT (256) vs HTML5 for optimization
• Real-time Updates: UPDATED mode regenerates visualization every frame
• Memory: Call dumpSound() on ABotVis when done to free resources

Platform Notes

• HTML5: Uses optimized browser FFT APIs
• Native: Uses custom FFT implementation with reduced FFT size (256)
• Audio Buffer: Only accessible while sound is actively playing
• Sample Rate: Defaults to 44100 Hz, detected from audio file

Advanced: Spectral Analysis

The ABotVis component uses SpectralAnalyzer from funkin.vis.dsp with these settings:

analyzer.minDb = -65;    // Minimum decibel threshold
analyzer.maxDb = -25;    // Maximum decibel threshold
analyzer.maxFreq = 22000; // Upper frequency limit (Hz)
analyzer.minFreq = 10;    // Lower frequency limit (Hz)
analyzer.fftN = 256;      // FFT size (native only)

These values are tuned for music visualization and can be adjusted for different visual effects or audio types.