Sound and music

How a phone game in plain JavaScript can make all its sound in code, effects, places and music, and give each world a sound of its own the way it already has eight inks of its own. Sound effects in games have been made from a dozen numbers since sfxr in 2007: a wave, a pitch and its slide, an envelope, a filter. ZzFX packs the same idea into 21 numbers and under a kilobyte, and renders each sound once into a buffer. Places are noise through filters (wind), many tiny pitched events (water, gravel, birds) and a slow signal steering both. Adaptive music has two moves, adding layers (vertical) and choosing the next section (horizontal), and a tune stored as scale degrees can change mode with the time of day without being rewritten. The finding: Tycho already has the hard half of this (Lodestar’s engine: the graph, the browser’s rules, voices, an offline bank, drones and noise bands for places). What it lacks is the musical half: a sample-loop effect synth driven by a few seeded numbers, a beat clock with a note scheduler, scales and modes, and a “sound palette” a quest declares like its palette, with a small, fixed voice budget, as the chips had.

Proposed: none of this is in Tycho yet. The demos run here in the page, written in plain JS with the study’s own kit (demos/kit.js), not Tycho; each says how it would run in Tycho: per frame (the GPU stand-in’s calls), pre-bake (painted once offscreen), or shader (per-pixel work every frame). For sound, pre-bake means rendered once into a buffer and per frame means scheduled live as the game runs. What would move into Tycho, and when, is in the roadmap.

Text marked (inference) is reasoning from the sources, not a sourced fact. Browsers allow sound only after a tap, so every demo here is silent until you press a button; the pictures run either way. Timings were measured in Chrome on a laptop; no phone was measured.

• What exists
• Sound effects from a dozen numbers
• Render once or build each time
• Footsteps, doors and places
• The chip as a style
• Music that follows the game
• A tune from a seed
• A sound palette per world
• Phones
• What it means for Tycho
• Sources

What exists

Tycho.audio is Lodestar’s sound engine, moved into Tycho bit for bit and held to Lodestar’s own by a test that traces every Web Audio call (the audio page, audio.ts). It has three parts:

Its voices are tone (an oscillator, gliding, through a low pass), noise (looped noise through a sweeping filter), fm and stack (detuned voices through a sweeping filter). Variation comes from the sound’s own seeded stream, never the game’s, “so sound never moves the world”.

Elsewhere’s js/snd.js is a thin shell over that engine: four cues of its own (a tap, a page of the log, a quest, a souvenir) and a pad, one note of a chord as a slow stack. Each quest brings its own sounds and a score: { bpm, beats, root, chords, w, lp, g }, played a bar at a time, 0.3 s ahead of the audio clock. The slice’s scores are already tiny sound palettes:

What neither has (inference, from reading the code): no effect synth that runs as a sample loop (every live voice is a node graph), no beat clock or note sequencer (Music follows a state, not a tempo), no scale or mode helpers beyond hz(semitones, root), and no recipe made from a seed. A score is chords only: no melody, no rhythm, no layers that follow play.

Sound effects from a dozen numbers

sfxr was written by DrPetter (Tomas Pettersson) for the 10th Ludum Dare in December 2007, as “a simple means of getting basic sound effects into a game” for entrants with 48 hours and no time to find sounds, and “a lot of the entrants used it” (sfxr). Its sound is about 22 numbers, each normalized to 0..1 (jsfxr’s sfxr.js):

The buttons are the clever part. Each category, pickupCoin, laserShoot, explosion, powerUp, hitHurt, jump, blipSelect, is a few lines of seeded choices within ranges. In jsfxr, pickupCoin is a sawtooth with a start between 0.4 and 0.9, a short sustain, a punchy decay and a 50% chance of an arpeggio jump. Jump is a square with a random duty and a rising slide of 0.1 to 0.3. hitHurt is square, sawtooth or noise with a steep falling slide. Explosion is noise at a low, squared random start with a falling slide, high punch, and optional repeat, phaser, vibrato or arpeggio. Mutate moves each parameter, with probability one half, by up to ±0.05. The loop renders at 8× oversampling, and its noise is a table of 32 random values refilled every period, so noise has a pitch (jsfxr).

The descendants added waves and effects, not new ideas. bfxr (increpare) added five waves (triangle, breaker, tan, whistle, pink noise), a one-bit noise after the BBC Micro’s SN76489, and compression, harmonics and a bitcrusher (rekkerd.org, a C++ port). ChipTone (Tom Vian, SFBGames, 2015, first in Flash) descends from bfxr by way of Vian’s own as3sfxr and releases its sounds as CC0 (ChipTone).

ZzFX (Frank Force, 2019, MIT) is the one built for the web: the micro version is “under 1 kilobyte” (ZzFX). A sound is an array of numbers in a fixed order, in seconds and hertz rather than 0..1 (ZzFX.js):

The 1.4 code takes 21, though its header still says “20 controllable parameters”. ZzFX writes samples at 44,100 Hz in a plain JS loop into a mono AudioBuffer and plays it with a buffer source, with no oscillator nodes, and sounds can be made ahead and played later (ZzFX). Keith Clark’s ZzFXM is a tracker-style player built on ZzFX instruments, for js13k games (ZzFXM).

The demo is a generator in this manner. A sound is up to four layers, each one voice described by sfxr’s kind of numbers plus one more, grains: the voice heard only in short random bursts, which is what turns noise into grass, gravel or the stick-slip of a creaking hinge. A category is a function from a seeded stream to layers, and Mutate nudges half the numbers. Each sound is rendered once into a Float32Array and played as a buffer.

Render once or build each time

Web Audio offers two ways to make a sound, and the choice is about cost and control:

Paul Adenot’s notes on Web Audio performance give the costs (Adenot): a BiquadFilterNode is “relatively cheap”, about five multiplies and four adds a sample; a ConvolverNode is “very expensive” (FFTs every block, in bursts); HRTF panning is “very expensive” (up to four convolvers while a source moves); equal-power panning is “rather cheap” and a StereoPanner cheaper; a GainNode is “very cheap to create and connect”. Blink and WebKit resample buffers linearly: cheap, a little rough. For phones he advises delays, all-passes and low passes instead of convolution reverb, a short reverb and an equal-power panner instead of HRTF, and a “degraded” mode for weak devices.

In the demo, a pickup (14,148 samples) renders in about 0.4 ms, a footstep in 3 to 5 ms, and a one-second, three-layer explosion in 6 to 11 ms once the code is warm (30 ms the first time). That is too long to do on the frame a sword lands, so effects are rendered at load or between frames, which is what Tycho’s Bank already does, then played instantly (inference from the timings). Ten effects with three variants each are under 3 MB of buffers (inference).

So effects with a fixed shape (pickups, hits, steps, doors) want rendering; sounds that follow a continuous value (wind, an engine, a rising danger) want live nodes, and music can be either: live oscillators per note are cheap at chip-like note rates (inference). Tycho already does both, live voices in Engine and offline renders in Bank. What a ZzFX-style synth would add is the compact, seedable description: 20 numbers a recipe, rendered in plain JS with no context at all, so a quest’s sounds could be generated from its seed and tested in Node.

Footsteps, doors and places

Andy Farnell’s Designing Sound (MIT Press, 2010, 688 pages) makes the case that any sound can be built from first principles, from the physics to a model to a Pure Data patch. His 2005 demo made “fire, water, wind, rain, some animals, some footsteps, some guns, some vehicles” for a first-person shooter, all in code (interview, 2012).

Footsteps. Perry Cook’s “Modeling Bill’s Gait” (AES, 2002) analyzes walking into gait timing and heel-toe events, finds a surface’s resonances by linear prediction, and resynthesizes loose surfaces with PhISEM, which treats them as particle collisions arriving as a Poisson process; gravel, grass and snow underfoot are among its uses (Cook 2002, Cook 2003). For a pixel game that reduces to two layers a step, heel then toe 50 to 90 ms apart (inference from Cook):

A door in the demo is a creak, a low sawtooth heard in bursts through a resonance (the stick-slip of a hinge), then the latch: a low thud and a click. The same recipe with a different resonance is a ship’s timber or a gate (inference).

Water is bubbles. Van den Doel’s “Physically Based Models for Liquid Sounds” (2005) builds streams, pouring, rain and waves from single bubbles: each a damped sine at the Minnaert frequency of its radius, rising in pitch as it rings, and a user study found a rise parameter near 0.1 sounded most real (van den Doel 2005, Moss et al. 2009). Minnaert’s formula puts a 1 mm bubble near 3.3 kHz and a 3 mm one near 1.1 kHz, so a babbling brook is a dozen or more tiny sines a second between 800 Hz and 4 kHz over a bed of low noise (the constant from Minnaert; inference to apply it).

Wind is noise through a band pass whose center and level follow a slow gust signal, and a second, narrow band for the whistle once it’s strong. Birds are short sine chirps, 40 to 90 ms, sweeping fast between 1.5 and 7 kHz, in phrases of two to five. Rain is high-passed noise plus more, smaller bubbles; thunder a low rumble that arrives after its flash (inference, in the manner of Farnell). Tycho’s Music.band is already the wind’s and the stream bed’s building block: a band of looped noise, silent until set.

The demo drives the picture and the sound from one seeded event stream at 30 steps a second: every bubble you hear is a sparkle on the stream, every phrase a bird crossing the sky. That is the rule worth keeping in the game (inference): sound events are the world’s events, so a quest that’s silent (sound off) or deaf (a phone on silent) shows exactly the same thing.

The chip as a style

The consoles’ sound chips gave every game the same tight budget, and composers made that budget a style:

From those limits came the chip idioms: a chord played as a fast arpeggio on one voice; the triangle as bass, its pitch dropping fast as a kick; noise shared between the hi-hat, the snare and the effects; each part keeping one timbre for the whole tune (inference).

This is the eight-ink rule’s twin. Eight inks force a world to choose what it is; four voices and a noise force a tune to choose what it says, and a player hears the limit as a style rather than a lack (inference). The game needn’t copy any one chip; each world can declare a budget, its voices and their waves, the way it declares its inks. Two worlds then sound as different as they look: a pulse-12.5% lead and a triangle bass at 128 bpm is a toy shop, sines and a slow pad at 70 bpm is a lake at night.

Music that follows the game

Adaptive music has two moves, and every system is some mix of them (Winifred Phillips at GDC 2021, hybrids):

• Vertical layering: synchronized parts switched on and off by the game (The Maw used three layers, LittleBigPlanet 2’s Toy Story level six).
• Horizontal re-sequencing: a piece built from segments chosen and ordered during play (Speed Racer, Spore Hero).

What shipped games did:

Behind the generative ones is Brian Eno. Music for Airports (1978) phases tape loops of different lengths; in 2/1 seven loops each carry one sung note, the C repeating every 20.1 s and the high F every 19.6 s, and “they all repeat in cycles that are called incommensurable — they are not likely to come back into sync again” (Reverb Machine). Generative Music 1 (1996) was a floppy of 12 pieces for SSEYO’s Koan player (Intermorphic).

Three lessons for a small game (inference). No Man’s Sky, the most “procedural” game on the list, streams nearly all its audio: the generative part is the choosing, not the synthesis. Mario’s bongos and Animal Crossing’s hours are the cheapest adaptive music there is, one layer and one clock, and both are remembered. And Spore’s line is the whole trick of generative melody: constrain the notes to a scale, and randomness starts to sound like music.

A tune from a seed

The demo makes a tune for a world in five steps, each small enough for a phone:

1. A palette from the seed: a family of three modes (day, dusk, night), a root, a tempo (70 to 130 bpm), swing or not, one wave for each of four voices, and a drum kit (a triangle kick falling fast, as on the NES; noise snare and hat).
2. Chords by a Markov step. Four bars of chord roots, each a move of a third, fourth or fifth likeliest; then the same four again ending home. A chord is three scale tones a degree apart.
3. Melody by a Markov walk over intervals: small steps likeliest, a leap followed by a step back, chord tones on the strong beats. That is the usual order-1 chain over scale-degree intervals (Melody Triangle).
4. Repetition. Two 2-bar motifs make an eight-bar song, A A′ B A″, with A″ ending on the tonic. Repetition with small changes is what makes a generated line sound written rather than random (inference).
5. Stored as scale degrees. Every note is a degree, mapped onto the current mode only when it plays, as Disasterpeace’s #January does (#January).

The modes, in order of brightness, each one note flatter than the last: Lydian, Ionian (major), Mixolydian, Dorian, Aeolian (minor), Phrygian. A family steps along that line at dusk and at night, Ionian → Mixolydian → Dorian, say, so the same tune darkens without being rewritten. The five-note families, major pentatonic → Japanese yo → minor pentatonic → in, give a world a place with few clashing notes. An equal-tempered approximation of other traditions’ scales (gamelan’s slendro or pelog, a maqam) is only a flavor of the real tuning, and should be called that (inference).

Layers by danger. Each voice has a threshold; the danger slider brings layers in at bar lines, as Mario’s bongos do: pad always, then bass (0.1), lead (0.25), hat (0.45), kick (0.55), arpeggio (0.7), snare (0.85). The bass also changes pattern as danger rises: half notes, then a syncopated figure, then octave eighths. Dusk drops the hat; night drops the drums (unless danger is near full), halves the lead and plays it an octave down, and slows the arpeggio to eighths. Tycho’s presence(s, at) would make the same entrances as crossfades rather than cuts.

Scheduling. Notes are scheduled on the audio clock a little ahead, never on the frame or setTimeout, following Chris Wilson’s pattern: “a good place to start is probably 100ms of ‘lookahead’ time, with intervals set to 25ms” (web.dev). The demo schedules each bar that starts within the next 150 ms, from its 30-a-second step, and makes bars again only from the next unscheduled one when the state changes, so a change lands on the next bar line, as iMUSE’s markers do.

A sound palette per world

The proposal (inference throughout): a quest declares a sound palette beside its palette, a small, fixed set from which every cue and every note is drawn, and from which Tycho can generate what the quest doesn’t write by hand.

Sketches for the slice and one proposed quest, extending what their scores already say:

The moor (in Elsewhere’s brief) is the case where sound is the play: the engine’s distance falloff and pan, which Tycho has, do the work.

Two rules from the eight-ink rule carry over. A palette is an identity, and a change of palette is an event: the lighthouse’s turn to major when the lamp is lit is the sound’s version of dusk at the frog stall. And the player is printed in every world’s inks by one recipe: the shell’s own cues (a tap, a page, a souvenir) could likewise be played in the quest’s accent wave and mode, so the Lantern’s chime is always the Lantern but always sounds like where you are.

Phones

What it means for Tycho

Tycho’s audio is a complete engine for one game’s hand-written sound. What Elsewhere needs on top is a way for each quest to have a whole sound from a few declared numbers. As options any game could use, in Tycho.audio (inference):

None of it changes Lodestar’s sound: Engine, Bank and Music stay as they are, and the new parts feed them (a rendered effect is a bank recipe; a tune’s notes are cues on the music bus) (inference). In Elsewhere, snd.js’s score would become a palette, with the four existing scores as its first examples.

Sources

• Tycho: the audio page and audio.ts
• DrPetter, sfxr and jsfxr
• bfxr on rekkerd.org and madeso’s C++ port
• ChipTone
• Frank Force, ZzFX and Keith Clark, ZzFXM
• Paul Adenot, Web Audio API performance and debugging notes
• Chris Wilson, A tale of two clocks
• Andy Farnell interview, Designing Sound, 2012
• Perry Cook, Modeling Bill’s Gait (2002)
• Kees van den Doel, Physically Based Models for Liquid Sounds (2005) and Moss et al., Sounding Liquids (2009)
• NESdev wiki, APU, Pan Docs, Audio, SNESdev wiki, S-SMP
• Wikipedia: MOD, Ultimate Soundtracker, iMUSE, Proteus
• Winifred Phillips at GDC 2021: horizontal re-sequencing and hybrids
• MarioWiki, Ground BGM (Super Mario World)
• Animal Crossing World, New Leaf’s 24 hourly tracks
• Game Developer, oral history of Rez
• PdCon, Using Pure Data in Spore and Darkspore
• A Sound Effect, Paul Weir on No Man’s Sky
• Reverb Machine, deconstructing Music for Airports and Intermorphic, Generative Music 1
• Melody Triangle and Disasterpeace, #January
• Chromium, autoplay policy
• MDN: AudioSession.type and BaseAudioContext.state; WebKit bugs 237322, 276687, 291892
• web-features, audio-worklet

The full research is in the note on sound and music.