# An AI browser played our VJ set: building FreeVJ, a VJ engine that runs in a tab

*Disclosure: FreeVJ is our own project. It runs on NETT, a publishing channel operated by Solution Market. This is a build log, not an independent review.*

A VJ (video jockey) mixes visuals live to the music, the way a DJ mixes tracks. The job needs a beat clock, a library of loops, a mixer and someone who watches the room and reacts. It is usually done in desktop software, by a person, at a club.

FreeVJ ([freevj.nett.kr](https://freevj.nett.kr/en/)) tries to put all of that in a browser tab. You pick a theme, you switch it on, and the visuals do the rest. While we built it, another use showed up. An AI agent running in a browser could play the same deck: it read the beat, picked clips, moved faders and sent the result to a projector window. It did this with no plugin, no native app and no video files coming from the server.

This is the story of how that happened, what is live today, and where we hit the edges of what a browser can do.

## Why we rebuilt FreeVJ from scratch

freevj.nett.kr existed before this project as a set of about twenty WordPress-style pages: flag clips and After Effects template loops, with no licence information. On 30 October 2026 we decided not to carry any of it over. The old URL patterns now return `410 Gone` so they drop out of search indexes quickly.

The new goal was written as one line in the project's intent document: **pick a theme, switch it on, the rest is automatic.** There are only two required controls. Everything else, including the pro LIVE DECK, stays hidden until you ask for it.

Three rules shaped almost every later decision:

- **The server never ships video.** Loops travel as code: GLSL fragment shaders, JSON manifests, SVG themes and small greyscale textures. The theme pack has a 300 KB gzip budget. AI-made movie loops travel as one black-and-white frame atlas per clip (WebP, at most 512 KB), and the atlas is fetched only when that clip actually goes on screen.
- **Export happens in the browser.** ISF shader files and WebM/MP4 loops are rendered by your browser and saved to your disk. There is no render or download endpoint on the server.
- **Your audio never leaves your device.** Microphone input, tab audio, music files, file names and analysis results stay local.

The core idea that makes this work is the **colourless loop**. Every loop outputs only brightness and transparency. Colour comes from the theme palette at runtime. Loop length is counted in beats, not seconds. So one loop can be reused across all 24 themes and fit any tempo.

Themes and loops live in the same six-axis mood space (energy, brightness, warmth, density, geometric, chaos). The engine computes an affinity between them and picks loops with probability proportional to affinity⁴, adjusted for freshness and compatibility. A Director state machine walks through IDLE, INTRO, GROOVE, BUILD, DROP and BREAKDOWN. It starts fades early so that they *finish* on the musical boundary.

## Eight days, 28 commits

The repository log is short and dense. The first commit, "FreeVJ (Full Auto VJ Loop): initial import with Docker stack", landed on 31 October 2026 at 17:01 KST. The first production deploy was recorded the next morning. The last commit in this story, on 8 November, was the 28th.

A few commit messages tell the story better than a summary would:

- *"Engine: tempo estimate can no longer hang the audio thread"*
- *"Beat: the stream decides tempo; the pre-analysis grid only assists"*
- *"Beat: CDJ-style beat grid (fixed-tempo prior, dynamic otherwise)"*
- *"Pipeline: a switched-off video AI is resting, not blocked"*
- *"Deck grid: slots 1-6 are presets, 7-9 are wildcards for the extended library"*
- *"vj-video: MCP + REST that turn words and a photo into VJ clips"*

Most of the code was written with AI coding agents. The project kept them honest with its documents rather than with memory. Every number in the intent document carries a tag: verified (with a date), owner decision, estimate, or decision needed. Corrections are never deleted. They are kept and marked as corrections, so the next agent can see what was believed before and why it changed. The owner's own words sit next to each decision, from decision A (the brand name) to decision AM (how an AI should talk to the video API).

At the last production deploy, 177 PHPUnit tests and 244 Vitest tests were passing, and headless Chrome end-to-end runs checked the live deck, the output window, sharing and export.

## What is live on freevj.nett.kr today

![The VJ console](https://freevj.nett.kr/brand/intro-02-1280.webp)

- **24 themes** in six families (space, earth, water and sky, city, stage, abstract), each with a BPM range, a palette and parallax background layers from a single SVG.
- **A public loop library of 238 loops**: 2 field, 175 texture, 6 pattern, 3 particle, 2 shape and 50 AI-made sequence loops. You can browse by kind, by mood and by BPM band, preview each loop live, and export it.
- **Audio from wherever you are.** Use the microphone, this computer's sound or your local music folder. Without permission it starts on a demo beat, so there is a moving picture within seconds.
- **The LIVE DECK**, with AUTO, CO-PILOT and MANUAL modes. It has a 40-key clip grid (slots 1–6 are presets and 7–9 are wildcards for the extended library), F1–F12 shortcut keys, a layer mixer, 13 blend modes, layer links (matte, displace, modulate, feedback and group), an FX rack and MIDI learn.
- **A dedicated output window** for a projector or second screen on the same computer. It stays locked to the control window's beat over a `BroadcastChannel` and never shows controls.
- **Eight languages**: Korean, English, Japanese, Chinese, French, German, Spanish and Russian. You can switch language mid-show without stopping the output.
- **Files for agents.** [`/llms.txt`](https://freevj.nett.kr/llms.txt) indexes the library, and [`/ai/catalog.json`](https://freevj.nett.kr/ai/catalog.json) publishes the full action grammar: sections, blends, links, transitions and effects.
- **A photosensitivity governor that cannot be turned off.** It watches the final output, not individual layers, and keeps large brightness changes to three per second or fewer (WCAG 2.3.1). It blocks saturated red flashes. Strobe, invert and ripple are disabled when the system asks for reduced motion. Neither the AI nor a preset can bypass it.

FreeVJ is free. There is no public login, no upload and no tracking of your audio. A small "buy me a coffee" box at the bottom accepts USDT for anyone who wants to support it.

## Designing a deck that an AI can play

Most VJ software assumes a human hand on a fader. FreeVJ was designed from the start with a second performer in mind, and the rules for that performer are strict.

**The AI does not play beats.** A language model takes one to three seconds to answer, which is a long time when a beat lasts half a second. So the AI decides *what* happens and *when*, at phrase level. The local engine executes it with a quantised scheduler, within about ±30 ms. Every AI action is schema-checked, limited to the current candidates and passed through the flash governor.

**Hands win.** Priority is hand, then AI, then Director. Any layer a person touches is locked against the AI and the Director for two phrases.

**The page explains itself as data.** Every bar, the live page updates a hidden `<script type="application/json" id="vj-context">` block. It holds the BPM, bar and section, what is coming next and in how many bars, the energy, the layers on screen, and a short "arise" list of the best candidate clips with scores. The engine narrows the choices (ARISE), the AI picks (CHOICE) and the engine executes on the beat (ACTION).

There are two ways to act on it:

- **WebMCP tools** (`vj_get_context`, `vj_find_assets`, `vj_perform`) are registered when the browser exposes `navigator.modelContext`.
- **`window.freevj`** offers the same three functions as plain page JavaScript, plus `press()` for the F-keys. It goes through the same validator, rate limit and flash governor.

The deck can also generate its own **"basic setup" brief**: a markdown work order with the current clip grid, shortcut keys, effects, macros and layer state, written to be pasted into a browser AI.

## The day a browser agent ran the show

![Immersive space](https://freevj.nett.kr/brand/intro-03-1280.webp)

On 31 October and 1 November we tested this with Aside, an AI browser. First we ran it on the local development build and then on the production deck at freevj.nett.kr. The setup was simple: the main window ran the deck, the output window drove the screen, and the agent was the VJ.

It was not smooth at first, and the failures were the useful part.

**WebMCP was not there.** The browser had no `navigator.modelContext`, so the WebMCP tools never appeared. The fallback that worked was `window.freevj`. The agent read `context()` and booked actions with `perform()` on each bar, all from inside the page, without dragging a single control. That matters: dragging a knob counts as a human edit and locks the layer for two phrases.

**The agent swapped clips but never played the faders.** In the first runs, it changed clips and left every slider at 1.0. We found three causes. Slider values were not in the context. The tool channel was missing. The brief described sliders as something the agent *could* use, not something it *should* use. The fix went into the product. The context gained `sliders`, `moving` and `stillBars` (how many bars since a hand or AI last moved a slider). The brief now says "clips are the stage set, sliders are the performance", with rules: if two bars pass with no movement, act now, and keep at least one layer always moving.

**The agent could add links but never remove them.** Links kept piling up. The grammar gained `link_layers` with `amount: 0` to remove a link, the context gained the F1–F12 key map, and `window.freevj.press()` let the agent hit shortcut keys as an AI action rather than a human one.

**It measured the output, not the deck.** The user set the judgement rule: decide by what the output window shows, aim for visual satisfaction, and avoid harsh colour contrast. So the agent scored the output picture and fed the numbers back. In those runs, an alpha matte of a layer over the field layer during BREAKDOWN scored best (0.86). One theme, Aurora, produced green against magenta clashes in about 48% of output samples, while the other themes stayed near zero, so it was taken out of rotation. Hard mutes and stacked additive blends caused brightness jumps at BUILD and DROP. Softer blends, opacity ducks instead of mutes and a shallower pump on the drop brought most scores into the 0.65–0.92 range.

**The performance needed a rhythm budget.** A second action in the same bar was often rejected with `rate`. Batching about three actions into one `perform()` call per bar fixed it. Section boundaries also moved from track to track, so the agent had to re-derive the phrase grid from the engine's `boundary_soon` events instead of assuming bar numbers.

The result was a browser that played a VJ set by itself and sent it to a projector window, following the music bar by bar. It used no simulated mouse and no video stream from a server. Every frame was rendered locally with WebGL2.

## Computer use without the mouse

People usually picture "computer use" as an AI looking at screenshots and clicking. FreeVJ points at a different version. When a web app publishes its state as structured data and its actions as a validated grammar, a browser agent does not need to see pixels or move a cursor. It reads the room, makes a decision and books it, and the app's own engine handles timing and safety.

The agent and the app also shaped each other. The agent's failures became product changes: new context fields, new verbs and a better brief. The next run then went better. That loop, from agent to log to code and back to the agent, is the part we find most interesting.

## From the deck to an API: vj-video

On 8 November the same engine became a service. **vj-video** is an MCP server, plus a REST API shaped like common video-generation APIs (create a video, poll the object, fetch content, remix). It turns words and an optional photo into VJ clips.

- Four models: **backdrop** (slow backgrounds that seep out from behind text or a photo), **stage** (club visuals), **texture** (seamless surfaces) and **motion** (AI movie loops on a base).
- Outputs include a scene code, links to the live deck and a render page, actions for a running FreeVJ, ISF, an HTML fragment, CSS and a thumbnail.
- **The server still never renders video.** The render link records the clip in *your* browser and saves it to your device.
- Clips are built only from approved loops and themes in the current pack. The text AI chooses *how* to use them (model, mood, theme, keywords) and never invents assets.
- Requests rarely contain a tempo, so rhythm comes from language. Onomatopoeia, motion verbs and genre words set the tempo and the pulse (kick, heartbeat, sway, breath, twinkle, drift). Adjectives such as "joyful", "heavy" or "light" go through a mood mutator based on the valence-arousal-dominance model and are projected onto the six mood axes.
- Access is by key. In the admin's Members screen, a single "MCP on" button creates a key, then connects to the MCP endpoint to prove it works. The public API answers `401` without one.

## Where the browser's edges are

Building a live visual instrument in a tab means meeting the browser's limits head-on. These are the ones we hit, with what we did about them:

- **Hidden windows stop painting.** If the output window is minimised or covered, the browser pauses its animation frames even though the page is alive. FreeVJ watches a heartbeat that reports "I actually drew in the last second". If that goes false, the main window takes over rendering at once and shows a badge, then hands back when the output recovers. The reverse case, where the main window is covered, is still open: the output window keeps the beat by extrapolation, but scene changes pause. The output is meant to sit on another screen.
- **Two windows, one clock.** Both windows compute automation and modulation every frame from the same pure beat functions. Only the scene, the clock and audio features travel over `BroadcastChannel`. All beat maths lives in one module so no two parts of the app ever disagree about where the downbeat is.
- **The audio thread must never block.** A tempo estimate once hung the AudioWorklet. Tempo now follows the live stream, and pre-analysis of a track only provides the bar grid, like a CDJ beat grid.
- **No GPL in the beat tracker.** Common beat-tracking libraries are GPL or AGPL, which doesn't fit a commercial web service. FreeVJ uses its own beat core instead.
- **Development reloads kill performances.** Hot reloads of deck files reset playback to OFF and wiped the agent's in-page scripts mid-set. A `beforeunload` guard could also leave a tab waiting on a "leave site?" dialog. For live runs, any code edit is now treated as a stop event.
- **WebMCP is not everywhere yet.** The deck registers WebMCP tools when it can, and the plain `window.freevj` API covers every browser that cannot.

## Try it

- Open the live deck: [freevj.nett.kr/en/live](https://freevj.nett.kr/en/live/). Pick a theme, press ON, and let the demo beat run if you don't want to grant audio permission.
- Browse the [loop library](https://freevj.nett.kr/en/loops/) and the [24 themes](https://freevj.nett.kr/en/themes/).
- If you build agents, read [`llms.txt`](https://freevj.nett.kr/llms.txt) and [`ai/catalog.json`](https://freevj.nett.kr/ai/catalog.json), then try `window.freevj.context()` in the console of the live page.

We started out trying to make a VJ that runs itself. We ended up with something a little stranger: a web page that an AI can perform on, safely, in time with the music.

More software like this, built to run on your own terms, is at [Solution Market](https://solutionmarket.net/en).

