Summary of 1005: Programatic and Skill based Video Creation with Remotion

Overview of Syntax #1005: Programmatic and Skill-Based Video Creation with Remotion

In this episode of Syntax, Wes Bos and Scott Tolinski talk with Remotion creator Johnny Berger about the evolution of programmatic video creation, the explosion of AI-driven video workflows, and new browser tech that could reshape motion graphics. The conversation covers how Remotion works, how it has been monetized sustainably as an open source project, how Claude “skills” drove massive growth, and why experimental web APIs like HTML in Canvas and WebCodecs are so exciting for the future of video tooling.

What Remotion Is and Why It Matters

Remotion is a framework for creating videos with code, primarily using React.

Core idea

• Build a video like a website.
• Use time as an input to change what appears frame by frame.
• Render the result into video.

Why it stands out

• Great for:
  • motion graphics
  • explainer videos
  • product launch videos
  • YouTube overlays
  • audio visualizers
  • reusable video systems
• It’s especially useful when layout and animation are easier to express in HTML/CSS/JS than in traditional video editors.

How Remotion Grew and How It’s Monetized

Johnny explained that Remotion has been open source since 2021, but with a licensing model that requires larger companies to pay.

Monetization model

• Free for individuals and small teams.
• Larger organizations pay once they cross a size threshold.
• This allows the project to stay open source while still funding maintenance and development.

Why it worked

• Keeps the “open source spirit.”
• Avoids alienating most users.
• Provides sustainable revenue for the team.
• Remotion has grown enough to support a small team.

Claude Skills and the AI Video Boom

A major turning point for Remotion was the release of a Claude skill that let users generate Remotion projects by prompting instead of writing code manually.

What changed

• Users can now describe a video in natural language.
• The AI generates the underlying React/JavaScript code.
• This made the product far more accessible to non-developers.

Impact

• Massive install growth after the skills release.
• Broader audience, including:
  • non-technical creators
  • marketers
  • product teams
  • design teams
• Common use cases now include:
  • product launch videos
  • explainer videos
  • YouTube graphics
  • reusable motion design systems

Big takeaway

AI made Remotion easier to approach, but it did not remove the need for good design and human judgment.

Human-in-the-Loop Workflows Still Matter

The hosts and Johnny agreed that fully prompt-driven video creation still has limits.

Current best workflow

• Prompt for broad ideas.
• Refine visually or in code.
• Use UI controls for fast iteration.
• Keep code as the source of truth.

Why this matters

• Video creation is still much faster when feedback is immediate.
• Pure agentic workflows are not ideal for low-latency motion tweaking.
• A hybrid approach is emerging:
  • prompt
  • inspect
  • edit visually
  • update code
  • render

Future Editing Workflow: UI + Code + Agent

Johnny described work toward a better editor experience inspired by After Effects.

Desired capabilities

• Drag-and-drop effects like:
  • blur
  • wiggle
  • vignette
• Adjustable controls with instant feedback.
• Visual editing that writes back to code.

Hard problem

• Remotion wants code to remain the source of truth.
• Visual changes must sync back into source cleanly.
• There are challenges with:
  • source maps
  • AST rewriting
  • fast refresh latency
  • concurrent agent edits

HTML in Canvas: A Potential Game Changer

One of the most exciting topics was Chrome’s experimental “HTML in Canvas” work.

Why it’s exciting

• Lets developers render DOM-like content inside a canvas.
• Preserves normal HTML behaviors like:
  • inputs
  • text selection
  • dropdowns
• Enables canvas-style effects on HTML content:
  • shaders
  • blur
  • grain
  • vignettes
  • other pixel-based effects

Why it matters for video

Traditional browser DOM and canvas are separate. This API could unify the two worlds, making it much easier to build motion graphics directly in web tech.

Current limitations

• Experimental only.
• Requires Chrome Canary and flags.
• Not production-ready.
• Nested canvas/HTML compositions have limitations.
• API may still change or disappear.

Johnny’s view

He sees it as “day one again” for browser-based motion graphics because it removes a major constraint.

How Remotion Actually Renders Video

The rendering process sounds simple on the surface, but has important engineering details.

Basic rendering model

• Use Puppeteer to render frames in the browser.
• Capture each frame at the correct time.
• Stitch frames together with FFmpeg.

Important constraints

• Remotion components should be deterministic.
• useCurrentFrame gives the current time/frame.
• The UI should behave like a pure function of time.
• Native CSS animations are not a good fit because they are not frame-deterministic.

Benefits of this model

• Frames can be rendered in parallel.
• Large renders can be split across many Lambda functions.
• More parallelism can make rendering faster than real time for long videos.

MediaBunny and WebCodecs

Johnny also discussed why Remotion moved toward MediaBunny, a lower-level media library.

Why MediaBunny matters

• Browser <video> tags are not frame-accurate enough for heavy video workflows.
• WebCodecs gives lower-level access to:
  • decoding
  • encoding
  • working with compressed video data
• It enables more advanced workflows such as:
  • transcoding formats
  • working around browser limitations
  • more precise video manipulation

Why it was a big deal

MediaBunny solved hard video infrastructure problems and made it easier for Remotion to work with modern browser media capabilities.

Rendering at Scale on Lambda

The discussion also covered how Remotion scales rendering in the cloud.

Key points

• Render jobs can be split into many parts.
• Remotion Lambda can parallelize across many functions.
• Warm Lambdas reduce startup time.
• Rendering millions of videos per month is already happening for users.

Bottlenecks

• Very long videos can run into Lambda limits.
• Audio stitching and encoding need care to avoid artifacts.

Motion Graphics, Lottie, and Reusable Video Components

The hosts discussed how Remotion fits into motion design workflows.

Remotion’s approach

• Build reusable components in React.
• Parameterize them with props.
• Edit them visually via forms/dropdowns in Studio.

Lottie integration

• Useful when importing After Effects-style templates.
• Lottie works as a pragmatic bridge.
• But it is limited because it’s data, not code:
  • less flexible than React
  • weaker logic/branching support
  • harder to build deeply dynamic layouts

Example use case

• Auto-sizing lower thirds
• Dynamic text length handling
• Video overlays and graphic systems for shows, sports, or content studios

Slop, Metadata, and Platform Detection

There was a brief discussion about “video slop” and automated content detection.

Interesting detail

• Remotion embeds metadata into rendered videos via FFmpeg.
• Some users reportedly want to remove that metadata, possibly because platforms may use it for detection.

Johnny’s stance

• He did not intend to support spammy or low-quality automated content.
• The tool is general-purpose, and users can misuse it like any other platform.

Sick Pick: DMX USB Pro and Smart Lights

For the “sick pick,” Johnny recommended the Enttec DMX USB Pro.

What it does

• Connects a MacBook to professional lighting via DMX.
• Lets you program lights from the browser using the Web Serial API.

Why he likes it

• Enables vibe-coded live light shows.
• Works with concert-style lighting gear.
• Can be combined with tools like Remotion and ElevenLabs for creative live-performance setups.

Main Takeaways

• Remotion is becoming a serious platform for programmable video, not just a developer toy.
• AI prompting lowered the barrier to entry and significantly accelerated adoption.
• The best workflows are still hybrid: code, prompt, and visual editing together.
• HTML in Canvas and WebCodecs could dramatically expand what browser-based motion graphics can do.
• Remotion’s frame-deterministic rendering model is a major strength for scalable video generation.
• Sustainable open source monetization can work when it’s structured thoughtfully.

Notable Practical Ideas

• Use Remotion for:
  • intros/outros
  • overlays
  • lower thirds
  • animated explainers
  • visualizers
  • reusable motion templates
• Use AI to draft the structure, then refine manually.
• Watch emerging browser APIs, especially HTML in Canvas, for future motion graphics breakthroughs.