diff --git a/.agents/skills/remotion-best-practices/SKILL.md b/.agents/skills/remotion-best-practices/SKILL.md new file mode 100644 index 0000000..471bee3 --- /dev/null +++ b/.agents/skills/remotion-best-practices/SKILL.md @@ -0,0 +1,46 @@ +--- +name: remotion-best-practices +description: Router for all Remotion skills +metadata: + tags: remotion, video, react, animation, composition +--- + +## New project setup + +If no Remotion project currently exists, load [Create a new Remotion project](remotion-create/REFERENCE.md) + +## React Markup Best Practices + +If you are writing Remotion React Markup, load [Remotion Markup Best Practices](remotion-markup/REFERENCE.md) + +## Maps + +For static maps, animated routes and markers, geographic explainers, Mapbox, MapLibre, MapTiler, GeoJSON, or 3D geographic flyovers, load [Remotion Maps](remotion-maps/REFERENCE.md). + +## Multimedia + +For achieving multimedia tasks in the browser, such as trimming, cropping videos, or getting metadata from them, load [Remotion Multimedia](remotion-multimedia/REFERENCE.md) + +## Improving Interactivity + +By structuring the Remotion markup well, we can allow users to interactively change things in the Studio and write back to code. If relevant: [Interactivity Best Practices](remotion-interactivity/REFERENCE.md) + +## Rendering + +For advanced rendering beyond simple `npx remotion render`, see: [Rendering Best Practices](remotion-render/REFERENCE.md) + +## Captions + +When working with Captions, load [Remotion Captions](remotion-captions/REFERENCE.md). + +## Creating a SaaS, automation or application + +Use the [Remotion SaaS skill](remotion-saas/REFERENCE.md) for knowledge about Remotion-powered SaaS apps, such as ``, rendering on Lambda, Vercel, Cloudflare, via Express.js, client-side rendering, or for finding the right SaaS template. + +## Looking up Remotion APIs and documentation + +To find and read current Remotion documentation, load [Remotion Docs](remotion-docs/REFERENCE.md). + +## Upgrading + +To upgrade Remotion, related packages, compatible Mediabunny packages, and installed Remotion Agent Skills, load [Remotion Upgrade](remotion-upgrade/REFERENCE.md). diff --git a/.agents/skills/remotion-best-practices/remotion-captions/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-captions/REFERENCE.md new file mode 100644 index 0000000..36d8da5 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/REFERENCE.md @@ -0,0 +1,36 @@ +--- +name: remotion-captions +description: Transcribing, displaying and animating captions +metadata: + tags: subtitles, captions, remotion, json +--- + +All captions must be processed in JSON. The captions must use the [`Caption`](https://www.remotion.dev/docs/captions/caption.md) type which is the following: + +```ts +import type { Caption } from "@remotion/captions"; +``` + +This is the definition: + +```ts +type Caption = { + text: string; + startMs: number; + endMs: number; + timestampMs: number | null; + confidence: number | null; +}; +``` + +## Generating captions + +To transcribe video and audio files to generate captions, load the [transcribe-captions.md](transcribe-captions.md) file for more instructions. + +## Displaying captions + +To display captions in your video, load the [display-captions.md](display-captions.md) file for more instructions. + +## Importing captions + +To import captions from a .srt file, load the [import-srt-captions.md](import-srt-captions.md) file for more instructions. diff --git a/.agents/skills/remotion-best-practices/remotion-captions/display-captions.md b/.agents/skills/remotion-best-practices/remotion-captions/display-captions.md new file mode 100644 index 0000000..f9cb0f0 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/display-captions.md @@ -0,0 +1,184 @@ +--- +name: display-captions +description: Displaying captions in Remotion with TikTok-style pages and word highlighting +metadata: + tags: captions, subtitles, display, tiktok, highlight +--- + +# Displaying captions in Remotion + +This guide explains how to display captions in Remotion, assuming you already have captions in the [`Caption`](https://www.remotion.dev/docs/captions/caption) format. + +## Prerequisites + +Read [Transcribing audio](transcribe-captions.md) for how to generate captions. + +First, the [`@remotion/captions`](https://www.remotion.dev/docs/captions) package needs to be installed. +If it is not installed, use the following command: + +```bash +npx remotion add @remotion/captions +``` + +## Fetching captions + +First, fetch your captions JSON file. Use [`useDelayRender()`](https://www.remotion.dev/docs/use-delay-render) to hold the render until the captions are loaded: + +```tsx +import { useState, useEffect, useCallback } from "react"; +import { AbsoluteFill, staticFile, useDelayRender } from "remotion"; +import type { Caption } from "@remotion/captions"; + +export const MyComponent: React.FC = () => { + const [captions, setCaptions] = useState(null); + const { delayRender, continueRender, cancelRender } = useDelayRender(); + const [handle] = useState(() => delayRender()); + + const fetchCaptions = useCallback(async () => { + try { + // Assuming captions.json is in the public/ folder. + const response = await fetch(staticFile("captions123.json")); + const data = await response.json(); + setCaptions(data); + continueRender(handle); + } catch (e) { + cancelRender(e); + } + }, [continueRender, cancelRender, handle]); + + useEffect(() => { + fetchCaptions(); + }, [fetchCaptions]); + + if (!captions) { + return null; + } + + return {/* Render captions here */}; +}; +``` + +## Creating pages + +Use `createTikTokStyleCaptions()` to group captions into pages. The `combineTokensWithinMilliseconds` option controls how many words appear at once: + +```tsx +import { useMemo } from "react"; +import { createTikTokStyleCaptions } from "@remotion/captions"; +import type { Caption } from "@remotion/captions"; + +// How often captions should switch (in milliseconds) +// Higher values = more words per page +// Lower values = fewer words (more word-by-word) +const SWITCH_CAPTIONS_EVERY_MS = 1200; + +const { pages } = useMemo(() => { + return createTikTokStyleCaptions({ + captions, + combineTokensWithinMilliseconds: SWITCH_CAPTIONS_EVERY_MS, + }); +}, [captions]); +``` + +## Rendering with Sequences + +Map over the pages and render each one in a ``. Calculate the start frame and duration from the page timing: + +```tsx +import { Sequence, useVideoConfig, AbsoluteFill } from "remotion"; +import type { TikTokPage } from "@remotion/captions"; + +const CaptionedContent: React.FC = () => { + const { fps } = useVideoConfig(); + + return ( + + {pages.map((page, index) => { + const nextPage = pages[index + 1] ?? null; + const startFrame = (page.startMs / 1000) * fps; + const endFrame = Math.min( + nextPage ? (nextPage.startMs / 1000) * fps : Infinity, + startFrame + (SWITCH_CAPTIONS_EVERY_MS / 1000) * fps, + ); + const durationInFrames = endFrame - startFrame; + + if (durationInFrames <= 0) { + return null; + } + + return ( + + + + ); + })} + + ); +}; +``` + +## White-space preservation + +The captions are whitespace sensitive. You should include spaces in the `text` field before each word. Use `whiteSpace: "pre"` to preserve the whitespace in the captions. + +## Separate component for captions + +Put captioning logic in a separate component. +Make a new file for it. + +## Word highlighting + +A caption page contains `tokens` which you can use to highlight the currently spoken word: + +```tsx +import { AbsoluteFill, useCurrentFrame, useVideoConfig } from "remotion"; +import type { TikTokPage } from "@remotion/captions"; + +const HIGHLIGHT_COLOR = "#39E508"; + +const CaptionPage: React.FC<{ page: TikTokPage }> = ({ page }) => { + const frame = useCurrentFrame(); + const { fps } = useVideoConfig(); + + // Current time relative to the start of the sequence + const currentTimeMs = (frame / fps) * 1000; + // Convert to absolute time by adding the page start + const absoluteTimeMs = page.startMs + currentTimeMs; + + return ( + +
+ {page.tokens.map((token) => { + const isActive = + token.fromMs <= absoluteTimeMs && token.toMs > absoluteTimeMs; + + return ( + + {token.text} + + ); + })} +
+
+ ); +}; +``` + +## Display captions alongside video content + +By default, put the captions alongside the video content, so the captions are in sync. +For each video, make a new captions JSON file. + +```tsx + + +``` diff --git a/.agents/skills/remotion-best-practices/remotion-captions/import-srt-captions.md b/.agents/skills/remotion-best-practices/remotion-captions/import-srt-captions.md new file mode 100644 index 0000000..333c962 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/import-srt-captions.md @@ -0,0 +1,69 @@ +--- +name: import-srt-captions +description: Importing .srt subtitle files into Remotion using @remotion/captions +metadata: + tags: captions, subtitles, srt, import, parse +--- + +# Importing .srt subtitles into Remotion + +If you have an existing `.srt` subtitle file, you can import it into Remotion using `parseSrt()` from `@remotion/captions`. + +If you don't have a .srt file, read [Transcribing audio](transcribe-captions.md) for how to generate captions instead. + +## Prerequisites + +First, the @remotion/captions package needs to be installed. +If it is not installed, use the following command: + +```bash +npx remotion add @remotion/captions # If project uses npm +bunx remotion add @remotion/captions # If project uses bun +yarn remotion add @remotion/captions # If project uses yarn +pnpm exec remotion add @remotion/captions # If project uses pnpm +``` + +## Reading an .srt file + +Use `staticFile()` to reference an `.srt` file in your `public` folder, then fetch and parse it: + +```tsx +import { useState, useEffect, useCallback } from "react"; +import { AbsoluteFill, staticFile, useDelayRender } from "remotion"; +import { parseSrt } from "@remotion/captions"; +import type { Caption } from "@remotion/captions"; + +export const MyComponent: React.FC = () => { + const [captions, setCaptions] = useState(null); + const { delayRender, continueRender, cancelRender } = useDelayRender(); + const [handle] = useState(() => delayRender()); + + const fetchCaptions = useCallback(async () => { + try { + const response = await fetch(staticFile("subtitles.srt")); + const text = await response.text(); + const { captions: parsed } = parseSrt({ input: text }); + setCaptions(parsed); + continueRender(handle); + } catch (e) { + cancelRender(e); + } + }, [continueRender, cancelRender, handle]); + + useEffect(() => { + fetchCaptions(); + }, [fetchCaptions]); + + if (!captions) { + return null; + } + + return {/* Use captions here */}; +}; +``` + +Remote URLs are also supported - you can `fetch()` a remote file via URL instead of using `staticFile()`. + +## Using imported captions + +Once parsed, the captions are in the `Caption` format and can be used with all `@remotion/captions` utilities. diff --git a/.agents/skills/remotion-best-practices/remotion-captions/transcribe-captions.md b/.agents/skills/remotion-best-practices/remotion-captions/transcribe-captions.md new file mode 100644 index 0000000..ba649d5 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-captions/transcribe-captions.md @@ -0,0 +1,70 @@ +--- +name: transcribe-captions +description: Transcribing audio to generate captions in Remotion +metadata: + tags: captions, transcribe, whisper, audio, speech-to-text +--- + +# Transcribing audio + +To transcribe audio to generate captions in Remotion, you can use the [`transcribe()`](https://www.remotion.dev/docs/install-whisper-cpp/transcribe) function from the [`@remotion/install-whisper-cpp`](https://www.remotion.dev/docs/install-whisper-cpp) package. + +## Prerequisites + +First, the @remotion/install-whisper-cpp package needs to be installed. +If it is not installed, use the following command: + +```bash +npx remotion add @remotion/install-whisper-cpp +``` + +## Transcribing + +Make a Node.js script to download Whisper.cpp and a model, and transcribe the audio. + +```ts +import path from "path"; +import { + downloadWhisperModel, + installWhisperCpp, + transcribe, + toCaptions, +} from "@remotion/install-whisper-cpp"; +import fs from "fs"; + +const to = path.join(process.cwd(), "whisper.cpp"); + +await installWhisperCpp({ + to, + version: "1.5.5", +}); + +await downloadWhisperModel({ + model: "medium.en", + folder: to, +}); + +// Convert the audio to a 16KHz wav file first if needed: +// import {execSync} from 'child_process'; +// execSync('ffmpeg -i /path/to/audio.mp4 -ar 16000 /path/to/audio.wav -y'); + +const whisperCppOutput = await transcribe({ + model: "medium.en", + whisperPath: to, + whisperCppVersion: "1.5.5", + inputPath: "/path/to/audio123.wav", + tokenLevelTimestamps: true, +}); + +// Optional: Apply our recommended postprocessing +const { captions } = toCaptions({ + whisperCppOutput, +}); + +// Write it to the public/ folder so it can be fetched from Remotion +fs.writeFileSync("captions123.json", JSON.stringify(captions, null, 2)); +``` + +Transcribe each clip individually and create multiple JSON files. + +See [Displaying captions](display-captions.md) for how to display the captions in Remotion. diff --git a/.agents/skills/remotion-best-practices/remotion-create/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-create/REFERENCE.md new file mode 100644 index 0000000..4ae3f65 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-create/REFERENCE.md @@ -0,0 +1,51 @@ +--- +name: remotion-create +description: Create a new Remotion video +metadata: + tags: remotion +--- + +These are instructions for making a new Remotion project and composition. +If this is not the next task, see [Remotion Best Practices](../SKILL.md) + +## Scaffold a project + +If a project already exists, skip this. +Ensure Node.js and Git is installed, and the current folder is appropriate for starting a new project. + +Scaffold one using: + +```bash +npx create-video@latest --yes --blank --no-tailwind my-video +cd my-video +npm i +``` + +Replace `my-video` with a suitable project name. + +## Designing a video + +Keep the scaffold and add React Markup. Follow [Remotion React Markup Best Practices](../remotion-markup/REFERENCE.md) and [Video Layout Rules](video-layout.md) for video-first layout and text sizing guidance. + +## Interactivity Best Practices + +By structuring the React Markup following [Remotion Interactivity Best Practices](../remotion-interactivity/REFERENCE.md), you allow the user to make edits in the Studio which write back to code. + +## TailwindCSS + +If Tailwind is requested, see [tailwind.md](tailwind.md) for using TailwindCSS in Remotion. + +## Starting preview + +```bash +npx remotion studio --no-open +``` + +This will start a long-running process and print the server URL for the preview. +If server is already started, it will print the URL. +You can visit a specific composition by navigating to `/[composition-id]`, for example `http://localhost:3000/MapAnimation`. + +## Follow-up + +The video creation process has finished. +For follow-up prompts, use [Remotion Best Practices](../SKILL.md) diff --git a/.agents/skills/remotion-best-practices/remotion-create/tailwind.md b/.agents/skills/remotion-best-practices/remotion-create/tailwind.md new file mode 100644 index 0000000..b902131 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-create/tailwind.md @@ -0,0 +1,11 @@ +--- +name: tailwind +description: Using TailwindCSS in Remotion. +metadata: +--- + +You can and should use TailwindCSS in Remotion, if TailwindCSS is installed in the project. + +Don't use `transition-*` or `animate-*` classes - always animate using the `useCurrentFrame()` hook. + +Tailwind must be installed and enabled first in a Remotion project - see https://www.remotion.dev/docs/tailwind. diff --git a/.agents/skills/remotion-best-practices/remotion-create/video-layout.md b/.agents/skills/remotion-best-practices/remotion-create/video-layout.md new file mode 100644 index 0000000..3e6bc43 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-create/video-layout.md @@ -0,0 +1,9 @@ +You are designing a video, not a webpage. + +- Decide what the viewer should notice first in each scene. Build the frame around that one thing. +- Keep important content inside a generous safe area. For 1080px-wide videos, keep key text at least 80px from the sides and 100px from the top and bottom. +- Do not add redundant elements. +- For 1080px-wide compositions, use these rough minimums: + - Main headline: 84px + - Important supporting text: 44px +- Scale those values with the composition width. diff --git a/.agents/skills/remotion-best-practices/remotion-docs/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-docs/REFERENCE.md new file mode 100644 index 0000000..35da8d2 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-docs/REFERENCE.md @@ -0,0 +1,47 @@ +--- +name: remotion-docs +description: Search Remotion documentation +metadata: + tags: remotion, docs, documentation, search +--- + +This skill teaches you how to discover and read current Remotion documentation. +If this is not relevant, load [Remotion Best Practices](../SKILL.md) instead. + +## Searching the docs + +Use the Algolia search API to find relevant documentation pages: + +``` +POST https://plsduol1ca-dsn.algolia.net/1/indexes/*/queries?x-algolia-api-key=3e42dbd4f895fe93ff5cf40d860c4a85&x-algolia-application-id=PLSDUOL1CA +Content-Type: application/x-www-form-urlencoded + +{ + "requests": [ + { + "query": "", + "indexName": "remotion", + "params": "attributesToRetrieve=[\"hierarchy.lvl0\",\"hierarchy.lvl1\",\"hierarchy.lvl2\",\"url\"]&hitsPerPage=10" + } + ] +} +``` + +Each hit contains a `url` field pointing to the documentation page. + +## Fetching a page as Markdown + +Append `.md` to any Remotion docs URL to retrieve its Markdown source (saves tokens): + +``` +https://www.remotion.dev/docs/use-video-config.md +https://www.remotion.dev/docs/sequence.md +https://www.remotion.dev/docs/lambda/rendermediaonlambda.md +``` + +## Workflow + +1. Search Algolia for the concept or API you need. +2. Pick the most relevant URL(s) from the results. +3. Fetch each URL with the `.md` suffix. +4. Implement using the current documentation rather than memorized API knowledge. diff --git a/.agents/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md b/.agents/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md new file mode 100644 index 0000000..74ea969 --- /dev/null +++ b/.agents/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md @@ -0,0 +1,226 @@ +--- +name: remotion-interactivity +description: Structure Remotion markup for interactivity +metadata: + tags: remotion, interactivity, studio, visual mode +--- + +By writing Remotion markup in a specific way, the Remotion Studio is able to recognize the structure of the code and makes it interactive: + +- Allowing items to be selected by clicking on them +- Allowing drag+drop, resizing and rotation +- Editing the CSS styles +- Making keyframes and easing values editable + +If the markup is too complex for the Studio to make it interactive, then the values become grayed out. + +## Make an HTML element interactive using `Interactive` + +Every HTML and SVG element such as `
` can be turned interactive using `Interactive`: + +```tsx title="Interactive elements" + + Hello + +``` + +This allows styles and keyframes to be set in the Studio. Be sensible, if a component has many elements, the timeline might get messy. + +## Give interactive elements a descriptive name + +Add a `name` prop to elements to make them easily identifyable. + +```tsx title="Interactive names" +<> + + Launch day + + +