Every video, signed.

Most video online has no sign language at all. SignLang.ai is an early prototype that adds an AI-generated American Sign Language interpreter to any video

Drag and drop a video or audio file here, or click to choose one

Tech stack

A typed React front end on Cloudflare, backed by a video-generation pipeline that runs locally on one consumer GPU (RTX 3070, 8 GB). No cloud APIs, no per-minute cost.

Front end

  • React 18 with TypeScript 5 (strict type-checking in every build)
  • Vite 5 bundler with a custom server-side pre-rendering step, hydrated on load
  • Static output served from Cloudflare Pages, no runtime server
  • Responsive, light/dark theming in plain CSS, accessible drag-and-drop upload UI

Video generation

  • Wan-Animate 2 (14B, distilled, int8) in ComfyUI 0.39
  • LightX2V step-distill LoRA for fast sampling
  • Wan 2.1 VAE, UMT5-XXL and CLIP-Vision encoders
  • PyTorch 2.5.1 + CUDA 12.1
  • One reference photo defines the signer

Sign motion pipeline

  • Python 3.10 orchestration scripts driving the ComfyUI API
  • OpenCV, NumPy, Pillow for analysis and frame work
  • Overlapping 49-frame chunks with cross-fades for long signs

Video assembly

  • FFmpeg 9: crop, scale, cross-fade, concat, overlay
  • Speech-onset detection from the audio's voice band, so each line starts on its first spoken word
  • Generated idle loop (breathing and blinking) for the pauses
  • Signing sped to 1.3x to reduce lag against speech

Speech and gloss app

  • Java 21, Spring Boot 3.5, Gradle 8.5 (Kotlin DSL)
  • Vosk offline speech-to-text
  • Local LLM through Ollama (Llama 3.1 8B) for English-to-ASL gloss
  • Pure-Java MP3 decoding

Demo: the NASA Perseverance Mars landing with an AI-generated ASL inset (bottom right). Video footage: NASA/JPL-Caltech.

Why it matters

Captions help many people, but for many Deaf people ASL is the first language, and captions are a second-language experience. Hiring interpreters for every video isn't possible, so almost no video has ASL. The goal here is to make that the default instead of the exception.

How the prototype works

1. Transcript to gloss

The spoken script is converted to ASL gloss, the written form of ASL word order.

2. Generated signer

A video model renders a consistent signer from a single reference photo.

3. Timed to the video

Signing starts when the first word is spoken, with a natural idle pose between lines.

What's next

  • Smoother transitions between signs.
  • Review and correction of generated signs.
  • A larger sign vocabulary.
  • Better timing against the spoken audio.
  • Expand to multiple signing languages.

About Ryan Dollar

Senior Full Stack Software Engineer

SignLang.ai is a personal project that explores how generative video can make sign language available on videos that would never get a human interpreter.

I built every layer myself: the typed React front end you are looking at, the Java speech and gloss app, and the Python pipeline that drives video generation on a single consumer GPU.

Contact