experiment2026
Monkey Gesture Detector
A real-time webcam experiment where I map face + hand gestures to live meme states
Overview
I built this in-browser gesture detector using MediaPipe Face Landmarker + Hand Landmarker to read facial expressions and hand poses frame by frame, then trigger meme states instantly. The biggest focus was making it feel smooth in real life, so I added confidence thresholds, gesture history smoothing, no-hand timing logic, and priority-based state transitions to cut false triggers and flicker.
What I Learned
- How to run a multi-model CV pipeline in the browser (face + hand) without killing responsiveness.
- How to turn raw landmarks into practical gesture rules for tongue-out, fist, middle finger, mouth-open, and two-hand detection.
- How much temporal smoothing matters: gesture history + no-hand frame thresholds made the UX way more stable.
- How to tune real-world detection with confidence gating, fallback logic, and live debug overlays.
- How to structure experimental code so it is still clean, modular, and easy to iterate on quickly.
Stack & Architecture
Frontend
HTML5 CanvasVanilla JavaScriptMediaPipe Tasks Vision
Backend
WebRTC getUserMediaFace LandmarkerHand Landmarker
Infra
GitHub PagesGPU Delegate (Web)Client-Side CV Pipeline