01WhatA rhythm game you play with one finger.
As a video game player, I think the most interactive games are rhythm games, so I set out to build one that uses a camera as the controller. It borrows the play mode of Beat Saber: bricks with arrows appear, and you hit each one in the direction of its arrow, on the beat. Instead of a VR headset and two sabers, the game tracks your hand and turns your finger into the lightsaber.
02HowCamera in, score out.
Every frame, OpenCV grabs an image from the camera and MediaPipe finds the hand. For each hand the game checks for collisions with the bricks on screen, updates the bricks, records the score and moves on to the next frame. The game itself is built with Pygame around three small classes:
- Button — the menu buttons: position, texture, and whether it was clicked.
- Brick — a note. It lives for one second, then removes itself, and changes texture depending on when it was hit.
- Sheet — the music sheet: a factory that knows where each brick appears and when.
- ± 0.4 s
- Perfect — green check, high score
- ± 0.8 s
- Good — blue, middle score
- > 1 s
- Miss — the brick self-destructs
- wrong way
- Red cross
03The trickInvisible bricks.
A webcam delivers separate frames, so a fast finger can jump across a brick's edge between two frames and the crossing is never seen. Long ago I heard about the invisible bunnies in World of Warcraft: untargeted spells are actually cast on a hidden target. I did the same with bricks. An invisible brick sits next to the correct edge of each target, so the boundary becomes an area instead of a line. If the finger touched the invisible brick on the previous frame, the hit came from the right direction.
To make it feel like Beat Saber, a lightsaber texture rides on the index finger: two MediaPipe hand landmarks and an arctan give the angle to rotate it.
04NextWhat I'd improve.
- Direction detection has an edge case: if one finger rests on the hidden brick, another finger can hit the target from any direction.
- I designed the “fast” direction check on intuition — it still needs measuring.
- Trade a little display delay for more frame time, to find the right balance between latency and FPS.
Adapted from my original write-up · more in the links above.