link：https://editor.p5js.org/yz10444/sketches/PLT-XwpFH
https://www.youtube.com/watch?v=1OpCWmf7aqA
This project is a real-time interactive machine learning sketch using p5.js and ml5.js. It trains a custom neural network to recognize five distinct hand gestures. Once trained, it acts as an AR filter, overlaying a corresponding Emoji directly onto the center of the user's palm in real-time.
Instead of relying on clunky mouse clicks on DOM buttons to collect data, the user simply holds a gesture in front of the webcam and presses keyboard keys1-5to rapidly stream training data. The training process is intuitively triggered by pressing T.
The sketch extracts all 21 keypoints generated by the hand tracking model. To make the neural network more robust and translation-invariant (so it works no matter where the hand is on the screen), it calculates the relative x and y distance of each keypoint from the wrist (keypoint 0).
The model is trained to classify 5 distinct categories simultaneously: Finger Heart (🫰🏻), Fireworks (🎆), Fist (👊🏻), Check/Hook (✅), and Peace Sign (✌🏻).
The sketch features a custom Heads-Up Display. The UI panel provides real-time feedback on the data collection count per category and the current system state (Collecting, Training, Predicting). The final prediction result is intuitively displayed as a large AR Emoji floating directly over the user's hand.