Qiongting Zhang

In A Blink


Seeing Machines
openFrameworks / Unity / Python / Machine Learning
Ongoing, New York.

OVERVIEW


In a Blink is an interactive movie for two players. The sole method of interaction is through blinking, with each player offering a distinct viewpoint of the story. The screen displays the perspective of the most recent blinker, and the storyline unfolds with each blink.


RESEARCH AND PROCESS


Eye Blink Detection


I customized the eye blink detection code by incorporating functionalities from this GitHub repository. It uses OpenCV and DLIB libraries to implement an eye blink detection system. The method involved calculating the Eye Aspect Ratio (EAR) to determine when a blink occurs.

EAR Equation


Open Eye Has Larger EAR Close Eye Has Less EAR


Test


Face Recognition


The previous blink recognition system successfully met the desired results when analyzing a single face. However, when testing the system with two individuals, it became apparent that using OpenCV on OpenFrameworks allowed for recognizing multiple faces and labeling them based on the order of detection. Despite this capability, the recognition process proved to be unstable during gameplay, as brief lapses in facial tracking caused player numbering to change.

To resolve this issue and ensure distinction in the game's perspective for each player, I incorporated facial recognition to identify the individual players. At the outset of the game, both players will face the camera and provide approximately 5 seconds (200 frames) of consecutively captured images while slowly rotating their heads. These images will serve as training data for a classification model, enabling it to discern between the players in subsequent gaming sessions. By implementing facial recognition, the program will maintain a clear distinction between the players, regardless of any momentary tracking interruptions that may occur during gameplay.

Face Recognition Demo Adapted Based On This GiuHub Repository


Connect OpenFrameworks With Python And Unity


When any player blinks, openFrameworks will send a message indicate which persome just blinked to Unity with KlakSyphon. 

For now, I am using camera movement in Unity as a test reaction to evaluate this feature. Whenever either player blinks, the Unity application accurately detects the signal and responds by smoothly moving the camera towards the respective player's direction.


Preliminary Test Result


Story Creation


Wait to be added.


Applications Used:
  • openFrameworks
  • Unity
  • Blender

Skills Used:
  • Machine Learning (Face Recognition, Blink Recognition with OpenCV)
  • Coding (C++, Python, C#)