Android apps will soon let you use your face to control your cursor

Developers can now integrate accessibility functionality into their apps, allowing users to control the cursor with facial gestures or by moving their head. For example, they can open their mouth to move the cursor or raise their eyebrows to click and drag.

Announced at last year’s Google I/O for desktop, the Gameface project uses the device’s camera and a database of facial expressions from MediaPipe’s Face Landmark Detection API to manipulate the cursor.

“Through the device’s camera, it seamlessly tracks facial expressions and head movements, translating them into intuitive, personalized control,” Google explained in its announcement. “Developers can now create apps where their users can configure their experience by customizing facial expressions, gesture sizes, cursor speed, and more. »

While Gameface was initially designed for gamers, Google says it has also partnered with Incluzza – an Indian social enterprise focused on accessibility – to see how they can extend it to other contexts like work, school and social situations.

Project Gameface was inspired by quadriplegic video game streamer Lance Carr, who has muscular dystrophy. Carr collaborated with Google on the project, aiming to create a more affordable and accessible alternative to expensive head tracking systems.