STMicroelectronics Offers Gesture Recognition for Time of Flight

• Pixelized Data Provides Precise Location
• Adjusting Thermostats, Stoves and Other Machinery
• Touch-Free Feature Reduces Infection Transmission

May 17, 2022Global electronic circuits firm STMicroelectronics has released a new feature for its time-of-flight (ToF) sensor that detects gestures in mid-air, such as a wave, a swipe or a tap, thereby enabling users to operate devices without touching them. The STGesture ToF recognition solution works with STMicroelectronics' VL53L5CX sensor, and several appliance, gaming and equipment manufacturers are testing or deploying the technology. According to the company, the gesture-recognition solution offers an alternative to solutions designed to recognize individuals and what they want done.

Most commonly, touchscreens are used for everything from phone access or laptop operation to viewing content on screens at stores or transit stations. Near Field Communication (NFC), ultrahigh-frequency (UHF) RFID, Bluetooth Low Energy (BLE) and ultra-wideband (UWB) technologies capture data regarding the identity of a person or thing, along with location data, and respond according to presence. Facial recognition and voice recognition are other options, though STMicroelectronics notes that though these can be complex.

Touchscreens in public places pose the challenge of requiring a physical touch, which STMicroelectronics says could pose infections risks. RFID technologies require a device that communicates with a sensor, thereby creating an additional expense. Throughout the past three years, the company's engineers have thus been developing a solution using its existing ToF optical sensors that could identify where a person or object was located, while adding gesture recognition. The VL53L5CX, released in 2021 (see STMicroelectronics Intros Time-of-Flight Multi-Zone Ranging Sensor), is a multizone ranging sensor with the company's ToF-enabled FlightSense technology.

Time of flight provides an alternative to traditional infrared (IR) sensors by detecting the presence of an object or person, then calculating distance from the sensor. With ToF, a sensor generally transmits optical photons periodically. When they hit a target, the photons at that specific spot are bounced back. The sensor receives that response and can calculate the time at which the light energy was returned, and thus how far away it is. With STGesture, the same sensor can also detect hand movements and convert that information into a computer command, according to Thomas Viart, STMicroelectronics' marketing engineer.

Pixelized Data Provides Precise Location

The sensor is an 8-zone by 8-zone matrix. Each zone or pixel (there are 64 in total) provides its own distance value so the system can understand the location at which a particular person or object is located in front of it. The sensor software can watch the object's or person's movement, such as a hand moving from one pixel to another, throughout the zones. A gesture can be detected at a distance of up to 50 centimeters (19.7 inches).

The sensor is relatively low-cost, Viart reports, and it leverages algorithms that provide that location data. With the ToF responses, the device can track a hand's movement at either slow or fast speed, based on a variety of motions including from left to right, or from top to bottom, as well as further or closer to the sensor. Users could thus perform specific gestures that have meaning to a device. Consider a typical left-to-right swipe, or double-taps with an open hand, never touching the sensor or device.

Adjusting Thermostats, Stoves and Other Machinery

In a real-world application, this could mean controlling an office's temperature settings, for example. The thermostat, equipped with the gesture-recognition feature, could be awoken via a hand gesture, then raise or lower the temperature based on the hand's specific movement. If an individual simply walked past the sensor, the algorithm would disregard that event.

Another use case could involve displaying content, such as a map, on a screen in a public place. Users could hold a hand above or in front of the screen to wake it up, then move the hand according to instructions to turn a page, select a tab, or zoom pages larger or smaller, without using a touchscreen. They could also browse through maps, pictures or videos to gain information without increasing the risk of transmitting or being exposed to an infection.

The system could be employed in industrial, medical and automotive spaces, the company notes, at which handling machinery or screens might be unsafe or unsanitary. In some cases, moving parts, cold temperatures, or the presence of grease or dirt makes it unrealistic for workers to touch screens. The gesture-recognition software could change the operation mode of machinery, turn equipment on or off, or adjust settings.

In a kitchen, the solution could be deployed for ovens, stovetops or other appliances. Individuals could use the technology in their homes by waving or gesturing instructions to their oven without having to touch the controls with dirty hands. "We have also seen a huge ask from the entertainment industry," Viart states, "where controlling your robotic toys or having [them] mimic your gestures became another common use case."

In addition, the system could enable gaming options allowing users to reliably understand movements without requiring gamers to hold controllers. To confirm an individual's presence, the solution typically would transmit ToF data periodically, such as every second, and if it detected an obstacle in front of it for several seconds, it would activate the screen and begin detecting gesture motions.

Touch-Free Feature Reduces Infection Transmission

For screens used in public places, the solution offers a benefit that has been in greater demand since the COVID-19 pandemic began. For instance, it could allow for a touchless soda machine, with customers placing a cup near a sensor in the machine to activate a pour. This sensor could detect the shape and position of the cup, based on the pixelated ToF data. The gesture-recognition sensor built into the machine would enable users to select a drink and prompt the pour without touching it.

The system has a few limitations, Viart notes. It does not require light, though he notes that ambient daylight can affect the recognition functionality. It does not identify a specific individual in the way RFID could, he says, but that feature offers a privacy benefit. Multiple use cases have been identified since the technology was announced, Viart reports, adding, "We don't know exactly which kinds of companies we can best target yet, because it's quite new. Until today, it was very difficult to achieve [gesture detection] with high accuracy."

According to Viart, the solution operates with less power than optical systems since it can go dormant upon detecting no one within its vicinity. "I'm always surprised to see a public screen that is always on," he says, "consuming a lot of power," with or without an audience. When the STGesture system is in use, Viart says, "It's very accurate and very robust, with about 95 percent of accuracy factor." Machines using the system typically would include a touch point or button for those who require it, but it could be a complimentary option.

Key Takeaways:

• Gesture recognition will enable people to operate equipment and screens without physically touching them, reducing safety or hygiene hazards.
• Equipment manufacturers are now sampling the system for integrating into everything from games to public displays and building thermostats.

Exhibitors at RFID Journal LIVE! 2022 will offer solutions for time of flight and other use cases. To learn more, visit the event's website.