AR and VR Create Remote Intelligence for Industrial Workers

• Augmenting Training with Smartphone App
• Virtual Inspections from Across the World
• Addressing Worker Shortages and Reducing Carbon Footprint

Jul 08, 2022The challenge is as old as industrial work: the person with the ability, knowledge and experience to fix a problem, or to provide instructions, is not physically there. With that in mind, workforce solutions technology company Librestream is providing augmented reality (AR) and virtual reality (VR) solutions to businesses, such as Tesco supermarkets in the United Kingdom, to manage remote inspections and maintenance procedures, using only a smartphone and the company's software-as-a-service (SaaS).

Companies that have facilities around the world, or that have field-based maintenance sites, are leveraging the technology to provide training enhancements for workers who may be new to the job or are unfamiliar with specific issues in front of them. In addition to a smartphone that can connect operators with relevant training or maintenance content, the solution can be used with connected, wearable devices. In either case, Librestream reports, the goal is to automatically arm individuals onsite at a work area with the knowledge they need.

Librestream's Onsight augmented worker platform has been gaining traction in the past few years, the company explains, as trained personnel are harder to come by and turnover rates increase. At the same time, its customers are attempting to reduce energy costs, leading them to seek a solution that can reduce how frequently trained personnel must travel to specific sites to inspect or maintain equipment. The Onsight solution consists of cloud-based software and an app that can be used on an individual's Android- or iOS-based phone to capture information regarding what is in front of them—whether they are at an assembly site workstation, in front of a piece of equipment on an offshore oil rig or maintaining an elevator.

The company calls this an augmented reality approach to manufacturing, maintenance and inspection. It can either link an operator to an expert or provide solutions automatically based on a person's location, that individual's experience level and other variables. Wherever users may be, the software can provide them with information specific to their location, then prompt actions. The system typically leverages whatever connectivity is available at a given site, whether it be 5G, Wi-Fi or another network.

Librestream has been providing solutions to industrial workers for the past two decades, according to Charlie Neagoy, the firm's senior VP of customer success. "We've had really one goal from the beginning," he says, "to ensure that industrial workers have the information they need at the moment they need it, wherever they are in the world."

Augmenting Training with Smartphone App

Most of Librestream's customers are Fortune 1000 companies with geographically distributed operations and assets. The company connects those at worksites with its experts, traditionally via a live call over a wireless network. With an increasingly global economy, and considering the manual labor shortages and pandemic quarantines of the past few years, the firm has seen an acceleration in the need for connectivity to experts, to address problems quickly and accurately.

Librestream offers a variety of artificial intelligence (AI) and AR technologies designed to bring individuals the help they require, even without a conversation with a subject-matter expert. Twenty years ago, the company, originating in Winnipeg, built the first version of this solution: a mobile collaboration system that could connect industrial workers with management, engineers or other experts.

Librestream's engineers saw the value after being acquired by Motorola. Once the firm was part of a larger company, with facilities in Taiwan, engineers sometimes had to fly to across the world to review or resolve challenges on a production line. "They would fly over there," Neagoy says, "and might spend about a half hour on the production line and solve whatever the issue was, then they fly back to Canada."

The company designed a camera-based solution to capture images and video on a production floor and forward that to engineers in Winnipeg. These days, the camera portion of the solution is ubiquitous, Neagoy says, since most people have their own pocket-size camera in the form of a smartphone. Approximately 90 percent of the Onsight solution's endpoints are smartphones, he adds. This is not the environment, however, for Zoom or Microsoft Teams conversations.

"Our customers don't operate in a carpeted space," Neagoy notes. "It's concrete. It's dirt. It's water. It's mud." And many sites, such as offshore oil platforms, have challenging network conditions. "We had to come up with ways to be able to handle a lot of that streaming payload, and in these really compromised network environments." One key tenet of the business, he says, is to provide a robust network that enables reliable access to data at industrial sites.

Virtual Inspections from Across the World

The Librestream software leverages data from a phone, such as its GPS location, along with camera images or video capture, to gather the necessary information. For instance, with AI technology, the system can identify a part on a piece of equipment, as well as what might have broken on that equipment. While Librestream was providing connectivity between subject-matter experts and trainees or maintenance operators in 2003, the software can now both gather data and provide potential solutions.

In the case of Tesco, the supermarket chain needed a way for auditors to view conditions at a variety of remote locations, such as food processing plants. The company first adopted the Onsight platform in 2020, and its auditors now use the system to conduct immersive virtual audits at remote locations, while workers leverage their smartphones or wearable devices.

To conduct an audit, plant or floor workers use their own mobile device, running the Onsight app, and follow instructions to capture images and video for an auditor. The auditor can complete a thorough audit of a product or process by marking up the video and images for clarity, Neagoy explains, and can take control of the camera during the audit process to adjust lighting and zoom in. The data captured from this audit is automatically saved to a centralized software platform known as the "knowledge base," where it can be shared with relevant team members.

This remote auditing technology with AR has made it easier for Tesco to collaborate with remote employees and suppliers, and it has helped to ensure full audits are conducted throughout the pandemic. Overall efficiency has improved as a result, Neagoy reports. For example, he says, one Tesco auditor completed virtual site visits in three different countries within a single day: a first product launch audit in Asia, a packaging approval in Ireland and a hygiene inspection in Spain. "Without Onsight," he states, "these audits would have taken the better part of two weeks," given the travel required.

Addressing Worker Shortages and Reducing Carbon Footprint

The common thread for companies coming to Librestream is that time is extremely valuable, Neagoy says. There's a demographic shift underway in which the most knowledgeable and experienced workers in many industries are approaching retirement age. "Our most knowledgeable people, who may have been with a company for 35, 40 years, are walking out the door for the last time," he notes, "and we're losing all that knowledge." The Onsight solution is designed to help businesses transfer that knowledge to new hires in their 20s, even if they leave the company within a few years.

When it comes to assisting workers, the goal is not to provide training, but rather to augment it. For instance, elevator service providers could use the collected data as a tool to identify maintenance or repair issues at an elevator site. For such companies, Neagoy says, the benefit is a faster repair of an engine or other critical equipment. If a maintenance company needs to send another worker to the site potentially the next day, he adds, "Well, that engine is now non-functional for 24 hours."

Going forward, the company is migrating Onsight to be more automated. If individuals are using a computer vision component, for instance, the system can capture the ID number on a particular piece of equipment, based on a smartphone image, and the software can then identify the part and provide content accordingly. It can also employ machine learning to leverage analytics for maintenance purposes, such as the need to replace the motherboard in a specific vending machine, based on its lifetime or the typical lifespan of similar vending machine motherboards.

"Our customers do roughly 8,000 remote inspections a week," Neagoy states, "on everything from goods being shipped out of factories in China to livestock and farm facilities that go to some large grocery retailers to effectively [collect data for] insurance claims."

Key Takeaways:

• Librestream's Onsight app and software platform provide access to data and experts for workers around the world or in remote locations.
• With AR and VR, the data being collected can provide intelligence enabling users to predict events and make recommendations, such as replacing a part before it fails.