Hunting Downtime: A Submariner’s Perspective on Predictive Maintenance

What can a nuclear submarine teach today's manufacturers in their quest to avoid costly downtime?
Published: June 12, 2019

“Dive! Dive!” Two simple words, punctuated with the familiar “ah-OOG-ah” alarm, were the signal throughout the ship that we were about to embark on another adventure under the sea. For many sailors, this quickly became routine, but in reality, there is nothing routine about submerging an 18,000-ton, 560-foot vessel like the USS Nevada (SSBN 733), containing 165 people, hundreds of feet beneath the surface of the ocean. One small defect or fault in any of dozens of systems, or a single valve out of position, and our crew would find itself on a one-way trip to the ocean floor.

Remarkably, since the U.S. Navy’s nuclear submarine program began, only two subs have been lost at sea, the last being the USS Scorpion (SSN 589), which sank in 1968, more than 50 years ago. This kind of uptime record is worth investigating. Let’s see what a nuclear submarine can teach today’s manufacturers in their quest to avoid costly downtime.

In the mid-1990s, I served as an officer onboard the Ohio-class nuclear submarine USS Nevada, before I began my career in manufacturing. In recent years, preventing downtime has created abundant interest in utilizing technology to make better maintenance decisions. At the heart of this technology are the fundamental principles of predictive maintenance: gathering data, analyzing it, predicting failures and taking proactive measures to prevent downtime. These fundamental principles are unchanged from how we operated the Nevada 20 years ago.

A crew of 165 people living in an underwater vessel for months at a time amplifies the meaning of “hazardous conditions.” Consider, for a moment, a transmission bearing failure causing a loss of propulsion, or a reactor coolant pump malfunction causing a meltdown, or a valve failure flooding the vessel with seawater. Clearly, nuclear submarine downtime costs lives, not just profits.

The U.S. Navy has never had a death aboard a U.S. submarine due to a radiation accident. With all the complexities of multiple systems crammed into a tiny space, operating in harsh environments, how have we maintained such an amazing safety record?

The two submarine tragedies in the 1960s resulted in the Navy’s SUBSAFE (Submarine Safety) program, which covers all systems exposed to sea pressure or that are critical to flooding recovery. The program was so successful that it was implemented throughout NASA following the Space Shuttle Columbia disaster. SUBSAFE and the Navy’s Nuclear Power program are based on a foundation of quality in design, material, fabrication and testing. They span the submarine’s life, from initial design and construction through ongoing maintenance and updates. Strict adherence to these programs has ensured safe and successful missions for half a century.

But for the most part, sailors don’t think about all the factors that went into the design and construction of the submarine on which they serve. We were far more focused on keeping it operating safely and efficiently. Navy submarines are always in a hostile environment. Beyond facing the potential threat of attack from another nation, the ship is also completely surrounded by saltwater and storms at sea, which can be extremely hazardous—not to mention the explosive ordnance being carried onboard. When you are dependent on every system running smoothly to harvest your oxygen and freshwater, and generally keep water out of the “people tank,” downtime must be hunted down and eliminated.

We used a variety of different maintenance practices to ensure everything worked at peak performance, including weekly and monthly preventative maintenance (PM) inspections and check-off sheets for routine items like oil filter changes or motor-generator brush repair. The principle here is to perform the maintenance at a specific interval, whether or not it needs to be done. When setup properly, this can be an effective, though conservative (and potentially wasteful) approach to carrying out maintenance.

However, there is no guarantee that just because you did the weekly or monthly PM, nothing would go wrong between the cycles resulting in a system failure. So we added another feature into our daily operations: ongoing data collection and trend analysis. This is essentially how predictive maintenance works in industry today: continuously evaluating how your equipment is working so you know when and how to maintain it optimally.

We had a team of 10 trained personnel who were responsible for operating the reactor plant, electrical and propulsion systems, 24 hours a day all the time we were under way. This team worked together in six-hour shifts (watches). If an issue or complex process arose, the team was there to address it, often with the help of another team of 10 who were off-duty. This team would record the readings of all the hundreds of gauges, dials and indicators throughout the engine room at least every hour—and, in some cases, every 15 minutes.

The watch team supervisors were expected to review these log readings every three hours, and the division officer (DO) and engineering officer (ENG) would review them daily. At first, this vast collection of numbers was overwhelming, but with experience the team gained a sense of what was normal and could quickly identify readings that signaled something amiss.

A key example of this is the bearing temperatures and oil flow bubblers of the main reduction gears. The reduction gears take the power generated by steam turbines turning at thousands of revolutions per minute and use it to drive the propulsion shaft at much slower speeds, which turns the screw and propels the submarine through the water. These gears are machined to extremely precise measurements and are carrying a lot of power. They are massive and can only be replaced in an extended dry-dock period, requiring that the submarine’s hull be cut open. They are lubricated and cooled in a continuous flow of purified oil.

If something were to happen to contaminate or stop the flow of oil, the gears could break or seize together, effectively shutting down the submarine’s propulsion system and rendering us dead in the water. There are dozens of bubbler sight glasses on the reduction gears that visually show both the flow and condition of the oil. These are recorded every hour, along with bearing temperatures to make sure that the reduction gears are being properly lubricated at all times. Using trend analysis to find a problem in this system before it becomes down-time could determine whether or not we make it back home.

What was the point of writing down and reviewing all these numbers? There are multiple things that come about by doing this.

• The operator writing them down has to look at the gauges every hour (or 15 minutes). This ensures that someone is paying attention. If the numbers are way out of their expected position, the operators would see that something was wrong before a long period of time passed, enabling them to respond quickly.

• The supervisors are looking to ensure that this is being done, but more importantly, they are looking at trends. What is changing over the course of the six-hour watch period, or the last day? Is there a bearing temperature that is slowly rising? Is something changing with the electrical system? Is pressure in the reactor coolant system changing? Is the temperature in the condenser rising? Do these changes match what I expect based on changing plant conditions?

• The DO and ENG are looking at longer trends to see that the plant is operating normally. Additionally, regular analysis is performed on the chemistry of the reactor coolant and radiation readings throughout the ship to verify that nothing unusual is happening in the reactor vessel.

• We used this data collection as a training tool as well. Occasionally, the ENG would replace normal log sets with artificial ones (for a drill or test) that had anomalies or other indications that something was happening. The watch team would then be evaluated on their ability to detect the problem and respond in the correct way.

• While this was a very manual process for collecting and analyzing data, it is a rigorous and detailed system to evaluate what is going on in the entire engine room. While some of this labor-intensive version of predictive maintenance is being replaced with today’s digital technology, the core principles remain unchanged. The USS Nevada was built in the mid-1980s. Newer submarines like the Virginia-class feature electronic logs. The log-taking process remains fundamentally the same, with watchstanders manually recording the data throughout the engine room on PDAs, which makes it instantly available for review by the supervisors. Some critical reactor plant readings are collected automatically and can be graphed to better identify trends.

While there is a clear case for predictive maintenance on life-critical systems like those on a nuclear submarine, modern manufacturing facilities also need this capability. For decades, companies have introduced process improvements like Lean Six Sigma and just-in-time manufacturing to streamline their operations, improve quality and minimize expenses, thus becoming more efficient and profitable. But maintenance has always lagged behind operations in receiving improvements. Many major manufacturers still rely on scheduled maintenance or run-to-failure approaches to managing maintenance. This strategy can result in maintenance resources being wasted by employees carrying out unnecessary repairs, or significant costly downtime if unexpected failures occur.

Only recently have companies started to consider smarter ways of performing predictive maintenance. The strongest trend in this area is to use data to develop indications and predictions of failure before they occur, which is exactly what we were doing with the data logs on the submarine. These days, this is done with sensors that are linked to an analytics platform which often resides in the cloud or on local servers. This data can then be evaluated and used to identify changes or anomalies that could be precursors to failures. This linking of data and machines is what is being referred to as the Industrial Internet of Things (IIoT) or Industry 4.0.

Technology has now reached a point at which remote condition monitoring and analysis can be done affordably and reliably. Rather than having an army of personnel manually recording gauge readings every hour and taking the time to study the data looking for trends, now this can be accomplished via sensors, machine learning and analytics. More advanced sensors now exist that enable the detection of changes in vibration that could indicate early symptoms of a future problem. These devices can also now communicate their findings directly to anyone who needs to know and take appropriate actions. Computing power continues to grow by leaps and bounds, enabling better and faster analysis and decision-making.

It’s been more than 20 years since I deployed on a nuclear submarine, but I imagine that the U.S. Navy is taking advantage of this new technology and data to make smarter, timelier decisions in maintaining and operating their equipment, and the same benefits can be applied to manufacturing plants. Predictive maintenance will bring huge dividends for everyone, even those who don’t live on a submarine.

Bryan Van Itallie truly spent time behind the wheel of an Ohio-class nuclear submarine. After obtaining a bachelor’s degree in aerospace engineering from the University of Colorado, Bryan joined the U,S, Navy and served aboard submarines. Leaving the Navy as a Lt. Commander, he launched into a career in the industrial sector. Along the way, he led teams in building processes, improving output and employing state-of-the-art manufacturing. Bryan earned an MBA degree from Duke University, as well as certificates from Northwestern and other institutions. Today, he serves as the COO of Grace Engineered Products, located in Davenport, Iowa.