Companies struggling with the large volumes of data their computer systems generate are turning to a new crop of software that uses algorithms to analyze their IT operations. It's a decidedly big data approach to the challenge of finding and fixing technology glitches that hamper service production.
Cardinal Health, a Fortune 21 health products company generating $125 billion in revenues across 60 countries, knows this scenario all too well. A glut of acquisitions had rendered Cardinal's technology footprint so broad that it was blind to problems plaguing its legions of servers and software systems. And while Cardinal gauged system health with monitoring tools and domain experts, it struggled with prolonged system outages because the volume and velocity of data produced made it impossible to get a holistic representation of IT operations.
"There are certain problems and patterns that we just aren't able to see with that approach," says Scot Lindsey, the company's senior vice president of enterprise IT shared services. "That bigger picture view is hard to see at our scale and complexity."
Cardinal turned to Splunk, whose algorithms are trained to analyze and correlate log files generated by computer systems to pinpoint the root cause of performance problems, providing IT a more holistic view of its IT operations."Splunk crunches huge amounts of data and presents the pattern to you," Lindsey says.
Embracing IT operations management as a big data problem to be analyzed with algorithmic software is a trend that's playing out across several industries, says Gartner analyst Colin Fletcher. He says that previous IT operations tools have been so coupled to domain-specific data collection that they were largely unaware of what was happening outside the systems they were built to monitor. "You can't make the same assumptions that you did in the past around what is relevant data, where will it come from and what are the skills of people I need to answer questions," Fletcher says.
Companies are turning to algorithm-based analytics software from the likes of Splunk and Sumo Logic, among other vendors. Fletcher expects that by 2020 50 percent of enterprises will use such platforms to provide insight into business execution and IT operations, an increase from fewer than 10 percent today.
Analyzing machine data optimizes performance
Cardinal first used the software three years ago to monitor log files created by its ecommerce software, which generates the bulk of its revenues. This platform daily endures what Lindsey calls a "Black Friday event," as healthcare providers rush to order medical and pharmaceutical products before the cut-off deadline to have it shipped to them the next day. Cardinal identified performance errors and made corrections to ensure that orders were processed and shipped in a timely fashion.
Expanding Splunk to other business lines paid dividends. The software found a curious error in the home health unit, which sells diabetes test strips and other aids. Warehouse workers receive orders electronically and scan merchandise with a scanner attached to their wrist before putting it into a tote to be packaged and shipped.
The company noted that its unit's warehouse management application was throwing off more than 500,000 "null scan" errors daily. Cardinal checked with the software vendor, which told it that the error occurs when a warehouse worker incorrectly scans a bar code on one of the packages it is supposed to ship. But a check of Splunk revealed that it was actually a software glitch that had been delaying shipments for years. "We had gotten used to the performance in the warehouse because we didn't know any better and because we couldn't see the error being meaningful," Lindsey says. The software vendor corrected the error and Cardinal's warehouse operations improved.
Splunk also helps with on-boarding new customers. When Cardinal's medical business scored a major new account, the IT staff used Splunk to identify potential problems that could arise as a result of the resulting 5 percent to 10 percent spike in order volume from the new customer. Lindsey says Splunk detected potential master data issues, which enabled Cardinal and the new customer to huddle in a "war room" to remediate the issue.
AWS, Splunk ninjas form formidable one-two punch
Cardinal hosts its Splunk implementation at Amazon Web Services, part of the company's effort to offload more functionality to public cloud software, Lindsey says. Splunk’s software aggregates, compresses and encrypts Cardinal log files before shipping them to AWS' servers.
Lindsey credits Cardinal's success to cultivating "Splunk ninjas," essentially IT staff with cross-functional backgrounds in hardware, database, middleware and programming backgrounds who have mastered the software. These ninjas partner closely with business stakeholders, teaching them how to use the technology.
Moreover, Lindsey says the approach has prompted other leaders in other business units to ask for Splunk, whose customers include Dunkin Donuts, Carnival Cruise Line, DirecTV and MasterCard. Gartner’s Fletcher says this is often the case when businesses find value in software that improves operations. "It unlocks a lot of creativity," Fletcher says.
Cardinal considers Splunk a key tool in its bid to refashion its IT architecture. "The closer you are to the system and the business process, the faster you'll get the insights and the more focused you'll be on getting down to the real value," Lindsey said.