How IBM Is Using Watson And An Innovative Workspace To Crunch Big Data For Big Solutions

(Originally published at Fast Co.Exist on March 11, 2014.)

IBM loves Big Data. The bigger it gets, the more servers, storage, and services Big Blue would like to sell you (a lot more, please). But the volumes involved have already grown so big that IBM’s own researchers struggle to get a handle on it.

Last year, for example, IBM fellow Laura Haas asked one of her colleagues at the company’s Almaden research center in Silicon Valley why he wasn’t using bigger data sets. Because, he replied, it takes 80% of my time just to prep the data I have. Haas realized that the more IBM’s research agenda was consumed by analytics, the more time and energy its experts would spend struggling with expanding data sets, slowing down the pace of discovery.

The obvious thing was to hand the volumes in question over to dedicated data scientists, but removing researchers from the loop would only make things worse. Plus, it seemed to cut against the grain of Big Data, whose value isn’t governed by some function of Moore’s Law or Kryder’s Law in terms of the linear expansion of storage capacity or the falling costs of sensors.

Rather, it’s more a function of Metcalfe’s Law, which states that the value of a network is the square of the number of connected devices; the value is in the exponentially increasing connections, not the nodes. The same is true of IBM’s people, too. Instead of sidelining its researchers, how could it bring more eyes—and different ones—to opaque data sets being crunched in the cloud?

The solution, unveiled at Almaden last fall, is the Accelerated Discovery Lab, a large, open space amply equipped with comfy furniture, whiteboards, and lots of screens, not to mention an ever-evolving mix of project teams, systems managers, visiting clients, corporate anthropologists, and drop-ins, not to mention a sliver of Watson IBM’s newest super computer. As the lab’s name implies, the goal is crack the code on the optimal combination of diversity, proximity, physical space, and cloud computing to spot opportunities in the gaps between disciplines faster and more often.

“We call it cultivating ‘strategic serendipity,’” says Haas, who is also the director of technology and operations for the lab. “It’s those ‘A-ha!’ moments you have in the shower or often around the water cooler. We want to bring people together in a rich enough environment they want to play in it, and then create serendipity by leveraging the connections in the room, the connections in the data, and our ability to see what users are doing.”

The lab’s first project was to apply Watson’s natural language-processing ability to new domains, with drug research at the top of the list. Working with computational biologists from the Baylor College of Medicine, IBM’s data scientists began plowing through millions of papers, patents, and clinical studies culled from databases and IBM’s pharma customers, before eventually narrowing their focus to the tumor-suppressing gene TP-53. Sifting through the literature for promising, but overlooked chemicals to treat mutated genes, within a few months the team found four candidates. According to Jeff Welser, the lab’s director of strategy and program development, “historically, you find about one per year.”

That’s pretty fast, but could it have been faster? Part of the lab’s mission is to test hypotheses about the space itself. “We’re trying to instrument our projects from the get-go, recording them from the day they start,” Haas says, benchmarking their progress against similar teams that aren’t in the lab to see whether all those whiteboards and multi-disciplinary teams yield better tangible results.

While there are currently no plans to build similar labs in any of IBM’s other research centers, Haas hopes to someday develop a software tool that might help the company manage its own far-flung resources. Imagine a version of Watson that recognizes who or what it is you’re searching for, then begins suggesting data sets and colleagues working in tangential fields the IBMer might have otherwise never thought of.

For now, however, when it comes to cross-pollination, “there is more than I expected,” she says. “And less than I want.”