Engineering Serendipity

WHEN Yahoo banned its employees from working from home in February, the reasons it gave had less to do with productivity than serendipity. “Some of the best decisions and insights come from hallway and cafeteria discussions, meeting new people, and impromptu team meetings,” explained the accompanying memo. The message was clear: doing your best work solo can’t compete with lingering around the coffee machine waiting for inspiration — in the form of a colleague — to strike.     

That same day, Google provided details of its new campus in Mountain View, Calif., to Vanity Fair. Buildings resembling bent rectangles were designed, in the words of the search giant’s real estate chief, to maximize “casual collisions of the work force.” Rooftop cafes will offer additional opportunities for close encounters, and no one in the complex will be more than a two-and-a-half-minute walk away from one another. “You can’t schedule innovation,” he said, as Google knows well, attributing the genesis of such projects as Gmail, Google News and Street View to engineers having fortuitous conversations at lunch.

Silicon Valley is obsessed with serendipity, the reigning buzzword at last month’s South by Southwest Interactive Festival. The term, coined by the British aristocrat Horace Walpole in a 1754 letter, long referred to a fortunate accidental discovery. Today serendipity is regarded as close kin to creativity — the mysterious means by which new ideas enter the world. But are hallway collisions really the best way to stoke innovation?

As Yahoo and Google see it, serendipity is largely a byproduct of social networks. Close-knit teams do well at tackling the challenges in front of them, but lack the connections to spot complementary ideas elsewhere in the company. The University of Chicago sociologist Ronald S. Burt calls these organizational gaps “structural holes.” In a 2004 study of 673 managers at the defense contractor Raytheon, Mr. Burt found that managers who serendipitously bridged such gaps were more likely to generate good ideas (and advance professionally as a result). “This is not creativity born of genius,” he wrote. “It is creativity as an import-export business.” In such cases, serendipity is the spontaneous plugging of these holes, over which good ideas flow.

Whereas Mr. Burt painstakingly assembled his analysis by hand, today sites like Facebook and LinkedIn contain enough information to do so automatically. Last month, researchers at Israel’s Ben-Gurion University detailed how they were able to construct social network maps of a half-dozen technology companies — including one with more than 50,000 employees — in a matter of hours using readily available data. Armed with such maps, says Michael Fire, the paper’s lead author, managers can spot isolated teams and structural holes, tweaking the organizational structure in real time. Rather than wait for their employees to cross paths, they could simply make the necessary introductions.

One reason structural holes persist is our overwhelming preference for face-to-face interactions. Almost 40 years ago, Thomas J. Allen, a professor of management and engineering at M.I.T., found that colleagues who are out of sight are frequently out of mind — we are four times as likely to communicate regularly with someone sitting six feet away from us as we are with someone 60 feet away, and almost never with colleagues in separate buildings or floors.

And we get a particular intellectual charge from sharing ideas in person. In a paper published last year, researchers at Arizona State University used sensors and surveys to study creativity within teams. Participants felt most creative on days spent in motion meeting people, not working for long stretches at their desks.

The sensors in the A.S.U. study were supplied by Sociometric Solutions, a spinoff company of the M.I.T. Media Lab’s Human Dynamics Laboratory that uses “sociometric badges” to measure workers’ movements, speech and conversational partners. One discovery, says Ben Waber, a co-founder of the company and a visiting scientist at M.I.T., was that employees who ate at cafeteria tables designed for 12 were more productive than those at tables for four, thanks to more chance conversations and larger social networks. That, along with things like companywide lunch hours and the cafes Google is so fond of, can boost individual productivity by as much as 25 percent.

“If you just think of serendipity as an interaction with an unintended outcome, you can orchestrate pleasant surprises,” says Scott Doorley, a creative director at Stanford University’s Institute of Design. He and his colleague Scott Witthoft have instituted simple measures like positioning couches near doorways and stocking rooms with multiple types of seating to encourage lingering conversations.

Dr. Waber goes further in his forthcoming book “People Analytics,” envisioning a sensor-strewn office that reconfigures itself each morning courtesy of algorithms that plug any nagging structural holes by reassigning seats. “We’re still in the very early stages of engineering serendipity,” he says. What comes next may make the data-driven Googleplex look touchy-feely by comparison.

Greg Lindsay is a visiting scholar at the Rudin Center for Transportation Policy and Management at New York University and co-author of “Aerotropolis: The Way We’ll Live Next.”