How to Build a Serendipity Engine

(This article was taken from the November 2013 issue of Wired [UK] magazine.)

Silicon Valley’s latest management fad is to not manage. “Some of the best decisions and insights come from cafeteria discussions, meeting new people, and impromptu team meetings,” said Yahoo! when it revoked working from home. Marissa Mayer, ex-Google, knew that hits such as Gmail and Street View were the product of engineers meeting serendipitously over lunch.

Google’s new campus is designed to encourage “casual collisions” at its rooftop cafés, and Facebook has hired Frank Gehry to build “the perfect engineering space: one giant room that fits thousands of people, all close enough to collaborate together,” Mark Zuckerberg explained.

Study after study has demonstrated the power of proximity when it comes to birthing ideas, and network theorists have shown how bridging “structural holes” between teams and disciplines consistently leads to better and more innovations. What’s ironic is how the world’s largest search companies and social network appear to have no better ideas of how to plug these holes than to toss everyone into a room with an espresso machine.

Google is obsessed with “engineering serendipity”, combining users’ search history, location and context to answer questions before they’re asked. What would it take to build a social serendipity engine—one that could identify who’s nearby, mine your hidden contexts and relationships and make the necessary introductions? After all, not everyone drinks coffee.

The first step is finding people, which location-based dating services such as OKCupid and Grindr manage well. They’ve been joined by “social discovery” apps such as Sonar and Circle, which link users’ networks to their smartphone’s GPS.

“Ten years from now, you will look at your phone or through Google Glass, and know everyone in the room,” says SocialRadar founder Michael Chasen. “I haven’t met anyone who disagrees with that.”

The next piece is harder. It’s not enough to know who’s in the room—you also need to care. Facebook “likes” aren’t accurate predictors of a match. What’s missing is context—why I want to meet this person now. Grindr works because its consideration is binary. Any serendipity engine with higher aspirations must grasp deeper motivations than social media supplies.

One solution is to build rich profiles manually. That’s the tack taken by Relationship Science, which maps potential connections between 2.3 million members of the 1 percent. Customers can trace the shortest path to their quarry via colleagues, corporate boards and alma maters, each link graded into strong, medium and weak ties—captions Glass might someday flash over their faces.

Another approach is to collect data using sensors. An MIT Media Lab spin-off, Sociometric Solutions, equips office workers with “sociometric badges” to measure movement, speech and conversational partners, using the results to plot maps of organisations. Its findings suggest Yahoo! should invest in large cafeteria tables: workers who eat at tables for 12 are more productive than those at tables for four.

The last step is the trickiest: given enough context about the strangers headed towards you, can an algorithm crunch enough data in time to spot connections? Another startup, Ayasdi, does that by rendering every big data set as a network map, revealing hidden connections that CEO Gurjeet Singh labels “digital serendipity”.

If yoked together, these approaches could help fill in the blank spaces on the org chart where work actually happens and new ideas are spawned. Although luring them into the same room with an espresso machine would probably speed things up a bit.