It's a field so new that few schools offer a degree in it. Yet demand is surging for data scientists, the hottest job title in Silicon Valley.
"For most companies right now," says Amrit Williams, "the real problem with getting value out of data is a lack of sophistication in talent. We have a major drought of qualified people who can ask the right questions."
Williams is chief technology officer of Emeryville-based Quantivo, one company in the vanguard of a new Big Data industry, developing analytical tools for business big and small.
"People are looking at all this data, going, 'I know it's interesting for me to collect Twitter data. I know it's interesting for me to understand what my customers are doing on Facebook. I know it's interesting for me to understand what they do online. But what's the question I'm trying to ask? And what's the answer I want?'"
For the first time, Big Data has its own convention, the Strata conference, inaugurated by O'Reilly Media of Sebastopol -- where Edd Dumbill's job is to stay in front of this trend.
"For a lot of people," he says, "the first Strata conference crystalized that they were data scientists, there were other people who were data scientists, so this is a profession, working with large datasets. And now it's a highly-in-demand job title."
For years, big retailers collected volumes of data about their customers. But they struggled to make sense of it. There was too much, and it was too unstructured. Then came the perfect storm: cloud computing, open source, and Internet companies that eat whole planets of data for breakfast. Now Google and others are sharing their technology.
