Is Big Data Too Big to Scale?

In the era of digital transformation, organizations have collected vast troves of data with the promise that these datasets would yield transformative insights. However, a growing sentiment in the tech industry questions whether the “big data” revolution has delivered on its promises.

The Myth of Universal Big Data

The tech industry has sold a compelling narrative about the value of data. As Derek Steer explains in his talk “We Don’t Actually Have Big Data,” BI tools typically demonstrate their value through a consistent story: an analyst finds a “blip” in the data, explores it, and uncovers valuable insights - but this narrative, while compelling in sales, rarely matches actual usage.

This narrative has been persistent across the industry. Steer points out how major BI platforms all use similar messaging: “quickly find meaningful insights within your data” or “discover and share insights that can change your business in the world.” But there’s a fundamental disconnect between this sales pitch and reality.

The Visualization Problem

The visualization challenge is particularly pronounced. When truly large organizations with massive datasets (like Target with $73 billion in revenue or Facebook with over a billion users) create visualizations, they can produce meaningful charts with clear patterns.

However, as Steer humorously points out: most companies don’t have charts that look like smooth, insightful curves - “your charts look like this,” showing a sparse, ambiguous graph. “What do you do with this? You don’t slice into it… you squint at it and you’re like ‘it’s up, is… I don’t know.‘”

This visualization problem is particularly evident when dealing with recent data. Most businesses typically analyze only the last 90 days of data, which often isn’t enough to establish meaningful patterns. While large tech companies might have billions of data points to create smooth, insightful visualizations, the average business looking at their last quarter of data is left with sparse charts that resist clear interpretation.

The Small Data Reality

The reality for most businesses is starkly different from the big data success stories we hear about. While we were “promised previously unimagined insights” (an actual line from Snowflake’s website according to Steer), most companies instead get “directional vibes” where you look at a chart and can only say “it’s uppish, I don’t know.”

A Path Forward

Rather than chasing the elusive promise of big data insights, Steer suggests several alternative approaches:

1. BI tools should focus more on helping interpret data rather than just exploring it - “interpretation of data is often a lot harder than exploration and what most tools focus on is exploration.”

2. Sometimes manually examining each data point is better than trying to find patterns in small datasets - “when you have reasonably small data, just looking at every example isn’t often that hard,” referencing how a colleague realized it was better to manually read seven articles rather than build a sentiment analysis system.

The big data revolution has undoubtedly transformed certain industries, but for most organizations, the promise remains unfulfilled. Perhaps it’s time to acknowledge that rather than continuing to invest in tools that promise to extract insights from supposedly “big” data, companies should focus on better understanding the limited data they actually have.