Big data is not required for big insights
Big Data vs. Small/Medium Data
And maybe Big Data will do that–for big organizations. If you’re Coke or the Fermilab or the National Security Agency, your products or services or spying naturally produce a lot of data. Tapping into and harvesting massive streams of continuously created data, which is the hallmark of Big Data, is a natural thing to do.
But for many of us who work at small and medium organizations, Big Data is an abstraction at best. We simply don’t have massive, ongoing data streams that we can dive into to learn about our markets, our products or services, our clients, or our organization. We’re not big enough to have Big Data. But that doesn’t mean we can’t learn from the principles behind this phenomenon and use them to our advantage.
The hype around Big Data is the data itself: massive, previously unattainable and unimaginable rivers of data pouring through your world. But the philosophy behind Big Data is actually more important. t’s about looking around to identify where those data flows are in your own environment and then tapping into them to gain insight. You don’t need Big Data to do that. It works just as well with Medium Data or Small Data, especially if you’re a medium or small organization. We too can tap into and harvest data; it just flows in smaller quantities at our scale.
Three sources from which to harvest data
So how can we start this harvesting? What can we collect? There are three main sources to consider, though we’ll concentrate mostly on the third one.
First, you can harvest data that already exists outside your organization and is updated regularly. For example, there are lots of federal surveys and data collection efforts out there, and they’re very cost-effective to retrieve if you know about them. The right ones can help you understand your environment.
Second, you can create data via ongoing special efforts, such as conducting a regular survey or instituting a special data collection effort that is not part of your daily operations. This is a bit of a different concept from harvesting data, but still falls within the realm of a streaming source of data you can use for analysis.
But the third concept is the core of where Small Data can help you. It’s the implementation of a system to collect and harvest on an ongoing basis the data that we produce in our daily operations. Or more precisely, it’s data that we do or could produce easily in our daily operations.
Thinking about that third concept, we all have opportunities to gather data on a daily basis. Most likely, we already do to some extent, even if it’s as simple as our client names or time sheets. So we’re already in the habit of creating data. But how are we using that data? As examples, I’m always surprised by the number of organizations that record their clients’ ZIP codes but then never use that data to examine their clients’ demographic and geographic makeup I’m also surprised by the number of nonprofits that don’t do research on their donor databases to identify their demographic sweet spots. These data are often collected but not often analyzed and leveraged to their full extent.
Beyond harvesting data that already exists, is there other data that we can efficiently build into our routines that can add value, either in understanding our clients, serving our clients, or improving our internal operations and efficiency? My company, for example, began tracking the origins of our consulting engagements a few years ago, and it has been very effective both in identifying inefficient means of marketing and effective ones. Our minor investment in that effort has paid itself back many times over.
There is value in data. We all know that. The key, of course, is to manage the process so you’re gathering valuable data in an efficient manner and then actually using it to your benefit. If you think about evaluating a program, a general rule of thumb is that 5 to 15 percent of the budget should be invested in evaluation, depending on the size of program. If you would make that investment in a program, why not follow the same rule for your organization as a whole? It may pay off handsomely.