Big Data for Big Companies. Where to Start?

THE STATE OF DATA

We live in a data-driven culture. And we’ve used it for centuries! Gathering data, analyzing data, and trying to make predictions based on historical data. The truth and challenge is that most of the “information” we have is chaotic and unstructured. The question is how do we organize it? How do we prepare usable data that can help us make better business decisions? What does the ideal data state even look like?

Data has different levels of maturity, ultimately resulting in optimized and mature data processes. In the following increasing order:

Descriptive data: “what happened?” 

Diagnostic data: “why did it happen?” 

Predictive data: “what will happen?”

Prescriptive data: “how can we make it happen?” 

The reality? Even the biggest technology companies aren’t all the way there!

The maturity states of data can be divided into the following 5 categories:

• initial
• managed
• standardized
• advanced
• optimized

As for the average large enterprise, most of them have not actionably concluded that they can drive decision-making with data, and thus, have not yet developed a data-driven culture. 

On the flipside, every large enterprise has data! As most in the 21st century would agree, it’s not a question of “if” data should be used for organizational decision-making;  perhaps a better question is “who” should care about the data, and “who” is chiefly responsible for maturing it within the enterprise? The answer is – ”everyone” – to some degree. But one of the most important groups, not including executives, are those employees & managers who are customer-facing. Data sets around customer behaviour, product use, verbal and non-verbal feedback, and transaction history is plentiful, and can be used to drive better business performance. 

DATA AND BIGGER DATA

“The main goal is not just to be able to use data, but to use it in a way to make better business decisions. That’s why this whole Big Data thing has been getting traction for the last 10 years.” 

– Andrew Bruce, Senior Data Architect & Cloud Solutions Architect, iRangers International Inc. (member company of DeepDive Technology Group)

Back in the olden days we had only “related data.” It was easy to put into many related tables, it dealt with transactional systems, it was structured, and it went directly into databases. With one trustworthy source, it was simple to analyze and report on.

But with the appearance of the internet, we began dealing with much larger amounts of data that wasn’t so related, or structured, or easy to analyze either.  And we faced datasets with much higher velocity, volume and variety – the three big V’s of Big Data . One example of non-related data is your social media feed. Eventually Big Data led us to new applications containing great amounts of information, such as simulation modeling, predictive analytics, neural networks, machine learning, and more.

Every big company has Big Data. And every big company should be leveraging it. Here is the trick of how big data actually works. In general, there are two types of data: transactional data and non-transactional data. One suggestion is to begin by improving non-transactional data first. In many 21st century companies, non-transactional data provides a goldmine for companies to extract, transform and load to have a high quality data set. This is any data scientists’ dream, to then go in and make predictive decisions from.

HOW TO GET STARTED WITH DATA

We know data is very important. And we want to use it. But where do we start?

Here’s what’s recommended when you’re starting any Big Data or data analytics project at an enterprise level. First of all, you need to have your experts prepared to spend 1 year studying Big Data…how it works, what can be done with it, and how to analyse it. In other words, it takes time. Companies who believe data can unlock better decision-making would be hypocritical not to see value in this particular time spent. Unfortunately, it happens all the time. 

The second important step is improving data quality. This is often a task for a data analyst is to go metaphorically upstream, where the data is entered to find out what it’s doing, how it’s working, and how to improve it. This is a process challenge more than a technical challenge. Generating useful, consistent data takes nurturing!

Additionally, data projects are lost without a clear data strategy. And this is not something that you should simply outsource; when creating data strategy, it’s important to own that data strategy in-house, and it’s important to know the business rules. Moreover, data scientists often need to be paired with someone who knows the business rules to contextualize the data and determine how data can best be used for predictive analysis. 

A Data Strategy is usually thought of as a technical process; however, a comprehensive Data Strategy addresses far more than just the data. It defines People, Process, Policies, Procedures, and Technology [1]. 

CONCLUSION

Data is a strategic asset. Often it’s unorganized, unstructured and scattered all over different systems. It will take time and money to learn how to use it, to best build your data strategy and find the best possible way to manage it. It’s a long-term initiative, no doubt; but if leveraged right, data can help improve operational efficiency, revenue, and profitability as a result of better decision-making. It’s all about getting started. 

—

References

Andrew Bruce, Senior Data Architect & Cloud Solutions Architect, iRangers International Inc. (member company of DeepDive Technology Group)

Andrew Bruce, DATA STRATEGY ROADMAP, – https://www.irangers.com/data-strategy-roadmap/