Big Data: Taking Stock of Canada's Situation
2018 marks five years since the term “big data” was added to the Oxford English Dictionary. Of course, large quantities of data are nothing new—Google was processing 20 petabytes of data a day in 2008—but the analytical practices surrounding big data are still a relatively recent development. For example, Hadoop technology has existed since 2005 but only became truly accessible and mainstream a few years ago. So what's the status of big data in Canada today?
Before exploring further, it's worth looking at some figures from the new 2018 study on the state of information technology by NOVIPRO and Léger. According to the study, 75% of Canadian decision-makers intend to use data for predictive analysis. And where predictive analysis is concerned, some form of big data is usually involved. Of these decision-makers, 46% plan to use business intelligence, 27% artificial intelligence, 26% the Internet of Things and 25% machine learning to continue transforming their businesses.
Large companies at the forefront
Unsurprisingly, companies that produce large quantities of data have been at the forefront of the big data transformation. Such companies include the major players in banking, insurance, telecommunications, the pharmaceutical industry, video gaming and multi-channel distribution. These players have been producing and storing data for decades, so it was logical for them to begin analyzing their data to extract valuable information.
In Quebec, most big data initiatives are still in their infancy. The major corporations are engaged either in proof-of-concept processes or in production, but generally only on a small or medium scale. For example, they might focus their big data usage on a specific concern for the business, like fraud prevention, client segmentation or path analysis. The maturity of their big data processes falls short of that of the American giants, but it's still a step in the right direction.
In Ontario, the big data landscape looks somewhat more mature. A simple online job search reveals far more offers related to big data in Toronto than in Montreal, considering all types of positions (management, development, sales, etc.). The term “full stack developer” is used below for comparison purposes.
Source of data: indeed.ca
A search for key words like “big data” and “artificial intelligence” can give a general idea of the vitality of a market's big data ecosystem. In this regard, a very active scene can be seen in the Toronto region. In all likelihood, this disparity can be attributed to the greater presence of national head offices in Toronto. They include the major banks, which are particularly fond of artificial intelligence.
While it is difficult to assess the exact state of big data analysis in the public sector, progress is likely in its early stages, comparable to the rest of the market. A few ministries seem to be opening up to more “modern" practices, but there is yet to be a broad, comprehensive shift in favour of big data. At the same time, it would not be surprising to find big data practices developing in Canada's intelligence services or armed forces.
Finally, a similar line of research comparing Canada and the United States demonstrates that the market south of the border is more active, with higher rates of job vacancies per 100,000 people.
Sources of data: indeed.ca, indeed.com
On a per-capita basis, the United States has nearly twice the number of offers in big data positions and its job market is nearly three times larger for data scientists, although this term is not always clearly differentiated. Interestingly, the United States’ lead is not nearly as substantial when it comes to artificial intelligence jobs. This may be due to the fact that artificial intelligence is still a relatively new field and the existing jobs tend to be highly specialized and research-focused. As a result, companies are not yet at the point where they need to recruit massive numbers of artificial intelligence experts.
Interest varies among smaller businesses
According to Statistics Canada, nearly 90% of Canada's working population worked for a small or medium-sized business in 2015. While it would be a mistake to generalize, it was observed that despite considerable interest in big data, relatively few of these businesses were actively implementing big data solutions. Naturally, it depends on the activity of the specific business and its managers’ penchant for technology. Current applications of big data include IT tools that assist businesses with their everyday operations, such as software packages and customer relationship management systems.
The field of analytics (not necessarily linked to big data) is gradually becoming a part of the small business landscape thanks to the more widespread availability of, among others, the Internet of Things and hyper-converged systems. In short, smaller businesses are focused on preparing for Industry 4.0 and improving their traditional operations. It isn't even big data in its simplest form, but machine data analysis methods have arisen at smaller businesses as a result of more complex practices, which is a positive sign.
According to our study, 24% of companies were already using predictive tools, 46% were implementing them and 25% were planning to adopt them. One aspect to keep in mind is that 42% of the respondents were medium-sized companies with less than 500 employees.
Eventually, we will reach a point in time when these technologies become universally accessible, even to businesses run by two individuals, in all spheres of economic activity. Soon, small farmers will have access to convenient tools that use historical weather data and connect to sensors placed in strategic areas of the farm. But for now, companies that don't have IT departments—be it a single IT expert—generally don't have plans to carry out data analysis.
Barriers to adoption
Around 2013, big data technology started to become more widely accessible, no longer restricted to web giants like Google and Yahoo!. Even back then, industry experts warned companies of the risks and challenges of taking on big data projects. Five years later, hundreds of attempts have been made, and the same problems generally tend to occur.
A previous study by Tata Consultancy Services noted that the vast majority of the challenges preventing businesses from capitalizing on big data were cultural rather than technological in nature. Unfortunately, this is still the case for large companies, where progress has often proved to be slow.
Some of the challenges identified in the Tata study are still relevant today. Companies are still unsure which business units should be the focus of their big data initiatives, and what structure these initiatives should take: agile development, DevOps or multidisciplinary teams? Managers also face the challenge of financing big data initiatives, recruitment, training and other expenditures. Where should the money come from? IT? The business units? What could we say to convince managers that big data is a sound investment? How do we measure the return on our investment? Once these questions are answered, companies often face an old dilemma: data retention. What will it take for business units to share their precious data? How do we break down the silo mentality in the organization? Who assumes the leadership for this process?
From a more local perspective, we see additional challenges in Quebec, such as a lack of qualified leadership. More and more managers want to adopt and embrace big data, despite the labour shortage in IT. However, their efforts are being slowed by the old guard, which has mastered other technologies and isn't always welcoming of the new databases and programming languages associated with big data. Furthermore, 28% of the study's respondents stated that hiring and training were among their top priorities for the coming year. In other words, companies currently lack the talent to undertake such projects. They would do well to partner up with service providers like NOVIPRO.
Also worth noting is the fact that big data initiatives are often run by IT departments. The problem is that IT departments tend only to consume data from their own systems. Meanwhile, other business lines still stand to benefit from big data analysis, whether it's improving their processes or facilitating decision-making. Excellent new technologies sometimes fail to reach their full potential due to a lack of users. This reflects negatively on the big data initiatives and puts their effectiveness into question. This is borne out by our study: 54% of companies occasionally, rarely or never consulted their stored data. This means that more than half of the business lines of the companies included in the study have not adopted solutions to aid decision-making in their operations.
We can also observe difficulties in change management. Big data means that at a given moment and in a given business process, a decision will be made on the basis of big data, which will have been processed and transformed into useful information. We can see resistance both to the use of the new data processing tools and to decision-making as such. But if we don't act on the information we extract, the investment in the initiative will have been wasted.
The greatest challenge is deciding what to do with the data itself; that is, identifying the needs of the business and deciding how they should be addressed. Businesses should consider practices in their industry and the initiatives of competitors when deciding what action to take. In reality, their dilemma is symptomatic of a deeper problem. Namely, the foundations of their data analysis, whether it's business intelligence or—everyone's favourite subject—descriptive statistics, are incomplete and poorly managed. They may have problems acquiring and storing data, or may lack the necessary IT resources. Perhaps no one at the company has taken the initiative to address these issues. What's certain is that we're far from embarking on new ventures in artificial intelligence. And if you aren't sure what to do, it will be hard to convince your boss of the utility of big data.
Five years later, we're still in the same situation:
That's why companies should reach out to external service providers like NOVIPRO. Instead of setting off without knowing your destination, you can rely on us to help you identify your needs and develop your strategies, whether it's big data, Industry 4.0 or the Internet of Things. Once we have developed a full understanding of your company's environment and situation, we will be able to help you adopt the technologies that will meet your needs and accomplish your new strategy.
Don't hesitate to contact us with any questions or ideas.