Latest News

3 Ways Big Data Makes Industry More Efficient

Completely shifting an industry paradigm poses many challenges, the most problematic being industry-wide acceptance. Many credit Henry Ford with revolutionizing the manufacturing world with his assembly line that allowed mass production of motor vehicles. However, similar methods are recorded hundreds of years earlier. There is evidence of assembly lines used in China in 2nd century BC for the production of crossbows. In 1320 the Venetian Arsenal production process was so efficient that it was one of the most powerful shipbuilding enterprises in the entire world. At its peak, it was capable of producing and outfitting a merchant or naval vessel every day when in the rest of Europe, creation of a similar sized vessel could take months.

More examples of utilizing this technique pepper the historical timeline all the way up to Ford in 1908, so why did it take so long to become popular? Rather than being embraced across many industries, the assembly line was ‘reinvented’ over and over to accomplish specific tasks and then set aside once that task was completed. Understanding the advantages and nuances of this process and applying them broadly in many different applications contributed significantly to the Industrial Revolution. The modern world is once again shifting. No longer is mass production/distribution of an item or service enough, there is now demand for specialized production and services on a massive scale. This new paradigm is being made possible with the application of Big Data.

Big Data Utility

‘Big Data’ has become the holy grail of many aspects of the current economy. From medicine to manufacturing, this mythical monolith promises significant gains to any industry that can harness its potential. However, utilizing the power of big data is not easy, especially considering how much it differs from current methodologies. The difficulty for many industry leaders is that they are unable to think outside the norms of current industrial and manufacturing processes. They work with tangible equipment performing easily observable tasks. Big data does not fit that mold, but it remains a formidable tool for increasing productivity and efficiency when it is properly understood.

Doug Laney is credited with first introducing the 3V’s of big data (1).

  • Volume – Big data, by its very definition, relies on high volumes of data to be useful. As such, the application of big data requires separating valuable data from mere noise. The amount of data created each day is on the order of exabytes (1018) – maximizing value per volume is important.
  • Velocity – The speed at which data is collected and analyzed is critical to modern-day big data applications. Gleaning reams of data doesn’t do any good unless it can be applied quickly and effectively. Speed of data processing can easily become a limiting factor, making streamlined analysis essential.
  • Variety – Modern big data practices call for gleaning data from as many sources as possible. Variety enhances analysis, and better analysis makes for better solutions. However, it must also be taken into account that a variety of data also means a variety of data formats, from PDF’s to Facebook posts

The three Vs are helpful in understanding how big data makes industry more efficient and what pitfalls should be avoided. Regardless of the industry your company is engaged in, here are three ways embracing big data concepts can lead to improvements:

1. Better Understanding of Your Market

It has been suggested that big data was birthed from the need to improve marketing. Whether or not that is true, there is no denying that big data can help companies better understand their markets. In this application, volume is particularly important to consider. With modern analytics offerings, a website owner can obtain a staggering amount of data on every person visiting their site. At this point one must consider quantity vs quality. When relevant analytical datapoints are distinguished from extraneous data, detailed insight into a customer base can be acquired. Industry leaders get a better understanding of who is buying their products, how those products are being used, what customers want in terms of improvements, etc.

A better understanding of a company’s market affords management the opportunity to refine organizational focus to meet market needs quickly. Less time and resources are wasted pursuing certain market aspects that do not really matter. Rather than scaling up or down to meet production needs, existing resources can simply be allocated to accommodate the market demands.

2. Better Insight into Your Process

Big data includes more than just tracking the number of website hits or customer demographics. Using sensor technology, manufacturing and production can be monitored with extreme precision. Toyota is known for the efficiency of the Toyota Production System, or as it is more commonly known, lean manufacturing (2). The goal of this system is to eliminate 7 categories of waste or “muda”.  These categories are:

  1. Overproduction
  2. Waiting (time on hand)
  3. Unnecessary transport or conveyance
  4. Over-processing or incorrect processing
  5. Excess inventory
  6. Motion
  7. Defects

Sensors of all varieties are constantly being improved and made more economical. Combining integrated sensor data collection with appropriate analysis tools can quickly identify any sources of waste. Using relevant data management structures and analysis tools, production can be monitored on a day by day or even hour by hour basis. Uncovering several small efficiency leaks that are less visible when viewing the process as a whole can lead to significant productivity gains. The easier it is to access and analyze detailed information about manufacturing processes, the more quickly waste is eliminated and resources are conserved.

3. Better Ability to Make Decisions

When making crucial decisions without data, the choice often comes down to what Harvard Business Review refers to the ‘HiPPO’ – the highest-paid person’s opinion. Through structured interviews with 330 executives from North American companies, McAfee and Brynjolfsson found that the more companies characterized themselves and their decisions as ‘data driven’, the better they performed on measures of financial and operational results (3). While personal experience and intuition do contribute to good decision making, they are substantially elevated by actual data. In daily operations, hard facts can replace long, drawn-out management meetings relying personal opinions to determine how to proceed.

Success in industry means not only profit in the present, but also in the future. Planning for expansion or product diversification involves many unknowns. While big data is by no means an analytical crystal ball, it gives a better overall view of the present in order to more reliably plan for the future. The ability to observe a production process or even an entire industry continually over a period of time allows for identification of patterns and trends. In R&D, detailed analytics can inform which projects or ventures are worth further investment and development. The more information a company has, the easier it is to see what future steps to take. Big Data not only records the minutia, it paints the bigger picture.

Though it seems to be mentioned by every tech blog written in the past few years, Big Data still has yet to reach its full potential in many industries. Rock West Solutions does not think of it as simply a buzzword, but as a powerful tool. As with any tool, its utility is dependent on the user. More advanced data necessitates more skilled data scientists to manage it. Through creation of data structures and custom analysis algorithms, Rock West is able to help our customers harness the power of this modern industrial paradigm.


  • 3D Data Management: Controlling Data Volume Velocity, And Variety. Doug Laney, Meta Group February 2001 –
  • Lean Manufacturing Made Toyota the Success Story it is Today. Laquita Harris, Capacity Magazine 2007 -
  • Big Data: The Management Revolution by Andrew McAfee and Erik Brynjolfsson, Harvard Business Review October 2012 -