Data Growth
Last updated
Last updated
“There were 5 exabytes of information created between the dawn of civilization through 2003, but that much information is now created every two days.” Eric Schmidt, Executive Chairman at Google
Data has been an integral part of human evolution on our path to overcoming problems, building a better future, and improving our civilization. The earliest records of data usage go as far back as 3000 BC, with the Ishango Bone in the Congo supposedly being used for counting and recording lunar cycles. We’ve now come a long way, and data is becoming an increasingly valuable key enabler for innovation across industries, with an enormous economic and societal potential. By properly understanding and using data, we enable the creation of new products and services, we advance the research of novel technologies, and we streamline production, thus building the tools for combatting societal challenges of the present and future.
We have become exceedingly good at collecting and processing data, having produced up to 120 zettabytes of data in 2023, with an estimated growth of 150% leading to 181 zettabytes in 2024:
Just so we can wrap our heads around how much data this is, we can approximate that 328.77 million terabytes of data are being produced every day, and this estimate is growing every year. If we were to talk in zettabytes, that would mean:
120 zettabytes / year
10 zettabytes / month
2.31 zettabytes / week
0.33 zettabytes / day
Current estimates show that 90% of the world’s data has been generated in the last two years, with over 70% being user-generated with videos accounting for more than 50% of the Internet data traffic, and the volume seems to be doubling every two years.
This has spurred a tremendous growth for the Big Data industry, shooting up 62% over four years, from $169 billion in 2018 to a staggering $274 billion in 2022, and a projection to generate more than $100 billion in revenue by 2027.
All of this data empowers individuals, organisations, governments and societies to get a better grasp of the world around them, and to make more informed decisions at all levels towards the benefit of humankind:
Medicine: access to more information enables medical researchers to get a better understanding of infectious diseases, helps them in identifying and predicting outbreaks, and enables them to discover proper treatment and prevention.
Education: more data empowers educators to get a better understanding of their students, revisit their teaching methodologies, and adopt a more personalised approach to the learning process.
Commerce: smart data helps retailers in analysing market influences. Historical sales, weather patterns, social media trends can help predict future demand for certain products considerably reducing overstocking and understocking.
Government: better data helps governments understand what interventions need to be made to alleviate poverty, and allows national authorities to improve predictions about natural disasters in order to minimise the impacts.
The path to more, better and smarter data requires data sharing, and getting vast quantities of data exactly where they need to be is not always easy. The U.S. is currently the undisputed leader in collecting data, with more than 5000 data centres, surpassing any other country in the world at least ten-fold. However, the U.S. also has the most restrictive data privacy laws and is currently lacking in much-needed technical standards for data sharing, mostly due to failed past experiments that have led them on the path of mistrust and data silos.
On the other hand, the European Union is focusing on reducing the legal, social, technical and economic barriers for the public and private sectors in an effort to drive innovations and new discoveries, through the Big Five acts. The Data Governance Act (DGA), which came into force in 2022, focuses on regulatory data sharing between member states and introduces the terms of data intermediaries and European data spaces, for boosting the data economy and data value chain through data altruism. An emphasis is added on data privacy for building trust with both public and private sectors, with additional safeguards being added on top of General Data Protection Regulation (GDPR) for ensuring trusted data sharing and re-use. While the DGA builds the foundation for a trusted data sharing ecosystem, the Data Act (DA) complements by introducing clear regulations for fair access, usage and remuneration of actors in the data value chain, by introducing data sovereignty and data governance.
In this ever-changing and ever-evolving landscape in which data is in the limelight, our mission is clear: to make more informed decisions we need better and smarter data. How do we obtain better and smarter data? The answer is simple: data processing, which will be discussed in depth in the next section.
