The Problem

Cloud computing is currently deemed as the proverbial key to unlocking the potential of automated data processing, DataOps, AI and ML. Thus, all of the vast amounts of data coupled with scalable computation power are being centralised into cloud platforms that carry out the data processing. By doing so, cloud technologies have yielded significant advances in data processing, and are giving data scientists the tools required to innovate in the field, with some of the fastest, most accurate and most reliable solutions to date.

Centralised data processing, despite its spectacular advancements, is fraught with challenges:

1. Cost Intensiveness: Although cloud computing uses elastic, or pay-as-you-go, revenue models, which are able to adapt to the workload of both large and small organisations, the sheer amount of data and processing currently required far surpasses the budgetary constraints of SMBs. It is no surprise that the key market players in automated data processing are Microsoft, Amazon, IBM and Salesforce, and the key market players in DataOps are Microsoft, IBM, Oracle and AWS.

2. Scarcity of Talent: The high price tag is not only reflected in services, but also in human resources. One of the biggest bottlenecks in the DataOps industry is talent shortage - engineers in this field need to be highly skilled in software development, data science & engineering, cloud computing and data visualisation, which is quite a rare mix. As we all know, high demand coupled with scarcity in offer drives up the price quite significantly. If large organisations are finding it difficult to hire qualified professionals, SMBs are having an even harder time attracting and retaining talent.

3. Integration Gaps: Even if small businesses find the budget to adopt an automated data processing system, they still have yet another hurdle to jump. Most SMBs still operate on older, legacy systems based on outdated software architectures, if any. With key players focusing on the latest technologies to attract business from cutting edge startups and large corporations, small businesses end up being siloed, or having to invest more in custom solutions that enable them to be compatible.

4. Privacy and Security: Opting for a centralised solution like cloud computing, also borrows some of its disadvantages. For instance, in DataOps, there are serious privacy and security concerns that need to be addressed for ensuring compliance with regulations. Since all of the data is stored in a centralised manner, the system must maintain rigorous security measures to ensure that sensitive data is not leaked into any of the systems and tools that are integrated in the platform.

If we look through the same lens at artificial intelligence and machine learning, we see the same narrative surfacing. The landscape is dominated by large tech companies such as Google, Meta and OpenAI which drive innovation and dictate the tone. Startups that attempt to capitalise on the recent rebirth of AI by building products on top of LLMs end up competing against key market players that are able to roll out new features and new AI models much faster than small companies. SMBs in AI end up jumping the same hurdles when it comes to recruiting talent from large companies and quickly discover that they cannot afford it - and it isn’t all about the cost of the talent, but mostly about the cost of the infrastructure required to train AI nowadays.

It is becoming exceedingly clear that small and medium businesses are becoming more limited in their use of data processing, artificial intelligence and machine learning, as they cannot afford them, nor do they have the technical capability to implement them. The landscape is increasingly dominated by large global companies that employ such technologies in their competitive repertoire, and use them to exploit their competitive position through the vulnerable dependencies on large digital platforms that SMBs create.

As an opposite reaction, the scientific community is retaliating through initiatives such as Hugging Face, in an effort to democratise machine learning, and to educate and open up machine learning to the software engineering community as a whole. The “Machine Learning For The Masses!” mantra is gaining traction with many AI enthusiasts of all technical backgrounds joining in.

Generative AI has already pierced the public consciousness, with over 180 million people using OpenAI’s ChatGPT on a daily basis. With looming uncertainties about the ethics behind AI, or how our data is being used, this is no longer a scientist’s game. Our collective society, technical and non-technical people alike, needs to get involved in the development of AI to ensure that it is ethical, fair and less biassed.

At Timeworx.io, we believe that data processing, artificial intelligence and machine learning are in dire need of democratisation and decentralisation. Furthermore, these efforts need to be pushed beyond the software engineering bubble, and focused on a societal level, since the problems we are faced with are starting to reach the fabric of our societies. Small and medium enterprises make up the lifeblood of our national economies, employing more people and outnumbering large companies by a wide margin, especially in developing countries. For our societies to thrive we need to focus on enabling the digital transformation of SMBs since they are the most responsible for driving innovation and competition across various industry sectors.