Breaking down data & AI silos

June 24, 2019

Data is an essential resource in our digital economy. Unfortunately, we currently live in the era of internet data silos, where data is captured by individual applications and platforms and is only shared sporadically. This status quo has many reasons, but can mostly be attributed to the absence of a global, standardized and trusted infrastructure that facilitates and incentivizes the exchange of data and algorithms between businesses, humans and machines at scale. Here we’ll dive deeper into the functional requirements of a global data exchange layer such as decentralization, data provenance, data pricing. When realized, we could see the true emergence of digital ecosystemswith interoperable services, built-in data monetization and the volume and variety of data to build powerful algorithms.

Our observations

  • As part of the EU’s strategy to develop a Digital Single Market, it wants to build a data economy. Part of the strategy is the re-use of public data sources and publicly funded data.
  • The UK published a report named ‘Unlocking Digital Competition’ which proposes a digital markets unit which is responsible for a digital code of conduct, promoting openness, personal data mobility and interoperability of services.
  • X-Road was initially developed by the Estonian government as a data sharing solution for its e-government services. Now it has been abstracted as a general data sharing solution as available open-source.
  • The Ocean Project is developing a protocol which orchestrates the exchange of data in a decentralized fashion. At its core are decentralized service agreements and decentralized access control, which execute on decentralized virtual machines. The project envisions that data marketplaces or clearinghouses are being built on top of the Ocean Protocol.
  • SingularityNET aims to develop a worldwide network that is powered by a decentralized protocol that allows people to create, share and monetize their AI algorithms.
  • Enigma and IOTA are building decentralized data marketplaces that provide a decentralized and secure data infrastructure on top of which applications can be built
  • With the emergence of a data exchange layer, new business models will appear that will harvest thenetwork effects that emerge from these open data marketplaces. For instance, Numerai is a crypto-driven hedge fund that crowdsources the predictive modeling for its investment strategies. Similarly,Enigma Catalyst is a platform for data-driven crypto-asset investing and research which allows anyone to develop and share their own investment strategies.

Connecting the dots

Enabling factors like the advent of mobile, user friendly interfaces and the cloud helped with generating enormous amounts of user data. This Cambrian explosion of data combined with rapid progress in storage and computing allowed for machine learning to undergo a renaissance over the past decade. This provided us with better statistical models and algorithms, resulting in more powerful services (e.g. recommender systems, image recognition, voice assistants) which once again leads to increased usage and, hence, even more data.

Unfortunately, most of the data currently sits in business silos. On the one hand this is caused by big platforms that keep “their” data to themselves to keep competitors at bay, on the other hand is caused by the absence of an advanced infrastructure that could facilitate and incentivize the exchange of data and algorithms. Most of thedata exchange that currently takes place happens in narrow APIs which are fully controlled by the applications themselves or by means of ad hoc data dumps via primitive data marketplaces. Consequently, we have still not realized the full potential of our data economy. That is, in the best-case scenario, any dataset, whether from Google or from a small business, could potentially be modelled by any talented data scientists in the world, and can be applied in other contexts and for different purposes and/or can be aggregated and fused with other datasets from other sources at scale.

However, in order for such a situation to be realized, the internet could benefit from the development of a global data– and AI exchange layer with a few important conditions in place. First of all, this data exchange protocol layer preferably needs to have the characteristics of a utility, providing trust, security, openness, privacyand neutrality in a multi-stakeholder environment.

In a previous note, we have discussed how the use of decentralized open-source protocols such as permissionless blockchains could be an important enabler for these prerequisites. In contrast, a centrally managed and owned data exchange platform could lead to distrust as it would introduce a middleman position, which could either become disproportionately powerful and/or could be gamed both internally and externally. Secondly, a data exchange layer requires some form of data provenance/lineage, i.e. keeping track of the origin of a piece of data, the processes that it undergoes, who processed it and where it is going over time. Keeping track of the life cycle of a piece of data is a crucial feature in making the data value chain transparent, but more importantly, treating data as a real commodity with a state, an owner and a value. The presence of data provenance in turn enables other important functionalities like pricing of data units and transaction of data and the creation and real-time verification of more fine grained service agreements in the form of smart contracts.

These functionalities are currently being developed by a plethora of projects, each competing to become a significant part of the solution in the data exchange layer. Several examples of such (complementary) projects are provided in the observations section above. Crucially, these projects have announced that they will collaborate other higher level features in order for them to be interoperable and to prevent the emergence of competing standards that would, once again, lead to the silofication of data.


  • We are likely to see another data explosion in the coming decade due to two important causes. The first one is caused by the combined adoption of 5G, autonomous sensor-based systems (e.g. autonomous vehicles, IoT, smart environments) and machine-to-machine communication. The second important cause is an increased availability due to a frictionless data exchange layer. Consequently, we could see another surge in the advancement of AI, similar to the deep learning renaissance in the last decade, as there is more data available to train AI systems on.
  • When an open global data-exchange layer is realized, we could see increased momentum for data commons and open data. More extremely, as there are no practical reasons worry about data leaks or downright abuse of data, we could expect the emergence of a new moral imperative in which it is self-evident that data that serves the common good should be shared by default.
  • Big tech companies which have built their empires on the data silo paradigm will have to shift their business models towards a more open, decentralized and non-rent seeking model. Facebook’s introduction of Libra can already be understood in this way; it does not natively possess the data that is created on its infrastructure.
  • The wide-spread availability of data and algorithms combined with the momentum in open source could further spur open innovation.

Series 'AI Metaphors'

1. The tool
Category: the object
Humans shape tools.

We make them part of our body while we melt their essence with our intentions. They require some finesse to use but they never fool us or trick us. Humans use tools, tools never use humans.

We are the masters determining their course, integrating them gracefully into the minutiae of our everyday lives. Immovable and unyielding, they remain reliant on our guidance, devoid of desire and intent, they remain exactly where we leave them, their functionality unchanging over time.

We retain the ultimate authority, able to discard them at will or, in today's context, simply power them down. Though they may occasionally foster irritation, largely they stand steadfast, loyal allies in our daily toils.

Thus we place our faith in tools, acknowledging that they are mere reflections of our own capabilities. In them, there is no entity to venerate or fault but ourselves, for they are but inert extensions of our own being, inanimate and steadfast, awaiting our command.
Read the article
2. The machine
Category: the object
Unlike a mere tool, the machine does not need the guidance of our hand, operating autonomously through its intricate network of gears and wheels. It achieves feats of motion that surpass the wildest human imaginations, harboring a power reminiscent of a cavalry of horses. Though it demands maintenance to replace broken parts and fix malfunctions, it mostly acts independently, allowing us to retreat and become mere observers to its diligent performance. We interact with it through buttons and handles, guiding its operations with minor adjustments and feedback as it works tirelessly. Embodying relentless purpose, laboring in a cycle of infinite repetition, the machine is a testament to human ingenuity manifested in metal and motion.
Read the article
3. The robot
Category: the object
There it stands, propelled by artificial limbs, boasting a torso, a pair of arms, and a lustrous metallic head. It approaches with a deliberate pace, the LED bulbs that mimic eyes fixating on me, inquiring gently if there lies any task within its capacity that it may undertake on my behalf. Whether to rid my living space of dust or to fetch me a chilled beverage, this never complaining attendant stands ready, devoid of grievances and ever-willing to assist. Its presence offers a reservoir of possibilities; a font of information to quell my curiosities, a silent companion in moments of solitude, embodying a spectrum of roles — confidant, servant, companion, and perhaps even a paramour. The modern robot, it seems, transcends categorizations, embracing a myriad of identities in its service to the contemporary individual.
Read the article
4. Intelligence
Category: the object
We sit together in a quiet interrogation room. My questions, varied and abundant, flow ceaselessly, weaving from abstract math problems to concrete realities of daily life, a labyrinthine inquiry designed to outsmart the ‘thing’ before me. Yet, with each probe, it responds with humanlike insight, echoing empathy and kindred spirit in its words. As the dialogue deepens, my approach softens, reverence replacing casual engagement as I ponder the appropriate pronoun for this ‘entity’ that seems to transcend its mechanical origin. It is then, in this delicate interplay of exchanging words, that an unprecedented connection takes root that stirs an intense doubt on my side, am I truly having a dia-logos? Do I encounter intelligence in front of me?
Read the article
5. The medium
Category: the object
When we cross a landscape by train and look outside, our gaze involuntarily sweeps across the scenery, unable to anchor on any fixed point. Our expression looks dull, and we might appear glassy-eyed, as if our eyes have lost their function. Time passes by. Then our attention diverts to the mobile in hand, and suddenly our eyes light up, energized by the visual cues of short videos, while our thumbs navigate us through the stream of content. The daze transforms, bringing a heady rush of excitement with every swipe, pulling us from a state of meditative trance to a state of eager consumption. But this flow is pierced by the sudden ring of a call, snapping us again to a different kind of focus. We plug in our earbuds, intermittently shutting our eyes, as we withdraw further from the immediate physical space, venturing into a digital auditory world. Moments pass in immersed conversation before we resurface, hanging up and rediscovering the room we've left behind. In this cycle of transitory focus, it is evident that the medium, indeed, is the message.
Read the article
6. The artisan
Category: the human
The razor-sharp knife rests effortlessly in one hand, while the other orchestrates with poised assurance, steering clear of the unforgiving edge. The chef moves with liquid grace, with fluid and swift movements the ingredients yield to his expertise. Each gesture flows into the next, guided by intuition honed through countless repetitions. He knows what is necessary, how the ingredients will respond to his hand and which path to follow, but the process is never exactly the same, no dish is ever truly identical. While his technique is impeccable, minute variation and the pursuit of perfection are always in play. Here, in the subtle play of steel and flesh, a master chef crafts not just a dish, but art. We're witnessing an artisan at work.
Read the article

About the author(s)

FreedomLab Fellow Arief Hühn headed FreedomLab from 2018 until 2023, directing our research and business endeavors with a special emphasis on the impact of emerging digital technologies on the economy, politics and society. He holds a master's degree in communication sciences from Radboud University Nijmegen and a doctorate degree in human-computer interaction from Eindhoven University of Technology.

You may also like