Frequently Asked Questions about Data Mesh & FME (Part 3 of 3)

Welcome to the third and final instalment of our data mesh series, in which we’ve previously explained what a data mesh is, and how you might go about implementing one.

As a new and somewhat abstract concept, you’ll likely have plenty more questions about the whats, whys and hows of data mesh. So in this article, we’ll round the subject out by answering some of the most common FAQs.

What is meant by data mesh?

In using the term ‘data mesh’ for her idea of a decentralised data architecture, Zhamak Dehghani was following in the footsteps of the systems that had come before. The data warehouse and the data lake were both concepts that used a real-world metaphor to describe their shape and function, and data mesh does the same.

While its decentralised nature makes it more difficult to imagine as a real-world object, the term ‘data mesh’ does a decent job of conceptualising the idea of decentralisation: a mesh, network, or fabric, made up of ‘nodes’, freed from a definitive structure with edges or a centre point.

How does data mesh support operations

Business units or domains typically digitally transform by automating processes at the department level by updating their line-of-business applications. These are increasingly cloud-based or SaaS solutions that are siloed from other business systems.

Data mesh connects these applications back into the organisation by broadcasting messages when information changes. These messages let other systems know to update so that information is available for analytics or to automate cross-department workflows.

How does data mesh support analytics?

Analytics is one area in which the data warehouse or lake may continue to have a role, albeit a slightly different and somewhat reduced one. Historically a business would use one of these monolithic architectures to centrally model and store all organisational information for the purpose of creating business intelligence reports. But as data quantities grow, this is becoming a massive (and in some cases impossible) job.

Data mesh relieves the data warehouse or data lake of the responsibility of being everything to everyone. Rather than storing everything, data warehouses and lakes can simply connect to the mesh and ingest the messages relevant to the reports and dashboards that executives or senior leadership requires.

What is a data mesh vs a data lake?

The predecessors to data meshes, and the most popular data architectures to this day, are data warehouses and data lakes. These are both centralised systems: a data warehouse holds structured data, while a data lake holds unstructured data. Both are controlled by specialised data teams. In a data warehouse or data lake, the information is centrally managed, modelled and stored by data system experts. 

In a data mesh, information is decentralised, handled not by data experts, but by the experts on the data. By organising data by domain, a data mesh grants a business unit more ownership of and control over the data it produces.

Is Snowflake a data mesh?

At this stage data mesh is a concept, not a singular technology. However, there are software solutions Kafka and FME that can help make data mesh a reality. We believe a data mesh can only be implemented by leaning on a number of technologies and by fostering a data mesh-friendly culture within your organisation.

For now, Snowflake is a data warehouse, so we would primarily see this participate in a mesh as the system of record for individual nodes or to support analytics workflows.

What problems does data mesh solve?

• Data silos: As organisations accumulate more data sources, they can become siloed. But a data mesh is predicated on all business domains making their data accessible to all others, which breaks down these silos.
• Centralised data governance: The centralised governance of data warehouses and lakes can be inefficient and inflexible. The distributed governance of a data mesh grants each domain the autonomy to make its own data decisions, allowing an organisation to extract more value from its data more efficiently.
• Scalability: Data warehouses and lakes can struggle to keep up with ever-growing data volumes. Data mesh, by comparison, is ultra-scalable, as each domain team is tasked with managing its own data infrastructure and services.
• Data quality: The challenges of maintaining data quality are heightened as volumes increase. But by handing this responsibility to individual domain teams, and building the right data governance culture, maintaining data quality, no matter the volume, is far more achievable.

What are the advantages of data mesh?

The benefits of data mesh are aligned with the problems that it solves, listed above. Compared to data warehouses and lakes, data mesh is scalable, promotes data accessibility, and can increase data quality. On top of that, data mesh can:

• Streamline development: By empowering domain teams to manage their own data, data mesh can streamline development processes. Ultimately this can result in faster delivery of new products and services.
• Improve collaboration: Data mesh promotes collaboration between domain teams. It’s in the best interests of every team to make data as accurate and easily accessible as possible. And better collaboration means better results for your business as a whole. 

Data Mesh is a new and somewhat abstract concept that aims to solve some of the problems associated with traditional data architectures such as data warehouses and data lakes. It is a decentralised data architecture that grants business units more ownership of and control over the data they produce. Data Mesh promotes scalability, data accessibility, and can increase data quality. It can streamline development processes, improve collaboration between domain teams, and ultimately result in faster delivery of new products and services. 

If you’d like to delve deeper into the topic, we recommend reading the first article of this series, “Unpacking the future of data management,” which introduces the concept of data mesh and its significance. Additionally, for insights into the role of FME and its support for data mesh principles, refer to the second article, “What is the role of FME in a data mesh?.”