What is the role of FME in a data mesh? (Part 2 of 3)

Data mesh is a new and potentially revolutionary concept in data architecture. As we discussed in the first article in this series, data mesh represents the evolution of the monolithic architectures that are standard today.

The centralised nature of data warehouses and data lakes means that they will struggle to keep pace with the continual, exponential growth of data. The decentralised nature of a data mesh, meanwhile, sees each business domain is responsible for its own data, while being granted the autonomy to access data from other domains. By spreading the load, a data mesh can see an organisation capitalise on more data more effectively.

As we mentioned in part one, data mesh isn’t a technology, but a concept; implementing one isn’t a matter of choosing the right product, as data mesh has too many people and process considerations to simply be ‘solved’ by technology.

That said, certain technologies can play key roles in facilitating data mesh, and one such solution is FME.

6 ways FME can help you align with data mesh principles

The concept of data mesh is built upon four guiding principles:

• Principle I: Domain-driven ownership of data.
• Principle II: Data as a product.
• Principle III: Data availability and self-service.
• Principle IV: Data is governed where it is.

The good news: FME can assist your organisation in aligning with every single one of these principles. Here are just six examples of how.

1. Low-code/no-code 

As a low-code/no-code platform, FME data flows are created using a drag and drop canvas. This allows all users – not just developers – to become the good data citizens that data mesh demands.

Traditionally data tools have been left in the hands of a specialist data team. The low-code/no-code functionality of FME puts the power of data management in the hands of all users, empowering business units to take greater ownership of their data: the ‘data ownership by domain’ outlined in principle I.

With the power of data management democratised and handed to non-specialists, there may be concern from business leaders that governance will be compromised. But the C-suite can still be confident that appropriate levels of governance will be maintained, per principle IV.

2. Webhooks and event streams

FME is capable of ingesting messages from webhooks and streams without code, allowing teams to treat data as a product, per data mesh principle II. The approach you use will depend on the volume of data:

• FME Server Webhooks are great for webhook sources or websockets that deliver one or two messages per second.
• FME Server Streams are designed for high volume throughput: thousands of messages per second from message brokers like Kafka, Azure Queue Storage or RabbitMQ.

FME thus provides the technical tooling for business units to get data in and out of data products across the data mesh.

3. Automations and topics

FME has its own messaging system called topics that is designed to extend webhooks and event streams. FME Server Automations can publish a message to a topic, which is then broadcast to any subscriber listening to the topic (sometimes called pub/sub).

In terms of data mesh, this allows events from one node to be broadcast to any number of other nodes independently, loosely coupling information flows. In this way FME supports principle III – data availability and self-service – as users can subscribe to FME topics and push self-service data into their own systems using automations.

4.   Data transformation

In a utopian data mesh, all data would be represented consistently across all the nodes – but here in the real world you’ll be constrained by the different software products you are connecting into the mesh. Enter FME.

FME can transform data so that information from one node on the mesh can be ingested into another – over 500 drag-and-drop activities called ‘transformers’ make the act of shaping data accurate and repeatable. The AttributeManager, for example, provides tools for mapping attributes that may be named or represented differently in different systems.

In this way, FME facilitates the pragmatic governance outlined in principle IV.

5. Batch data migration

FME is a powerful platform for batch processing data into new nodes on the data mesh. When a new system is added as a node, information from other systems will generally be required, especially when the new system is replacing an old one.

FME can connect to virtually any system within the data mesh architecture, transforming and reshaping the data within. This is yet another way in which FME supports data mesh principle III, by granting self-service access to historic information.

6. Mesh-wrapping legacy applications

It is more than likely that your organisation has legacy applications that lack the APIs or webhooks necessary to participate in a data mesh. But that doesn’t mean your data mesh hopes are dashed, nor that you’ll need to undertake resource-heavy upgrades.

FME is able to connect to over 400 files, formats and standards including major database management systems like MS SQL Server, PostgreSQL and Oracle. The direct connections that FME makes to these data sources effectively covers them in data mesh-friendly wrapping, allowing otherwise incompatible legacy platforms to participate in the mesh.

Bringing these legacy systems into a data mesh is a powerful way to loosely couple them to other systems in the enterprise, making it much simpler to lift and shift, migrate or decommission.

Data mesh architecture challenges beyond FME

To reiterate: data mesh is not a specific technology solution, but rather a concept, and one that has only recently emerged. It is therefore important to say that while FME can play a significant role in many of the technical aspects of employing a data mesh within your organisation, it cannot turn this concept into reality by itself.

Many challenges remain, including:

• Products without APIs: Significant challenges arise from legacy systems that lack event streams or APIs as part of their product roadmap, as these may be unable to be ‘mesh-wrapped’. Organisations should be careful when selecting technology, choosing only those solutions that feature APIs in their product roadmap.
• Replaying Messages: While FME is ideal for data migration, it currently lacks a simple way to replay past messages in Automations – it’s possible, just not simple. 
• Mesh catalogue: No easily accessible catalogue exists that unifies the knowledge about the mesh – a fact that complicates the task of realising true self-service. This means that a greater emphasis needs to be placed on governance, documentation and collaboration.

The good news is that none of these issues are show-stoppers for small to medium businesses, provided teams are being responsible data citizens and working together for the good of the wider organisation. Training and culture is critical: all business units need to understand exactly what to do, and they need to know that doing the right thing is in their own best interests.

As the scale of the organisation gets larger, there will be more systems and a higher volume of messages exchanged on the mesh. In a large organisation, it’s more likely that FME will participate in individual nodes on the mesh, rather than forming the centralised message exchange platform for the otherwise decentralised system. Large-scale may be the realm of a complex, code-based system like Apache Kafka.

With data mesh still an abstract concept for most organisations, you’ll likely have plenty of other questions, considerations and concerns that you’ll want to address before committing to mesh implementation.

To gain a deeper understanding of data mesh, we recommend reading the first article, “Unpacking the future of data management,” which introduces the concept of data mesh and its significance. Additionally, if you’d like to get answers to commonly asked questions around data mesh and FME, refer to the third article, “Frequently Asked Questions about Data Mesh & FME.“