Making Real-Time Data Work For Your Business With FME Server

Real-time data is already a valuable organisational asset, but it’s one that will get exponentially more valuable in the future. Data holds the answer to almost any business question you care to ask, and the more data you have, and the more current it is, the greater the actionable insights you can gain.

In this article we’ll take a closer look at real-time data: what it is, why your organisation might benefit from it, and how you can capitalise on it using FME Server.

What Is Real-Time Data?

Real-time data is exactly what it says on the packet: information that is generated, delivered and processed instantly. Real-time data can offer up information in any number of time-critical situations, delivering the location of vehicles, operational issues in machinery, message notifications and more.

Real-time data can be generated by a wide variety of sources, including:

• Sensors: Examples include temperature sensors, intruder alarms, flood gauges, equipment malfunction sensors and proximity alerts.
• Applications: Including stock trading systems, social media analytics, IT monitoring systems and server status notifications.
• Mobile devices: Smartphones and other mobile devices are becoming ever more valuable real-time data sources, as they combine both sensors and applications, and can be found in almost everyone’s pockets.

The rise of real-time data has been powered by the growth of the ‘internet of things’ (IoT) – the ever-increasing number of devices, sensors and other physical objects that are now capable of connecting to the internet and transmitting data.

Real-time data is becoming ubiquitous, and it will only become more prevalent and important in years to come. The potential of real-time data is huge; it can be used to identify issues earlier, it can reveal opportunities that you may not have known existed, and it can help you create better connections with customers, team members and other user communities.

Types of real-time data

Real-time data can take two forms:

• Stream data: Stream data is continuous. Data streams deliver a non-stop flow of unbound data. While individual data records tend to be small, the volume of data is large.
• Event data: Event data is discrete. It comes in the form of messages generated when a specific set of conditions are met. Event data records can be small or large, depending on the event they describe, but the total volume of data will generally be lower than that of data streams over a given period.

To gain a better understanding of the difference between event and stream data, let’s take a look at a few comparable examples:

Both event and stream data should be processed quickly in order to deliver full value. The sooner data is processed, the more relevant and actionable it is. If there is even a slight delay, you may no longer be able to take immediate action, and value will be limited to insights that can be gained from historic data analysis.

How FME Server Works With Real-Time Data

One of the most important real-time data metrics is data velocity: how quickly the data is generated. Data velocity dictates how the information will be processed.

Stream data tends to have a high velocity that demands a purpose-built data stream processing architecture capable of dealing with the flow of real-time information. Event data, meanwhile, tends to have a lower velocity, being that it is only generated when certain conditions are met. These discrete events can be handled using a more basic system of real-time event processing.

Let’s take a closer look at these processing methods.

Data stream processing

Data stream processing is designed for high-velocity data streams, and is capable of delivering insights quickly (in literal milliseconds) and continuously across often massive datasets. Data stream processing is an ongoing task, and can be handled within the Streams function of FME Server (learn more here).

Real-time event processing

Real-time event processing handles each event separately then connects it to other relevant events, with these connections kept in persistent storage. This system, also known as complex event processing, can be handled by Automations within FME Server, where an incoming event is used as the input data and trigger to deploy a workflow.

Real-time event processing can range from the simple to the complex, such as combining event information with other data. FME Server’s ‘pay by CPU time’ model is ideal for all types of real-time event processing, as you only pay for what you use.

Batch processing

There is another, near real-time processing method that may be fast enough for certain business cases. In batch processing, real-time event data is stored and processed at a given interval, such as a retailer who monitors transactions every half hour.

The length of the interval will depend on what is being monitored – if you’re looking to gain actionable insights, the shorter the better, but attempting to use batch processing as a true real-time data processing method will prove inefficient and expensive. It nevertheless forms a solid first step toward processing real-time data.

3 Benefits Of Using Real-Time Data In FME Server

Why should you invest in the use of real-time data in FME Server? We think these three are the most compelling.

1. A deeper understanding of internal and external operations

These days everyone keeps some form of mobile device close by. This offers an organisation an incredible opportunity to be in closer contact and gain a deeper understanding of the internal and external operations

• You can monitor internal operations to better manager staff and assets.  Whether it is getting realtime data from you staff in the field or reports back from expensive equipment that something has failed there is huge potential to save money.
• You can better work with external suppliers or customers by sharing reatime data on your assets and your most pressing operations.  This type of information creates a far more responsive process that means tasks are getting completed quickly and with far less human interaction.

Seamless Data worked with  FME Server and Nelson City Council to dispatch jobs to environmental contractor Nelmac in real time. A job is created within the council’s internal Infor system. The job is processed by FME Server then delivered to Nelmac’s internal systems for finance (Microsoft Business Central) and timesheeting (Solar Workspace). The job is then assigned to an employee via smartphone, who completes it through Nelmac’s field management tool (ArcGIS Field Maps), generating actionable, real-time data in the process.

2. Enhanced monitoring of environments

Real-time data in FME Server grants users a greatly enhanced view of the current situation in a wide array of systems and environments. Such a complete, real-time view facilitates both increased awareness of even subtle changes and faster responses to those changes.

In Canada, The Weather Network provides lightning alerts to viewers and subscribers who may be affected by a coming storm. They use FME Server to process these alerts from the Pelmorex lightning detection network, then broadcast them across local TV, The Weather Network website, SMS, email and apps. 

3. Quicker responses to changes in system operations

By processing real-time data in FME Server, an organisation is far better placed to detect and respond to changes in operational systems, which can lead to faster identification of potential issues and serious gains in operational efficiency.

Waka Kotahi, New Zealand’s Transport Agency, worked with Seamless Data to  receive real-time notifications from EVROAM, delivered directly into FME Server, about the use of electric vehicle charging infrastructure. This offers incredible insight into the adoption and utilisation of electric vehicles, which in turn facilitates collaboration with organisations like Transpower and the Energy Efficiency & Conservation Authority, leading to data-driven decision making on infrastructure and funding. 

Working With Real-Time Data In FME Server

FME Server offers two powerful solutions for ingesting real-time data.

Automations

FME Server Automations can be triggered by event data received from another system via a WebSocket or webhook message, typically in JSON format. This method is effective if events in the source system are well-defined – a form submission, for example – and data throughput is not excessive. Message intervals down to every few seconds are generally acceptable, though this depends on the size of the data.

If the system you are consuming data from can’t connect via a WebSocket or webhook, you can still use other FME Server Automation triggers to perform near real-time processing with frequent polling, e.g. an automation can poll a REST API for changes or a SQL database for new records and trigger an FME data process in response. 

Streams

If your real-time data is arriving in a constant flow, up to thousands or even hundreds of thousands of messages every second, such as the high volume data readings from devices like sensors, FME Server Streams are the solution. A stream allows you to keep a workspace running continuously to receive a data stream from a message broker, like Kafka, MQTT or RabbitMQ. Streams can also use WebSocket connections.

Are You Ready For Real-Time Data?

The potential of real-time data is incredible, and in FME Server you have a solution that is capable of realising the endless opportunities that real-time data offers up.

You can enjoy a deeper understanding of all organisational stakeholders, from team members to customers. You can monitor environments like never before, identifying both issues and opportunities. And you can make quicker decisions to adapt to changing conditions.

Your choice of data processing will reflect the needs and realities of your organisation. But no matter whether you choose batch processing, real-time event processing or data stream processing, FME Server Automations and Streams are capable of delivering.

If you’re having trouble setting up real-time data processing, head to the FME Server Troubleshooting Guide, which guides you through the process of setting up processing within Automations and Streams.

Alternatively, speak to us! At Seamless we know the difference that real-time data can make to an organisation, as we’ve used it to help a wealth of Kiwi businesses grow. If you’re ready to capitalise on real-time data processing, get in touch today.