1. Home
  2. Science
  3. Semantic Folding for Sensor Fusion

Semantic Folding for Sensor Fusion

How Semantic Folding powers real-time processing of sensor fusion data while enhancing the reliability of autonomous systems

What is Sensor Fusion?

Sensor fusion is the key to autonomous operations. It involves combining data from multiple sensors to obtain a more accurate and comprehensive understanding of a system or environment. Sensor fusion analyzes patterns to capture unexpected anomalies, thereby enhancing control and predictability significantly. Sensor fusion is commonly used in smart applications across many industries including manufacturing, automotive, aerospace, utilities and healthcare.

The Challenges of Sensor Fusion

The goal of sensor fusion is both to improve the reliability of the status information about a system, and to provide an exhaustive overview of complex systems in environments with constantly varying conditions. To make this vision a reality, however, requires addressing specific challenges related to the significant complexities and computational demands of sensor fusion systems:


A growing number of sensors deliver a steady flow of data in various formats and inconsistent quality.


Real-time processing is not feasible in many cases because it affects performance and slows a system’s response.


Sensor fusion is not yet scalable, as adaptation to a new environment creates additional costs.


The different sensor systems are difficult to integrate in one single sensor fusion platform.

What is Semantic Folding?

Semantic Folding originates from text data analysis, where it is used to capture the semantic properties of textual data within topographical representations called semantics fingerprints. The similarity of semantic fingerprints is computed by measuring their overlap, a method coming from set-theory that has proven to be both highly accurate and efficient for high volumes of data.


By conceptual analogy we extended the Semantic Folding method to numerical data, presenting a novel approach to sensor data integration that overcomes the limitations of conventional sensor fusion methods as described above.

How does Semantic Folding for Sensor Fusion work?

The first step is to create a semantic space based on a reference collection representing the use case. For automobile sensor fusion, for example, a data stream from the car’s sensors capturing all anticipated driving conditions and environments is used as training material. A set of concurrent sensor values is recorded every second and stored in a time-series training file.

Each set of sensor values represents a car status at a given moment. We describe this set as a “context”, analogous to a sentence in textual data, where each sensor value is a “word”.

Creating a semantic map is a fully unsupervised training, where each set of sensor values (=each context) is assigned a specific position within the map, in such a way that similar records are placed close to each other, and dissimilar records are placed far from each other. This spatial organization within a metric space effectively highlights the similarity and dissimilarity among measurement records.

Once the distribution of the sensor contexts within the semantic map is established, the semantic fingerprints for every single sensor value can be generated: for any given value, the system checks in which contexts (or sets of values) it is contained and marks these positions as “1” on an empty map. The semantic fingerprint of a sensor value is a map, where all contexts containing this value are activated. The collection of these semantic fingerprints builds the sensor data dictionary.

How can Semantic Folding solve the current sensor fusion challenges?

Semantic Folding addresses the current issues faced by sensor fusion systems by transforming sensor data into semantic fingerprints – compact, structured representations that reflect the system’s status at any given moment. These semantic fingerprints facilitate real-time processing as they require comparatively little computing resources, simplify the integration process, and enhance the overall system reliability.

By using a unified and meaningful representation for all sensor data, Semantic Folding can efficiently integrate information from multiple sensors, providing a more accurate and robust estimate of the state of the system being monitored. If one sensor fails or provides erroneous data, the system can rely on data from other sensors to maintain performance and accuracy (redundancy).


By using a unified and representation for all sensor data, Semantic Folding enables a simple computing architecture.


Processing semantic fingerprints requires orders of magnitude less compute resources than processing raw data from sensors.


New sensors and features can be added as they become available or as needs evolve without redesigning the entire system.


All sensor data are converted into a unified format that simplifies the integration process.

Use Case Example: Automotive

Modern cars contain a multitude of sensors, monitoring all important subsystems like the engine, the electric system, the lightning, the driver’s cabin etc. All these measurements are relayed to an onboard computer, which detects any potential anomaly and triggers any required follow-up action. Typically, one can expect to receive at least 50-100 measurements per second. With autonomous vehicles, it’s likely that the number of sensor streams will increase even further, making sensor fusion and hence system reliability a real challenge.

What happens when a temperature of 140°C is measured in the engine?

Without a sensor fusion system, the board computer simply compares the measured value with the accepted range of temperatures and triggers a failure procedure if the temperature falls outside this range. But the fact that the engine temperature reaches 140°C can have several causes and not all of them necessitate to trigger an alarm signal – for example, if the car is driving up-hill or if the ambient temperature is particularly high.


Accurately detecting an anomaly requires to put the measured value in its context, which is what Semantic Folding for Sensor Fusion makes possible.


The reference semantic fingerprint of the temperature sensor value of 140 encapsulates the different contexts where such a high temperature manifests, like mountaing driving, highway travel, or summer weather. When driving in real conditions, the car system will compare in real time the actual temperature fingerprints with the reference fingerprint and only trigger an alarm signal if it detects unexpected cluster combinations – for example, a temperature of 140°C while driving down-hill in winter.

Predictive Maintenance with Semantic Folding

To improve both the reliability and efficiency of maintenance strategies, a library of reference fingerprints representing all possible kinds of malfunction can be created – either by capturing actual failure states or synthetically generating sets of values representing specific malfunctions.

When a status fingerprint signals an anomaly, it can be immediately compared to this catalogue of malfunction fingerprints to help determine the possible causes of the anomaly and establish an informed diagnostic.

Given the minimal computational demands, anomaly detection and resolution can occur in real time, even on edge processing platforms with limited computational capabilities. Moreover, since the overlap between the status and the diagnostic fingerprints may evolve over time, this incremental anomaly detection can be leveraged for predictive maintenance, allowing for early intervention before failures become critical. 


How Semantic Folding can help monitor the operational status of municipal water pumps