Research and Publications

Introduction

RMIDDM introduces a fully unsupervised and interpretable concept drift detection method that uses Radial Basis Function networks to form evolving concept groups and a Markov Chain to track transitions between them, allowing the system to detect both abrupt and gradual distribution shifts without labels, windows, or error monitoring. By modeling how concepts emerge and change over time, the method provides transparent, graph‑based explanations and demonstrates strong performance on synthetic benchmarks as well as real‑world data such as COVID‑19 case and vaccination trends.

Problems addressed

• Most drift detectors rely on supervised error signals, making them unusable when labels are delayed or unavailable.

• Existing unsupervised methods often lack interpretability, making it difficult to understand how or why a drift was detected.

• Window‑based approaches require manual tuning and may fail to capture nonlinear changes in the data distribution.

• Real‑world streaming applications need lightweight, real‑time drift detection that adapts as concepts evolve.

Methods overview 

RMIDDM uses Radial Basis Function networks to cluster incoming data into evolving concept groups and models transitions between these groups using a Markov Chain. Drift is detected when transition probabilities indicate a significant change in how concepts relate over time. This fully unsupervised, window‑free approach captures both abrupt and gradual shifts while providing interpretable, graph‑based insights into how concepts emerge, evolve, and disappear.

Key outcomes

• Provides a fully unsupervised, interpretable drift detection method.

• Avoids window tuning through a model‑based, nonlinear approach.

• Offers transparent visual explanations via Markov transition graphs.

• Performs strongly on both synthetic and real‑world data streams.

• Detects real‑world distribution shifts, including pandemic‑related changes.

Experimental findings

Across synthetic benchmarks and real‑world datasets, RMIDDM performs competitively with leading drift detectors despite requiring no labels or window tuning. The method successfully identifies meaningful distribution changes—such as shifts in COVID‑19 case, death, and vaccination patterns—demonstrating its practical value for monitoring dynamic, evolving systems.