Real-Time Bayesian Risk Intelligence AI-Augmented Threat Detection in Cloud–Lakehouse Systems for Data-Limited Environments

Abstract

As organizations increasingly adopt cloud–lakehouse architectures to unify analytical and operational workloads, the need for timely and reliable threat detection becomes critical—especially in environments with limited labeled data. This work proposes a Real-Time Bayesian Risk Intelligence framework that integrates probabilistic modeling, streaming analytics, and AI-augmented inference to detect cyber-security threats under uncertainty. The system leverages Bayesian networks and hierarchical priors to dynamically update risk estimates as new telemetry arrives from distributed lakehouse components, enabling robust reasoning even when observations are sparse, noisy, or partially missing. We incorporate lightweight edge-side feature extraction, generative models for imputing incomplete signals, and online learning mechanisms to maintain model calibration while respecting compute and cost constraints. Experimental evaluations across simulated and real-world cloud telemetry streams demonstrate that the Bayesian approach outperforms conventional anomaly detectors in low-data regimes, reducing false positives while improving early detection of stealthy behaviors such as lateral movement and privilege escalation. The proposed architecture offers a principled, explainable, and resource-efficient pathway for operationalizing cyber risk intelligence in modern data ecosystems.