SRE-Driven Reliability Framework for Enterprise Data Platforms
Main Article Content
Abstract
Business data platforms, which exist today, can enable major business processes through the use of massive data streams, analytics platforms, dashboards, reports and products derived through big data. The growing amounts of data, distributed architecture and even intricate dependencies have however rendered it difficult to ensure that there is a consistent reliability and operational stability. This work suggests an SRE-based Reliability Framework of Enterprise Data Platforms, which would use the concepts of Site Reliability Engineering (SRE) to data engineering and analytics experience. Data reliability is operationally represented and numerically measured according to the proposed framework with the help of Service Level Indicators (SLIs), Service Level Objectives (SLOs) and error budgets. It evaluates the key metrics that include availability of pipelines, freshness of the data, quality of the data, processing latency, availability of the dashboards, report delivery and endurance of the analytics service. It is also seen within the framework that there is also a repetitive reliability lifecycle that includes reliability measurement, observability, anomaly detection, incident management, automated remediation and post incident learning. Under an error-budget mechanism, the organization is able to trade features development and stability in the operation by balancing out the changes as reliability goals are met each time. Dependency-aware monitoring and reliability scoring between interconnected data products and pipelines, is also introduced by the framework. The goals of this integration of engineering-data governance-operational analytics will be to decrease the number of incidents and enhance Mean Time to Detect (MTTD) and Mean Time to Recover (MTTR) and trust of enterprise data services. The schema offers a growing base towards attaining quantifiable, preventive and enduring information platform trustworthiness.
Article Details
Section
How to Cite
References
[1] M. R. Mekala, “Observability in AI-driven pipelines: A framework for real-time monitoring and debugging,” International Journal of Research in Computer Applications and Information Technology, vol. 8, no. 1, pp. 686–702, 2025.
[2] N. Bangad, V. Jayaram, M. S. Krishnappa, A. R. Banarse, D. M. Bidkar, A. Nagpal, and V. Parlapalli, “A theoretical framework for AI-driven data quality monitoring in high-volume data environments,” arXiv preprint arXiv:2410.08576, 2024.
[3] S. Patel, R. Gupta, and T. Shaw, “Standardizing AI-driven data quality checks within mesh architectures,” Journal of Data Science and Quality Assurance, vol. 12, no. 1, pp. 70–85, 2023.
[4] L. Hummer, S. Green, and M. Lee, “Data lineage techniques for data-intensive applications,” Data Engineering Review, vol. 15, no. 4, pp. 210–225, 2023.
[5] X. Liu and J. Smith, “Real-time AI applications for monitoring data integrity,” Journal of Data Integrity and Quality, vol. 5, no. 2, pp. 34–45, 2023.
[6] International Organization for Standardization, ISO/IEC 25012:2008—Software Engineering—Software Product Quality Requirements and Evaluation (SQuaRE)—Data Quality Model. Geneva, Switzerland: ISO, 2008.
[7] IEEE, IEEE Standard for an Architectural Framework for the Internet of Things (IoT), IEEE Std 2413-2022. Piscataway, NJ, USA: IEEE, 2022.
[8] IEEE, IEEE Standard for Intercloud Interoperability and Federation (P2302). Piscataway, NJ, USA: IEEE, 2021.
[9] E. Tabassi, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1. Gaithersburg, MD, USA: National Institute of Standards and Technology, Jan. 2023, doi: 10.6028/NIST.AI.100-1.
[10] IEEE, IEEE Standard for Transparency of Autonomous Systems, IEEE Std P7001-2022. Piscataway, NJ, USA: IEEE, 2022.
[11] K. Shah, S. Gami, and A. Trehan, “An intelligent approach to data quality management: AI-powered quality monitoring in analytics,” International Journal of Advanced Research in Science Communication and Technology, vol. 4, no. 3, 2024.
[12] Gartner, Gartner AI Maturity Model. Stamford, CT, USA: Gartner, Inc., 2017.
[13] European Union, Artificial Intelligence Act. Brussels, Belgium: European Commission, 2024.
[14] European Union, General Data Protection Regulation (GDPR). Brussels, Belgium: European Union, 2018.