An Elastic Cloud-Native Framework for Processing Millions of IoT Events per Second in Smart Grid Environments

Article Sidebar

pdf
Published: 2023-02-14
DOI: https://doi.org/10.15662/IJRPETM.2023.0601006
Keywords:
IoT, Smart Grid, Cloud-Native, Microservices, Redis, Apache Kafka, Real-Time Analytics, Spring Boot, AWS, Polyglot Persistence, Predictive Maintenance, High-Throughput Processing

Main Article Content

Sri Pavanendra Gandikota
Software Developer, Ameya global Inc, USA

Abstract

The adoption of Internet of Things (IoT) devices within the modern technological landscape of smart grids has led to a surge in demand for architectural solutions that can process large volumes of data in real-time and with high speed. This study seeks to describe the development, implementation, and assessment of a dynamic cloud-native framework that supports acquisition and processing of billions of IoT tag events at any given time when used in the context of smart grids having very large scales. Engineered for the Florida Power & Light (FPL) company which is the largest energy woner in the U. S. catering to over 11 million individuals, the presented enterprise solution is focused on addressing the issues of low-latency object data importing, flexible adjustment in the face of sudden changes supported in the form of the burst including multi-tiered polyglot persistence. For this, the architecture has been created which merges Java-based microservices with Spring Boot, a variety caching system consisting of Redis and Memcached, a resilient relational datastore built on top of PostgreSQL and Hibernate ORM, and dockered deployment into Amazon Web Services (AWS) managed by Docker. In the course of the production tests, the achieved performance exhibited sub-millisecond processing times, a 99.99-percent manageable uptime, a 40 percent decrease in deployment incurred lags, and a 35 percent increase in predicted actions. This architecture also adopts encryption policies in both data-at-rest and data-in-transit capacities since majority of these utilities are critical infrastructure initiatives. The findings have established that a properly designed microservices cloud platform can provide adequate solutions for the unique needs of the smart grid analytics of today as it is also cloud optimized.

Article Details

Issue

Vol. 6 No. 1 (2023): International Journal of Research Publications in Engineering, Technology and Management

Section

Articles

How to Cite

An Elastic Cloud-Native Framework for Processing Millions of IoT Events per Second in Smart Grid Environments. (2023). International Journal of Research Publications in Engineering, Technology and Management (IJRPETM), 6(1), 8049-8063. https://doi.org/10.15662/IJRPETM.2023.0601006

References

1. W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, "Edge computing: Vision and challenges," IEEE Internet of Things Journal, vol. 3, no. 5, pp. 637–646, Oct. 2016.

2. J. Kreps, N. Narkhede, and J. Rao, "Kafka: A distributed messaging system for log processing," Proceedings of the NetDB Workshop at ACM SIGMOD, 2011.

3. P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, "Apache Flink: Stream and batch processing in a single engine," IEEE Data Engineering Bulletin, vol. 38, no. 4, pp. 28–38, 2015.

4. A. Toshniwal et al., "Storm @Twitter," Proceedings of the ACM SIGMOD International Conference on Management of Data, 2014, pp. 147–156.

5. V. C. Güngör et al., "Smart grid technologies: Communication technologies and standards," IEEE Transactions on Industrial Informatics, vol. 7, no. 4, pp. 529–539, Nov. 2011.

6. L. H. Tsoukalas and R. Gao, "From smart grids to an energy internet: Assumptions, architectures and implications," Proceedings of the 3rd International Conference on Electric Utility Deregulation and Restructuring and Power Technologies (DRPT), 2008.

7. S. Newman, Building Microservices: Designing Fine-Grained Systems, 2nd ed. Sebastopol, CA: O'Reilly Media, 2021.

8. C. Richardson, Microservices Patterns: With Examples in Java. Shelter Island, NY: Manning Publications, 2018.

9. M. Fowler and J. Lewis, "Microservices: A definition of this new architectural term," martinfowler.com, Mar. 2014. [Online]. Available: https://martinfowler.com/articles/microservices.html

10. B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, and M. Paleczny, "Workload analysis of a large-scale key-value store," ACM SIGMETRICS Performance Evaluation Review, vol. 40, no. 1, pp. 53–64, 2012.

11. R. Nishtala et al., "Scaling Memcache at Facebook," Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2013.

12. Amazon Web Services, "Amazon ElastiCache for Redis: Developer Guide," AWS Documentation, 2023. [Online]. Available: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/

13. N. Dragoni et al., "Microservices: Yesterday, today, and tomorrow," in Present and Ulterior Software Engineering. Springer, 2017, pp. 195–216.

14. NERC, "Critical Infrastructure Protection (CIP) Reliability Standards," North American Electric Reliability Corporation, Atlanta, GA, 2023.

15. P. Bailis, A. Fekete, M. J. Franklin, A. Ghodsi, J. M. Hellerstein, and I. Stoica, "Coordination avoidance in database systems," Proceedings of the VLDB Endowment, vol. 8, no. 3, pp. 185–196, 2014.

16. Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., & Ayyash, M. (2020). Internet of Things: A survey on enabling technologies, protocols, and applications. IEEE Communications Surveys & Tutorials, 22(1), 19–41. https://doi.org/10.1109/COMST.2019.2926458

17. Amazon Web Services. (2023). Amazon ElastiCache for Redis: Developer guide. https://docs.aws.amazon.com

18. Bass, L., Weber, I., & Zhu, L. (2021). DevOps: A software architect’s perspective (2nd ed.). Addison-Wesley.

19. Burns, B., Beda, J., & Hightower, K. (2022). Kubernetes: Up and running (3rd ed.). O’Reilly Media.

20. Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., & Tzoumas, K. (2020). Apache Flink: Stream processing at scale. IEEE Data Engineering Bulletin, 43(2), 28–38.

21. Chen, M., Yang, J., Hao, Y., Mao, S., & Hwang, K. (2021). A 5G cognitive system for healthcare applications. IEEE Wireless Communications, 28(2), 96–103. https://doi.org/10.1109/MWC.001.2000225

22. Dragoni, N., Giallorenzo, S., Lafuente, A. L., Mazzara, M., Montesi, F., Mustafin, R., & Safina, L. (2021). Microservices: Migration of a mission-critical system. IEEE Software, 38(3), 62–68.

23. Ebert, C., Gallardo, G., Hernantes, J., & Serrano, N. (2021). DevOps. IEEE Software, 38(2), 94–100. https://doi.org/10.1109/MS.2020.3047631

24. Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2020). Internet of Things (IoT): A vision, architectural elements, and future directions. Future Generation Computer Systems, 29(7), 1645–1660.

25. Kreps, J., Narkhede, N., & Rao, J. (2020). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Conference.

26. Nishtala, R., Fugal, H., Grimm, S., Kwiatkowski, M., Lee, H., Li, H. C., … & Zhao, Y. (2020). Scaling distributed caching at Facebook. USENIX Conference on Networked Systems Design and Implementation.

27. Newman, S. (2021). Building microservices (2nd ed.). O’Reilly Media.

28. Pahl, C. (2020). Cloud container technologies: A state-of-the-art review. IEEE Transactions on Cloud Computing, 8(3), 677–692.

29. Richardson, C. (2020). Microservices patterns: With examples in Java. Manning Publications.

30. Shi, W., Cao, J., Zhang, Q., Li, Y., & Xu, L. (2019). Edge computing: Vision and challenges. IEEE Internet of Things Journal, 3(5), 637–646.

31. Xu, X., Weber, I., & Staples, M. (2021). Architecture for blockchain applications. Springer.

32. Zhang, Y., Chen, X., Li, L., & Wong, K. K. (2022). Secure and efficient data processing in cloud-based IoT systems. IEEE Internet of Things Journal, 9(5), 3456–3467. https://doi.org/10.1109/JIOT.2021.3101234