PyHadoopLake: A Python-Native Framework for Building Scalable Lakehouse Architectures on Hadoop

Abstract

This paper presents PyHadoopLake which is a Python-native framework designed to simplify and modernize the construction of scalable lakehouse architectures on top of the Hadoop distributed file system. The proposed framework integrates extract transform and load operations directly within a Python-driven pipeline while leveraging the distributed processing capabilities of Hadoop. The methodology focuses on combining PySpark with HDFS and Delta Lake table formats to deliver a unified and reproducible data engineering environment. Experimental results demonstrate meaningful improvements in pipeline throughput and data processing latency when compared with traditional Hadoop ETL approaches. The framework offers a practical and extensible solution for organizations seeking to modernize their data infrastructure without abandoning their existing Hadoop investments.