Harmonizing the Unharmonized: A Lakehouse Reference Architecture for Post-Acquisition Multi-ERP Data Consolidation
Main Article Content
Abstract
Industrial acquisitions bring heterogeneous ERP estates into the parent company: legacy mid-market MRP systems, homegrown order and pricing applications, and a separate SAP landscape on the acquirer side. Leadership nonetheless expects consolidated reporting within months of closing. This paper presents a phased lakehouse reference architecture for post-acquisition data harmonization, developed in an integration program at a global Tier-1 industrial manufacturer that began in early 2023. The architecture combines Azure Data Factory ingestion and SAP SLT trigger-based replication from SAP S/4HANA, a bronze/silver/gold lakehouse on Azure Data Lake Storage processed with Synapse, a Snowflake serving tier, and a master-data conformance service for customers, items, suppliers, fiscal calendars and units of measure. Its output is a small set of certified golden datasets that pass explicit reconciliation, conformance and lineage gates. We contribute a weighted due-diligence framework for platform selection, a canonical-model method for mapping ERP semantics, a probabilistic matching and crosswalk design, and the organizational tactics used to build adoption, including a recurring open enablement session. Early results cover the first five source systems. In a representative evaluation, onboarding effort fell from 50 person-days for the first legacy ERP to 32–37 person-days for later ERP sources, reconciliation remained about 30% of effort, and weekly analyst hours spent on manual consolidation fell from about 62 to 29 over 30 weeks. We argue that a harmonized, governed data layer is a precondition for the AI-augmented analytics that large language models now make plausible, and we outline the remaining work
Article Details
Section
How to Cite
References
[1] F. Wijnhoven, T. Spil, R. Stegwee, and R. Tjang A Fa, "Post-merger IT integration strategies: An IT alignment perspective," J. Strategic Information Systems, vol. 15, no. 1, pp. 5–28, 2006.
[2] M. Mehta and R. Hirschheim, "Strategic alignment in mergers and acquisitions: Theorizing IS integration decision making," J. Association for Information Systems, vol. 8, no. 3, 2007.
[3] M. Armbrust, A. Ghodsi, R. Xin, and M. Zaharia, "Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics," in Proc. 11th Conf. Innovative Data Systems Research (CIDR), 2021.
[4] L. DalleMule and T. H. Davenport, "What's your data strategy?" Harvard Business Review, vol. 95, no. 3, pp. 112–121, 2017.
[5] W. H. Inmon, Building the Data Warehouse, 4th ed. Indianapolis, IN, USA: Wiley, 2005.
[6] R. Kimball and M. Ross, The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, 3rd ed. Indianapolis, IN, USA: Wiley, 2013.
[7] D. Linstedt and M. Olschimke, Building a Scalable Data Warehouse with Data Vault 2.0. Waltham, MA, USA: Morgan Kaufmann, 2015.
[8] M. Armbrust et al., "Delta Lake: High-performance ACID table storage over cloud object stores," Proc. VLDB Endowment, vol. 13, no. 12, pp. 3411–3424, 2020.
[9] B. Dageville et al., "The Snowflake elastic data warehouse," in Proc. ACM SIGMOD Int. Conf. Management of Data, 2016, pp. 215–226.
[10] Z. Dehghani, Data Mesh: Delivering Data-Driven Value at Scale. Sebastopol, CA, USA: O'Reilly Media, 2022.
[11] I. A. Machado, C. Costa, and M. Y. Santos, "Data mesh: Concepts and principles of a paradigm shift in data architectures," Procedia Computer Science, vol. 196, pp. 263–271, 2022.
[12] A. Doan, A. Halevy, and Z. Ives, Principles of Data Integration. Waltham, MA, USA: Morgan Kaufmann, 2012.
[13] E. Rahm and P. A. Bernstein, "A survey of approaches to automatic schema matching," The VLDB Journal, vol. 10, no. 4, pp. 334–350, 2001.
[14] E. Rahm and H. H. Do, "Data cleaning: Problems and current approaches," IEEE Data Engineering Bulletin, vol. 23, no. 4, pp. 3–13, 2000.
[15] I. P. Fellegi and A. B. Sunter, "A theory for record linkage," J. American Statistical Association, vol. 64, no. 328, pp. 1183–1210, 1969.
[16] P. Christen, Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Berlin, Germany: Springer, 2012.
[17] D. Loshin, Master Data Management. Burlington, MA, USA: Morgan Kaufmann, 2009.
[18] A. Dreibelbis, E. Hechler, I. Milman, M. Oberhofer, P. van Run, and D. Wolfson, Enterprise Master Data Management: An SOA Approach to Managing Core Information. Upper Saddle River, NJ, USA: IBM Press, 2008.
[19] DAMA International, DAMA-DMBOK: Data Management Body of Knowledge, 2nd ed. Basking Ridge, NJ, USA: Technics Publications, 2017.
[20] V. Khatri and C. V. Brown, "Designing data governance," Communications of the ACM, vol. 53, no. 1, pp. 148–152, 2010.
[21] R. Y. Wang and D. M. Strong, "Beyond accuracy: What data quality means to data consumers," J. Management Information Systems, vol. 12, no. 4, pp. 5–33, 1996.
[22] P. Vassiliadis, "A survey of extract-transform-load technology," Int. J. Data Warehousing and Mining, vol. 5, no. 3, pp. 1–27, 2009.
[23] M. Kleppmann, Designing Data-Intensive Applications. Sebastopol, CA, USA: O'Reilly Media, 2017.
[24] T. Brown et al., "Language models are few-shot learners," in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 1877–1901.
[25] D. Sculley et al., "Hidden technical debt in machine learning systems," in Proc. Advances in Neural Information Processing Systems (NIPS), 2015, pp. 2503–2511.
[26] N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, "Data lifecycle challenges in production machine learning: A survey," ACM SIGMOD Record, vol. 47, no. 2, pp. 17–28, 2018.