Applying data engineering principles to build distributed, scalable, and fault-tolerant data systems

Over the past two decades, we have seen the rapid adoption of distributed data management systems in industry, with many early adopters leading the way. Preferences are shifting from smaller, local, centralized, monolithic systems towards larger, distributed, concurrent, global systems that are scalable, fault-tolerant, and can provide diverse functionalities over a wider range of data types. Organizations are building next-generation data services using many innovations in distributed systems and information management technology: peer-to-peer and web services architectures, industrial-strength clustering and fault-tolerance technologies, large scale reliable storage systems, and efficient indexing and retrieval methods for unstructured data (Armbrust et al., 2010; Bass et al., 2012; Gollapudi, 2021).

At the same time, research efforts in data systems are focusing increasingly on the development of distributed, scalable, and fault-tolerant techniques that can support services such as web-search, information-sensors, click-stream analysis, peer-to-peer storage and publish/subscribe services. There have been interesting ideas, especially in the areas of scalable data access and retrieval services, reliable storage, and high-performance data dissemination services. Today, large amounts of data are being generated and collected by organizations. Simultaneously, businesses are realizing that enormous improvement in profitability can be achieved by employing new tools and approaches for data analysis: mining for knowledge; learning predictive models; performing trend analysis over historical data; performing on-line, real-time analysis and filtering of current data.

Many of these organizations are beginning to analyze transaction data from their business processes in conjunction with traditional data-analysis techniques. By employing technology to analyze and filter the data, businesses can make intelligent decisions about managing customer relationships. With increasingly complete data repositories, businesses are increasingly looking to drive customer relationships by collecting explicit data from customers and analyzing their behavior. A strong tool-set for large scale data analysis will allow organizations to automatically enhance their intelligence infrastructure. As these organizations have entered into the hype cycle for such analysis tools, they expect an understanding ecosystem (Krishnan, 2013; Kleppmann, 2017).