Scalable Data Integration and Distributed Processing Frameworks

The enormous, and still rapidly increasing, volume of data produced and collected by organizations introduces the need for scalable systems that integrate and process this data efficiently and reliably. A scalable data integration framework is primarily responsible for bringing together data from distributed, heterogeneous sources in a way that minimizes the burden on data producers while maximizing data consumers’ ability to find and use data for their applications. Reusable, high-quality data can then be made available to users and applications throughout the data life cycle, supporting business intelligence, machine learning, data science, and other tasks. It may even be possible for end users to compose queries on demand without custom integration while ensuring data quality and consistency at query time. Yet these techniques cannot completely satisfy all demands, so data producers are also under pressure to move towards a centralized model, enabling data scientists to more easily share and explore data. The benefits of centralized storage are accompanied by a need for scalable data processing frameworks that can respond quickly to user queries and data-analysis jobs despite the ever-increasing volume of stored data.

To meet this diverse set of requirements, a variety of scalable data integration and processing systems and tools have been conceived and implemented over the past years. The aim of such scalable frameworks is for users to be largely unconcerned about how or where the data is stored and to be able to analyze or visualize the data with minimal delays—either by processing the combined data sets in a single job or by using a gateway to quickly access recently used, central storage. Consequently, integrated system architectures are often characterized by a clean separation of data storage, data management and integration, and data processing, allowing different processing frameworks to be used at different stages of the data-science life cycle and giving systems the freedom to grow in different directions.