Building a scalable and resilient data infrastructure to power intelligent financial services

We live in a world of data, one whose volume is growing at an exponential rate. Recent estimates indicate that the volume of data created globally doubles approximately every two years, an increase driven in equal parts by the growing digitization of all sectors of the economy and by increasing improvements in the speed, resolution, and functionality of digital recording instruments. Financial services and technology firms have recently constructed large-scale data infrastructures to process the data from this explosion, in efforts to lower costs, improve revenues, and mitigate against risks (Cilliers & de Waal, 2017; Inoue & Matsubara, 2018; Chan & Mukherjee, 2021).

These data infrastructures collect, store, analyze, and visualize data from different sources, in different formats, and of different types, including structured, schemaless, and unstructured data. The different sources include the financial or economic activity of firms, households, institutions, and the government; the thousands of financial and macroeconomic variables reported by various governments and multilateral institutions; the millions of financial transactions recorded every day around the world; the millions of messages, reviews and information published online by users on social media and collaborative platforms; and the news articles published daily by press agencies. The data infrastructures analyze these data to produce alerts on the changes in the states of the firms, customers, or institutions; visualize dashboards to monitor the key variables and trends; and compound signals to be used in risk models or forecasts. Insights from these insights help decision-makers in financial services firms and corporations, such as chief executive officers, chief financial officers, chief risk officers, and trading desks, make complex strategic decisions daily. As with any other technology, the adoption, integration, and deployment of these data infrastructures may be governed by technological, economic, behavioral, financial, institutional, political, or regulatory factors ( Pfohl & Gomm, 2019; Kowalski & Rungta, 2020).