Modern data engineering architectures for financial decision systems

Financial decision systems have gained great interest over the last two decades, as computerized trading, credit risk management, risk hedging in asset derivatives, and credit card fraud detection became widespread. Among the numerous automated systems being provided, the highly volatile financial markets and the requirements for technology upgradability forced most firms to develop their own financial decision systems. Overall, the financial domain has a significant market demand for automatic decision systems. As such systems can be very complex, there is a need for systematic approaches to their design and programming. This paper presents the new emerging area of data engineering for financial decision systems, discussing its market demand and introducing its new analysis, design, and implementation principles based on model-driven architecture and composite applications. It presents results from an extensive empirical study of a large European investment bank, examining its data engineering processes. This research was inspired by the business requirements of data engineering in this specific financial institution, providing a comprehensive view of the field. It represents novel academic scholarship based on rigorous detailed empirical research.

This paper focuses on data engineering for financial decision systems, an area not yet investigated in the literature, indicated as a new important emerging area of research. Data engineering has gained importance with the emergence of data warehousing and data mining technologies. Over the last two decades, there has been great interest in decision systems in the financial domain, propelling the commercial development and academic study of financial decision systems. The explosive growth of these two new emerging application areas has sparked the need for a systematic approach to the engineering of complex financial decision systems, thereby paving the way for the emergence of data engineering for financial decision systems, a new area of research and practice.

Thus, the specter of outperformance by years by the shunning of data still begs to be conjured. For financial decision systems requiring procedural analyses of real-time or sequential data, this paper proposes a new systems-level architectural framework together with semi-aggregative methods to reveal characteristics of the data to aid financial decisions. With this framework and these applicable methods, new class-level decision systems can be made manifest. The proposed framework and methods define a synergistic yet modularized way to help build a better understanding of the inferential interaction between covariates (features) and responses (targets) within financial data.