The integration of big data pipelines into financial decision-making processes

The world of financial decision-making is entering the big data age; desk-based decision-making will shortly be out of date. New desk-free ways of informing decisions, triggered and delivered by data exhausts, will emerge. Data itself will not be the problem—data exhausts, both structured and unstructured, will flow at unprecedented rates. Making intuitive judgments using available analysis will be the new normal. Big data pipelines have the potential to change financial decision-making processes and to unlock previously unseen added value.

Traditionally, analysts operated at their desks with server-based terminal technology; they would use a watch-list of potential financial opportunities, which would arrive with little or no indication of significance. Where opportunities were spotted, the analyst would then run analyses priced at fixed costs against judgment-based buy/sell decisions. These analyses were relatively superficial, covering no more than several hundred rows of data against a backdrop of unconventional and overwhelmingly more important narratives. The recognition of new questions that had value added was a function of experience and subject domain knowledge. Full narrative texts, valued at thousands of dollars per analysis, contained systematic analysis of potential opportunities with likely buy/sell timing attached. The chain of events leading to potential prizes was evidenced in the data; however, these subjects were not only selected by less than perfect agents but also would become rapidly exhausted.

Decisions by analysts, almost all of whom were financially and academically high status, were typically accepted; some were dropped by brokers inducing a place of declined requests. Ad hoc analyst-generated questions were seldom but grew in popularity. Analyst-driven data fall-offs were often partial in nature and could result in substantial flows of unreturned funds. Query interfaces were seldom designed to deal with the mostly unstructured data items on the periphery of stakeholder interest. Prioritization was an issue. Analysts routinely added users and cases to query lists made by predecessors, but after a time, diminishing marginal returns set in. In addition, the need for information transformation, normally chained all at once, was an issue. The results generated bore little resemblance to what was needed.