Building scalable and secure data platforms for advisory services

Real-world data is gone unexploited at various companies and organizations, when shared but under controlled access. Data exchange and sharing can result in valuable insights in affected domains, and may boost cooperation and stimulate novel services. Despite many upsides of sharing and exchanging data, significant challenges must be addressed. The importance of data sharing and exchange in DeFi is further fueled by the current architecture of DeFi protocols and related risks of collecting and maintaining data in one location. Full homogenization of data at one location may lead to security issues, as reflected by the various high-profile breaches in crypto. New solutions must safeguard parties involved when sharing and exchanging their data between either analysis or analytics. The lack of both standard representation of knowledge and interconnectedness is a bottleneck for scaling organization-wide and ecosystems-wide applications with the following three interrelated areas as key challenges. A lack of robustness of computational back-end systems, supporting interfaces and analytics of growing complexity and capacity, becomes more challenging with the aforementioned growing complexity of (and demand for) data-processing pipelines. The choice of a primary programming language is pivotal when designing a data platform in terms of performance, the tooling and libraries available, and scalability. Many software engineers have faced the challenge of either designing a new back-end system, or improving and scaling an already existing one. One such area where the design choices are crucial is that of educational and business solutions revolving around the one-off passing, storing and processing of huge amounts of data. In this paper, a common problem in both areas is defined: the lack of heterogeneity of the data sets sent, and the impossibility to practically expand on either system. Many companies and organizations collect, store and analyze on-chain and protocol activity data. Drafts of information representation are standardized and agreed upon for most widely used datasets. However, even while two companies may obtain their data from the same data sources, they still may not be able to profitably query the same datasets.This session is inspired by and describes a fast, efficient programming model for the future of the finance industry to support scalable computation on the data and input formats of the next decade. This platform will allow clients to capture and store large text/image/audio inputs into a set of document stores and arrays. These document stores will allow clients to perform chunk-level data manipulation taking document-less data into future-compliant formats. Arrays will allow clients to run elastically scalable computation keeping data at rest within their data-compliant storage solutions.