Distributed Data Processing and Storage Architectures

The rapid proliferation of cheap storage and high-speed networks has rendered the construction of distributed data processing and storage systems not only feasible but also essential. Such systems are key not just for data sharing or fault-tolerance purposes but also to cope with the immense quantities of data produced and stored during the normal functioning of Internet services. Currently, few applications can afford to use a single machine for data processing, and for those that cannot, the problems of redundancy in processing are much less critical than the problems of latency and availability. For even fewer applications is a single machine remotely sensible for data storage, bandwidth and maintenance costs being measured in life’s hours. Consequently, distributed data systems are evolving according to the principles of decoupling storage and compute decoupling being so obvious that any group developing and scaling its workloads naturally reaches for such an architecture. At some point in the development of a workload, data locality no longer can be taken for granted, and when it does and task scheduling becomes a consideration a new cluster-based batch-processing framework emerges.