Memory-Based Multi-Processing Method For Big Data Computation (1908.03617v1)

Abstract: The evolution of the Internet and computer applications have generated colossal amount of data. They are referred to as Big Data and they consist of huge volume, high velocity, and variable datasets that need to be managed at the right speed and within the right time frame to allow real-time data processing and analysis. Several Big Data solutions were developed, however they are all based on distributed computing which can be sometimes expensive to build, manage, troubleshoot, and secure. This paper proposes a novel method for processing Big Data using memory-based, multi-processing, and one-server architecture. It is memory-based because data are loaded into memory prior to start processing. It is multi-processing because it leverages the power of parallel programming using shared memory and multiple threads running over several CPUs in a concurrent fashion. It is one-server because it only requires a single server that operates in a non-distributed computing environment. The foremost advantages of the proposed method are high performance, low cost, and ease of management. The experiments conducted showed outstanding results as the proposed method outperformed other conventional methods that currently exist on the market. Further research can improve upon the proposed method so that it supports message passing between its different processes using remote procedure calls among other techniques.