The Join task between Spark large tables takes a long time to run and produces a lot of disk
I/O, network I/O and disk occupation in the Shuffle process. This paper proposes a …

Currently, the estimated amount of data created daily have reached the threshold of
petabytes or even zettabytes globally. It is no wonder that traditional data processing …

In the era of information explosion, the amount of data generated is increasing day by day,
reached the threshold of petabytes or even zettabytes. In order to extract useful information …

Currently, we are dealing with large scale applications, which in turn generate massive
amount of data and information. Large amount of data often requires processing algorithms …

In recent years, the Spark memory computing framework has risen rapidly, and the data
processing speed has been greatly improved. However, the upper limit of speed is limited by …

The theta-join between tables is a common operation in the data query and statistical
analysis. When dealing with large amounts of data, it will produce a great deal of cost. The …

The join operation is a fundamental building block of parallel data processing. Unfortunately,
it is very resource-intensive to compute an equi-join across massive datasets. The …

This paper presents a methodology for the optimization of parallel join execution. Past
research on parallel join methods mostly focused on the design of algorithms for partitioning …

The ratio of disk capacity to disk transfer rate typically increases by 10× per decade. As a
result, disk is becoming slower from the view of applications because of the much larger …

In order to interpret, enrich, and analyze the streaming data, stream applications often
access the data stored in an external database. Although there has been a lot of studies on …