Selection and replacement algorithms for memory performance improvement in spark

M Duan, K Li, Z Tang, G Xiao… - … and Computation: Practice …, 2016 - Wiley Online Library
Concurrency and Computation: Practice and Experience, 2016Wiley Online Library
As a parallel computation framework, Spark can cache repeatedly resilient distribution
datasets (RDDs) partitions in different nodes to speed up the process of computation.
However, Spark does not have a good mechanism to select reasonable RDDs to cache their
partitions in limited memory. In this paper, we propose a novel selection algorithm, by which
Spark can automatically select the RDDs to cache their partitions in memory according to the
number of use for RDDs. Our selection algorithm speeds up iterative computations …
Summary
As a parallel computation framework, Spark can cache repeatedly resilient distribution datasets (RDDs) partitions in different nodes to speed up the process of computation. However, Spark does not have a good mechanism to select reasonable RDDs to cache their partitions in limited memory. In this paper, we propose a novel selection algorithm, by which Spark can automatically select the RDDs to cache their partitions in memory according to the number of use for RDDs. Our selection algorithm speeds up iterative computations. Nevertheless, when many new RDDs are chosen to cache their partitions in memory while limited memory has been full of them, the system will adopt the least recently used (LRU) replacement algorithm. However, the LRU algorithm only considers whether the RDDs partitions are recently used while ignoring other factors such as the computation cost and so on. We also put forward a novel replacement algorithm called weight replacement (WR) algorithm, which takes comprehensive consideration of the partitions computation cost, the number of use for partitions, and the sizes of the partitions. Experiment results show that with our selection algorithm, Spark calculates faster than without the algorithm, and we find that Spark with WR algorithm shows better performance. Copyright © 2015 John Wiley & Sons, Ltd.
Wiley Online Library
以上显示的是最相近的搜索结果。 查看全部搜索结果