查看文章

With the advent of big data processing frameworks, the performance of data-parallel applications is heavily affected by the time it takes to read input data, making it important to improve data locality. Existing methods in achieving data locality have primarily focused on selecting machines to place tasks of applications. Nevertheless, the set of machines that an application can choose from is determined by a cluster manager, which is oblivious to the location of data in existing resource sharing frameworks. In this paper, we design, implement and evaluate Custody, a new cluster management framework that helps to maximize data locality by allocating the executor processes with local access to data to those applications in need. Custody achieves this objective by dynamically collecting runtime information of an application's input data and by effectively allocating executors among and within applications through …