Graham E Fagg, Thara Angskun, George Bosilca, Jelena Pjesivac-Grbovic, and Jack J Dongarra (2005)
Scalable Fault Tolerant MPI: Extending the Recovery Algorithm
In: Proceedings of Recent Advances in Parallel Virtual Machine and Messaging Passing Interface Users' Group Meeting Euro PVMMPI 2005, chap. Volume 3666, pp. pp 67-75, Springer Heidelberg, Lecture Notes in Computer Science.
Fault Tolerant MPI (FT-MPI)[6] was designed as a solution to allow applications different methods to handle process failures beyond simple check-point rest art schemes. The initial implementation of FT-MPI included a robust heavy weight system state recovery algorithm that was designed to manage the membership of MPI commun icators during multiple failures. The algorithm and its implementation although robust, was very conservative and this effected its scalability on both very large cluste rs as well as on distributed systems. This paper details the FT-MPI recovery algorithm and our initial experiments with new recovery algorithms that are aimed at being b oth scalable and latency tolerant. Our conclusions shows that the use of both topology aware collective communication and distributed consensus algorithms together produ ce the best results.