Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems

doi:10.1109/TPDS.2010.154

	Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems
	Xiong, Jin 1; Hu, Yiming 2; Li, Guojie 1; Tang, Rongfeng 1; Fan, Zhihua 3
	2011-05-01
发表期刊	IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
ISSN	1045-9219
卷号	22 期号:5 页码:803-816
摘要	Most supercomputers nowadays are based on large clusters, which call for sophisticated, scalable, and decentralized metadata processing techniques. From the perspective of maximizing metadata throughput, an ideal metadata distribution policy should automatically balance the namespace locality and even distribution without manual intervention. None of existing metadata distribution schemes is designed to make such a balance. We propose a novel metadata distribution policy, Dynamic Dir-Grain (DDG), which seeks to balance the requirements of keeping namespace locality and even distribution of the load by dynamic partitioning of the namespace into size-adjustable hierarchical units. Extensive simulation and measurement results show that DDG policies with a proper granularity significantly outperform traditional techniques such as the Random policy and the Subtree policy by 40 percent to 62 times. In addition, from the perspective of file system reliability, metadata consistency is an equally important issue. However, it is complicated by dynamic metadata distribution. Metadata consistency of cross-metadata server operations cannot be solved by traditional metadata journaling on each server. While traditional two-phase commit (2PC) algorithm can be used, it is too costly for distributed file systems. We proposed a consistent metadata processing protocol, S2PC-MP, which combines the two-phase commit algorithm with metadata processing to reduce overheads. Our measurement results show that S2PC-MP not only ensures fast recovery, but also greatly reduces fail-free execution overheads.
关键词	Distributed file systems metadata management
DOI	10.1109/TPDS.2010.154
收录类别	SCI
语种	英语
资助项目	Natural Science Foundation of China[60970025] ; National High-Technology Research and Development Program of China[2006AA01A102] ; National High-Technology Research and Development Program of China[2009AA01Z139] ; National High-Technology Research and Development Program of China[2009AA01A129] ; National Science Foundation[CCF-0541103]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
WOS记录号	WOS:000288755400009
出版者	IEEE COMPUTER SOC
引用统计	被引频次：36[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/12606
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Xiong, Jin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Cincinnati, Dept Elect & Comp Engn & Comp Sci, Engn Res Ctr 542, Cincinnati, OH 45221 USA 3.NetEase Com Inc, Beijing 100084, Peoples R China
推荐引用方式 GB/T 7714	Xiong, Jin,Hu, Yiming,Li, Guojie,et al. Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2011,22(5):803-816.
APA	Xiong, Jin,Hu, Yiming,Li, Guojie,Tang, Rongfeng,&Fan, Zhihua.(2011).Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,22(5),803-816.
MLA	Xiong, Jin,et al."Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 22.5(2011):803-816.