Distributed Metadata Management of Mass Storage System in High Energy Physics

doi:10.1088/1742-6596/898/6/062003

	Distributed Metadata Management of Mass Storage System in High Energy Physics
	Du, Ran; Cheng, Yaodong; Shi, Jingyan; Kan WX(阚文枭); Chen G(陈刚); Shi JY(石京燕); Cheng YD(程耀东); Du R(杜然); Huang QL(黄秋兰); Kan, Wenxiao
刊名	Journal of Physics: Conference Series
	2017
卷号	898 期号:6 页码:062003
ISSN号	1742-6588
DOI	10.1088/1742-6596/898/6/062003
文献子类	Proceedings Paper
英文摘要	In this contribution, we design and implement a dynamic and scalable distributed metadata management system(StarFS) to High Energy Physics (HEP) mass storage system. Particularly, we discuss the key technologies of the distributed metadata management system. We propose a new algorithm named Adaptive Directory Sub-tree Partition(ADSP) for metadata distribution. ADSP divides the filesystem namespace into sub-trees with directory granularity. Sub-trees will be stored on storage devices in flat structure, whose location and file attributes are recorded as extended attributes. The placement of sub-tree is adjusted adaptively according to the workload of metadata cluster so that the load balance could be improved and metadata cluster could be extended dynamically. Experiments show that ADSP achieves higher metadata performance and scalability compared to Gluster and Lustre. We also propose a new algorithm called Distributed Unified Layout(DULA) to improve dynamic scalability and efficiency of data positioning. A system with DULA could provide uniform data distribution and efficient data positioning. DULA is an improved consistent hashing algorithm which is able to locate data in O(1) without the help of routing information. Experiments prove that the better uniform data distribution and efficient data access can be achieved by DULA. This work is validated in YangBaJing Cosmic Ray(YBJ) experiment. © Published under licence by IOP Publishing Ltd.
电子版国际标准刊号	1742-6596
会议地点	San Francisco, CA, United states
会议日期	October 10, 2016 - October 14, 2016
语种	英语
内容类型	期刊论文
源URL	[http://ir.ihep.ac.cn/handle/311005/285435]
专题	中国科学院高能物理研究所
作者单位	1.University of Chinese, Academy of Sciences, Beijing, China 2.IHEP Computing Center, P.O.Box 918-7, 19B Yuquan Road, Beijing; 100049, China;
推荐引用方式 GB/T 7714	Du, Ran,Cheng, Yaodong,Shi, Jingyan,et al. Distributed Metadata Management of Mass Storage System in High Energy Physics[J]. Journal of Physics: Conference Series,2017,898(6):062003.
APA	Du, Ran.,Cheng, Yaodong.,Shi, Jingyan.,阚文枭.,陈刚.,...&Chen, Gang.(2017).Distributed Metadata Management of Mass Storage System in High Energy Physics.Journal of Physics: Conference Series,898(6),062003.
MLA	Du, Ran,et al."Distributed Metadata Management of Mass Storage System in High Energy Physics".Journal of Physics: Conference Series 898.6(2017):062003.