QQ客服 官方微博 反馈留言

	S-Storm: A Slot-aware Scheduling Strategy for Even Scheduler in Storm
	Qian, Wenjun ; Shen, Qingni ; Qin, Jia ; Yang, Dong ; Yang, Yahui ; Wu, Zhonghai
	2016
关键词	stream data processing storm scheduling load-balancing slot-aware scheduling
英文摘要	Storm has been a popular distributed real-time computation system for stream data processing, which currently provides an even scheduler to distribute all executors and workers of topology among all worker nodes. In this paper, we find that the even scheduler ignores the allocation and dependence relationship among slots. This would bring the load-unbalancing problem when the topology run failed and is killed by its user, or more new machines are extended in Storm cluster. Aiming at solving them, we design the S-Storm, a slot-aware scheduling strategy for even scheduler in Storm, which achieves a fine-grained EvenScheduler using the slot-aware sorting queue and merger factor. S-Storm has the following desirable features: 1) It evenly allocates slots for multitopologies in load-balancing cluster. 2) When load-unbalancing happen, it distributes workers to slots among light-load worker nodes. 3) It achieves a fine-grained EvenScheduler by accessing one slot to the slot-aware sorting queue, and the slot is on the minimum load worker node. 4) It uses the merger factor to avoid over competition of the same resources among the same Spout/Bolt tasks in machine. We implemented S-Storm based on Storm 0.10.0 and used two different applications for performance evaluation. Extensive experimental results show that compared to Storm, S-Storm can achieve a better performance in the average processing time and throughput, while recovering balance from load-unbalancing.; National High Technology Research and Development Program ("863" Program) of China [2015AA016009]; National Natural Science Foundation of China [61232005, 61672062]; Science and Technology Program of Shen Zhen, China [JSGG20140516162852628]; CPCI-S(ISTP); 623-630
语种	英语
出处	18th IEEE International Conference on High Performance Computing and Communications (HPCC) / 14th IEEE International Conference on Smart City (Smart City) / 2nd IEEE International Conference on Data Science and Systems (DSS)
DOI标识	10.1109/HPCC-SmartCity-DSS.2016.191
内容类型	其他
源URL	[http://ir.pku.edu.cn/handle/20.500.11897/470178]
专题	软件与微电子学院
推荐引用方式 GB/T 7714	Qian, Wenjun,Shen, Qingni,Qin, Jia,et al. S-Storm: A Slot-aware Scheduling Strategy for Even Scheduler in Storm. 2016-01-01.

个性服务

查看访问统计

相关权益政策

暂无数据

收藏/分享

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。