Evaluating Methods of Transferring Large Datasets

Kopeć, Jakub

Evaluating Methods of Transferring Large Datasets

dc.contributor.author	Kopeć, Jakub
dc.contributor.organization	Interdisciplinary Centre for Mathematical and Computational Modelling, University of Warsaw	en
dc.date.accessioned	2022-11-22T11:52:45Z
dc.date.available	2022-11-22T11:52:45Z
dc.date.issued	2022
dc.description.abstract	Our society critically depends on data, Big Data. The humanity generates and moves data volumes larger than ever before and their increase is continuously accelerating. The goal of this research is to evaluate tools used for the transfer of large volumes of data. Bulk data transfer is a complex endeavour that requires not only sufficient network infrastructure, but also appropriate software, computing power and storage resources. We report on the series of storage benchmarks conducted using recently developed elbencho tool. The tests were conducted with an objective to understand and avoid I/O bottlenecks during data transfer operation. Subsequently Ethernet and InfiniBand networks performance was compared using Ohio State University bandwidth benchmark (OSU BW) and iperf3 tool. For comparison we also tested traditional (very inefficient) Linux scp and rsync commands as well as tools designed specifically to transfer large datasets more efficiently: bbcp and MDTMFTP. Additionally the impact of using simultaneous multi-threading and Ethernet jumbo frames on transfer rate was evaluated.	en
dc.identifier.citation	Kopeć, J. (2022). Evaluating Methods of Transferring Large Datasets. In: Panda, D.K., Sullivan, M. (eds) Supercomputing Frontiers. SCFA 2022. Lecture Notes in Computer Science, vol 13214. Springer, Cham. https://doi.org/10.1007/978-3-031-10419-0_7	en
dc.identifier.doi	10.1007/978-3-031-10419-0_7
dc.identifier.isbn	978-3-031-10418-3
dc.identifier.isbn	978-3-031-10419-0
dc.identifier.uri	https://open.icm.edu.pl/handle/123456789/21888
dc.language.iso	en
dc.publisher	Springer Nature	en
dc.rights	Uznanie autorstwa 4.0 Międzynarodowe	*
dc.rights.uri	http://creativecommons.org/licenses/by/4.0/	*
dc.subject	I/O	en
dc.subject	file systems	en
dc.subject	data management	en
dc.subject	data transfer	en
dc.subject	file transfer protocols	en
dc.subject	network evaluation	en
dc.subject	I/O benchmarking	en
dc.subject	Elbencho	en
dc.title	Evaluating Methods of Transferring Large Datasets	en
dc.type	bookPart	en