Papers for

large-scale scientific data centers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

IceCube switches data system to Pelican platform improving performance

IceCube Takes Flight with Pelican - A First Experience

Abstract: The IceCube Neutrino Observatory has removed GridFTP and x.509 certificate authentication for data transfers, migrating to the Pelican Platform, the Open Science Data Federation, and WLGC tokens. While this is a common solution on the computing infrastructure we use, we required several customizations to work with our existing data storage structure and make it easier for scientists to use. We wrote a custom WLCG token issuer to support our POSIX filesystem with custom user and group permissions across the entire filesystem. We also made several modifications to HTCondor to ease job submission using tokens, including a custom credmon. After an initially bumpy transition due to several now-resolved Pelican issues, the Pelican-based system has already proven superior to GridFTP in several ways.

Fri 25 SeptDistributed, Parallel, and Cluster Computing

The gist

The IceCube Neutrino Observatory used to rely on an older method called GridFTP with complicated certificates to move data. The authors switched to a newer system called Pelican that uses tokens, making it easier and safer to transfer data. They had to create custom tools to fit their unique file setup and help scientists use it more easily. After some problems at the start, the new system now works better than the old one.

Open → 2609.31851v1