PCP (Parallel Copy)
A parallel copy program for Lustre.
pcp is similar to cp -r ; simply give it a source directory and destination and pcp will recursively copy the source directory to the destination in parallel. It has several features that make it useful for copying data in a cluster computing environment with Lustre distributed filesystems:
- files are chunked and each chunk is copied in parallel
- Lustre striping information can be copied (or optionally set based on file size)
- built-in checkpointing to safely resume transfers that are interrupted by nodes failing, jobs being killed for running over queue limits, etc.
- checksumming of files after transfer to verify integrity of the copied data
pcp works best in a cluster environment with MPI support using the Lustre filesystem, but it can also copy files to/from other filesystems (without Lustre-specific functionality like stripe-awareness).
Downloads
Development and release versions of the source code for pcp are available in the GitHub Repository
Further information
pcp is available under the GNU General Public License, version 2 (GPLv2).
External Contributors
