Storage and Data Transfer to/from ARCHIE-WeSt
ARCHIE-WeSt has a 825TB high-performance GPFS filesystem for all data storage, running across a dedicated 10Gb/s ethernet network. Benchmarks have demonstrated that the filesystem is capable of writing data at 12GB/s.
Although the ARCHIE-WeSt filesystem is backed up daily, this is for disaster recovery purposes only. Data is not archived and therefore the ARCHIE-WeSt should be not used for long-term storage. All essential data and computation results should be stored elsewhere, for example on the University's central storage facilities (e.g. H: & I: drives) .
Long term storage
Data should be transferred to your H: drive or I: drive for long term storage (Strathclyde), or the equivalent at your institution.
Quotas
Quotas are applied to the user's storage allocation in order to enable us to manage the overall storage effectively. By default users are allocated a soft quota of 350GB, which they can temporarily exceed for 30 days up to the hard quota limit of 1TB. In well-justified cases the user disk allocation may be increased.
Exceeding storage quotas
The soft quota can be temporarily exceeded up to the hard quota limit for 30 days. The user will then have 30 days to return under the soft quota limit.
Exceeding hard quota or soft quota for > 30 days
If the hard disk quota is exceeded, or the soft quota is exceeded for longer than 30 days, the user will be unable to create any new files. This may cause problems for running jobs and connecting to ARCHIE-WeSt.
Transfer to/from a Windows Desktop
For data transfer between ARCHIE-WeSt and a Windows desktop MobaXterm can be used, or alternatives like WinSCP or FileZilla. Simply drag and drop the files between windows. Users should access ARCHIE using MobaXterm in the same way as when connecting for terminal access. Note that the user's folders on ARCHIE are visible on the LHS and files can be dragged and dropped from here to the local workstation (and vice-versa).
Transfer from/to Linux(Mac) Desktop
From a Linux or Mac Terminal
Example
A typical command from local Linux/Mac terminal would be:
scp -pr DS_User_ID@login.hpc.strath.ac.uk:/work/DS_User_ID/MY_DATA .
Where DS_User_ID is an userneme issued by the University of Strathclyde Options: -p: preserves file attributes and timestamps -r: recursive copy - transfers the directory and its contents.
Note the space between 'MY_DATA' and ' . '
' . ' represents the current working directory.
In this example, the folder MY_DATA and its contents is copied from the ARCHIE-WeSt filesystem to the user's workstation.
To the User's Desktop (Mac/Linux)
Example
A typical command from local Linux or Mac terminal would be:
scp -pr MY_DATA DS_User_ID@login.hpc.strath.ac.uk:/work/DS_User_ID
Where DS_User_ID is an userneme issued by the University of Strathclyde Options: -p: preserves file attributes and timestamps -r: recursive copy - transfers the directory and its contents.
In this example, the folder MY_DATA and its contents is copied from the user's workstation to the folder /work/
Transfer to H: or I: Drives (Strathclyde Users only)
The H: and I: drives are located on the centrally managed storage facilities at the University of Strathclyde.
The H: drive can be accessed from the ARCHIE login nodes.
The I: drive is currently only accessible from our dedicated datamover node.
Make sure you transfer data directly between ARCHIE & the H: & I: drives
ARCHIE and the H: & I: drives are both connected to a fast 10Gb/s network.
It is not necessary (or advisable) to use your local workstation as an intermediate step in a two-stage copy process.
To be able to access the H: or I: drives from ARCHIE, they must first be "mounted" (this is analagous to the "Map Network Drive" procedure on Windows").
Using datamover
We currently have a dedicated server for transferring data to and from the I: drive.
This is accessible by ssh only from the login nodes or from any machine on campus. To access it, use:
ssh datamover.hpc.strath.ac.uk
Files can be copied to and from the I: drive via the command line only, using the commands below.
Using the mount_hdrive / mount_idrive command
This is the preferred method
To mount your I: drive space on datamover, type the following command after ssh'ing to datamover:
mount_idrive
Note you will be prompted for your DS password. Then the I: drive will be mounted at ~/i_drive
To mount your H: drive on the login nodes, type:
mount_hdrive
The H: drive will be mounted at ~/h_drive
Tip
~ refers to your home directory i.e. so ~/i_drive refers to /users/DS_User_ID/i_drive
Copying files to the H: or I: drive
You can then copy files to ~/i_drive or ~/h_drive . The cp command will work as usual. For example, to copy the directory my_stuff and all its contents, type
cp -r my_stuff ~/i_drive
where the -r flag means copy recursively i.e. the directory and its contents. Using -rp will also preserve timestamps.
Copying files from the H: or I: drive
To copy the directory i_drive/my_stuff and all its contents, to your current working directory, use:
cp -r ~/i_drive/my_stuff .
Note the " . " which refers to your current directory on ARCHIE.
For more information on the cp command visit https://www.howtoforge.com/linux-cp-command.
Unmounting the H: and I: drives
Once finished, you should unmount the drive(s). Type (as appropriate):
umount_hdrive
umount_idrive
Using the file manager
Generally, we don't advise relying on the file manager available with the GUI (via ThinLinc) as it has caused problems in the past. However, it is useful as a backup method for transferring large files between the H: and I: drives.
Incorrect reporting of H: drive and I: drive quotas
The H: drive and I: drive quotas reported by the file manager will probably be incorrect. You can ignore this and can copy data up to the quota limit set by IT services.
Connecting to the H: drive
Open the file manager and select "Location" from the "Go" menu
In the "Go To:" location bar that appears, type the path to your hdrive, using the form
smb://ds.strath.ac.uk/hdrive/XX/DS_User_ID
where XX are the last two digits from your DS_User_ID e.g.
Enter your DS username and password when prompted:
You will now be able to drag-and-drop files between the H: drive (or I: drive) and archie using the GUI file manager.
Transfer Speeds
Transfer speed are typically 70-80 MB/s - much faster than transferring via your desktop.
Connecting to the I: drive
Repeat as for the H: drive but enter smb://ds.strath.ac.uk/idrive in the address bar.
Disconnecting
To disconnect (unmount) the folder, click on the "eject" icon from within the file browser, or right click on the folder icon (on your desktop) and select "Unmount".
or 