Skip to content

Storage and Data Transfer to/from ARCHIE-WeSt

ARCHIE-WeSt has a 230TB high-performance GPFS filesystem for all data storage, running across a dedicated 10Gb/s ethernet network. Benchmarks have demonstrated that the filesystem is capable of writing data at 12GB/s.

Although the ARCHIE-WeSt filesystem is backed up daily, this is for disaster recovery purposes only. Data is not archived and therefore the ARCHIE-WeSt should be not used for long-term storage. All essential data and computation results should be stored elsewhere, for example on the University's central storage facilities (e.g. H: & I: drives) .

Long term storage

Data should be transferred to your H: drive or I: drive for long term storage (Strathclyde), or the equivalent at your institution.

Quotas

Quotas are applied to the user's storage allocation in order to enable us to manage the overall storage effectively. By default users are allocated a soft quota of 350GB, which they can temporarily exceed for 21 days up to the hard quota limit of 1TB. In well-justified cases the user disk allocation maight be increased.

Exceeding storage quotas

The soft quota can be temporarily exceeded up to the hard quota limit for 21 days. The user will then have 21 days to return under the soft quota limit.

Exceeding hard quota or soft quota for > 21 days

If the hard disk quota is exceeded, or the soft quota is exceeded for longer than 21 days, the user will be unable to create any new files. This may cause problems for running jobs and connecting to ARCHIE-WeSt.

Transfer to/from a Windows Desktop

For data transfer between ARCHIE-WeSt and a Windows desktop MobaXterm can be used, or alternatives like WinSCP or FileZilla. Simply drag and drop the files between windows. Users should access ARCHIE using MobaXterm in the same way as when connecting for terminal access. Note that the user's folders on ARCHIE are visible on the LHS and files can be dragged and dropped from here to the local workstation (and vice-versa).

File Transfer with MobaXterm

Transfer from/to Linux(Mac) Desktop

From a Linux or Mac Terminal

Example

A typical command from local Linux/Mac terminal would be:

scp -pr DS_User_ID@login.hpc.strath.ac.uk:/work/DS_User_ID/MY_DATA .

Where DS_User_ID is an userneme issued by the University of Strathclyde
Options:
-p: preserves file attributes and timestamps
-r: recursive copy - transfers the directory and its contents.

Note the space between 'MY_DATA' and ' . '

' . ' represents the current working directory.

In this example, the folder MY_DATA and its contents is copied from the ARCHIE-WeSt filesystem to the user's workstation.

To the User's Desktop (Mac/Linux)

Example

A typical command from local Linux or Mac terminal would be:

scp -pr MY_DATA DS_User_ID@login.hpc.strath.ac.uk:/work/DS_User_ID

Where DS_User_ID is an userneme issued by the University of Strathclyde
Options:
-p: preserves file attributes and timestamps
-r: recursive copy - transfers the directory and its contents.

In this example, the folder MY_DATA and its contents is copied from the user's workstation to the folder /work/ on ARCHIE-WeSt.

Transfer to H: or I: Drives (Strathclyde Users only)

The H: and I: drives are located on the centrally managed storage facilities at the University of Strathclyde.

The H: and I: drives can only be accessed from the ARCHIE login nodes.

Make sure you transfer data directly between ARCHIE & the H: & I: drives

ARCHIE and the H: & I: drives are both connected to a fast 10Gb/s network.
It is not necessary (or advisable) to use your local workstation as an intermediate step in a two-stage copy process.

Using the mount_hdrive / mount_idrive command

This is the preferred method

To copy files to your I: drive space, type the following command in a terminal on ARCHIE:

mount_idrive

Note you will be prompted for your DS password. Then the I: drive will be mounted at ~username/i_drive

To copy files to your H: drive, type:

mount_hdrive

The H: drive will be mounted at ~username/h_drive

You can then copy files to ~username/i_drive or ~username/h_drive . The cp command will work as usual. For example, to copy the directory my_stuff and all its contents, type

cp -r my_stuff ~acs0315XX/i_drive

where you should use your DS UserID instead of acs0315XX. The -r flag means copy recursively i.e. the directory and its contents.

Tip

~acs0315XX refers to the home directory of user acs0315XX (that is DS_User_ID)

For more information on the cp command visit https://www.howtoforge.com/linux-cp-command.

Once finished, type (as appropriate):

umount_hdrive

umount_idrive

Using the file manager

Generally, we don't advise relying on the file manager available with the GUI (via ThinLinc) as it has caused problems in the past. However, it is useful as a backup method for transferring large files between the H: and I: drives.

Incorrect reporting of H: drive and I: drive quotas

The H: drive and I: drive quotas reported by the file manager will probably be incorrect. You can ignore this and can copy data up to the quota limit set by IT services.

Connecting to the H: drive

Open the file manager and select "Location" from the "Go" menu

Location->Go

In the "Go To:" location bar that appears, type the path to your hdrive, using the form

  smb://ds.strath.ac.uk/hdrive/XX/DS_User_ID

where XX are the last two digits from your DS_User_ID e.g.

Location Bar

Enter your DS username and password when prompted:

DS Credentials

You will now be able to drag-and-drop files between the H: drive (or I: drive) and archie using the GUI file manager.

Transfer Speeds

Transfer speed are typically 70-80 MB/s - much faster than transferring via your desktop.

Connecting to the I: drive

Repeat as for the H: drive but enter smb://ds.strath.ac.uk/idrive in the address bar.

I Drive

Disconnecting

To disconnect (unmount) the folder, click on the "eject" icon from within the file browser, or right click on the folder icon (on your desktop) and select "Unmount".

or