If you have already reached out to us to prioritize transfer of a specific subset of your folders or files, no further action is required. These files will be copied to same-named folder in your active /project share on the new Project storage system.

If you have not yet contacted us with a list of priority folders or files to transfer, or if there are additional folders and files that you urgently need for your active work, please reach out to RC with a specific list of those folders/files and we will add them to the file transfer queue. See “How can I get help with migration of my data?" for details.

Questions about the data migration process should be directed to our user services team.

Until February 26, all files are shown in the new Project storage system, including those that have already been transferred and those that are still on the old Project storage system. Files that are still on the old system present as empty stub files in the new system. When accessed for the first time, the empty stub file is replaced by the original file as it is transferred on-demand from the old system to the new system.

On February 26, the empty stub files will be deleted from the new system as they are not needed for the new migration process. This is a gradual process that may take a few weeks to complete, so you may see different files, depending on when you access the new system. However, the original files behind the stub files still exist and are secure on the Old Project system.

See “How do I find out what files are on the old Project storage system?"

Until February 26, files that you have already requested be transferred will be copied to the same-named directories in your active /project share on the new Project storage system.

Beginning February 26, all your files, including files that are still on the old system and files that have already been transferred to the new system, will start being copied to the same-named directories in a new /stagedproject share. The /stagedproject share was created for your to-be-migrated files to prevent potential interference with your active work in /project. Note: Your folder in /stagedproject will be empty on February 26, but will gradually fill with your files as they are copied over.

On February 26, a new /stagedproject folder will become available in read-only mode on the Rivanna login nodes and the UVA Standard Security Storage data transfer node (DTN). It will not be available on compute nodes. This folder will be used as destination to stage data transferred from your old Project storage to the new storage system.

On Feb 26, your folder in /stagedproject is set up as read-only on the Rivanna login nodes, compute nodes, and the UVA Standard Security Storage data transfer node (DTN).

Option 1 (preferred):

For compute jobs we recommend you first copy files from /stagedproject into your /project or /scratch folder. For transfer of large folders see “How can I consolidate my files in /stagedproject and /project?”.

If you do not need any of the files affected by any failed copy operation immediately, you may continue to work out of /project and /scratch folders as usual. We will inform you when all stub files have been deleted and you may consolidate the remaining files from /stagedproject to /project then, following the copy instructions one more time. See “How can I consolidate my files in /stagedproject and /project?”

Option 2:

If the copy of any needed files to /project fails, you can update your job scripts to read the necessary input files from your /stagedproject folder and write the output to a new folder in your existing /project share. We will inform you when all stub files have been deleted and you may consolidate the remaining files from /stagedproject to /project then by following the copy instructions one more time, See “How to consolidate files from /stagedproject to /project?”.

Performance of the old Project storage system is severely degraded. Any exploratory search for folders or file listings by users would create additional strain on the system, which would further reduce the already limited data transfer rates from the old to new Project storage system.

Because of these performance issues, RC set up a managed process that transfers all files from the old Project storage system to a new /stagedproject folder for you in the new system. See “How do I access the new /stagedproject folder?”.

You can reach out to RC to request a list of your files on the old Project storage system. See “How do I find out what files are on the old Project storage system?"

RC will work with you to prioritize the list of your files so that those files most urgently needed for your active work can be transferred first. See “How can I get help with the migration process?"

After February 26, you will not be able to connect to the old Project storage system. See “Why can’t I access the old Project storage system directly to copy my own files?"

However, we have placed a list of your old Project storage files in the top-level folder of your new share on /stagedproject (i.e. /stagedproject/my_share/old-project-file-list.txt). You may use this list to prioritize folders and files for your data migration (see “Can I pick which of my files are transferred first?").

Please keep in mind that the list in old-project-file-list.txt represents all your files stored on the old Project storage system, some of which have already been transferred to the new system. Eventually all files will be transferred from the old to the new Project storage system. If you already have all the data needed for your active work on the new Project storage system, no action is required.

No. We are prioritizing transfer of files that you actively need for your research. You may reach out to us to provide a specific list of your high priority, essential folders and files.

The severe performance degradation of the old storage system will remain a bottleneck regardless of the change in data transfer method. However, the more selective this list, the better we can help you with this transition. See “Can I pick which of my files are transferred first?" for details.

Once the transfer process has been stabilized, engineers will begin transferring any remaining files that users did not explicitly request to be moved.

Yes. Please complete this web form to provide us with a list of specific, high priority folders and files for migration. Please indicate as precisely as possible which folders or files should be transferred first so our storage engineers can prioritize these items. The more selective this list, the better we can help you with the transition.

If you need help with your file prioritization, you may reach out to RC to request a list of files that you still have on the old Project storage system.

Though the new Project storage system provides vastly improved performance, the overall transfer of files is limited by the degraded performance of the old system. This issue with the old storage system will remain a bottleneck regardless of the change in data transfer method.

However, you can facilitate the data transfer process by providing us with a narrowed down list of files that you need for your research over the next few months. This will allow us to deprioritize less urgently needed files.

See “Can I pick which of my files are transferred first?" for details.

All data will be migrated eventually, but this process is expected to take several months to complete. We will post weekly progress of data migration on this page.

We will notify the PI of the storage allocation when all their folders have been copied over. We will not purge any files on the old Project storage system until the PI has had an opportunity to verify that their files have been migrated to the new Project storage system.

The old Project storage hardware (GPFS) is in a degraded state. Due to this state, transferring data from the system is unusually slow even in otherwise ideal conditions. With a wide range of researchers and research workflows attempting to simultaneously access the system, the system becomes overloaded resulting in the “IO Error” and similar errors that researchers have been experiencing. As the system gets overloaded the transfer rates drop even further.

By having RC manage the data transfer from the old to the new system we will be able to limit the load on the system and keep it closer to ideal conditions in order to maximize the transfer rate going forward.

The old GPFS hardware was beyond its expected end-of-life and due to historical sporadic financial investment in RC the hardware was not replaced. That changed recently with substantial University investments in RC and replacement hardware was immediately purchased. Unfortunately, we were not able to get the new GPFS hardware up and running before we started to experience failures on the old GPFS hardware. Given that the hardware was already end-of-life, we prioritized completing installation of the new hardware quickly and transferring from the old hardware to the new. Because the old GPFS hardware is still experiencing failure, these transfers have been slow with ~35% of the data transferred to-date.

With recent University investments in Research Computing we have planned annual purchases of new hardware with a model that allows for the seamless addition of new hardware and decommissioning of old hardware as it reaches its end-of-life. This will reduce the need for downtimes and manual data migrations as we ensure the integrity of our infrastructure.

Research Computing is also working to improve its approach to user communications. If a major disruption occurs to an RC system, we will be using pages like this one to make sure there is a single location researchers can go to see both the current status and the history of the incident.

Looking forward, Research Computing’s service roadmap includes filling in some services currently missing from our portfolio. Critically, there should be options available for researchers to elect for redundant access to their key datasets. This means not only additional levels of protection in case of a disaster but also an alternate location where data can be retrieved.

At this time, about 35% of files have been successfully migrated. These files were moved as part of prioritized research datasets or were transferred over by AFM when accessed by a researcher. 

The prioritized datasets were known to be actively used and were prime candidates to transfer when extra transfer capacity was available.

Stub files are “placeholders” for files that exist on the old project storage system but had not been copied over to the new project storage system. They are not needed for the new data migration process. We began with deletion of these empty placeholder stub files on February 26. This process is still ongoing. The original files are still intact and secure on the old system.

You may create a list of stub files currently present in your /project folder by running this command:

find /project/MY_SHARE -ls > regular-files.log 2> stubfiles.log

This produces two files, regular-files.log and stubfiles.log. The stubfiles.log contains all files that the system cannot list which is indicative of being stub files.

Stub files may be present which are placeholders that linked the new storage system to the legacy storage system. As a part of the data migration process, stub files linking to the legacy system were also attempted to be deleted. A subset of these stub files remains visible on the new system, but attempting to access them will result in File Not Found Errors, as they are no longer coupled with the old system. These files are scheduled for deletion through an automated process eventually.

To verify the list of files in /stagedproject you can run the following on the command line:

check_stagedproject MYSHARE

Replace MYSHARE with the name of your project’s share. This will create ~/stagedproject-file-list.txt which contains a list of files that have been copied from /oldproject to /stagedproject. This list is compared with /stagedproject/MYSHARE/old-project-file-list.txt to ensure that all files have been transferred from /oldproject.

~/stagedproject-file-list.txt will change if your data transfer is still in progress. The share’s owner will be notified once all the data is transferred.

To organize your files efficiently:

1. If you relocated files to new folders to avoid issues with file access performance issues, or if you have duplicates in project storage, please put them back in their original folders using the “mv” command. This keeps things neat and prevents duplicates. Be cautious not to overwrite newer files with older ones while moving them.

2. Use the “rsync” command to copy files from the /stagedproject folder to the main project folder (/project). This ensures that all essential files are consolidated in one location (/project).

If you require assistance with these steps, please contact us via our support webform or during office hours for help.

Submit the following script to copy large directories in bulk:

#!/bin/bash
#SBATCH -A your_allocation  # to find your allocation, type "allocations"
#SBATCH -t 12:00:00   # up to 7-00:00:00 (7 days)
#SBATCH -p standard
STAGEDPROJECTFOLDER=/stagedproject/MYSHARE/      #replace MYSHARE  with your share name

PROJECTFOLDER=/project/MYSHARE/                  #replace MYSHARE with your share name
rsync -uav ${STAGEDPROJECTFOLDER} ${PROJECTFOLDER} 1> ~/rsync.log 2> ~/rsync-error.log

The script will also be available through the Open OnDemand Job Composer:

1. Go to Open OnDemand Job Composer
2. Click: New Job -> From Template
3. Select demo-copy-stagedproject
4. In the right panel, click “Create New Job”
5. This will take you to the “Jobs” page. In the “Submit Script” panel at the bottom right, click “Open Editor”
6. Enter your own allocation. Edit the MY_SHARE placeholder in the script as needed. Click “Save” when done.
7. Going back to the “Jobs” page, select demo-copy-stagedproject and click the green “Submit” button.

As we expect a high volume of data migration, please refrain from doing so directly on the login nodes but instead submit it as a job via the provided Slurm script as described above.

rsync -uav  --no-owner --no-group --no-perms --no-times ${STAGEDPROJECTFOLDER} ${PROJECTFOLDER} 1> ~/rsync.log 2> ~/rsync-error.log

We have placed a list of your old Project storage files in the top-level folder of your new share on /stagedproject (i.e. /stagedproject/my_share/old-project-file-list.txt). You may use this list to prioritize folders and files for your data migration (see “Can I pick which of my files are transferred first?”).

Researchers with an urgent need to access files that have not been migrated to the new Project storage system yet may submit a support request through our webform. Please indicate as precisely as possible which folders or files should be transferred first so we can prioritize these items.

All files will be transferred eventually.