Overview

Understanding Your Storage Options with exceRpt

We have recently implemented some storage quotas for using exceRpt.
In particular, these quotas apply when you select the "Upload Full Results" option.
Please read the information below to better understand the storage quotas
and how you can clear up space for your Genboree Group.

Basics on Storage Quotas

Each Genboree Group has two different storage quotas associated with it.
These storage quotas include local space and FTP-backed space.
Note that the quotas are associated with a Group and not a user!
Please do not create multiple Groups as a single user to bypass the quota restriction.
You should only use a single Group for your exceRpt output (unless discussed with us previously).

By default, a Genboree Group is allotted 100 GB of local space and 100 GB of FTP-backed space.
Local space is defined as any space outside of the virtual FTP areas in your Group.
You can learn more about creating virtual FTP areas on the Workbench here.

As mentioned above, these space requirements only apply if you launch an exceRpt job with
the "Upload Full Results" option enabled. This option will upload a large results archive for each
sample containing alignment .bam files (and other similar files).
You can learn more about the specific files present in the full results archive on the exRNA Data Analysis page.

The default storage quotas should be sufficient for most users, especially if you normally run your samples
through exceRpt without the "Upload Full Results" option.

What Files Contribute to Your Quotas?

Any files in your Genboree Group will contribute to either the local or FTP-backed quota.
For example, if I uploaded a large number of FASTQ files that I wanted to process, those would contribute to one of the quotas.
This is one reason why it is very important to compress your files on the Workbench.
You can learn more about this idea below.

When you submit your files for processing through exceRpt (with "uploadFullResults" enabled),
the following factors will be considered:

1) How much space you're currently taking up in your Group (discussed above)
2) An estimation of how much space your current submission will take up when fully uploaded
3) An estimation of how much space your other, currently-running submissions will take up when fully uploaded

In other words, if I launch a huge job with 200 samples, I will likely get an error message when I try to launch another job,
even if my first job hasn't finished yet.

If you do receive an error message, it will clearly indicate how much space is taken up by each of the enumerated factors above.

Recommendations for Meeting the Quotas

Because we only recently instituted these quotas, many Genboree Groups have far exceeded the numbers given above
and cannot launch exceRpt jobs until space is freed up.
We have a number of suggestions for meeting the quotas:

Deleting Files

If you have any files that you no longer need on the Workbench, you can delete those files to free up space.
You should use the Remove File(s) tool (found under Data -> Files) to do so.
Simply drag your files into the Input Data panel and then run the tool to delete them.
The tool also accepts folders as inputs, so if you want to delete an entire folder at once, you can do that also.
Remember that exceRpt is constantly being worked on and improved, so if you have some old results, you could
potentially delete those and re-run your samples through the newest version of exceRpt.

Compressing Files

Another way to make space is to compress your files if they are currently uncompressed.
For example, many users have uncompressed FASTQ files stored on Genboree.
These FASTQ files can be very large, but they get much smaller when they're compressed.

Furthermore, we recently added a restriction to exceRpt so that it no longer accepts uncompressed FASTQ files as input.
This means that if you want to use FASTQ files in exceRpt, you will need to compress them first.
You can compress the files on the Genboree Workbench using the Prepare Archive tool (found under Data -> Files).
Simply drag your uncompressed files into the Input Data panel and the Database where you want your archive created
into the Output Targets panel.

IMPORTANT: After you run the tool and receive an email informing you that the job was successful,
you must manually delete your old uncompressed files. We will implement an option in the near future
that will do this deletion automatically, but for now, you'll need to do it using the Remove File(s) tool (discussed above).

Moving Files to Different Storage Type

As mentioned above, we have two different quotas: one for local storage and one for FTP-backed storage.
Thus, if one quota is completely full, it might make sense to move some files to a different storage type to free up space.
You can accomplish this by using the "Copy / Move File" tool (found under Data -> Files).
In most cases, users will want to move their files from local storage to FTP-backed storage, so we'll explain how to do that below.

  1. Create a virtual FTP area if you haven't already. You can follow the directions here.
  2. Drag the files you want to move into the Input Data panel.
  3. Drag the folder associated with the virtual FTP area into the Output Targets panel.
  4. Use the Copy/Move File tool and select the MOVE option (we do not want to copy the files).

If you do have files in your FTP-backed area that you want to move to local storage, the method is very similar.
Drag the files you want to move into the Input Data panel, and then drag the Database where you want your files to be stored
into the Output Targets panel.

Requesting More Space for Your Group

Before requesting more space, you should try all of the methods given above.
If you still need more space, email Emily and let him know your Genboree Group and why you need more space.
For example, if your Group stores files associated with a collaborative effort between several different labs,
then it might make sense to increase that Group's storage quota.

Also available in: HTML TXT