Data Submission

Overview

The data submission portal enables laboratories to submit data to the Brain Image Library. The key processes to submit data are:

1. Create an account.
2. Log in to the data submission portal.
3. Create a submission: A submission is a collection of image sets associated with the metadata.
4. Upload image data: Because this is the time-limiting step, we suggest this step be started before uploading metadata for the submission.
5. Upload metadata: Submission metadata can be uploaded through the portal in spreadsheet format.
6. Validate data: Users request to validate the data and correct any errors discovered.
7. Data is made public: Once data is validated and metadata is curated, the submission is made public.

How to submit data

Once you have created your account by creating an ACCESS portal account, requested access to the BIL submission portal, and set your initial password, you are ready to continue to the submission portal.

1. Enter the submission portal

Visit submit.brainimagelibrary.org and enter your PSC username and password.

2. Define your projects (PIs and Data Managers Only)

The PI dashboard allows PIs and Data Managers to define and manage projects. This dashboard will allow the PI to add authorized users to the project and see the status of in-progress, unsuccessful, and successful submissions, regardless of the users who uploaded and submitted the data.

You will need to create a project that all your BIL submissions will belong to. Your projects and sub-projects can represent different grants and different experimental projects and are a way for you to organize data being submitted at different times. You will only need to create a project/sub-project once, not every time you submit data to BIL.

Each project at BIL operates based on a “Parent” to “Child” relationship, similar to a family tree. Any project or sub-project will either be a parent or a child of a linked project.

Add/edit projects

To enter (or edit) project information, select the Manage Projects option. From this menu, you can create a new project and view both the personnel and submissions associated with each project.

To define a main project:

1. Select the Create a New Project button.
2. In the Project Name field, enter a name for your project For your main project, this name will be your grant.
   • Ex: R24 Ropelewski: A Confocal Fluorescence Microscopy Brain Data Archive
3. Enter the grant number associated with the project in the Funded By: field.
   • Ex: 5-R24-MH114793-07
4. Choose the Consortia Affiliation
5. For your main project, leave the Project Affiliation field empty.
6. Select Submit New Project

Your main project for your grant will be created. Now, you can create a sub-project for your experiments or other groupings of submissions.

To define a sub-project:

1. Select the Create a New Project button.
2. In Project Name enter a name for your sub-project.
   • For BICAN Only: This will become your Data Project name.
3. Enter the grant number associated with the sub-project in the Funded By: field. This should be the same as the grant for the main project
   • Ex: 5-R24-MH114793-07
4. Choose the appropriate Consortia Affiliation
   • Options include BICCN, BICAN, and SSPsyGene. Please contact bil-support@psc.edu if you are part of a different consortia that is not listed.
5. From Project Affiliation, choose the main project/grant that this sub-project is associated with from the drop-down menu.
6. Select Submit New Project

Add users to a project

To add a user to the project (such as a data submitter) select the Manage Projects link from the main dashboard then select View Personnel for the project you would like to add the user to. Select the user from the list and select Submit at the bottom of the page.

3. Create a submission

A submission contains one or more related datasets and associated metadata. Submissions will inherit project metadata (such as grant), thus all datasets within a submission must belong to the same project.

A submission can contain single or multiple datasets, but smaller submissions are recommended because all datasets within a submission must pass the validation process for the datasets within the submission to be published.

To create a submission, select the Submission sub-option of the New menu.

Next, enter the required metadata associated with the submission (see image below).

When you are done filling out the form, select the Save button. You will then be automatically taken to the metadata submission step.

When created, each BIL submission will have a unique 16-digit identifier associated with it (e.g. 6247417d691a4548) and a unique dropbox-like landing zone directory (e.g. /bil/lz/username/6247417d691a4548. This landing zone directory is where the datasets belonging to the submission must be transferred for validation and ingestion processing.

4. Upload metadata

After the submission is created, you will automatically be taken to the metadata submission page. Alternatively, you can also access the metadata submission page from the submission portal by selecting New and then New Metadata in the drop-down menu.

If you are NOT ready to upload metadata at this time, click Cancel to exit the upload metadata spreadsheet step. When you are ready to load your metadata, return to the portal and select New and then New Metadata in the drop-down menu.

First, you will choose the submission collection from the drop-down menu that you are ready to upload metadata for and Submit.

At this point, you can download the metadata sheet from the submit portal. Next, you will select the appropriate metadata schema for your data. Be sure to choose the appropriate option as this will affect the way your metadata is ingested.

Metadata upload options

1. Each dataset was generated from a single sample and donor (e.g. whole brain imaging) on the same instrument. Multiple datasets can be accommodated in a single submission. The lines of the specimen tab in the metadata spreadsheet must match exactly to the lines in the dataset tab (i.e. the specimen listed in specimen tab row 5 is the specimen for dataset row 5). Each dataset tab should have a corresponding image tab. There should be exactly one line in the instrument tab.
   

2. The dataset was generated from multiple samples, specimens, or donors on the same instrument. Only a single dataset can be accommodated per submission. Multiple specimens should be listed in the metadata spreadsheet, but only a single dataset. There should be one line in the image tab. There should be exactly one line in the instrument tab.
   

3. All datasets were generated from one specimen, but are from unique regions of interest on the specimen. Multiple datasets can be accommodated in a single submission. List the specimen multiple times in the specimen tab. Identify the specific region of interest in the locations column. Multiple donor lines must match exactly in the metadata spreadsheet (i.e. the specimen listed in specimen tab row 5 is the specimen for dataset row 5). Each dataset tab should have a corresponding image tab. There should be exactly one line in the instrument tab.
   

4. All datasets were generated from one specimen – but used different experimental (machinery) parameters. Multiple datasets can be accommodated as a single submission. There should be exactly one entry in the specimen tab. The lines listed in the Instrument tab in the metadata spreadsheet must match exactly to the lines in the dataset tab (i.e. the instrument listed in instrument tab row 5 is the instrument for dataset row 5). Each dataset tab should have a corresponding image tab.
   

5. Reconstructed neurons in standardized SWC format. All reconstructions were generated from one specimen. There should be one line listed in the specimen tab. There should be one line listed in the dataset tab. No information should be listed in the image tab. File format specification: http://www.neuronland.org/NLMorphologyConverter/MorphologyFormats/SWC/Spec.html
   

After a metadata spreadsheet is prepared, select the submission it will be associated with. Then select the Upload Metadata button to select your metadata spreadsheet and upload it. This button will not be available until a metadata upload option is selected.

Note that while preparing the metadata spreadsheet, separate data directories need to be listed for each dataset in the submission.

5. Upload image data

The submission portal creates an upload landing zone space for you to transfer your image data to. To find this landing zone, select the Submissions sub-option of the View menu. The field "Data Path" shows the landing zone space for the submission.

Note that separate data subdirectories are required for each dataset in the submission. For example, if you had an experiment containing 5 mouse datasets that you wanted to to include as a single submission, you would create 5 subdirectories in the landing zone for the submission, one for each mouse dataset. e.g:

/bil/lz/testuser/abcdef0123456789/mouse1
/bil/lz/testuser/abcdef0123456789/mouse2
/bil/lz/testuser/abcdef0123456789/mouse3
/bil/lz/testuser/abcdef0123456789/mouse4
/bil/lz/testuser/abcdef0123456789/mouse5

To tie metadata to an image dataset, each image dataset must be uploaded in a separate subdirectory.

Due to size, image data can not be uploaded through the submission portal (submit.brainimagelibrary.org). It must be uploaded separately through the BIL data transfer nodes, which are available at the virtual host: upload.brainimagelibrary.org. All users authorized to use the data submission portal (submit.brainimagelibrary.org) are also authorized to use the data transfer nodes (upload.brainimagelibrary.org). The username and passwords are the same on both systems.

Data Upload Methods

There are many supported ways to upload files into the landing zone directory through the BIL data transfer nodes including rsync, Globus, sftp, and scp.

$ rsync -lrtpDvP mouse1 testuser@upload.brainimagelibrary.org:/bil/lz/testuser/abcdef0123456789
sending incremental file list
mouse1/
mouse1/data1.tiff
     1356122 100%  126.20MB/s    0:00:00 (xfer#1, to-check=0/2)

sent 1356392 bytes  received 35 bytes  2712854.00 bytes/sec
total size is 1356122  speedup is 1.00

$ sftp testuser@upload.brainimagelibrary.org
The authenticity of host 'upload.brainimagelibrary.org (128.182.108.164)' can't be established.
ECDSA key fingerprint is 32:cf:46:44:3d:9c:8e:b2:1d:14:03:66:45:0b:11:29.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'upload.brainimagelibrary.org,128.182.108.164' (ECDSA) to the list of known hosts.
testuser@upload.brainimagelibrary.org's password:
Connected to upload.brainimagelibrary.org.
sftp> cd /bil/lz/testuser/abcdef0123456789
sftp> mkdir /bil/lz/testuser/abcdef0123456789/mouse1
sftp> cd /bil/lz/testuser/abcdef0123456789/mouse1

sftp> put data1.tiff
Uploading data1.tiff to /bil/lz/testuser/abcdef0123456789/mouse1/data1.tiff
data1.tiff                                      0%    0     0.0KB/s   --:-- ETA
data1.tiff                                    100% 1324KB   1.3MB/s   00:00

sftp> exit

$ scp data1.tiff testuser@upload.brainimagelibrary.org:/bil/lz/abcdef0123456789/mouse1/data1.tiff
testuser@upload.brainimagelibrary.org's password:
data1.tiff                                                                      0%    0     0.0KB/s   --:-- ETA
data1.tiff                                                                    100% 1324KB   1.3MB/s   00:00

rclone copy your_bucket:data_directory/file_name.zarr brainimagelibrary:/bil/lz/testuser/abcdef0123456789/ffile_name.zarr -P

5. Validate and submit publish request

Once all data has been uploaded to the landing zone and the metadata has been submitted, request that the data be validated and made publicly available. This can be done through the submission portal by selecting Submit Publish Request from the main menu of the submission portal. If an embargo period is being requested, please send an email to bil-support@psc.edu along with the submission id and brief note.

To submit your datasets for publication:

1. Log in to the submission portal https://submit.brainimagelibrary.org/
2. On the top navigation bar, select "Submit Publish Request"
3. Select the submission you would like to publish from the list by selecting the check box on the left
4. Select "Submit Validation Request"

If data validation fails, the data submitter will be notified by email. The submitter should address the validation issue(s) and re-submit the validation request. Both data and metadata need to pass the validation checks – datasets that fail the validation process are considered incomplete and will not be made publicly available.

Glossary

The following terms in this document have specific context, as defined below:

Dataset
At BIL a dataset is a stand-alone entry of an image-volume or image-set associated with a single subject or experimental unit with unique metadata. A single dataset is usually associated with a single donor or subject when submitting multiple subjects in a submission, or a single part of the brain when imaging many parts of the brain in a submission. Many datasets make up a submission. A dataset typically contains many 2d image files that are assembled to form a more complex two or three-dimensional volume.

Submission
A submission contains one or more related datasets and the associated metadata. Submissions will inherit project metadata (such as the NIH project, grant number, laboratory name, etc.), thus all datasets within a submission must belong to the same project. A submission can contain a single or multiple datasets. In general, smaller submissions are recommended because all datasets within a submission must pass the validation process for the datasets within the submission to be published. Each "level" of data should be uploaded in separate submissions (e.g. The set of raw data and the same data aligned to a reference are considered two separate submissions).

Project
A way of grouping submissions to ensure that (1) People working with the data gain appropriate access and credit for their contributions and (2) Data is linked to the proper funding sources.

Sub-Project
A way of grouping data within a BIL Project to link data of the same funding source, and to group data from the same emperiment or scientific project.

Publish
The act of making a submission that has passed all validation checks publicly available.