Data Transfer Tool Command Line Documentation

Downloads

Downloading Data Using a Manifest File

A convenient way to download multiple files from the GDC is to use a manifest file generated by the GDC Data Portal. After generating a manifest file (see Preparing for Data Download and Upload for instructions), initiate the download using the GDC Data Transfer Tool by supplying the -m or --manifest option, followed by the location and name of the manifest file. OS X users can drag and drop the manifest file into Terminal to provide its location.

The following is an example of a command for downloading files from GDC using a manifest file:

gdc-client download -m  /Users/JohnDoe/Downloads/gdc_manifest_6746fe840d924cf623b4634b5ec6c630bd4c06b5.txt

Downloading Data Using GDC File UUIDs

The GDC Data Transfer Tool also supports downloading of one or more individual files using UUID(s) instead of a manifest file. To do this, enter the UUID(s) after the download command:

gdc-client download 22a29915-6712-4f7a-8dba-985ae9a1f005

Multiple UUIDs can be specified, separated by a space:

gdc-client download e5976406-473a-4fbd-8c97-e95187cdc1bd fb3e261b-92ac-4027-b4d9-eb971a92a4c3

Resuming a Failed Download

The GDC Data Transfer Tool supports resumption of interrupted downloads. To resume an incomplete download, repeat the download of the manifest or UUID(s) in the same folder as the initial download. Failed downloads will appear in the destination folder with a .partial extension. This feature allows users the ability to identify quickly where the download stopped. For large downloads this feature can let the user identify where the download was interrupted and edit the manifest accordingly.

gdc-client download f80ec672-d00f-42d5-b5ae-c7e06bc39da1

Download Latest Version of a File

The GDC Data Transfer Tool supports file versioning. Our backend data storage supports multiple file versions so older and current versions can be accessible to our users. For information about accessing file versioning information with our API and finding older UUID information from current UUIDs please check out the the API User Guide section in our API documentation. When working with older manifests or older lists of UUIDs the latest version of a file can always be download with the --latest flag.

gdc-client download 426de656-7e34-4a49-b87e-6e2563fa3cdd --latest -t gdc-user-token.2018.txt
Downloading LATEST versions of files
Latest version for 426de656-7e34-4a49-b87e-6e2563fa3cdd ==> 6633bfbd-87f1-4d3a-a475-7ad1e8c2017a
100% [#############################################################################################################################] Time: 0:01:16  14.10 MB/s
Successfully downloaded: 1

Downloading Controlled-Access Data

A user authentication token is required for downloading Controlled-Access Data from GDC. Tokens can be obtained from the GDC Data Portal (see instructions in Obtaining an Authentication Token). Once downloaded, the token file can be passed to the GDC Data Transfer Tool using the -t or --token-file option:

gdc-client download -m gdc_manifest_e24fac38d3b19f67facb74d3efa746e08b0c82c2.txt -t gdc-user-token.2015-06-17T09-10-02-04-00.txt

Directory structure of downloaded files

The directory in which the files are downloaded will include folders named by the file UUID. Inside these folders, along with the the data and zipped metadata or index files, will exist a logs folder. The logs folder contains state files that insure that downloads are accurate and allow for resumption of failed or prematurely stopped downloads. While a download is in progress a file will have a .partial extension. This will also remain if a download failed. Once a file is finished downloading the extension will be removed. If an identical manifest is retried another attempt will be made to download files containing a .partial extension.

C501.TCGA-BI-A0VR-10A-01D-A10S-08.5_gdc_realn.bam.partial  logs

Uploads

Uploading Data Using a Manifest File

GDC Data Transfer Tool supports uploading molecular data using a manifest file to the Data Submission Portal. The manifest file for submittable data files can be retrieved from the GDC Data Submission Portal, or directly from the GDC Submission API given a submittable data file UUID. The user authentication token file needs to be specified using the -t or --token-file option.

First, generate an upload manifest, either using the GDC Data Submission Portal, or using a call to the GDC Submission API manifest endpoint (as in the following example):

export token=ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTOKEN-01234567890+AlPhAnUmErIcToKeN=0123456789-ALPHANUMERICTO

curl --header "X-Auth-Token: $token" 'https://api.gdc.cancer.gov/submission/CGCI/BLGSP/manifest?ids=460ad2fe-5a7f-4797-9e18-336d33e21444' >manifest.yml
gdc-client upload --manifest manifest.yml --token-file token.txt

Uploading Data Using a GDC File UUID

The GDC Data Transfer Tool also supports uploading molecular data using a file UUID. The tool will first make a request to get the filename and project id from GDC API, and then upload the corresponding file from the current directory.

gdc-client upload cd939bdd-b607-4dd4-87a6-fad12893932d -t token.txt

Resuming a Failed Upload

By default, GDC Data Transfer Tool uses multipart transfer to upload files. If an upload failed but some parts were transmitted successfully, a resume file will be saved with the filename resume_[manifest_filename]. Running the upload command again will resume the transfer of only those parts of the file that failed to upload in the previous attempt.

gdc-client upload -m manifest.yml -t token

Deleting Previously Uploaded Data

Previously uploaded data can be replaced with new data by deleting it first using the --delete switch:

gdc-client upload -m manifest.yml -t token --delete

Troubleshooting

Invalid Token

An error message about an 'invalid token' means that a new authentication token needs to be obtained from the GDC Data Portal or the GDC Data Submission Portal as described in Preparing for Data Download and Upload.

 403 Client Error: FORBIDDEN: {
      "message": "Your token is invalid or expired, please get a new token from GDC Data Portal"
      }

dbGaP Permissions Error

Users may see the following error message when attempting to download a file from GDC:

 403 Client Error: FORBIDDEN: {
      "message": "You don't have access to the data: Please specify a X-Auth-Token"
      }

This error message indicates that the user does not have dbGaP access to the project to which the file belongs. Instructions for requesting access from dbGaP can be found here.

File Availability Error

Users may also see the following error message when attempting to download a file from GDC:

 403 Client Error: FORBIDDEN: {
    "message": "You don't have access to the data: Requested file abd28349-92cd-48a3-863a-007a218de80f does not allow read access"
    }

This error message means that the file is not available for download. This may be because the file has not been uploaded or released yet or that it is not a file entity.

GDC Upload Privileges Error

Users may see the following error message when attempting to upload a file:

 Can't upload: {
     "message": "You don't have access to the data: You don't have create role to do 'upload'"
     }

This means that the user has dbGaP read access to the data, but does not have GDC upload privileges. Users can contact The database of Genotypes and Phenotypes (dbGaP) to request upload privileges.

File in Uploaded State Error

Re-uploading a file may return the following error:

 Can't upload: {
      "message": "File in uploaded state, upload not allowed"
      }

To resolve this issue, delete the file using the --delete switch before re-uploading.

Microsoft Windows Executable Error

Attempting to run gdc-client.exe by double-clicking it in the Windows Explorer will produce a window that blinks once and disappears.

This is normal, the executable must be run using the command prompt. Click 'Start', followed by 'Run' and type 'cmd' into the text bar. Then navigate to the path containing the executable using the 'cd' command.

Help Menus

The GDC Data Transfer Tool comes with built-in help menus. These menus are displayed when the GDC Data Transfer Tool is run with flags -h or --help for any of the main arguments to the tool. Running the GDC Data Transfer Tool without argument or flag will present a list of available command options.

gdc-client --help
usage: gdc-client [-h] [--version] {download,upload,settings} ...

The Genomic Data Commons Command Line Client

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

commands:
  {download,upload,settings}
                        for more information, specify -h after a command
    download            download data from the GDC
    upload              upload data to the GDC
    settings            display default settings
The available menus are provided below.

Root menu

The GDC Data Transfer Tool displays the following output when executed without any arguments.

gdc-client
usage: gdc-client [-h] [--version] {download,upload,settings} ...
gdc-client: error: too few arguments

Download help menu

The GDC Data Transfer Tool displays the following help menu for its download functionality.

gdc-client download --help
usage: gdc-client download [-h] [--debug]
                           [--log-file LOG_FILE]
                           [--color_off] [-t TOKEN_FILE]
                           [-d DIR] [-s server]
                           [--no-segment-md5sums]
                           [--no-file-md5sum]
                           [-n N_PROCESSES]
                           [--http-chunk-size HTTP_CHUNK_SIZE]
                           [--save-interval SAVE_INTERVAL]
                           [--no-verify]
                           [--no-related-files]
                           [--no-annotations]
                           [--no-auto-retry]
                           [--retry-amount RETRY_AMOUNT]
                           [--wait-time WAIT_TIME]
                           [--latest] [--config FILE] [-u]
                           [-m MANIFEST]
                           [file_id [file_id ...]]

positional arguments:
file_id               The GDC UUID of the file(s) to download

optional arguments:
-h, --help            show this help message and exit
--debug               Enable debug logging. If a failure occurs, the program
                      will stop.
--log-file LOG_FILE   Save logs to file. Amount logged affected by --debug
--color_off           Disable colored output
-t TOKEN_FILE, --token-file TOKEN_FILE
                      GDC API auth token file
-d DIR, --dir DIR     Directory to download files to. Defaults to current
                      dir
-s server, --server server
                      The TCP server address server[:port]
--no-segment-md5sums  Do not calculate inbound segment md5sums and/or do not
                      verify md5sums on restart
--no-file-md5sum      Do not verify file md5sum after download
-n N_PROCESSES, --n-processes N_PROCESSES
                      Number of client connections.
--http-chunk-size HTTP_CHUNK_SIZE, -c HTTP_CHUNK_SIZE
                      Size in bytes of standard HTTP block size.
--save-interval SAVE_INTERVAL
                      The number of chunks after which to flush state file.
                      A lower save interval will result in more frequent
                      printout but lower performance.
--no-verify           Perform insecure SSL connection and transfer
--no-related-files    Do not download related files.
--no-annotations      Do not download annotations.
--no-auto-retry       Ask before retrying to download a file
--retry-amount RETRY_AMOUNT
                      Number of times to retry a download
--wait-time WAIT_TIME
                      Amount of seconds to wait before retrying
--latest              Download latest version of a file if it exists
--config FILE         Path to INI-type config file
-u, --udt             Use the UDT protocol.
-m MANIFEST, --manifest MANIFEST
                      GDC download manifest file

Upload help menu

The GDC Data Transfer Tool displays the following help menu for its upload functionality.

gdc-client upload --help
usage: gdc-client upload [-h] [--debug]
                                            [--log-file LOG_FILE]
                                            [--color_off] [-t TOKEN_FILE]
                                            [--project-id PROJECT_ID]
                                            [--path path]
                                            [--upload-id UPLOAD_ID]
                                            [--insecure] [--server SERVER]
                                            [--part-size PART_SIZE]
                                            [--upload-part-size UPLOAD_PART_SIZE]
                                            [-n N_PROCESSES]
                                            [--disable-multipart] [--abort]
                                            [--resume] [--delete]
                                            [--manifest MANIFEST]
                                            [--config FILE]
                                            [file_id [file_id ...]]
positional arguments:
 file_id               The GDC UUID of the file(s) to upload

optional arguments:
 -h, --help            show this help message and exit
 --debug               Enable debug logging. If a failure occurs, the program
                       will stop.
 --log-file LOG_FILE   Save logs to file. Amount logged affected by --debug
 --color_off           Disable colored output
 -t TOKEN_FILE, --token-file TOKEN_FILE
                       GDC API auth token file
 --project-id PROJECT_ID, -p PROJECT_ID
                       The project ID that owns the file
 --path path, -f path  directory path to find file
 --upload-id UPLOAD_ID, -u UPLOAD_ID
                       Multipart upload id
 --insecure, -k        Allow connections to server without certs
 --server SERVER, -s SERVER
                       GDC API server address
 --part-size PART_SIZE
                       DEPRECATED in favor of [--upload-part-size]
 --upload-part-size UPLOAD_PART_SIZE, -c UPLOAD_PART_SIZE
                       Part size for multipart upload
 -n N_PROCESSES, --n-processes N_PROCESSES
                       Number of client connections
 --disable-multipart   Disable multipart upload
 --abort               Abort previous multipart upload
 --resume, -r          Resume previous multipart upload
 --delete              Delete an uploaded file
 --manifest MANIFEST, -m MANIFEST
                       Manifest which describes files to be uploaded
 --config FILE         Path to INI-type config file

Data Transfer Tool Configuration File

The DTT has the ability to save and reuse configuration parameters in the format of a flat text file via a command line argument. A simple text file needs to be created first with an extension of either txt or dtt. The supported section headers are upload and download which can be used independently of each other or used in the same configuration file. Each section header corresponds to the main functions of the application which are to either download data from the GDC portals or to upload data to the submission system of the GDC. The configurable parameters are those listed in the help menus under either download or upload.

Example usage:

gdc-client download d45ec02b-13c3-4afa-822d-443ccd3795ca --config my-dtt-config.dtt

Example of configuration file:

[upload]
path = /some/upload/path
http_chunk_size = 1024


[download]
dir = /some/download/path
http_chunk_size = 2048
retry_amount = 6

Display Config Parameters

This command line flag can be used with either the download or upload application feature to display what settings are active within a custom data transfer tool configuration file.

gdc-client settings download --config my-dtt-config.dtt
[download]
no_auto_retry = False
no_file_md5sum = False
save_interval = 1073741824
http_chunk_size = 2048
server = http://exmple-site.com
n_processes = 8
no_annotations = False
no_related_files = False
retry_amount = 6
no_segment_md5sum = False
manifest = []
wait_time = 5.0
no_verify = True
dir = /some/download/path