How to Upload

There are two basic features to the ProteomeCommons.org Tranche network: adding data and getting data. This document describes how to add data to the network using the command line tool and the Java API.

If you are a researcher who wishes to use the GUI to upload data to the ProteomeCommons.org Tranche network, you may wish to visit the tutorial on using the Tranche GUI. However, to illustrate how the servers work, we will walk through the process of running the server using the GUI, creating a user, setting permissions, and uploading our first file on our localhost.

Brief network introduction

Anyone can host a Tranche server, and it is remarkably simple to do so.

The simplest way is to run the Tranche GUI tool, which uses Java Web Start so that installation and updates are automatic. Two other options include using the command line tool, or by running the server in a custom tool using our Java API. The latter two are discussed in this document.

Note that ProteomeCommons.org Tranche requires authentication. If you are interested in using the ProteomeCommons.org Tranche network, you will need to contact an administrator to generate a certificate and digital keys, or to sign an existing certificate. Until you register with ProteomeCommons.org Tranche, any attempt to upload will result in a security exception, preventing a file upload.

However, you can still use Tranche in networks where you are trusted. To make this scenario simple, you can run the tranche software on your localhost (127.0.0.1). The process of uploading requires:

To follow along with this exercise, launch the Tranche GUI tool. To begin, select the "Server Configuration" tab under the "Preferences" menu located at the top of the application.

The default port of 443 is listed in the port field. Decide if this is the port you would like to use or change it to another of your choosing, then click Start the server. (I will use 1500 instead of the default, 443, since it is less likely to be closed by various operating systems, though you can use whatever port you deem appropriate.)

An error message will be displayed if the port you have chosen is blocked by your OS or currently in use. The best way to resolve this issue is to determine whether or not your OS has closed the port. The easiest way to resolve this issue is to simply use another port, which is why I chose 1500 for this exercise.

If the server starts successfully you will be able to see the addition of the server under the "Server" tab. The new server will be displayed in the form tranche://ip_address:port.

Creating a user

To create a user, click on Create User Tool. This tool is located under the "Users" tab on the left hand menu. This will launch the Create User dialog box.

Fill in the required information as well as any relevant optional information, and click on Create User. The client will take a few seconds to create the user file, so wait for a confirmation message.

When the client is done creating the user file, it will alert you with a message saying "User file successfully made". You probably want to store this file somewhere readily accessible because it is required everytime you wish to upload to Tranche.

Now that you have created your user file, you can close the Create User dialog box.

Registering the user with the network

Since you are running the server on your own computer, you can grant permissions to users (via their user files) at your discretion.

By default any time a user starts a server they are logged on as that server's administrator; however, this also requires that before a server can be started, a "user file" must be created for the user and said user must also be logged in. If you wish to create additional administrators this can be accomplished fairly easily; however, it is not recommended that when creating new "user files" that they are given administrator priveledges.

To add an administrator, click on Add New Administrator, which is found on the left hand menu under the "Servers" tab. A choose user file dialog box should appear. Find and select the user file. When Tranche has successfully added the user, you will be alerted with a message that states that the user was added as an administrator for the server.

Next, we want to view the server configuration. Under the "Servers" tab find and select your server. On the left hand menu you can select "Configure Server". A Server Configuration dialog box should open.

If you again select Configure Server from the left hand menu of the "Servers" tab after selecting your server, and this time select the "Data Directories" tab, you can change the directory where uploaded data is stored as well as set size limits. This is especially useful for testing as well as dedicating separate drives for server storage.

When you are satisfied with the settings, click Save Changes. Assuming you set this particular user as an administrator, you should receive no objections. If you get an error message about priveledges, you need to add the user as an administrator.

After saving the changes, you can close this server configuration dialog box.

Testing the Upload Tool

This document doesn't aim to cover the GUI upload function, but to complete our exercise, click on Upload Tool. A Tranche Uploader dialog box will appear.

If we want the upload to work with our server we must modify the appropriate fields:

  1. Upload: select a test file to upload
  2. User: load the same user file we've been using
  3. Title: each uploaded file must have a title
  4. Description: if a directory of files is being uploaded a description must also be input
  5. In order to upload to only your server, you must specify/configure the uploader appropriately. To do this select "Options" from the top menu and then click on "Configuration". A dialog box will be displayed with all available servers, their status, and the user permissions for those servers. From this list, ensure that only your server is selected.

You can optionally uncheck Register this upload with ProteomeCommons.org's network. Since we supplied a server IP, the client won't attempt to upload to the Proteome Commons Tranche network.

Click "Save Changes" and you will be returned to the uploader dialog box.

Here's a screenshot of a sample dialog box before clicking Upload File(s):

Click Upload File(s). After a few seconds (assuming the file size is small), you should get a confirmation dialog box with the hash, which you should store. (Expect more time if file(s) is large.) The hash is used to download the file.

The key to exploring the software on the server running on your computer is to supply the URL for localhost (i.e., tranche://127.0.0.1:1500). This will prevent Tranche from attempting to upload to the ProteomeCommons.org Tranche network. Before you continue, you may want to explore the client software to get an idea of the scope of the Tranche project.

Uploading files using the command line tool

Download the Tranche command-line tool

Due to the fact that Tranche was designed for command-line code and with security features focused on keeping data safe, some features of this project are not well suited for a web browser yet. Currently command-line code (well suited for scripts) exists, and GUI tools for non-technical users are in the works.

If you are looking for instructions on using the GUI upload tool, please visit the GUI upload tutorial in the User section.

Directly adding files to the network

Anyone that is trusted by the data network may directly upload data and have that data be publicly available for others to use -- the data may optionally be encrypted so only collaborators can access it. The upload process is very similar to submitting data; however, instead of just referencing the file(s) to upload a public X.509 certificate and matching private key are required. A commercial certificate and key may be purchased from a number of different vendors (GoDaddy.com sells them for $30). Alternatively, you can have the ProteomeCommons.org Tranche network auto-generate an appropriate certificate and key for you. In any case, whoever administers the data network servers you are accessing must have previously registered your public certificate as trusted in order for you to be able to directly add data.

You will note that no direct upload page exists on the demo web application. This functionality was purposely omitted because it would require a user to upload their private key – a bad idea because the owner of the web site could then steal the private key. Instead, the file upload GUI tool and the command line client tool are provided so a user can use a private key locally without ever sending it to a server.

The command line client tool can be run by executing the AddFileTool.jar class in a directory that contains your private key (named 'private.key') and public certificate (named 'public.certificate'). The command can take several options, but requires only one: the name of the file or directory to upload.

java -jar AddFileTool.jar C:/my-files-to-upload

The code will execute and automatically upload the file(s) using your local private key for credentials.

Connecting to: file:///ProteomeCommons.org-DFS
Trying to upload file(s): C:/my-files-to-upload
17% Done, uploaded: C:/my-files-to-upload/51958.zip, MD5: feffccc94ccffcd576fb7c71f0b9d2e9
38% Done, uploaded: C:/my-files-to-upload/28629.zip, MD5: e995b45ae60caa7045cedb8ec91a97c9
67% Done, uploaded: C:/my-files-to-upload/53178.zip, MD5: 4b4715dc4641d827df801470a204575d
100% Done, uploaded: C:/my-files-to-upload/mag2005.zip, MD5: 018c91ce06f5f3c1552401701b33812f
Upload Complete

After uploading, the files will immediately be available for download using any of the file download options.

Securely sharing pre-publication data

Securely sharing data is identical to publicly sharing data except the secure data is encrypted before it is uploaded. The encryption can be done manually, but the ProteomeCommons.org Tranche network provides excellent support for most all of the popular encryption algorithms. To encrypt data, simply use the '--useEncryption true' option when uploading data.

java -jar AddFileTool.jar --useEncryption true C:/my-files-to-upload

The output will look very similar to the normal AddFileTool.jar output but with a notice that provides the encryption algorithm used along with the passphrase required to decrypt the data.

Using encryption.
  - Auto-generated passphrase: A+T95oCWlp8B9DDEewCW
  - Encryption algorithm: Blowfish
Connecting to: file:///ProteomeCommons.org-DFS
Trying to upload file(s): /dfs-data/abrf-prg2006
17% Done, uploaded: abrf-prg2006/51958.zip, MD5: feffccc94ccffcd576fb7c71f0b9d2e9
38% Done, uploaded: abrf-prg2006/28629.zip, MD5: e995b45ae60caa7045cedb8ec91a97c9
67% Done, uploaded: abrf-prg2006/53178.zip, MD5: 4b4715dc4641d827df801470a204575d
100% Done, uploaded: abrf-prg2006/mag2005.zip, MD5: 018c91ce06f5f3c1552401701b33812f
Upload Complete

In the above example, the Blowfish algorithm – a very strong encryption – is used, with a passphrase of 'A+T95oCWlp8B9DDEewCW' required to decrypt the data. It is critical that you remember this passphrase (write it down!) as it is not stored with the data and you will need it to be able to retrieve the data. As with the normal method of adding data to the network, the uploaded data will immediately be available for download; however, the data is encrypted. While anyone will be able to download it, only those with the passphrase can decrypt and use it. This strategy works well for collaborations, as the passphrase can easily be shared with anyone who needs access to the data. The strategy also works well for letting people download pre-publication data while preventing them from using it until the passphrase is published.

Besides security, the command line offers many possibilities that can make working with data much simpler, including the possibility to...

The last two can leverage varying degrees of tools, but are more easily accomplish using the unix find tool. For example, say I wrote a small script (perhaps shell, perhaps a very high level language like Perl or Python) that takes in a list of files from the standard input, puts them all in an upload directory, and calls the upload command client. We could use find in combination as a powerful way to quickly upload data:

find . -name "*.t2d" -mtime -7 | ./custom_script

In this example, we are asking find to look in this directory, and recursively through all subdirectories, for any file ending in ".t2d" that was created (or added) within the last week. We are piping the output to a script we wrote, which will gather the data in one directory and execute the upload command line client to upload the directory, ideally appending the output hash in a text file of hashes. This cuts out a lot of overhead, including repeatedly navigating through GUI selections and having to copy the hash every time you commit an upload.

Java API

There are many reasons you may wish to create your own Java upload tools:

A complete simple sample upload client and a more verbose upload client exists that you may wish to look at now. However, we will walk through the steps of creating your own custom upload client.

If you are looking for sample code illustrating the Tranche Java API for downloading a file, or other common activities, visit the code snippets page. The upload code found above will also be cross-listed on the code snippets page.

Since user uploads are validated using X.509 certificates and encryption using your private key, writing upload code could involve handling the security issues using a security API. However, to abstract, the creation of "users" simplifies the validation process by encapsulating security.

An overview of the upload process involves codifying these steps:

AddFileTool

AddFileTool offers a simple API for uploading data to a Tranche server. Let's look at a code snippet to illustrate its use:

AddFileTool addTool = new AddFileTool(user.getCertificate(), user.getPrivateKey());
addTool.addServerURL("tranche://127.0.0.1:1500");
addTool.setTitle("The title.");
addTool.setDescription("The description.");
BigHash hash = addTool.addFile(new File("/path/to/file"));
System.out.println("The file's hash: " + hash);

We stated earlier that the creation of a "user" encapsulates our security concerns. In the case of this code snippet, our "user" offers simple access to the certificate and private key, which AddFileTool needs for authentication and (potentially) for security. (The "user" is actually an object of the class UserZipFile, which we will discuss shortly.)

This code snippet assumes we have a server running on our localhost, accessible via port 1500. The default port is 443, but I chose 1500 since unpriveledged users will more likely have access to that port. However, if you are hosting a server on the Tranche network, you will most likely use the default port to simplify usage.

The method addFile(File) uploads a file to the server we selected, and returns a hash, which you can print out. You should store the value of the hash, since it is needed to download the file from the server.

If you are developing the tool as a command line tool, you may find it convenient to simply redirect the output to a file for convenient storage, or, if you choose, you can use a file writer to create a new file or append to an existing file, which simplifies the record-keeping process of retaining hashes for future reference. In either case, if you use Tranche frequently, using a custom tool can greatly simplify the process of uploading data.

Users and UserZipFile

Say you have already created a user file with administrative priveledges. You can load that file in your custom application:

UserZipFile user = new UserZipFile(new File("/path/to/user_file.zip.encrypted"));
user.setPassphrase("super_secret_password");

Working with existing users is easy. It is easy to create a new user, but since we are uploading a file, we will have to set the permissions so the upload will be successful:

maker = new MakeUserZipFileTool();
maker.setName("John Smith");
maker.setPassphrase("super_secret_password");
maker.setUserFile(new File("John.zip.encrypted"));
UserZipFile user = (UserZipFile) maker.makeCertificate();
// Set user permissions as admin
user.setFlags((new SecurityUtil()).getProteomeCommonsAdmin().getFlags());

It is assumed that if you upload multiple times, you will only create the user once. To upload to the Tranche network, you will need to contact an administrator at ProteomeCommons.org, who will generate a certificate and key for you, or authenticate a certificate for you if you already have one that you wish to use. As long as you are uploading to a server you control however, you do not need to do this. See the introduction above for more information.

Servers

If you already have a server running, you do not need to run a server from your code. In fact, if you attempt to run a second server on the same host using the same port, the server will be unable to connect, citing a java.net.BindException. We can use this to our advantage — we can assume a server isn't running on a specific port, and even if it is, our application will still work, capitalizing on the advantage of exception handling.

FlatFileTrancheServer ffserver = new FlatFileTrancheServer(new File("/path/to/dir"));
ffserver.getConfiguration().getUsers().add(user);
try {
  Server server = new Server(ffserver, 1500);
  server.start();
}
catch (BindException e) {
  System.out.println("Port 1500 already bound, continuing assuming a Tranche server is already running...");
}

// ... code handling upload

try {server.setRun(false);} catch (Exception e){}
try {ffserver.close();} catch (Exception e){}

Notice that we are registering the user with the flat file server, and we are wrapping that server in a Server object. We will actually run the server as a thread, and notice that we catch a BindException. If you are confident that this exception will almost always be the result of an instance of Tranche already running, you may even leave out the message, meaning the user doesn't really need to know whether a server is running or not.

After we upload the file, we will want to close the server so that the operating system can free the socket.

The complete upload tool

There are two examples of the upload tools. The first assumes that you know the Tranche server that you would like to upload to and that you have an appropriate user file. The second example assumes that you want to use the Java API to set up a new server as well as create a new user file. Here is the first, simple code example. Here is the complete sample upload client Java source code, which is a modified version of our code snippets. Note that there are boolean variables you can change depending on whether you have an existing user file and whether you have a server running. (Even though the code will work regardless of whether or not a server is running, you may not want the application to attempt to create a server process, choosing for your application to fail if a process doesn't already exist.)

You may find it easier to adapt the above code rather than to start from scratch. If that is the case, there are many ways you can make a custom tool more useful. For example you could modify the code to:



Home   •   FAQ   •   Contact Us   •   Join email list   •   Collaborations   •   Credits