Skip to content

How-To: Use File Exchange

Obtaining File Boxes

At the moment, file boxes are only available to members of core facilities (e.g., genomics, bioinformatics, or metabolomics) for exchanging files for their collaboration partners. Currently, HPC users cannot use the file box mechanism on their own.

BIH HPC IT provides a file exchange server to be used by the BIH core facilities and their users. The server is located in the BIH DMZ in Buch. Users authenticate using their Charite/BIH (user@CHARITE) or MDC accounts (user@MDC-BERLIN). File exchange is organized using "file boxes", directories created on the server to which selected users are granted access. Access control list maintenance is done with audit-trails ("Revisionssicherheit") and the file access itself is also logged to comply with data protection standards.

Access from Charite Network

Access from the Charite network (IP ranges 141.x.x.x and 10.x.x.x) must connect through the Charite proxy (http://proxy.charite.de:8080). Depending on the client software that you are using, you might have to configure the proxy.

File Box Management

File boxes are created by the core facilities (e.g., the genomics facilities at Charite and MDC). The facility members also organize the access control. Please talk to your core facility contact on file exchange.

External users must obtain a Charite or MDC account first. Account creation is handled by the core facilities that the external user is a customer of.

File Access

Generally, you will be given a URL to your file box similar to https://file-exchange.bihealth.org/<file-box-id>/. The files are served over an encrypted connection using WebDAV (which uses HTTPS).

The following describes how to access the files in the box from different platforms.

From Linux

We describe how to access the files on the command line using the lftp program. The program is preinstalled on the BIH (and the MDC cluster) and you should be able to just install it with yum install lftp on CentOS/Red Hat or apt-get install lftp on Ubuntu/Debian.

When using lftp, you have to add some configuration first:

# cat >>~/.lftprc <<"EOF"
set ssl:verify-certificate no
set ftp:ssl-force yes
EOF

In case that you want to access the files using a graphical user interface, search Google for "WebDAV" and your operating system or desktop environment. File browsers such as Nautilus and Thunar have built-in WebDAV support.

Connecting

First, log into the machine that has lftp installed. The login nodes of the BIH cluster do not have it installed but all compute and file transfer nodes have it. Go to the data download location.

host:~$ mkdir -p ~/scratch/download_dir
host:~$ cd ~/scratch/download_dir

Next, start lftp. You can open the connection using open -u <user>@<DOMAIN> https://file-exchange.bihealth.org/<file-box-id>/ (NB: there is a trailing slash) where

  • <user> is your user name, e.g., holtgrem,
  • <domain> is either MDC or CHARITE, and
  • <file-box-id> the file box ID from the URL provided to you.

When prompted, use your normal Charite/MDC password to login.

host:download_dir$ lftp
lftp :~> open -u holtgrem@CHARITE https://file-exchange.bihealth.org/c62910b3-c1ba-49a5-81a6-a68f1f15aef6
Password:
cd ok, cwd=/c62910b3-c1ba-49a5-81a6-a68f1f15aef6
lftp holtgrem@CHARITE@file-exchange.bihealth.org:/c62910b3-c1ba-49a5-81a6-a68f1f15aef6>

Browsing Data

You can find a full reference of lftp on the lftp man page. You could also use help COMMAND on the lftp prompt. For example, to look at the files of the server for a bit...

lftp holtgrem@CHARITE@file-exchange.bihealth.org:/c62910b3-c1ba-49a5-81a6-a68f1f15aef6> ls
drwxr-xr-x  --  /
drwxr-xr-x  --  dir
-rw-r--r--  --  file1
lftp holtgrem@CHARITE@file-exchange.bihealth.org:/c62910b3-c1ba-49a5-81a6-a68f1f15aef6> find
./
./dir/
./dir/file2
./file1

Downloading Data

To download all data use mirror, e.g. with -P 4 to use four download threads.

lftp holtgrem@CHARITE@file-exchange.bihealth.org:/c62910b3-c1ba-49a5-81a6-a68f1f15aef6> mirror .
Total: 2 directories, 3 files, 0 symlinks
New: 3 files, 0 symlinks
lftp holtgrem@CHARITE@file-exchange.bihealth.org:/c62910b3-c1ba-49a5-81a6-a68f1f15aef6> exit
host:download_dir$ tree
.
├── dir
│   └── file2
├── file1
└── file.txt

1 directory, 3 files

Ignoring gnutls_record_recv errors.

A common error to see is mirror: Fatal error: gnutls_record_recv: The TLS connection was non-properly terminated.. You can just ignore this.

Uploading Data

To upload data, you can use mirror -R . which is essentially the "reverse" of the mirror command.

lftp holtgrem@CHARITE@file-exchange.bihealth.org:/c62910b3-c1ba-49a5-81a6-a68f1f15aef6> mirror -R
mirror: Fatal error: gnutls_record_recv: The TLS connection was non-properly terminated.
mirror: Fatal error: gnutls_record_recv: The TLS connection was non-properly terminated.
mirror: Fatal error: gnutls_record_recv: The TLS connection was non-properly terminated.
Total: 2 directories, 3 files, 0 symlinks
Modified: 3 files, 0 symlinks
4 errors detected

From Windows

We recommend to use WinSCP for file transfer.

  • Pre-packaged WinSCP on Charite Workstations. Charite IT has packaged WinSCP and you can install it using Matrix24 Empirum on Windows 10 using these instructions in the Charite intranet.
  • Installing WinSCP yourself. You can obtain it from the WinSCP Download Page. A "portable" version is available that comes as a ZIP archive that you just have to extract without an installer.

Connecting

After starting WinSCP, you will see a window titled Login. Just paste the URL (e.g., https://file-exchange.bihealth.org/c62910b3-c1ba-49a5-81a6-a68f1f15aef6/) of the file box into the Host name entry field. In this case, the fields File protocol etc. will be filled automatically. Next, enter your user name as user@CHARITE or user@MDC-BERLIN (the capitalization of the part behind the @ is important). The window should now look similar to the one below.

The WinSCP Login Window

Proxy Configuration on Charite Network

If you are on the Charite network then you have to configure the proxy. Otherwise, you have to skip this step.

Click Advanced and a window titled Advanced Site Settings will pop up. Here, select Connection / Proxy in the left side. Select HTTP for the Proxy type. Then, enter proxy.charite.de as the Proxy host name and set the Port number to 8080. The window should nwo look as below. Then, click OK to apply the proxy settings.

The WinSCP Advanced Site Settings Window

Finally, click Login. You can now transfer files between the file exchange server and your local computer using drag and drop between WinSCP and your local Windows File Explorer. Alternatively, you can use the two-panel view of WinSCP to transfer files as described here.

From Mac

For Mac, we you can also use lftp as described above in From Linux. You can find install instructions here online.

Proxy Configuration on Charite Network

If you are on the Charite network then you must have configured the proxy appropriately. Otherwise, you have to skip this step.

You can find them in your System Preference in the Network section, in the Advanced tab of your network (e.g., WiFi). The Mac Os X Proxy Setting

If you want to use a graphical interface then we recommend the usage of Cyberduck. After starting the program, click Open Connection on the top left, then select WebDAV (HTTPS) and fill out the form as in the following way. Paste the file box URL into the server field and use your login name (user@CHARITE or user@MDC-BERLIN) with your usual password.

The Cyberduck Login Window

If you need to perform access through a graphical user interface on your Mac, please contact hpc-helpdesk@bihealth.org for support.

Security

The file exchange server has the fail2ban software installed and configured (Charite, MDC, and BIH IPs are excluded from this).

If you are entering your user/password incorrectly for more than 5 times in 10 minutes then your machine will be banned for one hour. This means someone else that has the same IP address from the side of the file exchange server can get you blocked. This can happen if you are in the same home or university network with NAT or if you are behind a proxy. In this case you get a "connection refused" error. In this case, try again in one hour.


Last update: April 20, 2022