SFTP - Bigdata.com API developer docs

When it comes to getting your data into our system, SFTP upload is currently the only supported input method. It’s simple, secure, and easy to implement even for less technical data providers. That said, we’re actively working on expanding our input mechanisms - an API-based option is on the roadmap, along with other improvements to make integration even smoother down the line. Setting up your SFTP connection is as easy as 1-2-3:

1. Generate your SSH public key

SSH authentication is widely regarded as the most secure and reliable method for accessing an SFTP server. It uses the Secure Shell (SSH) protocol to establish an encrypted connection, ensuring that data transferred between the client and server remains private and protected from interception. Unlike basic password authentication, SSH key-based authentication adds an extra layer of security by requiring a private key on the client side that matches a public key stored on the server. This eliminates the risk of brute-force attacks or stolen credentials, as the private key never gets transmitted. Additionally, SSH keys can be further secured with passphrases and are easier to automate safely in scripts and scheduled tasks, making them ideal for both human and system-level access. Overall, SSH authentication combines strong encryption, identity verification, and secure automation, making it the best practice for accessing SFTP servers.These steps will help you generate a new SSH key pair using your terminal on Linux, macOS, or WindowsWe will need only your public key (that is safe to share and helps us set up authentication). Your private key will also be created (and protected by the passkey) - keep it to yourself and never share it with anyone!

1.1 Open your terminal

macOS: Open the Terminal app
Linux: Use your default terminal
Windows: Use PowerShell or Terminal

1.2 Check for existing SSH keys

Run this command to see if you already have SSH keys:

ls ~/.ssh

Look for files like:

id_rsa / id_rsa.pub
id_ed25519 / id_ed25519.pub

If you find them, proceed to 1.6 View your public key If not, continue reading ;)

1.3 Generate a new SSH key pair

Use the ssh-key command to generate a new SSH key pair:

ssh-keygen -C "your_email@example.com"

Adding your_email@example.com is just the argument for -C, which allows you to specify the comment attached to the generated key. The comment is simply text appended to the key in your public key file, and is typically used as a label for your key.The default comment is your username @ the hostname of the system you generate your key on, but it can be any string you wish.

1.4 Choose the file location

You’ll be prompted:

Enter file in  which to save the key (/home/youruser/.ssh/id_ed25519):`

Press Enter to accept the default - we recommend this option
Or enter a custom path if needed

⚠️ If the file exists, you’ll be asked to overwrite - only do this if you’re sure you want to do that!

1.5 Set a Passphrase

You’ll see:

Enter passphrase (empty for no passphrase):

Type a secure passphrase (you won’t see characters as you type)
Press Enter, then confirm

🔐 A passphrase protects your private key, you will need it to authenticate

At this point, you should have both your public and private keys created. Typically, your private key is the file without the extension (id_ed25519), while your public key has .pub as the file extension (id_ed25519.pub)

1.6 View your public key

Whether you already had it or just created it, let’s quickly check if everything went well. Run:

cat ~/.ssh/id_ed25519.pub

(in case you changed the key file name, make sure to point to the correct file in your command instead of id_ed25519.pub)You’ll see something like:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMF... your_email@example.com

You can use this key on servers, services like GitHub or any other place that supports SSH authentication. For our needs, you would need to share .pub file with us, so we import it on our SFTP server. Keep reading to see how to do that.

2. Send the public key our way

We are working on a user friendly way to upload your public key but, until we have it, please send the .pub file (your SSH public key) to data@bigdata.com. We will confirm the reception of the key by responding to your email and immediately get started with setting up the SFTP for you.What we need to do is import your public key and configure your access - it typically takes us one day to get this organized and get back to you with the connection details. Once you receive that from us, proceed to the next step and test your SFTP connection.Meanwhile, while you wait for our response, you can start working on understanding our Bigdata Document Format (BDDF) and preparing the files. Go here to get started with creating your first BDDF file.

3. Test your SFTP Connection

Once you have received a confirmation from us that your public key is imported and SFTP configured (with connection details) it’s time to test your connection.Note that we will share two separate usernames for real-time and historical data delivery - it doesn’t make a difference which one you use to test the connection, if it works for one, it should work for the other as well.Below you can find three ways to do this, pick the one that suits you the most:

Using open-source FTP client

1. Download and install FileZilla

Go to the official FileZilla website
Click on Download FileZilla Client (not Server)
Choose the version for your operating system (Windows/macOS/Linux)
Download the standard version (you don’t need Pro unless you require cloud access features)
Run the installer:

on Windows, run the .exe file.
on macOS, open the .dmg and drag FileZilla into the Applications folder
on Linux, use your package manager or download the .tar.bz2 archive.

2. Convert SSH private key to PPK (if using Windows)

FileZilla on Windows needs a .ppk key (PuTTY format), not OpenSSH by default but we’ll help you convert it.Please note this is about your private key, not the public key that you have shared with us previously. If you don’t have your SSH key pair created, please go here and follow instructions.

Download PuTTYgen
Open PuTTYgen.
Click Load, and in the file browser, select:

File type: All Files
Navigate to C:\Users\yourname.ssh\id_ed25519 (or wherever your private key is)

Click Save private key to export it as a .ppk file.

📝 You can now use this .ppk file in FileZilla.

3. Configure SFTP connection in FileZilla

Open FileZilla
Go to File → Site Manager
Click New Site and give it a name (e.g., RavenPack SFTP)
Set the following fields:

Protocol: SFTP - SSH File Transfer Protocol
Host: use the hostname we provided to you (most likely sftp.prod.nvirginia.common.ravenpack.com)
Port: leave empty or use 22 (default SFTP port)
Logon Type: key file
User: Your username (provided by RavenPack - use either real-time or historical data username, both should work equally)
Key file: — On Windows: browse and select the .ppk file — On macOS/Linux: browse and select your private key (e.g., ~/.ssh/id_ed25519)

Click Connect

4. Verify the connection

If successful, you’ll see the remote directory structure on the right side of FileZilla. Congratulations!

Using command line interface

Testing by using command line interface could be a bit simpler because you don’t need to install any additional software. To do this, follow these steps:

1. Test the connection

Open Terminal
Run the following command:

sftp -i ~/.ssh/id_ed25519 your_username@your.sftp.host

replace your_username with the username provided to you (use either real-time or historical data username, both should work equally)
replace your.sftp.host with the actual SFTP server address (most likely sftp.prod.nvirginia.common.ravenpack.com)

💡 On some systems, sftp uses ssh underneath and may not accept the -i flag. If that’s the case, use this workaround:

ssh -i ~/.ssh/id_ed25519 your_username@your.sftp.host

Then try sftp again, assuming ~/.ssh/config is properly configured.

2. Add an SSH config entry

To simplify future connections, add the following to your ~/.ssh/config:

Host ravenpack-sftp
HostName your.sftp.host
User your_username
IdentityFile ~/.ssh/id_ed25519

Now you can just run:

sftp ravenpack-sftp

3. Use the FTP

Once connected via SFTP, you can run interactive commands: View remote files:

ls

Upload a file:

put local_file.txt

Downaload a file:

get remote_file.txt

Exit:

bye

Using python script

Here’s a Python script that automates the process of testing an SFTP connection using SSH key authentication with the help of the paramiko library.

1. Install paramiko library

Install paramiko library, if you haven’t already:

pip install paramiko

2. Save the script file

Save the code from the snipped below in a file named sftp_text.py:

import paramiko
import os

# === Configuration ===
HOST = "your.sftp.host"            # Replace with your SFTP server
PORT = 22                          # Default SFTP port
USERNAME = "your_username"         # Replace with your username
PRIVATE_KEY_PATH = os.path.expanduser("~/.ssh/id_ed25519")  # Path to your private key
REMOTE_TEST_FILE = "test_remote.txt"
LOCAL_TEST_FILE = "test_local.txt"

# === Connect and Perform SFTP Actions ===
def sftp_connect():
    try:
        # Load private key
        key = paramiko.Ed25519Key(filename=PRIVATE_KEY_PATH)

        # Create Transport and SFTP client
        transport = paramiko.Transport((HOST, PORT))
        transport.connect(username=USERNAME, pkey=key)

        sftp = paramiko.SFTPClient.from_transport(transport)
        print(f"✅ Connected to {HOST} via SFTP")

        # List remote directory
        print("📁 Remote directory listing:")
        for filename in sftp.listdir():
            print(" -", filename)

        # OPTIONAL: Upload a test file
        if os.path.exists(LOCAL_TEST_FILE):
            print(f"⬆️ Uploading '{LOCAL_TEST_FILE}'...")
            sftp.put(LOCAL_TEST_FILE, REMOTE_TEST_FILE)
            print(f"✅ Uploaded as '{REMOTE_TEST_FILE}'")

        # OPTIONAL: Download the same file back
        print(f"⬇️ Downloading '{REMOTE_TEST_FILE}' to 'downloaded_{REMOTE_TEST_FILE}'...")
        sftp.get(REMOTE_TEST_FILE, f"downloaded_{REMOTE_TEST_FILE}")
        print(f"✅ Downloaded successfully")

        # Close connection
        sftp.close()
        transport.close()
        print("🔌 Disconnected.")

    except FileNotFoundError:
        print(f"❌ Private key not found at {PRIVATE_KEY_PATH}")
    except paramiko.AuthenticationException:
        print("❌ Authentication failed. Check your SSH key and username.")
    except Exception as e:
        print(f"❌ Connection failed: {e}")

# === Run Script ===
if __name__ == "__main__":
    sftp_connect()

3. Edit the configuration

Edit the configuration at the top of the script:

HOST (most likely sftp.prod.nvirginia.common.ravenpack.com)
USERNAME (use the username we provided to you - use either real-time or historical data username, both should work equally)
PRIVATE_KEY_PATH (most likely you will just keep the default one)

4. Run the script

Just run the script and, if everything goes well, you will see the status message confirming the connection was established successfully:

python sftp_test.py

Next Steps

All done and ready with the SFTP? Head to Quick Start Guide to start preparing your first file.

​Next Steps

Next Steps