Previous: Step 8 - Start uploading real-time documents

Real-time data is flowing - now it’s time to deliver historical data so we can perform the backfill. Please use the FTP username for historical data delivery that we have provided you as it will store the files in a slighly separate location - they will not be automatically picked up by our pipeline but we will get notified and plan (and perform) the backfill in a controlled manner (not to overload our system)

We would prefer if you would deliver all of the BDDF documents separately but, if it’s more convenient for you, you can also bundle them in one big file (or a set of files, partitioned by year or by source or however it suits you) - as long as they follow our BDDF specification we will be able to ingest them with ease.

You don’t need to inform us about the delivery - we will get notified as soon as your files are uploaded and plan the backfill as soon as possible. If you have any questions or would like to know more on when exactly will the data become available - feel free to reach out to data@bigdata.com