Bigdata.com API subscriptions have two quotas:

  • API Query Unit: Allow customers to retrieve data from Bigdata.com.
  • Pages of uploaded files: Allow customers to upload files to Bigdata.com.
    • PDF format files
    • Other format files (CSV, XML, JSON, HTML, TXT, DOCX)
      • In standard contracts the platform considers a page as group of 3000 characters.

We encourage you to monitor usage and contact us if you would like assistance in choosing the right subscription plan for your organization.

You can monitor usage at the Subscription level or at Search level.

We recommend to try this how-to guide on

Open in Colab

Otherwise follow the how-to guide Prerequisites instructions to set up your Python SDK environment in your local environment.

​
Subscription level

Use the method get_details to retrieve subscription details:

from bigdata_client import Bigdata

bigdata = Bigdata()
subscription_details = bigdata.subscription.get_details()

​
API Query Unit

Bigdata.com measures the amount of retrieved data with API Query Units; each unit allows retrieval of 10 text chunks.

subscription_details = bigdata.subscription.get_details()
print(f"API Query Unit monitoring")
print(f"Total: {subscription_details.organization_quota.query_unit.total}")
print(f"Used: {subscription_details.organization_quota.query_unit.used}")
print(f"Remaining: {subscription_details.organization_quota.query_unit.remaining}")

Output:

API Query Unit monitoring
Total: 50000
Used: 2025
Remaining: 47975

​
Pages of uploaded files

Bigdata.com measures the amount of uploaded files in pages.

​
PDF format

subscription_details = bigdata.subscription.get_details()
print(f"PDF files pages monitoring")
print(f"Total: {subscription_details.organization_quota.pdf_upload_pages.total}")
print(f"Used: {subscription_details.organization_quota.pdf_upload_pages.used}")
print(f"Remaining: {subscription_details.organization_quota.pdf_upload_pages.remaining}")

Output:

PDF files pages monitoring
Total: 20000
Used: 13923
Remaining: 6077

​
Other format (CSV, XML, JSON, HTML, TXT, DOCX…)

subscription_details = bigdata.subscription.get_details()
print(f"Generic file pages monitoring")
print(f"Total: {subscription_details.organization_quota.file_upload_pages.total}")
print(f"Used: {subscription_details.organization_quota.file_upload_pages.used}")
print(f"Remaining: {subscription_details.organization_quota.file_upload_pages.remaining}")

Output:

Generic file pages monitoring
Total: 20000
Used: 13923
Remaining: 6077

​
Search level

Each new search tracks the amount of retrieved data, and you can consult it at any time with the method get_usage()

Initially, the usage of a new search is 0

from bigdata_client.query import Keyword

some_search = bigdata.search.new(query=Keyword("AI in finance"))
print(f"Initial search usage {some_search.get_usage()}")

After executing the method run(), the method get_usage() returns the used Query Units.

The response might contain a smaller number of chunks due to discarding duplicates, so the usage could be lower. Check the how-to guide Retrieve limited chunks for more details.

from bigdata_client.search import ChunkLimit

some_search.run(ChunkLimit(100))
print(f"Current search usage {some_search.get_usage()}")

Please get in touch with your Account Manager or support@bigdata.com if you have any questions or would like guidance in selecting the best subscription for your needs.