Importing your Data

πŸ’‘

Overview

At Datagrid we are fully committed to upholding stringent privacy standards, our platform is SOC2 compliant and follow regulations set by the California Consumer Privacy Act (CCPA).

We ensure that your data is not only secure but also managed in respect to your privacy rights. Our platform's advanced security protocols are designed to safeguard your information throughout your entire data import process into Datagrid.

When setting up your data import, we recommend streamlining this process by being selective and importing only the data you need. This practice minimizes your overall data consumption costs and accelerates processing time. By focusing on your essential data, you enhance both efficiency and data quality within Datagrid.


General Import Configuration Guidelines

Start by gathering the credentials or API token(s) you need for your choice of data sources. In most software, your API token/key is typically found in the settings or admin panel which can be used for configuration purposes. Once you have gathered all the necessary information, you can begin connecting your sources to Datagrid.

When configuring your data sources, there are a couple things you should note;

  • Import timing: You can maximize your operations when scheduling your data imports during off-peak hours of your chosen system. Generally off-peak hours are when most users are not working and the system is least accessed. This helps to minimize the load on both your system(s) and Datagrid.
    • We also suggest verifying the update cycles of your databases when scheduling your data imports. This will ensure that you schedule your imports accordingly to capture the most relevant data, preventing potential disruptions to your workflow.
  • Frequency of data import: Determine how often you need to import data depends on the nature of the data sources and how dynamic they are. Balance the need for up-to-date information with system performance, keeping in mind the limitations of the source system that you're extracting data from.

Example:

  • If you would like your Data Pipeline or Copilot to have the most up-to-date data at the start of each day, and your data source updates it's databases between the hours of 12am-3am, it would be best to schedule your import from 3-7am. Depending on what data source and how heavily used it is in your organization, scheduling multiple data imports during peak hours is generally not recommended in order to ensure that your imports run as efficiently as possible.

Troubleshooting Common Issues

Even with the best setup, you may encounter hiccups in your data import flows. Here are a couple things to look out for;

  1. Not seeing the source options you selected; confirm you have the correct permissions on your source system's user account in order to extract the data.
  2. Your data is not fully updated;
    1. Double check your schedule for import timing. This would make the difference to the completion time of your data.
    2. Simply, wait. When you first set up your pipeline to import your data the initial run is retrieving all of the selected source options in the system, known as a full load. Depending on how much data you have in your system and the number of source options selected during pipeline set up, this could take Datagrid some time to import.