We approach this challenge through two primary mechanisms:
- Data sources we store locally in our own environment: these sources provide us updates of the files on a predetermined schedule (e.g., weekly or monthly). In these instances, we are diligent about selecting an update schedule that makes the most sense for our customers, and then making that data available to our customers as soon as we receive the updates.
- Data points that are updated rapidly in the real world (e.g., proxy IP addresses): we leverage real-time APIs that receive these updates as they happen. This allows us to query for and receive the most up-to-date data available. APIs are not always ideal due to the additional latency they add for our customers, but in certain instances, it is a tradeoff we make in order to access data in real time.