FileVantage queryChanges paging

[renatoferreirarenatoferreira]

Hi!

I am trying to retrieve changes detected by FileVantage, but the Python API does not seem to work like the HTTP API as described in the documentation (Support and resources / Documentation / Falcon FileVantage APIs).

No matter what I do, I never receive the paging token ("after" parameter), but only offset, limit, and total. Those parameters seem to be useless as I didn't understand how to use them to build a filter to retrieve a next page of items. Due to this, I am using the last ingestion_timestamp and filtering using something like "ingestion_timestamp:>last_ingestion_timestamp". This method seems to be a little susceptible to losses due to the seconds precision of timestamps. It is possible to note that the amount of changes detected is never the same as the one I can see in the dashboard.

Is there a known method to retrieve this information eliminating, or at least reducing, data losses?

Thank you! 👍

[jshcodes]

Hi @renatoferreirarenatoferreira -

Thank you for the question! 😄

The method query_changes points to the API operation queryChanges which provides the offset key for pagination as an integer. This operation is limited to 10K records total.

I believe you are looking for the scrolling API operation, highVolumeQueryChanges which does use the after key for pagination. This token is a string, and should not be susceptible to the precision concern. The PEP8 name for this method is query_changes_scroll.

[renatoferreirarenatoferreira]

I was having problem because I was using filters like "now-1d", so, after some requests, the first object was moving together, what was making the offset useless.

Now I am using a fixed time, what keeps the first object always the same one, so, even if I have additional data being added to the list, the pagination moves together until the very last item without losses during the process. The scrolling API is useful especially for the fist time when I had to download everything stored in our falcon account. It worked well and I intend to keep the same technique even for the incremental retrieves.

I had to do this in order to comply with the PCI log retention period that is not supported by crowdstrike in the filevantage licensing package we have. It is a contrab script that puts the detected changes in a logfile to be sent by filebeat to our elasticsearch cluster.