You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
artifact collection state has an EndTime property - this is the time up to which we are confident we have collected ALL data
the end time is calculated as the lastEntryTime-granularity
i.e. if the graunlarity is 1 day and the last entry time is jan 23 we are confident we have collected all data up to jan 22
This assumes OnCollected is called in date order
file discovery events are ordered - so assuming the artifact source discovers files in date order, we assume the the collection state should track the valid last entry time (and therefore end time)
However the downloads are async - which means the download events come out of order - this means OnCollected is called out of order, which breaks our logic
For now we have work around this by setting a min granularity of 1 day and hoping this works around the problem
Strategic solution is for colleciton stater to track discovered events then with an artifact state map
Each time an artifact is discovered add to the map
when an artifact is dowloaded, update the state int he map and determine the new end time (i.e. if the downloaded artifact is the EARLIEST in the map, use to update the end time)
The text was updated successfully, but these errors were encountered:
artifact collection state has an EndTime property - this is the time up to which we are confident we have collected ALL data
the end time is calculated as the lastEntryTime-granularity
i.e. if the graunlarity is 1 day and the last entry time is jan 23 we are confident we have collected all data up to jan 22
This assumes OnCollected is called in date order
file discovery events are ordered - so assuming the artifact source discovers files in date order, we assume the the collection state should track the valid last entry time (and therefore end time)
However the downloads are async - which means the download events come out of order - this means OnCollected is called out of order, which breaks our logic
For now we have work around this by setting a min granularity of 1 day and hoping this works around the problem
Strategic solution is for colleciton stater to track discovered events then with an artifact state map
Each time an artifact is discovered add to the map
when an artifact is dowloaded, update the state int he map and determine the new end time (i.e. if the downloaded artifact is the EARLIEST in the map, use to update the end time)
The text was updated successfully, but these errors were encountered: