SYVT Google Maps Issue #1: Invalid Vehicle Position Stale Timestamp

[unreasonableman]

[mostly copy&paste from slack]

Ali:
looking at SYVT’s realtime feed on Google’s Partner dash and it seems everything is going well. Going into the individual reports, there are some issues that pop up such as Invalid_vehicle_position_stale_timestamp (screenshot attached).
Cause The VehiclePosition’s timestamp is too old, implying that it’s either stale, or from a bad clock.
Effect The VehiclePosition is discarded.
Fetches affected 8% (771 of 10010 fetches)
Possible resolutions

• Stale/unassigned VehiclePositions will be ignored. This is not necessarily problematic, unless this vehicle was expected to be in service.
• Consider filtering out-of-service vehicles from the feed.
• Double-check that the clock source is synchronized, and that the timezone conversions are correct.

[unreasonableman]

[Scott on slack]

Reading through this, and one thing that comes to mind
@Kay Neuenhofen
is that, as a part of the "batch position update" process you put together, aren't there some amount of dummy positions updates sent to the server? I wonder if this could be related

[unreasonableman]

[Kay on slack]

I don't think so. IIRC, the way it works is that position updates aren't always immediately written through to the DB if they come in fast and furious. But we are talking a delay of a couple of seconds at most here. There aren't any dummy position updates, except for in the server tests to make sure that the actual updates all get flushed out. Those have their own agency IDs, and even if they didn't, there's no way that would account for 8% of updates.

My suggestion would be to:
see if there are any details available in the dash regarding just how stale the updates are
get SYVT position updates from the DB for some relevant days and look at the timestamps
Is it possible that one of the tablets used by SYVT has an incorrect date/time set?

Okay, logged into the dash and looked at the actual data. You can drill down into individual updates that were marked problematic. Stale updates are off by ~20 mins on average, and this happens infrequently for more than one vehicle, so it's unlikely to be an off clock setting.
Looking at it, it seems more like the following scenario: vehicle X submits and update and no other updates from X come in for ~20 mins (spotty coverage?). The server keeps the last update around and includes it in later feed messages.
As mentioned above, the logs are very detailed, so it would be possible to compare with actual DB data to confirm.
@Ali Attari
should we be spending more time on this? And if so, should it be me or
@Scott Owades
?

The other recurring issue I saw is "bad start time", which sounds like the observed start time for a trip is different than what is given in the static feed. Again, we can look at what's actually in the DB to confirm, and then check with SYVT to see if they occasionally have trips that start 10-15 mins late

To continue this thread of talking to myself, if it turns out that spotty coverage leads to stale position updates remaining in feed updates, a possible fix is to just retire vehicle updates once they hit a certain age.