Incomplete Visits Table

[MohammadMahdiJavid]

Hi,

Would it be possible to explain how does incomplete Visits Table works?
In an experiment including 2K websites, I have all site_rank provided for command_sequence and near 400 visit IDs
So incomplete Visits contains both visit IDs and site_rank
But what is this table trying to say?
And what is the exact meaning of incomplete visit? An exception occurred during execution of Commands?
Do we lose the whole visit_id? Or just part of the data for that specific visit_id

Thanks

[vringar]

Hey,

I just realized that we never documented this anywhere.

The short version is that you should discard all data from incomplete visits.

There are two ways a visit can be considered a failed visit, if there was an error while executing the command sequence or if the command sequence was interrupted by a shutdown.

While there might be some data saved but due to the incompleteness we never considered it as part of our analysis.

If you are running larger crawls with lots of failing websites, I would recommend you take a look at the crawler.py, which is what we previously used. Retrying each website up to three times if there was a failure helped us significantly reduce the failure percentage.