-
Notifications
You must be signed in to change notification settings - Fork 3
Data refresh (overnight scripts)
Data in Species+ is subject to changes happening more or less frequently. Some of these changes are immediately visible everywhere on the site, while some others need to be processed before being correctly shown to the public. This processing is better known as the "Overnight scripts", as they run during the night (UTC time). This means that changes made to data affected by the overnight scripts will be available the day after the change has been made.
Here is a very high level list of data types involved by the overnight scripts processing:
- Taxonomy
- CITES accepted flags
- Listing changes
- CITES listing
- EU listing
- CMS listing
- Taxon Concepts (Species data)
- Taxon Concepts annex validity date range
- Taxon Concepts appendix validity date range
- Non-compliant shipments data (ComplianceTool)
- TradePlus/TradeView data
The overnight scripts are managed via the Sapi module, and specifically running the Sapi.rebuild command.
This command is also available as a rake task:
bundle exec rake db:migrate:rebuild
Species+ satellite applications, like CITES Checklist, Species+ API, ComplianceTool and TradeView are also dependant on the overnight scripts for some data changes. For example, changes to the listings might affect the results of the Species+ API queries and the timeline charts on the CITES Checklist website.
The overnight scripts run every night, with the exception of the scripts related to the ComplianceTool and TradeView data, which is pre-processed more on an on demand basis.
As already mentioned, there are other cases where changes won't need to go trough extra processing before being visible on the site. This is the case for Countries name changes for example. If some changes made to data not listed above are not displaying correctly, that might be related to caching. If the cache hasn't been refreshed automatically, a manual intervention is always possible.