broken resource url/file size scripts

[maxFredenburgh]

No description provided.

[wardi]

please use vscode, pycharm or some editor that lints your python code for you, there are lots pep8 problems here like spaces missing around == and =, spaces missing after , and unnecessary parenthesis after if statements.

Commit messages are important too, just 60 characters or so about what this change does to help make sense of our git logs.

[wardi]

Here's how I think this should work:

1. script that reads urls on stdin and updates a url database (could be a simple csv file) containing:

   • url (primary key)
   • utc timestamp (may be used for removing stale requests and retry in the future)
   • server response
   • file size (if successful)
   • file type (if successful)

   if a url is already present in the database this script won't attempt to fetch it again, this allows repeated runs to refresh the url database and prevents requesting the same url multiple times if present in many resources

   this script can run parallel requests with multiple processes or by using async requests.

2. a script that combines the url database and the od-do-canada json lines file to generate a report on broken links and incorrect file types

3. a script that combines the url database and the od-do-canada json lines file to update file types and sizes in ckan using ckanapi (this should be python2 so it can update directly from the config file instead of over http + api key)

We can feed script # 1 with a command like zcat od-do-canada.jl.gz | jq .resources[].url there's no need for another script to pull out urls.

[wardi]

looking good just a few comments. Please assume each comment applies to all files below it (I didn't bother repeating at each location)

[wardi]

the content-type returned from a web server is not reliable because it gets ignored by users. Can you collect the file type from the Content-Disposition filename=... header instead? https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition

[wardi]

this duplicates a list from the metadata schema. Is it necessary to check against this? If so can you reference the one in ckanext/canada/schemas/presets.yaml instead?

[maxFredenburgh]

the content-type returned from a web server is not reliable because it gets ignored by users. Can you collect the file type from the Content-Disposition filename=... header instead? https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition

@wardi The headers returned from each request do not contain content-disposition. Here's an example of what is returned:
{'Content-Length': '968', 'Content-Language': 'en', 'X-AspNet-Version': '4.0.30319', 'Set-Cookie': 'BIGipServer~NRCAN-FGP-PROD~NRCAN-FGP-VIP1.app~NRCAN-FGP-VIP1_pool=rd1o00000000000000000000ffffa72883cco80; path=/; Httponly; Secure', 'X-Powered-By': 'ASP.NET', 'Server': 'Microsoft-IIS/8.5, ', 'Connection': 'close', 'Cache-Control': 'private', 'Date': 'Thu, 11 Jun 2020 16:27:40 GMT', 'Content-Type': 'text/html;charset=utf-8'}