add connection error handling

[lsloan]

This issue was inspired by lack of connection error handling in MiVideo code, both for UDP's BigQuery and Kaltura's API, but it probably applies to code for other data sources and to the application's database as well.

Connection error handling should log messages and either retry the operation a specific number of times or exit with an error status.

When testing connection error handling, helpful techniques include (but not limited to):

• Specify the wrong connection parameters (hostname, database name, port number, etc.)
• Specify the wrong credentials (username, password, token, etc.)
• Forcibly shut down the application DB
• Forcibly shut down the network connection

[ssciolla]

This is too broad. We have exception catching for all (I believe) Canvas API calls and retry mechanisms for most of them. There is an issue for handling database transactions already (#135). This issue should be split into multiple issues for things I didn’t mention (mivideo, initial db and network connections). It’s not realistic for a single PR to address all these.

[ssciolla]

It's possible I'm getting mixed up here and conflating connection errors with other kinds of API/DB errors. Still, I kinda felt like the reason for opening this issue was to make sure MiVideo/Kaltura API calls would retry if they got an irregular status code (besides the one you're already handling).

I understand the idea of creating this broad issue, and there are probably several connection-specific areas that could be improved, but more specific issues would probably serve us better.

[zqian]

@lsloan seems this issue is too broad, and may not fit the timeline of 2020.02.01 release. Can we move revise the scope and move it to the next release issue list?

[lsloan]

@zqian: Yes, as we discussed a few days ago, I'm fine with moving it to another release.

Also, when @ssciolla suggested more error handling for the MiVideo code, I agreed. As I was creating this issue, I noticed that some of the other code doesn't seem to have enough error handling, which is why I broadened the scope of the issue. I don't mind narrowing the scope of this issue to MiVideo code only. However, if I'm correct in my analysis that other parts of the code need more error handling, issues for that should be opened. Issues for improved error handling should be linked together for ease of reference.