Gradient check example

[m-philipps]

• add a documentation example for performing gradient checks (closes Add gradient check methods to notebooks #1481)
• fix Conditional variable initialisation in AmiciObjective.check_gradients_match_finite_differences #1494

I'd be happy about suggestions on the "Best practices" and "How to fix my gradients".

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.34%. Comparing base (e9a969e) to head (c0ef8cb).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1497      +/-   ##
===========================================
- Coverage    84.37%   84.34%   -0.03%     
===========================================
  Files          163      163              
  Lines        14037    14037              
===========================================
- Hits         11844    11840       -4     
- Misses        2193     2197       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dweindl]

Thanks. I think I would just skip over check_grad and directly introduce check_grad_multi_eps. The latter performs much better.

[m-philipps]

I think I would just skip over check_grad and directly introduce check_grad_multi_eps. The latter performs much better.

I agree that check_grad_multi_eps is convenient to use in practice; I like showcasing check_grad first to build it up like a tutorial, such that it is easier to understand what check_grad_multi_eps is doing for people who aren't so familiar with gradient checks. I'm also fine with changing it though.

[PaulJonasJost]

Looks good to me, thanks for this.

[dweindl]

I think it would be good to include some rationale for why to check it afterwards and what to look for. I.e. except for parameters with active bounds, the values should be close to 0. At the same time, this might make it difficult to get good FD approximations.

[m-philipps]

You mean, a 0 gradient makes the FD approximation difficult?

[dweindl]

Yes, that, and one might miss gradient entries that are (incorrectly) always zero.

[m-philipps]

Shouldn't these already show up before optimisation?

[dweindl]

Yes, they should.

[Doresic]

Added some comments, more in the direction of how would one interpret the results of gradient checks. Because sometimes we get high absolute errors, but relative are still ok, so is this an issue, or not? Some guidance for users.

Not sure if we want to add that type of content to this notebook tho, or just want to leave it only as -- "This can be done and it's done using these functions."

[Doresic]

It might be good to add what this represents and why it's here. It is in the name (error) but it shows in some way how much does the gradient change in the local area of the point that's being checked. So it is in some way a showing of whether the step size is small enough -- in cases the function is smooth enough.

However, that's also not completely true. If you're at the optimum then the forward gradient will be positive, and the backward negative, possibly with high values, so fd_err will be very high, almost for any choice of eps.

[Doresic]

Do you think we should add some conclusion on this or one of the gradient checks? In the sense what would we think if we saw this gradient check: the absolute errors are rather high but can be expected with finite differences on random points. We see this also with fd_err, so changing the eps might make sense.
But what's reassuring is that the relative error is not too high in most cases.

Would redoing the gradient check with another eps make sense to show how it can affect the check?