Gradient optimizers

[demroz]

adds a few gradient optimizers

• gradient descent
• nesterov
• and L-BFGS

everything is policy based, so these should be quite extensible. I may add a few more optimziers in the future, but these are the main ones i think. If there are any specific ones that you guys think should be added, I'd be happy to do so. Also although everything is reverse-mode autodiff centric, as long as you provide the objective function, a way to evaluate it, and a way to evaluate the gradient, everything should work correctly.

For some examples on how to use the optimziers: test_gradient_descent_optimizer.cpp, test_nesterov_optimizer.cpp, and test_lbfgs.cpp should be good starting points.

I'm working on the documentation currently. I wanted to hold off to see if any major revisions are necessary

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.32%. Comparing base (2697069) to head (2a63c30).
⚠️ Report is 82 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1328      +/-   ##
===========================================
+ Coverage    95.24%   95.32%   +0.07%     
===========================================
  Files          814      825      +11     
  Lines        69309    67994    -1315     
===========================================
- Hits         66015    64812    -1203     
+ Misses        3294     3182     -112     

see 509 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2697069...2a63c30. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[demroz]

At this point the code is something I'm relatively happy with. I also updated the docs. Would be great if this could get reviewed cc: @mborland @NAThompson

[mborland]

@demroz in general this looks very good! Left some comments, and opinions that you can feel free to push back on. I pushed changes to CI and spelling rather than leaving those as TODOs for you.

[demroz]

@mborland

I updated the PR according to your suggestions. A few points :

1. I think adding support for C++17 execution policies should be done in a separate PR with dedicated benchmarks.
2. using <random> instead of <boost/random.hpp>. I'm not sure what should be done here. I agree with your point that this breaks the ability of this being a standalone package. However, <random> doesn't support multiprecision types. We could static_cast to double for multiprecision types, and static_cast back to multiprecision but thats not behavior I would expect as a user. Practically, i doubt it makes much difference as this is just an initialization policy. Any thoughts?

[mborland]

@mborland

I updated the PR according to your suggestions. A few points :

1. I think adding support for C++17 execution policies should be done in a separate PR with dedicated benchmarks.
2. using <random> instead of <boost/random.hpp>. I'm not sure what should be done here. I agree with your point that this breaks the ability of this being a standalone package. However, <random> doesn't support multiprecision types. We could static_cast to double for multiprecision types, and static_cast back to multiprecision but thats not behavior I would expect as a user. Practically, i doubt it makes much difference as this is just an initialization policy. Any thoughts?

I mentioned it above but I think generating a random double and then casting it to a multiprecision type should be fine. I would not expect the lost digits to create much of a difference.

[demroz]

@mborland

I updated the docs and removed the boost::random dependency. Let me know if anything else is needed from my end to merge