Learning rate

[batson]

If sometimes happens that there is a huge increase in loss in a single step. Increasing the penalty changes the objective function, but it may also be possible that the objective is well behaved but the optimization procedure is not. Is it possible to take more conservative steps somehow to mitigate divergences without changing the prior? (In analogy with deep learning/sgd, where controlling the stepsize can be crucial.)

Iteration: 0 | deviance=1.7062E+7
Iteration: 1 | deviance=1.6108E+7
Iteration: 2 | deviance=1.2395E+7
Iteration: 3 | deviance=5.1439E+15
Iteration: 4 | deviance=2.1030E+49
Iteration: 5 | deviance=3.5205E+44
Iteration: 6 | deviance=5.8820E+39
Iteration: 7 | deviance=9.8265E+34
Iteration: 8 | deviance=1.6415E+30
Iteration: 9 | deviance=2.7422E+25
Iteration: 10 | deviance=8.0801E+21

[willtownes]

Yes, I agree this is a big problem and I am becoming increasingly convinced that the diagonal Fisher scoring approach in the current implementation is not a good optimizer. I recently benchmarked diagonal Newton's method on a simplistic 2-dimensional problem and it performed badly compared to popular first-order methods like gradient descent with momentum. I will try rewriting the code to use something like avagrad and see if it solves this problem. An added bonus is this would facilitate scaling to larger datasets via stochastic gradients.

To more directly answer this question, there is not currently any step size parameter in the implementation. However, the place to put it in would be here and here. However, I don't think this would fundamentally solve the problem, as the learning rate would be just altering the diagonal of the Hessian by a multiplicative constant, and if the diagonal approximation to the inverse of the full Hessian is bad (it probably is), this would still cause the update to blow up like it is in your example. That's why I think moving to a more standard first-order method would be a better long-term solution.

[batson]

Thanks, Will. Looking forward to trying out a more stable version.