If predictions are 0 (therefore on the linear predictor scale log(0)=-Inf), then getting derivatives for use in variance estimation is tricky. Numerically (grad()), it falls over.
This seems like it would be relatively rare (predictions of exact zeros seem unlikely due to the smooth nature of the model).