Question: How was the step hint calculated?

[tom-jod]

I would like to ask how the step hint was calculated and whether this could be changed to create a LR schedule based on runtime rather than steps, as this would affect an algorithm like shampoo with a longer time per step negatively if the step hint was based off of NadamW.

Thanks!

[fsschneider]

Hi Tom,

tldr; The step hint is based on NAdamW, but there is no need to use it. It is just a convenience function to help submitters define LR schedules more easily.

Details: The step hint provides an estimate of how many update steps the baseline (NAdamW) can perform within the given runtime budget. It was introduced to make it easier and more convenient for submitters to define learning rate (LR) schedules, though it's entirely possible to build LR schedules based on runtime instead. In fact, that used to be the default before we added the step hint. However, since most LR schedules are typically step-based in practice (where you also have a rough idea for how long you want to train for), we introduced the step hint as a convenience feature.

That said, using the step hint is entirely optional. You're free to base your LR schedule on runtime or any other method (or get rid of it, like Schedule-Free). For instance, if your algorithm takes twice as long per step, you might choose to use half the step hint as the LR schedule horizon. Just make sure that any such stretch factor is workload-agnostic, for example, (1) use a fixed factor across workloads, (2) treat it as a tunable hyperparameter, or (3) determine it automatically by estimating how many steps your algorithm can complete within the budget.

Also note: the step hint does not indicate how many steps the baseline needs to reach the target. It only reflects how many steps it could take within the allocated budget. It may have reached the target well before that.

Hope this clarifies things. Let me know if you have any further questions,
Frank