Add Cosine LR Scheduler for Fine-Tuning

[azahed98]

Have you read the Contributing Guidelines?
Yes

Issue # ENG-23270

Adds support for Cosine LR Scheduler in the fine-tuning api as well as simplifies future support of additional schedulers (https://github.com/togethercomputer/together-finetune/pull/1349)

[mryab]

Suggested change

	help="Number of cycles for cosine learning rate scheduler.",
	help="Number of cycles for the cosine learning rate scheduler.",

[VProv]

nit: Maybe also add what fractional values mean.

[azahed98]

Updated to "Number or fraction of cycles".

[VProv]

nit: maybe also add some meaningful upperbound?

[azahed98]

Hard to know what a reasonable upperbound would be without knowing the number of steps afaik. I think it makes sense to follow the hf implementation and let the cosine alias if the user inputs something unreasonably large for their job.

[VProv]

Let's call it scheduler_num_cycles for clarity?

[azahed98]

The backend expects the field in the request to be num_cycles, but in the scheduler args. Would just changing the CLI arg be ok?

[azahed98]

Changed CLI arg to scheduler_num_cycles. Worth noting FinetuneCosineLRSChedulerArgs still has the field as num_cycles