Skip to content

Added per epoch time in log files#1

Closed
quic-abhamidi wants to merge 27 commits intomainfrom
ft_experimental
Closed

Added per epoch time in log files#1
quic-abhamidi wants to merge 27 commits intomainfrom
ft_experimental

Conversation

@quic-abhamidi
Copy link
Owner

@quic-abhamidi quic-abhamidi commented Mar 10, 2026

Added a TrainerCallback function called epoch timer to log the per epoch timing.

quic-meetkuma and others added 26 commits January 2, 2026 13:14
@quic-meetkuma @quic-dhirajku
[QEff. Finetune]: Added logger and its test cases. (quic#644)
a732d02 
- Added a logger which will log onto console and file. This code is
similar to existing QEff. Finetuning logger code.
- Also added dist_utils which serves as utility code when dealing with
distributed training.
- Added logger test cases for sanity checks.

---------

Signed-off-by: meetkuma <meetkuma@qti.qualcomm.com>
@quic-meetkuma @quic-dhirajku
[QEff. Finetune]: Added component registry and factory functionality. (…
3564e27 
…quic#645)

- Added functionality to register dataset, model, optimizer, trainer
objects in a registry and fetch the class of given object based on
configuration provided.
- Also, added simple test cases to verify the functionality.

---------

Signed-off-by: meetkuma <meetkuma@qti.qualcomm.com>
@tchawada @quic-dhirajku
[QEff. Finetune]: Adding optimizer registry and its test cases (quic#649
9715b1b 
)

Adding a Script for Registering and Retrieving Optimizer Classes
The script includes:


get_optimizer()
Returns the optimizer class and kwargs.
Additionally, there is a test_optimizer.py script that validates the
functionality of the optimizer registration and retrieval process.

---------

Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
@quic-dhirajku
[QEff. Finetune]: Added Base dataset class and SFT dataset classes al…
52af924 
…ong with its test cases. (quic#647)

Edited the SFTDataset class to enable custom dataset loading.
Updated the dataset.py file to only enable support for SFTDataset type.
Created test file to check the functionalities.

---------

Signed-off-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
@tchawada @quic-dhirajku
[QEff. Finetune] Adding callback and its test cases. (quic#652)
5f834bd 
Adding a Script for Registering and Retrieving Callback Classes
It has create_callback() function which creates an instance of callback.
Additionally, there is a test_callbacks.py script that validates the
functionality and retrieval process.

---------

Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
@tchawada @quic-dhirajku
"[QEff.finetuning] Adding config_manager and its test cases." (quic#656
46fca42 
)

Added Config_manager to parse the training, model and dataset related
arguments.

---------

Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
@quic-akuruvil @quic-dhirajku
Revert " "[QEff.finetuning] Adding config_manager and its test cases.…
c80d0a8 
…"" (quic#666)

Reverts quic#656
@tchawada @quic-dhirajku
"[QEff.finetuning} Rebasing: hf_config_mananger." (quic#667)
d9cc6fe 
Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
@quic-swatia @quic-dhirajku
[QEff. Finetune]: Adding base class and HF class (quic#658)
389f15a 
-  Added Base Model class and HF model class.
- Base Model class will support FT for any custom model and will be a
common skeleton for any model, including any HF model. 
- Added unit tests for these.

---------

Signed-off-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm.com>
@quic-dhirajku
Added Trainer classes and tests for FT (quic#697)
edfef59 
This PR contains all the changes of PR quic#660 along with all the comments
being addressed. The new PR was created due a rebase issue.

Signed-off-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
@tchawada @ochougul
@mohiso22 @tv-karthikeya @vjanfaza @quic-akuruvil @smedhe @asmigosw @abukhoy @quic-amitraj @quic-dhirajku @quic-rishinr @abhishek-singh591 @qcdipankar @quic-meetkuma @quic-swatia
[QEff.finetuning] Adding sample config and ReadMe file (quic#692)
d072a9c 
Added Readme file for the parameters used in sample config.

---------

Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vtirumal <vtirumal@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <quic_akuruvil@quicinc.com>
Signed-off-by: Abukhoyer Shaik <abukhoye@qti.qualcomm.com>
Signed-off-by: Amit Raj <amitraj@qti.qualcomm.com>
Signed-off-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Abhishek Kumar Singh <sabhis@qti.qualcomm.com>
Signed-off-by: abhishek-singh591 <sabhis@qti.qualcomm.com>
Signed-off-by: Abhishek kumar singh <sabhis@qti.qualcomm.com>
Signed-off-by: asmigosw <asmigosw@qti.qualcomm.com>
Signed-off-by: Dipankar Sarkar <dipankar@qti.qualcomm.com>
Signed-off-by: meetkuma <meetkuma@qti.qualcomm.com>
Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
Signed-off-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Onkar Chougule <168134249+ochougul@users.noreply.github.com>
Co-authored-by: Mohit Soni <quic_mohisoni@quicinc.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: vtirumal <vtirumal@qti.qualcomm.com>
Co-authored-by: vjanfaza <vjanfaza@qti.qualcomm.com>
Co-authored-by: Ann Kuruvilla <quic_akuruvil@quicinc.com>
Co-authored-by: smedhe <smedhe@qti.qualcomm.com>
Co-authored-by: asmigosw <asmigosw@qti.qualcomm.com>
Co-authored-by: Abukhoyer Shaik <abukhoye@qti.qualcomm.com>
Co-authored-by: Amit Raj <amitraj@qti.qualcomm.com>
Co-authored-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Abhishek Kumar Singh <sabhis@qti.qualcomm.com>
Co-authored-by: Dipankar Sarkar <dipankar@qti.qualcomm.com>
Co-authored-by: Meet Patel <meetkuma@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <quic_sallabad@quicinc.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm.com>
@tchawada @ochougul
@vjanfaza @quic-akuruvil @smedhe @quic-rishinr @abhishek-singh591 @qcdipankar @mohiso22 @tv-karthikeya @asmigosw @abukhoy @quic-amitraj @quic-dhirajku @quic-meetkuma @quic-swatia
['QEff.finetuning'] Changing some params from training config to mode…
b78efe6 
…l config (quic#747)

This PR contain:
1.documentation for new finetune experimental stack.
2. Updates inconfig_manager.py

---------

Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <quic_akuruvil@quicinc.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Abhishek Kumar Singh <sabhis@qti.qualcomm.com>
Signed-off-by: abhishek-singh591 <sabhis@qti.qualcomm.com>
Signed-off-by: Dipankar Sarkar <dipankar@qti.qualcomm.com>
Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vtirumal <vtirumal@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Abukhoyer Shaik <abukhoye@qti.qualcomm.com>
Signed-off-by: Amit Raj <amitraj@qti.qualcomm.com>
Signed-off-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
Signed-off-by: Abhishek kumar singh <sabhis@qti.qualcomm.com>
Signed-off-by: asmigosw <asmigosw@qti.qualcomm.com>
Signed-off-by: meetkuma <meetkuma@qti.qualcomm.com>
Signed-off-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Onkar Chougule <168134249+ochougul@users.noreply.github.com>
Co-authored-by: vjanfaza <vjanfaza@qti.qualcomm.com>
Co-authored-by: Ann Kuruvilla <quic_akuruvil@quicinc.com>
Co-authored-by: smedhe <smedhe@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Abhishek Kumar Singh <sabhis@qti.qualcomm.com>
Co-authored-by: Dipankar Sarkar <dipankar@qti.qualcomm.com>
Co-authored-by: Mohit Soni <quic_mohisoni@quicinc.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: vtirumal <vtirumal@qti.qualcomm.com>
Co-authored-by: asmigosw <asmigosw@qti.qualcomm.com>
Co-authored-by: Abukhoyer Shaik <abukhoye@qti.qualcomm.com>
Co-authored-by: Amit Raj <amitraj@qti.qualcomm.com>
Co-authored-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
Co-authored-by: Meet Patel <meetkuma@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <quic_sallabad@quicinc.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm.com>
@quic-akuruvil @ochougul
@mohiso22 @tv-karthikeya @vjanfaza @smedhe @asmigosw @abukhoy @quic-amitraj @quic-dhirajku @quic-rishinr @abhishek-singh591 @qcdipankar @quic-meetkuma @tchawada @quic-swatia
Ft experimental rebasing with main (quic#785)
68c4a98 
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vtirumal <vtirumal@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <quic_akuruvil@quicinc.com>
Signed-off-by: Abukhoyer Shaik <abukhoye@qti.qualcomm.com>
Signed-off-by: Amit Raj <amitraj@qti.qualcomm.com>
Signed-off-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Abhishek Kumar Singh <sabhis@qti.qualcomm.com>
Signed-off-by: abhishek-singh591 <sabhis@qti.qualcomm.com>
Signed-off-by: Abhishek kumar singh <sabhis@qti.qualcomm.com>
Signed-off-by: asmigosw <asmigosw@qti.qualcomm.com>
Signed-off-by: Dipankar Sarkar <dipankar@qti.qualcomm.com>
Signed-off-by: meetkuma <meetkuma@qti.qualcomm.com>
Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
Signed-off-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Onkar Chougule <168134249+ochougul@users.noreply.github.com>
Co-authored-by: Mohit Soni <quic_mohisoni@quicinc.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: vtirumal <vtirumal@qti.qualcomm.com>
Co-authored-by: vjanfaza <vjanfaza@qti.qualcomm.com>
Co-authored-by: smedhe <smedhe@qti.qualcomm.com>
Co-authored-by: asmigosw <asmigosw@qti.qualcomm.com>
Co-authored-by: Abukhoyer Shaik <abukhoye@qti.qualcomm.com>
Co-authored-by: Amit Raj <amitraj@qti.qualcomm.com>
Co-authored-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Abhishek Kumar Singh <sabhis@qti.qualcomm.com>
Co-authored-by: Dipankar Sarkar <dipankar@qti.qualcomm.com>
Co-authored-by: Meet Patel <meetkuma@qti.qualcomm.com>
Co-authored-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <quic_sallabad@quicinc.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm.com>
@quic-swatia
[QEff. Finetuning] Adding text field and some other changes in datase…
4414951 
…t file (quic#787)

1) Adding text field required by TRL's scripts.
2) Passing config_name in the load_dataset_builder
3) Updated test_dataset accordingly.

Signed-off-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm.com>
@quic-swatia
[QEff. Finetuning]: Adding FinetuningPipeline (finetune_experiemental…
ab5918e 
….py) and related code (quic#791)

1) Added FinetuningPipeline (finetune_experiemental.py) which integrates
all the components added for HF-trainer and enable running fine tuning
through it.
2) Added files to handle PEFT and training config. 
3) Made changes in the config_manager and callbacks files. 
4) Added unit tests for the FinetuningPipeline (test_finetune.py)
5) Updated tests in test_callback and test_config_manager based on above
changes.

Finetuning on openai/gsm8k for 5 epochs on single SOC gave the following
numbers:

{"eval_loss":1.0224987268447876,"eval_runtime":484.8933,"eval_samples_per_second":2.72,"eval_steps_per_second":2.72,"eval_entropy":0.9871161538059735,"eval_num_tokens":6525025.0,"eval_mean_token_accuracy":0.7452040632806826,"epoch":5.0,"num_input_tokens_seen":6525025,"global_step":37365}

{"train_runtime":32856.1501,"train_samples_per_second":1.137,"train_steps_per_second":1.137,"total_flos":3.8132170931712e+16,"train_loss":1.0178058738101043,"epoch":5.0,"num_input_tokens_seen":6525025,"global_step":37365}

Training loss at the start of training :1.5146,

Signed-off-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm.com>
@quic-akuruvil
Ft experimental rebasing with main (quic#793)
356355c 
Signed-off-by: Ann <quic_akuruvil@quicinc.com>
@quic-akuruvil
Aligning with main (quic#794)
e8cc003 
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
@quic-swatia
[QEff. Finetuning]: Adding PP support in HF trainer stack (quic#813)
c0ef8cc 
* Added PP support in HF trainer stack. 
* Updated the documentation for the same. 
* Sample command to test PP : QAIC_VISIBLE_DEVICES=0,1 python -m
QEfficient.cloud.finetune_experimental
QEfficient/finetune/experimental/configs/sample_pp_config.yaml

---------

Signed-off-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm.com>
Co-authored-by: Swati Allabadi <sallabad@qti.qualcomm>
@tchawada
[QEff.finetuning] Hf config update (quic#795)
0c49669 
To run integrated_test for DDP use following command:
QAIC_VISIBLE_DEVICES=0,1 torchrun --nproc-per-node=2 -m pytest -q
QEfficient/finetune/experimental/tests/test_integrated.py

---------

Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
Restructure and added info in docs
5942ccc 
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
Cleanup
00a4dbd 
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
Cleanup
bcf8d34 
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
@tchawada
[QEff.finetune]Test finetune (quic#826)
d191e8a 
Modified test_finetune.py
Changed optimizer names

---------

Signed-off-by: Tanisha Chawada <tchawada@qti.qualcomm.com>
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@quic-akuruvil
Docs Updated (quic#833)
cecd2c2 
Signed-off-by: Ann Kuruvilla <akuruvil@qti.qualcomm.com>
@smedhe
[QEff. Finetuning]: adding example scripts to demonstrate custom data…
9751148 
…set registration (quic#835)

Added example script for registering seq_completion dataset_type and
also updated the hf_finetune.md.

---------

Signed-off-by: Sharvari Medhe <smedhe@qti.qualcomm.com>
@quic-abhamidi
Added per epoch time in the log files
f2ab8a3 
Signed-off-by: Anusha Bhamidipati <abhamidi@qti.qualcomm.com>
@quic-abhamidi quic-abhamidi force-pushed the ft_experimental branch 2 times, most recently from 753fab8 to 35f2ff1 Compare March 16, 2026 10:12
@quic-abhamidi
Added per epoch time in the log files
db2a803 
Signed-off-by: Anusha Bhamidipati <abhamidi@qti.qualcomm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@quic-abhamidi @quic-meetkuma @tchawada @quic-dhirajku @quic-akuruvil @quic-swatia @smedhe