Plan of the 'game'

[PLtier]

In this project we are tasked with building the pipeline given code, mainly notebooks, from DS team.
I propose we:

1. Deploy MVP as fast as possible i.e. make it run as fast as possible i.e. a) put the notebooks code to single .py code b) setup simplest dagger workflow which pulls the data and executes .py code c) setup simplest github workflow with manual code. d) after it's done, give the tag to the version
2. Then focus on refactoring the code with readability and scalability in mind
3. Then do the non-required tasks i.e. tests and so on.

[PLtier]

Hi, let me do a short update for you all.
I'm going to test a potential new github workflows.

My plan for now is to implement

• a workflow testing a model on PR to main
• a workflow of attaching a tag after successful PR to main.

For testing it, I will create two branches to create testing commits. I will then cherry-pick commits related to workflows' yaml files.

@SebWae @adam-rosenorn any comments?

Also I would like @SebWae to prepare fine-grained partition of the python file. Could you make it for the next Tuesday, or preferably Monday? You could give us a short presentation on that?

[SebWae]

Sounds great! I think it makes sense for the workflow to be triggered by a PR to main.

I can split the full_script into more fine-grained Python scripts. I can make a PR when I think it is ready to be shared.

[PLtier]

Just wanted to write a plan:
@SebWae -> testing. Write any tests for our code, but preferably pipeline (like, if file was created etc., read learnit)
@adam-rosenorn -> Write documentation. And if we decide to remove xgboost, you would do that. See Paolo's answer to Jakub inquiry.
@PLtier -> for now nothing, let's catch up Friday or Sunday.

Let me remind you the deadline is on Thursday.

[PLtier]

OK, I have found tasks for myself:

• https://learnit.itu.dk/mod/forum/discuss.php?d=38131#p84369 => we can write tests to ensure all of the files in artifacts are output somewhere. We will run dagger
• Use the make file to install dependencies on the machine.
• I will add the precommits to check for quality of our code AND run all of the tests after every commit.
• I will do something with this log artifacts, I don't see any reason to store them in mlflow run (we have this completely random storage of all artifacts in mlflow runs directory, it's fine, but we should not store xgboost model in LR mlflow dir and vice versa)

🐐

[PLtier]

Ok, for the next three days I will be mostly unavailable, and I will do it on Wednesday.
What I would like you to accomplish:

• Documentatio
• Cleaning pipeline.go (@PLtier)
• Code refactor. What I mean is reading very thoroughly through first part and checking if there are redundant operations carried out. Also, please remove this flow artifact logging as we already log them to artifacts dir.
• Cleaning repo if needed? (Please do not remove init files ! :) )
• Now @SebWae has done a pretty cool test which checks if model has indeed been logged. What I want to do now is to do the same for all artifacts folder: upload all artifacts and then check if all of them have indeed been uploaded. It's a very important test as Jeppe said we needed to ensure all artifacts are logged!

[PLtier]

We should not include xgboost

After really 'tough' decision we decided on not doing the comparison of the xgboost. There are strong incentives to do this, but we really don't need to take that risk. (@adam-rosenorn )
I think @SebWae has done a great work of wrapping the code beautifully, so the data science team can in one line-change just perform it.

Why yes for xgboost?

• there is plausible intent to include in mlflow comparison xgboost
• it's just a few lines to do this, to the extent such that one could say it was an oversight.

But actually why not?

• We realised that xgboost usessubsample parameter which introduces randomness. How should we handle this randomness, especially when the tests are failing? Retry? What is a plausible fail or just because of randomness? And tests for xgboost indeed fail, actually often
• They didn't include xgboost in their validation action deps suggesting it's not supposed to be tested.
• In the comparison between xgboost and LR a weighted f1_score is used whereas in MLflow comparison (best experiment) is made based on binary f1_score. We are not entirely sure which metric to use.
• Furthermore, xgboost's weighted f1_score was computed on training data whereas for LR on test data. We are muc certain it has been a mistake, but again it introduces uncertainty.
• We actually don't have any information on what is happening with this model afterwards. Maybe xgboost should not be served /used due to some constraints or wishes we are not aware.
• data science and guessing are not our tasks ;) We are here to refactor code such that it runs efficiently.

Common ground:

• Just wrap and refactor the code, but don't run it. If they want - they can easily use our code.

---

I think we should better go safe and indeed take for now LR until we hear more from our Data Science team ;)
We wrapped their intent, since there were strong incentives that this is what they wanted to do, and actually it was easy to implement, but let's go safe.
This is already a great piece of software we have crafted.

Originally posted by @PLtier in #45 (review)

[PLtier]

We have done amazing job! Yeah, we could have done more, but you can do this forever :)