[Re] Teaching CLIP to Count to Ten

[Harshvardhan-Mestha]

Original article: Teaching CLIP to Count to Ten (https://arxiv.org/pdf/2302.12066)

PDF URL: https://github.com/SforAiDl/CountCLIP/blob/main/resc/ReScience.pdf
Metadata URL: https://github.com/SforAiDl/CountCLIP/blob/main/resc/metadata.yaml
Code URL: https://github.com/SforAiDl/CountCLIP/blob/main/model.ipynb

Scientific domain: Computer Vision
Programming language: Python
Suggested editor:

[rougier]

Thanks for your submission and very sorry for the delay. We'll soon assign an editor.

[rougier]

@benoit-girard @gdetor @koustuvsinha Can any if you edit this paper ? I may have some names for reviewer.

[karannb]

Hello reviewers,

This is Karan, one of the authors, are there any updates for us?

Thanks.

[rougier]

Very sorry for the long delay. I'll edit your submission. Don't hesitate to post here to remind us.

[rougier]

@anthony-strock @ilemhadri @bengioewould @hhihnyou be interested in reviewing this submission:
Teaching CLIP to Count to Ten (https://arxiv.org/pdf/2302.12066)

[Harshvardhan-Mestha]

Hello
Are there any updates on this ?

[gdetor]

@rougier I can review this submission.

[rougier]

@gdetor Thanks. I'll ask anthony-strock by email.

[a-strock]

@rougier Happy to review this submission.

[gdetor]

Hi @rougier and @Harshvardhan-Mestha
Here is my report:

Text

Overall, the text is coherent and well-written. There are a few typos here and there; for instance, the parameter \lambda sometimes appears as \lambda and others as lambda. Please fix that that. Moreover, Figures 2 and 3 should be larger.
My primary concern is that the authors claim in the main text that the baseline accuracy in the work of (Paiss et al., 2023) is 27.5%. Paiss reports the official baseline accuracy on the CountBench dataset to be 31.67% (CLIP-B/32) (Table 1A in Paiss).
I didn't manage to run the code due to some errors and broken dependencies (see my comments below).

Source Code

1. The script experiment.py does not run due to a missing merged.json file.
2. The google-colab library is not included in the requirements.txt. You can add a line like this: pip install git+https://github.com/<project_owner>/<project_name> in your requirements.txt.
3. Also, what version of Google Colab did you use? There were some broken dependencies on my installation.
4. There is a missing license file. Please add the appropriate license.
5. I recommend cleaning all cells' outputs in your notebooks. Moreover, adding some comments/text in the notebooks explaining the basic steps and operations would be better.
6. In the usage section of the README file, the requirements.txt file is located in the parent directory, not in the scripts.
7. Please pass all your Python scripts through the `autopep' to comply with PEP.
8. Finally, cleaning up and organizing the repository would be nice.