Clarification on Linear Probing: Discrepancy between Paper Description and Implementation/Results

[freezing-index]

Thanks for the impressive work. I found a discrepancy regarding the Linear Probing implementation in the paper versus the current codebase.

1. Feature Source: Bottleneck vs. Last Layer Tokens
The paper (Sec. 4.1) states: "we probe only the reduced-dimensionality features from the bottleneck... directly evaluating the inherent properties of the latent features." This implies LP is done on the 64-dim latent features.

However, the current implementation seems to use concatenated tokens:

• In the README example, it suggests using cls token + patch tokens.
• In tools/test_linear_probing_hf.py (Line 143), it also appears to use concatenated features instead of the bottleneck.

2. Performance Gap
I conducted LP experiments using the VTP-L-d64 model, specifically mapping the $16 \times 16 \times 64$ bottleneck features to 1,000 classes:

• Paper Result: 85.7% Top-1 Acc.
• My Result: 73.896% (trained for 100 epochs).

Questions:

1. Was the 85.7% accuracy reported in the paper achieved using the 64-dim bottleneck or the 768-dim transformer output?
2. If it was the 64-dim bottleneck, could you please provide the specific LP training hyperparameters (e.g., learning rate, weight decay, optimizer) to help reproduce the result?

Looking forward to your clarification. Thanks!

[JingfengYao]

That’s a great question.

In the linear probing (LP) of the DINO series, the standard approach involves training multiple probe heads simultaneously using different learning rates, various features, and combinations of class tokens and pooling features, then selecting the best result. (We can refer to this as DINO-style LP.) Refer here.

However, the issue with this approach is that only the 64-dimensional features are truly relevant to the generative process. Therefore, while exploring the relationship between understanding and generation, we specifically used those 64-dim features for LP to measure understanding performance (which explains why most of those values are not particularly high). When reporting the final performance of 85.7, we utilized the DINO-style LP to evaluate the VTP encoder’s performance as a general feature extractor, looking beyond its role as just a generative VAE.

[freezing-index]

Thanks for the clarification.