Potential Similarity Implementation Issue during Retrieval

[mshen2]

Hi, thanks for the work and the released the code. These are very helpful. However, I feel like there might be something off and really appreciate some clarification. Please correct me if I am misunderstanding the code. Thank you!

1. In here, this is calculating the similarity score between instruction and observation instead of actions, since emb is calculating from here. And 'Actions' in memory is the trajectory always starting with Action: search[] since it is passed by here with actions[2:-1]. So basically emb is for obs (page after first search), act, obs, act, ... and np.argmax(cos_scores_act)*2] is always getting the even position which is an obs. This issue is also related to all the other places dealing with even and odd positon.
2. In lines, what is the purpose of adding 2.

[PrincetonThong]

Hi @mshen2, thanks for showing interest in our work!
In L437, the very first action for webshop (which is "reset") is not saved, so the 1st entry in the actions list is the 1st observation seen in webshop. Using [1:] in L329 will change the status such that the emb becomes act, obs, act, obs, ...

For Q2, we previously added +2 to also include the next action of the target observation. However, since we're using sufficiently long actions in the prompt, I think the impact on performance will be minimal.