Answer confidence of Generated Knowledge Prompting

[yishangupenn]

Copied from upstream: dair-ai/Prompt-Engineering-Guide#57
Original author: @NiDHanWang
Originally created: 2023-03-20

---

Example of Generated Knowledge Prompting in the guide of advanced usage of prompt, mentioned confidence of answer:

Answer 1 (confidence very high):

No, the objective of golf is not to get a higher point total than others. Rather, the objective is to play a set of holes in the least number of strokes. The total number of strokes is used to determine the winner of the game, not the total number of points.

...
Answer 2 (confidence is a lot lower):

Yes, part of golf is trying to get a higher point total than others. Each player tries to complete the course with the lowest score, which is calculated by adding up the total number of strokes taken on each hole. The player with the lowest score wins the game.

How to calculate any answer's confidence ? I checked the paper by Liu et al. 2022, and the confidence's formula from the paper explained part of my question and meanwhile brought me more questions:

$$p_I(a|q) = \frac{exp \, s_I(a|q)}{\sum_{a' \in A_q} exp \, s_I(a'|q) }$$

where

$$\, s_I(a|q) = \sum_{i=1}^{|a|} log \, p(a_i|a_{ \textless i}, q),$$

and $$a_i$$ is the i-th token of choice a

I can understand that the denominator is kind of normalization factor and what really bothers me is the numerator.

To be more precise, I have two questions about it:

1. It said that "a_i is the i-th token of choice a ". What does "token" mean? Is it each word/phrase of the answer?
2. What is $$p(a_i|a_{ \textless i}, q)$$ trying to do?
   Does it calculate the probability of the i-th token based on the presence of tokens before it? If so, how to calcaluate this probability?