Integrative experiment of statement types

[markwhiting]

Based on the idea of #61, we want to conduct an integrative experiment across statement design points. Here we outline the experiment and a registration for it.

Preparation procedures

• Check that statements are clean
• Check that they are labeled
• Check that we have coverage of the design space, e.g., at least 15 statements per design point
• Integrative experiment registration #87
• Copy edit platform pages #44

Sampling procedure

1. Sample 10 starting design points at random.
2. Sample 15 statements for each design point.
3. Assign 100 participants to each design point using inverse weighting
4. When a design point is finished, evaluate PQ common sense and train a model to predict other design points with all completed data.
5. Compute model certainty on unsampled design points and select the next design point based on the one about which we are least certain.
6. Register predictions for the new design point before starting to sample it.

Required features

We should finish these other things before launching

• GPT Statement generation commonsense-statements#11
• GPT Statement cleaning commonsense-statements#9
• Automated statement labeling commonsense-statements#10
• View for results and sharing #21
• Page for adding statements #23
• Sampling procedure and modeling infrastructure #89

[amirrr]

Dimensions of statement and their definition:

behavior

• Social: it refers to beliefs, perceptions, preferences, and socially con- structed rules that govern human experience; it can be “real” or opinion, but is intrinsically of human origins. e.g., I exist and am the same person I was yesterday. He yelled at me because he was angry. There are seven days in the week.
• Physical: it refers to objective features of the world as described by, say, physics, biology, engineering, mathematics or other natural rules; it can be measured empirically, or derived logically. e.g., Men on average are taller than women. The Earth is the third planet from the Sun. Ants are smaller than Elephants.

everyday

• Everyday: people encounter, or could encounter, situations like this in the course of their ordinary, everyday experiences, e.g., Touching a hot stove will burn you. Commuting at rush hour takes longer. It is rude to jump the line.
• Abstract: this claim refers to regularities or conclusions that cannot be observed or arrived at solely through individual experience, e.g., Capitalism is a better economic system than Communism. Strict gun laws save lives. God exists

figure_of_speech

• Figure of speech: it contains an aphorism, metaphor, hyperbole, e.g., Birds of a feather flock together. A friend to all is a friend to none.
• Literal language: it is plain and ordinary language that means exactly what it says. e.g. The sky is blue. Elephants are larger than dogs. Abraham Lincoln was a great president.

judgment

• Normative: it refers to a judgment, belief, value, social norm or convention. e.g., If you are going to the office, you should wear business attire,not a bathing suit. Treat others how you want them to treat you. Freedom is a fundamental human right.
• Positive: it refers to something in the world such as an empirical regularity or scientific law, e.g., hot things will burn you; the sun rises in the east and sets in the west.

opinion

• Opinion: it is something that someone might think is true, or wants others to think is true, but can’t be demonstrated to be objectively correct or incorrect; it is inherently subjective. e.g., FDR was the greatest US president of the 20th Century.. The Brooklyn Bridge is prettier than the Golden Gate. Vaccine mandates are a tolerable imposition on individual freedom.
• Factual: it is something that can be demonstrated to be correct or incorrect, independently of anyone’s opinion, e.g., the earth is the third planet from the sun (this is correct and we know it is correct), Obama was the 24th president of the United States (this is incorrect, but we know it’s incorrect). It will be sunny next Tuesday (we don’t yet know if this is correct, but we will be able to check in the future).

reasoning

• Knowledge: the claim refers to some observation about the world; it may be true or false, opinion or fact, subjective or objective e.g., The sun rises in the east and sets in the west. Dogs are nicer than cats. Glasses break when they are dropped.
• Reasoning: the claim presents a conclusion that is arrived at by combining knowledge and logic, e.g., The sun is in the east, therefore it is morning. My dog is wagging its tail, therefore it is happy. The glass fell off the table, therefore it will break and the floor will become wet.

[markwhiting]

@amirrr How should I specify the experiment for this?

[amirrr]

@amirrr How should I specify the experiment for this?

You can create a new file that ends with *.expaeriment.js under server/survey/experiments/ or modify an already existing one and it will added to the records. For what to put in the file you can look at this example here:
integrative.experiment.js