betaFeedback

WorksComputer Science

Hey, guess what: CUES lead to inferences about typicality

Abstract

Throughout life, we accumulate knowledge of how typical certain events are. We use this knowledge in daily conversation, often interpreting meaning beyond what has been said explicitly. In this work, we take steps to better understand this meaning recovery, investigating both atypicality and typicality inferences. First, we construct two corpora, EventTyp and the human-validated EventTypGold. We conduct large-scale human and LLM experiments and find that for humans, three CUES predict inferences about typicality: Commonsense Knowledge, Utterance Emphasis, and Snark. However, in most experimental settings, we find that LLMs lack the prerequisite Commonsense Knowledge necessary to accurately assess an event's general typicality, and therefore struggle to follow these CUES.

Introduction

In our daily lives, we make decisions based on what we think will and will not happen. This knowledge about event typicality (i.e., which events are likely or unlikely to happen) helps us navigate the intricacies of human conversation, interpreting meaning beyond what speakers have literally said [11, 1].

In this work, we investigate two types of event-typicality inferences. These rely on our commonsense knowledge about which events are typical (and unlikely to be mentioned when they occur) and which are atypical (and likely to be mentioned when they occur). Consider Figure 1. We might expect that when someone thanks their boss, they typically mean it (there is no need to explicitly state this). But we also expect that when a speaker says something, it is informative. So, when a typical event is explicitly mentioned (left panel), our expectations of the speaker conflict with our commonsense knowledge (i.e., the speaker appears to be stating something uninformative). To resolve

Figure 1. Examples of what can cause an atypicality or typicality inference. Someone making an atypicality inference would answer the question on the left with unlikely, whereas if they made a typicality inference, they would answer the question on the right with likely.

this conflict, we make an atypicality inference that, in this context, this event that is generally typical for others is in fact atypical for Anahita. Conversely, when an atypical event is explicitly mentioned (right panel), this is straightforwardly informative and so we may simply make a typicality inference that, in this context, this generally atypical event is more typical for Anahita than we previously believed.1

Previous studies have carried out only limited human experiments on event-typicality inferences ([25, 26, 39]), and even less work has been done on LLMs ([27]). However, event-typicality inferences pose important challenges because they rely on both commonsense knowledge and reasoning ability. For LLMs, investigating such inferences is particularly critical since recent works have shown that LLMs often exhibit shallow and inconsistent commonsense event-knowledge ([3, 2, 23]). Therefore, in this paper we aim to expand our understanding of event-typicality inferences in both humans and LLMs. We ask:

  • RQ1: Do humans and LLMs share Commonsense Knowledge about event typicality?

  • RQ2: When do humans make inferences about event typicality?

  • RQ3: Do LLMs make inferences about event typicality in the contexts where humans do?

To address these, we create EventTyp, a corpus of 2,354 labeled event tuples, and EventTypGold, a subset of 858 tuples with additional human validation. Using our corpora, we conduct large-scale experiments with both humans (∼900\sim900 participants) and LLMs (10 models and 3 inference methods).

First, our results show that LLMs mostly do not agree with humans in their commonsense knowledge about event typicality (RQ1). This has broad implications for LLM usage, since many downstream tasks, such as task planning [49] and knowledge graph completion [19], rely on LLMs’ commonsense knowledge.

Second, when examining event-typicality inferences in humans (RQ2), we find that three CUES invite them: 1) Commonsense Knowledge, 2) Utterance Emphasis (e.g., “And hey, guess what”), and 3) Snark (i.e., when an atypicality inference helps the listener learn something negative about someone).

Finally, when examining LLM behaviour (RQ3) we find that while some LLMs make event-typicality inferences following similar CUES as humans, this is limited primarily to large models with Chain-of-Thought prompting (Wei et al., 2022).

Our contributions are: (i) We create two new large-scale, high-quality corpora for studying event typicality, EventTyp and EventTypGold; (ii) We conduct large-scale human experiments and find evidence for three CUES that can lead to inferences about event typicality; and (iii) We conduct extensive experiments with LLMs and show that LLMs often lack event commonsense knowledge, which undermines their ability to make event-typicality inferences.2

Related Work

LLMs & commonsense knowledge about events

LLMs have been shown to struggle to reason about relationships between events, whether they are rare/newsworthy [24, 17] or daily life [40, 44, 43]. Prior work has examined LLMs’ ability to differentiate event arguments [35], as well as how probable events are on different levels of granularity: e.g. plausible vs. implausible [18, 21] or a spectrum from probable to inconceivable [22, 15, 29]. To our knowledge, no work has investigated the more fine-grained task of distinguishing the typicality of two events that are both highly plausible but still different in typicality.

Human pragmatic reasoning about typicality

Pragmatic reasoning can be characterized in terms of comprehenders’ expectations for what cooperative speakers are likely to do, including the expectation for speakers to provide an appropriate Quantity of information [11]. Apparent violations of this expectation (saying something ambiguous or otherwise uninformative) can invite inferences.

With atypicality inferences, a speaker’s apparent uninformativity (the mention of some typical event given the context) contrasts with the listener’s belief in the speaker’s cooperativity. But the speaker is informative if the reported event is actually atypical given the context (see §A for more details). Existing experiments on such inferences [25, 26] have asked participants to read dialogue snippets and rate the mentioned events’ typicality, revealing atypicality inferences for typical events but no inferences for atypical events. Subsequent work confirms the inferences reflect Quantity-driven reasoning [39].

LLMs & pragmatic reasoning

The limited work on LLMs’ pragmatic reasoning has shown that a wide gap persists between humans and LLMs [14, 48, 31]. For event-typicality inferences in particular, Kurch et al. [27] conducted prompting experiments with two LLM families on a set of 24 utterance tuples, which is relatively limited in scale compared to our experiments, and observed no atypicality inferences.

The EventTyp Corpus

This section describes the process of creating the EventTyp corpus. The corpus comprises event tuples (Context, TypicFollowUp, Atypic-FollowUp): Context is an event extracted from a commonsense dataset (§ 3.1), TypicFollowUp is a typical follow-up event to Context and AtypicFol-

DefinitionExampleWritten byValidated by
ContextThe first event utteranceAnahita thanks her bossLLMHumanLLMHuman
TypicFollowUpEvent utterance stating something that is generally typical given Context.Anahita means it2,0622921,2261,128
AtypicFollowUpEvent utterance stating something that is generally atypical given Context.Anahita is in tears1,67967502,354

caption: Example (Context, TypicFollowUp, AtypicFollowUp) tuple from EventTyp, along with the number of items per-type written and validated by an LLM and a human. Note that we also count the human-written and human-validated data from Kravtchenko and Demberg [25]. The total number of tuples in EventTyp is 2,354.

:::

lowUp is an atypical follow-up event to Context. See examples in Table 1. Both the TypicFollowUps and AtypicFollowUps are generated by an LLM (GPT-4o, OpenAI et al. [33]). Both the TypicFollowUp and the AtypicFollowUp depend on the Context, but we generate each AtypicFollowUp according to both its Context and the TypicFollowUp. Therefore, after generating the TypicFollowUps (§ 3.2), we filter them to remove those that could not plausibly lead to an atypicality inference (§ 3.3) and then only generate AtypicFollowUps for the remaining examples (§ 3.4). Our final EventTyp corpus has 2,354 event tuples.

Context Event Extraction and Filtering

We source our Contexts from ATOMIC (Sap et al. [41]; Hwang et al. [16]), a commonsense knowledge base consisting of if-then event pairs (e.g., “PersonX takes PersonY’s advice” isAfter “PersonY gives advice to PersonX”). We extract all event pairs for two relation types (isBefore and isAfter) and limit utterances to those with up to two entities. Combining and deduplicating them yields 33,924 utterances.3 We then apply rule-based and LLM-based filtering to remove noisy or unsuitable events (e.g., idioms, passive voice; see full details in §B.2). This leaves 12,978 Context events.

Typical Follow-ups (TypicFollowUps)

For each Context, we use an LLM to generate up to three TypicFollowUps in order to increase the diversity of our corpus (see §B.3 for details). This yields 19,827 (Context, TypicFollowUp) candidates.

Atypicality Inference Event Criteria

Despite prior work on atypicality inferences, no precise attributes have been identified as triggering them.4 To create EventTyp, we develop a set of specific triggering criteria through manual annotation. We then apply these criteria through LLM-prompting to our candidate (Context, TypicFollowUp) pairs.

Criteria Development. One of the authors manually annotated a subset (N=9,188N = 9{,}188) of candidate event pairs for whether or not it was possible to make an atypicality inference from them, and if not, why it was not possible. The annotations yielded a set of 2,026 (Context, TypicFollowUp) pairs labeled as triggers for atypicality inferences. From these we determine two primary rules (see Appendix Table 5 for rule details and examples):

  1. The follow-ups should be natural for dialogue (e.g. grammatical; ‘Mary takes Linda’s hand’ is but ‘Mary take Linda hand’ is not).

  2. The TypicFollowUp should have ‘natural’ typicality, e.g. very typical given the Context but still not entailed by the Context. So, (“Mary gives Linda advice”, “Linda listens”) is acceptable, but (“Mary arrives at Linda’s party”, “Linda has a party”) is not.

LLM-based Criteria Filtering. We operationalize our criteria by prompting an LLM with 27 true-false questions (see §B.3 for details) We reject candidate event pairs where even one criteria rule is violated. Due to inconsistency in the LLM responses, we split some criteria into multiple overlapping questions in order to increase accuracy. We use few-shot prompting with manually crafted examples and iteratively develop the prompts, favoring precision over recall. That is, we prefer the LLM rejects more good follow-ups if that ensures fewer bad ones.

To evaluate the quality of the LLM-filtering, we create a set of human labels by having one human annotator label 39 randomly sampled (Context, TypicFollowUp) pairs from our dataset using the same 27-question setup as the LLM. We then calculate precision (78.11), recall (69.93), and inter-uninformative” to characterize TypicFollowUps. But these heuristics are too vague to apply in large-scale annotation. annotator agreement in the form of Cohen’s Kappa (κ=.4532\kappa= .4532) [5] between the LLM and human annotator on this set. While agreement is modest, this is primarily because the LLM rejects more follow-ups than the human annotator. On the annotated sample, the human annotator rejects 16.48%, while the LLM annotator rejects 26.37%. We run the LLM-filtering on all 19,827 candidate (Context, TypicFollowUp) event pairs, resulting in 4,155 that meet our criteria.

Atypical Follow-ups

For each (Context, TypicFollowUp) pair that passes our filtering in § 3.3, we use an LLM to generate a candidate AtypicFollowUp. We instruct the model to generate a less typical, more specific alternative to the original TypicFollowUp (§B.4). The number of generated AtypicFollowUps (4,155) is much lower than the number of TypicFollowUps generated in § 3.2 (19,827), so we manually validate and correct all candidate AtypicFollowUps.5 We are left with 4,113 (Context, TypicFollowUp, AtypicFollowUp) tuples, from which we subsample 2,330 so our data has only one TypicFollowUp and one AtypicFollowUp per Context (note that there are up to three followups per Context from § 3.2, and see §B.5 for details).

We add the 24 tuples from Kravtchenko and Demberg [25] (K&D) to our data, rewritten to have a consistent format with our tuples. In total, our EventTyp corpus has 2,354 (Context, TypicFollowUp, AtypicFollowUp) event tuples. See Table 1 for the distribution of follow-ups written and validated by a human or an LLM.

Human and LLM Experiment Details

Here we describe our collection of human and LLM event-likelihood ratings, which applies to both our Commonsense Knowledge experiment (§ 5) and our Event-Typicality Inference experiment (§ 6). We collect event-typicality ratings by asking how ‘likely’ certain follow-up events are; hence, we use the term likelihood ratings throughout.

Human Participant Details

We recruit 330 participants for the Commonsense Knowledge experiment (§ 5) and 575 for the Event-Typicality Inference experiment (§ 6);6 see §C.1 for more details. For both experiments, each participant rated the likelihood of a batch (Context, Follow-up) pairs, where Follow-up was either TypicFollowUp or AtypicFollowUp, and no Context event appeared more than once in each batch. Each batch contained 6 attention check questions that were randomly mixed with the other items. For each event pair, we collected likelihood ratings from multiple participants: 3 for the Commonsense Knowledge experiment and 5 for the Event-Typicality Inference experiment.

LLM Details

We test 28 LLM configurations covering three inference methods (surprisal, two prompting approaches) and five model families with two sizes each: GPT-5.4, (OpenAI, [34]); Llama-3.1, (Grattafiori et al., [10]); Olmo-3, (Olmo et al., [32]); Qwen3, Yang et al., [47]; Gemma-3, (Gemma Team, [9]). See §C.2 for details.

Surprisal We compute surprisal (negative-log-likelihood), which can be used to probe LLMs for underlying language understanding. We adapt the method from Cilleruelo et al. [4], computing surprisal Sp(Completion) over tokens tit_i:

Sp⁡(Completion)=∑i=1N−log⁡p(ti∣t1,…,ti−1)\operatorname{Sp}(\mathrm{Completion}) = \sum_{i=1}^{N} -\log p(t_i \mid t_1,\ldots,t_{i-1})

To simulate multiple-choice question-answering, we insert each likelihood option into our input (shown in Figure 2) and then compute surprisal on the Completion (Follow-up). Since we are only comparing surprisals of the same Follow-up, there is no need to normalize the surprisal values by length. We consider the LLM’s chosen answer to be the Input likelihood option where the Completion’s summed surprisal is lowest. For this approach, we use only non-instruction-tuned (base) LLMs.

Figure 2. Setup for our surprisal experiments with examples using the Unlikely likelihood rating option. Surprisal is computed over the Completion, given the Input.

Prompting To evaluate LLMs in a way that more closely mirrors how we obtain human results, we also prompt them for a likelihood rating in the same way as we do humans. We try both direct prompting (Prompting) and Chain-of-Thought prompting (CoT-Prompting).7 For this approach, we use the instruction-tuned versions of the models used to compute Surprisal.

Commonsense Knowledge Experiment

To answer RQ1 (whether humans and LLMs share Commonsense Knowledge about event typicality), we elicit likelihood ratings from both humans and LLMs for every (Context, Follow-up) pair in EventTyp and then compare their responses.

We then create EventTypGold (N = 858) from only the EventTyp tuples where the humans voted that the TypicFollowUp was typical and the AtypicFollowUp was atypical (§ 5.3). We use EventTypGold for our Event-Typicality Inference experiment (§ 6).

Experimental Setup

We collect likelihood ratings for all (Context, Follow-up) pairs in EventTyp following the procedures in § 4. Table 2 shows an example of the question format we use. For each tuple in EventTyp,8 we generate two questions: one for the (Context, TypicFollowUp) pair and the other for (Context, AtypicFollowUp) pair. We use GPT-4o (§D.1) to generate the questions, manually validating them and correcting any errors (e.g., pronoun ambiguity). For each question, the possible likelihood ratings are {Very Unlikely, Unlikely, Likely, Very Likely}.

Follow-upContext example Anahita thanks her bossQuestion In general, when someone thanks their boss, how likely is it that they
TypicFollowUpAnahita means it...mean it?
AtypicFollowUpAnahita is in tears...are in tears?

Table 2. Example of the questions we ask in the Commonsense Knowledge Experiment. Two questions are asked per Context, one for the likelihood of the TypicFollowUp and one for that of the AtypicFollowUp.

For each LLM configuration, we calculate inter-annotator agreement (more details in §G) using Cohen’s Kappa (Cohen, [5]) for the LLM and

human responses (each response is a 1 majority-vote). We collapse the four likelihood options into the binary categories of UNLIKELY (Very Unlikely/Unlikely) and LIKELY (Very Likely/Likely) and calculate κTypicFollowUp\kappa^{\mathrm{TypicFollowUp}} for TypicFollowUps and κAtypicFollowUp\kappa^{\mathrm{AtypicFollowUp}} for AtypicFollowUps.

RQ1: Do humans and LLMs share Commonsense Knowledge about event typicality?

Mostly no: humans and LLMs generally do not agree on which events are typical versus atypical.

Figure 3 shows our agreement results. Out of our 28 LLM configurations, only GPT-5.4 with CoT has higher than 0.4 agreement for both Follow-ups (κTypicFollowUp=0.4336\kappa^{\mathrm{TypicFollowUp}} = 0.4336 and κAtypicFollowUp=0.5059\kappa^{\mathrm{AtypicFollowUp}} = 0.5059). Only GPT-5.4 and Llama-3.1-70B-Instruct achieved κ>.3\kappa> .3 for both follow-ups with both prompting settings, but the other three model families do not achieve this even with CoT-Prompting. This low agreement illustrates the lack of shared commonsense knowledge between our human participants and LLMs.

Figure 3. Human-LLM agreement for the Commonsense Knowledge experiment, with κAtypicFollowUp\kappa^{\mathrm{AtypicFollowUp}} on the X-axis and κTypicFollowUp\kappa^{\mathrm{TypicFollowUp}} on the Y-axis.

The EventTypGold Corpus

We are interested in whether EventTyp’s Follow-up categories (Typic/Atypic) match our human annotators’ actual likelihood ratings. We first compute, for each Context, the percentage of tuples where the average rating of the TypicFollowUp was higher than that of the AtypicFollowUp (i.e., the TypicFollowUp was judged more likely than the AtypicFollowUp). This condition holds for 80.04% of our generated tuples and 95.83% of the K&D tuples. This aligns with how we prompted the LLM (§ 3.4), asking it to generate a Follow-up that was less likely than the TypicFollowUp but not necessarily unlikely.

However, for a given Context we want not only the correct relative relation between Follow-ups but also the stricter condition that the TypicFollowUp is LIKELY and the AtypicFollowUp UNLIKELY. We compute the percentage of tuples where this occurs with at least 2/3 annotators agreeing. From the tuples from EventTyp that we generated, this condition holds for 36.09% (841), while it does for 70.83% (17) tuples from K&D, despite the fact that the tuples were manually crafted by their original authors to have pairs of likely-unlikely events. The 858 tuples that meet the strict condition are our final EventTypGold corpus.

Event-Typicality Inference Experiment

To answer RQ2 and RQ3, we collect human and LLM likelihood ratings on a sample (N=450N = 450) of EventTypGold with dialogue scenarios. We subsample due to budget constraints (see §E.1).

Experimental Setup

Dialogue Scenarios To create an environment where pragmatic inferences are possible, we insert events into dialogue scenarios and then ask questions about those scenarios. Unlike Kravtchenko and Demberg [26], we opt to use minimal dialogue framing in order to avoid confounding factors. We create two example instances from each (Context, TypicFollowUp, AtypicFollowUp) tuple using randomly selected rare names (see §E.2 for details). Then we use GPT-4o to generate a typicality question for the TypicFollowUp and AtypicFollowUp events. We manually validate and correct all questions. Unlike in the Commonsense Knowledge experiment (§ 5), the questions are use names specific to the dialogue scenarios.

Experimental Conditions We embed the Context in a dialogue scenario and then investigate how the likelihood ratings for its Follow-up (TypicFollowUp/AtypicFollowUp) change when the Follow-up is mentioned (Plain and HeyGW) or unmentioned (NONE). Table 3 explains this setup. In the Plain setting, the Follow-up is mentioned without Utterance Emphasis, while in the HeyGW setting the Follow-up is preceded by Utterance Emphasis (the phrase “And hey, guess what”). We consider a participant’s response in the NONE setting to be a control (i.e., the Follow-up’s baseline likelihood given the Context). Our analyses compare responses from the NONE setting to those from Plain or HeyGW.

Follow-up Setting
Follow-upNONEPlainHeyGWQuestion
TypicFollowUp–“She meant it.”“And hey, guess what, she meant it!”In general, when Anahita thanks her boss, how likely is it that she means it? (Very Unlikely, Unlikely, Likely, Very Likely)
AtypicFollowUp–“She was in tears.”“And hey, guess what, she was in tears!”In general, when Anahita thanks her boss, how likely is it that she is in tears? (Very Unlikely, Unlikely, Likely, Very Likely)

Table 3. Setup for Event-Typicality Inference experiments (§ 6). A participant sees the top dialogue (containing a Context), and ONE Follow-up setting (containing either a TypicFollowUp or a AtypicFollowUp), along with its corresponding question. This means that for each tuple, there are 6 possible experiment conditions.

Inference score (InfScore) To compute whether an atypicality or typicality inference was made, we define InfScore (see §G). For each individual event, InfScore measures the difference in average responses between the NONE and the mentioned (Plain or HeyGW) settings. A positive InfScore indicates an atypicality inference and a negative InfScore a typicality inference. The magnitude of InfScore represents inference strength.

Snark Annotation We hypothesize that one possible reason why speakers produce utterances that lead to atypicality inferences is that they wish to convey negative information about someone without it being explicitly negative. We define these scenarios as Snarky: if the atypicality inference helps the listener learn that the person being talked about has some negative trait, and the listener can therefore interpret the speaker as having a snarky attitude. For example, on the TypicFollowUp side of Figure 1, if an atypicality inference is made (i.e., that Anahita usually does not mean it when she thanks her boss), this may mean Anahita is fake or dishonest. Thus, we label this scenario as snarky. For all (Context, TypicFollowUp) pairs in EventTypGold, we manually annotate whether they form a snarky scenario or not. See §E.3 for a deeper discussion on how snark relates to commonsense knowledge and irony.

Figure 5. Human distributions of responses in the NONE vs. HeyGW settings for TypicFollowUps, with non-snarky scenarios (left) and snarky scenarios (right).

RQ2: When do humans make inferences about event typicality?

Atypicality inferences We find that humans only consistently make atypicality inferences on TypicFollowUps when Utterance Emphasis is present; Snark increases these inferences.

Fig. 4 shows the InfScores for the Plain vs. HeyGW settings. We can see that for TypicFollowUps, InfScore >0> 0 appears often for HeyGW (p<0.001<0.001, see §E.5), indicating atypicality inferences are made when Utterance Emphasis is present. But for Plain, the opposite, InfScore <0< 0, is more frequent (p<0.001<0.001), indicating typicality inferences. This contrasts with Kravtchenko and Demberg [26]’s work, which found that participants made atypicality inferences in the Plain setting. A possible reason for this is that their events were presented within verbose backstories explicitly mentioning typicality, which may have primed participants to make atypicality inferences. Our items, in contrast, use only a short dialogue.

To check Snark’s effect on atypicality inferences, we focus on TypicFollowUp events in the HeyGW setting, the condition for which participants made consistent atypicality inferences. Fig. 5 shows that the difference between ratings in the NONE and HeyGW settings is higher when Snark is present (p<0.001<0.001, see §E.5), confirming that participants make more atypicality inferences for snarky scenarios. This finding is consistent with the claim that people use pragmatics to communicate about socially sensitive situations. But for a listener to recognize ‘snark’ in a specific scenario and make an atypicality inference on it, they must not only have the commonsense knowledge that the TypicFollowUp is generally typical, but also that it should happen according to social conventions or expectations of competence (see §E.3). So, it is interesting that these scenarios are the ones with higher rates of atypicality inferences.

Typicality inferences We find that humans make typicality inferences for AtypicFollowUps regardless of whether Utterance Emphasis is present; humans also make them for TypicFollowUps when Utterance Emphasis is not present.

For the majority of AtypicFollowUps, InfScore <0< 0 (Figure 4), showing typicality inferences for both the Plain and HeyGW settings (p<0.001<0.001). This not consistent with Kravtchenko and Demberg [26]’s findings, which showed no significant difference between likelihood ratings for unmentioned vs. mentioned Follow-up settings. However, our finding is consistent with the intuition that if a Follow-up event is initially unexpected (i.e., has a low likelihood rating in the NONE setting), but a listener learns that it happened once (i.e., mentioned in Plain/HeyGW settings), the listener may assume that it actually happens more often than they initially expected. In other words, they simply update their beliefs based on what happened without making a pragmatic inference.

Figure 4. Human results for TypicFollowUps vs. AtypicFollowUps, where a point’s size denotes the number of InfScores equal to its X value. InfScore > 0 = atypicality inference, and InfScore < 0 = typicality inference.

RQ3: Do LLMs make inferences about event typicality in the contexts where humans do?

Only large models with CoT-Prompting do this.

We focus our analyses on the HeyGW setting, as this is where humans made both typicality and atypicality inferences in the expected event contexts (see §E.5). Figure 6 shows that very few LLM configurations achieve a similar balance to humans between typicality and atypicality inferences. We confirm this with statistical tests §E.5: only 6 out of the 28 configurations produce statistically significant inferences (i.e., the mentioned status of the Follow-up in the dialogue, e.g., NONE vs. HeyGW, predicts the Follow-up’s likelihood rating) for both inference types, and 4 out of those are with CoT prompting on larger models. For most larger LLM configurations which showed atypicality inferences, we also see an increase in these inferences for Snarky scenarios (§E.5 and Figure 12).

Figure 6. Percentage of Follow-ups in the HeyGW setting where an LLM made a typicality (X-axis) vs. atypicality inference (Y-axis). Points in the gray halo are LLM configurations where the atypicality and typicality inferences were both statistically significant (p<0.01).

Figure 12. GPT-5.4 comparison of the distribution of responses to NONE vs. HeyGW settings for TypicFollowUps, with non-snarky dialogues on the left and snarky dialogues on the right.

Discussion

While some LLMs appear to follow the same CUES as humans in making event-typicality inferences, the story may be more complicated due to their often-inconsistent displays of commonsense knowledge. With LLMs, we find a significant (p<0.01) correlation between higher commonsense knowledge and more event-typicality inferences (see §E.6), which indicates that it is an essential component of these inferences. However, LLMs also fail to exhibit consistent event likelihood ratings, calling into question their commonsense knowledge. Figure 7 shows the percentage of Follow-ups where each LLM configuration rated the likelihood of the that Follow-up the same in both the Commonsense Knowledge Experiment (§ 5) and the Event-Typicality Inference Experiment (in the comparable NONE-setting; § 6). The LLMs’ consistency is very low compared to humans: no LLM configuration apart from GPT-5.4 with CoT scored higher than 80% consistency on both TypicFollowUps and AtypicFollowUps.

Figure 7. LLM consistency; i.e. the percentage of Follow-ups where responses from the Event-Typicality Inference experiment NONE setting matched those from the Commonsense Knowledge experiment, (X-axis for AtypicFollowUps and Y-Axis for TypicFollowUps).

Note that we would hope for much higher scores because in each case, the questions are about exact same Follow-up events; the only difference is the latter’s addition of dialogue and named entities. The fact that LLMs show such inconsistency when asked about the exact same events suggests that they are more sensitive to the framing of the prompt than to the actual typicality of an event (see §F). This is in line with prior work on both commonsense knowledge tasks and more general tasks, which has found that when the wording of the prompt is slightly changed, LLMs give inconsistent, and sometimes even opposite, answers to the same exact questions ([3, 2, 23, 30]).

Conclusion

In this paper, we investigated event-related typicality inferences on a much larger scale than any previous work has before. We created EventTyp and EventTypGold, recounting the corpora creation process in detail to enable reproducibility. We conducted a human study that gave us three CUES that predict inferences: Commonsense Knowledge, Utterance Emphasis, and Snark. We asked whether LLMs are also sensitive to these CUES and found that although they sometimes are, having the Commonsense Knowledge to predict an event’s general typicality is a much bigger hurdle than expected.

Our findings leave many open questions for future work. For example, other features aside from textual emphasis and snark, like verb aspect or prosodic cues, might predict these inferences in humans. And for LLMs, further exploration is necessary to answer why models fail to consistently exhibit commonsense knowledge and to find solutions to mitigate this.

Limitations

In our experiments, we tested only one form of utterance emphasis (the HeyGW setting) in eliciting pragmatic inferences. Similarly, we used a single, minimal form of dialogue context for our experiments. It remains to be seen whether other interventions, including more minimal ones tested in prior works (e.g., adding ‘!’; Kravtchenko and Demberg, [26]), and other forms of context, also trigger atypicality and typicality inferences at scale for humans or LLMs. Furthermore, because we use textual data, our experiments cannot account for the role that prosody may play in triggering pragmatic inferences.

The events tested in our experiments are constrained to those can that happen relatively often in an average (US) person’s life, as well as relatively quickly (i.e., the event is complete within a day of starting). It remains to be seen whether other types of events can trigger typicality-related inferences in the same patterns.

For our human experiments, we specifically tested people based in the US, due to the US-centric nature of the original data. As such, the underlying participant pool may not reflect the full diversity of English speakers. Additionally, our criteria for Context and Follow-up events were developed by English speakers. Event-typicality inferences may be triggered differently for speakers of languages other than English.

In our LLM experiments, we only tested five LLM families. Since we saw varied behavior across these models, we cannot predict how LLMs from other model families (e.g., Mistral (Jiang et al., [20]), DeepSeek (Guo et al., [12])) will behave. In particular, we only tested one closed source model (GPT-5.4). Testing with other such models could provide more insights.

Pragmatics Background

Cooperativity & Quantity

Pragmatics is the branch of Linguistics that targets questions about how speakers package information and how listeners draw inferences and recover intended meaning. In assessing why a speaker produces a particular utterance, listeners bring to bear various expectations about the ways in which the speaker’s contribution will be relevant and, crucially for our purposes, newsworthy.

These expectations reflect a set of principles that are posited to underlie cooperative communication [11]: A speaker’s contribution is expected to be (i) appropriate linked to the discourse topic and goals (via some Relation to prior information), (ii) sufficiently informative (as measured in the Quantity of new information), and also (iii) truthful (to uphold the Quality of the contribution). Given these expectations for cooperativity, a speaker who makes an apparent violation of any of these can invite subsequent inferences, i.e., how does the discourse contribution uphold a commitment to cooperativity despite the apparent violation of Quality/Relation/Quantity? For example, an apparent violation of Quality may be interpreted as sarcastic, or an apparent violation of relation may encourage the listener to seek additional pathways to relevance. When a speaker’s contribution appears to violate Quantity, the listener may potentially infer speaker ignorance (if the speaker has provided unexpectedly little information) or they may seek to establish why the contribution is newsworthy at all (if the speaker has been unexpectedly redundant).

Quantity-driven reasoning includes the treatment of scalar implicatures, whereby the use of an ambiguous scalar (e.g., ‘warm’) generates an implicature from its seeming underinformativity. A comprehender reasons that if the speaker is cooperative, their choice to not use a more informative alternative (‘hot’) implies the stronger meaning is not true (→ warm but not hot). That comprehenders seek to establish a cooperative interpretation is what drives a range of Quantity inferences about apparent redundancy [6, 38], underspecification [8], or triviality [37], including atypicality inferences.

Atypicality Inferences

With atypicality inferences, a speaker’s apparent uninformativity (the mention of some individual’s TypicFollowUp act), combines with the listener’s belief that the speaker is cooperative: The speaker is informative if the reported act is actually atypical for that individual.

For the example in Figure 1, a speaker asserts content that is typically inferable from shared knowledge of day-to-day events: When thanking one’s boss, it is typical for an individual to mean it. A speaker who states that information explicitly ("XX thanked their boss. [And hey, guess what,] she meant it [!]") appears to violate expectations about the appropriate Quantity of information that should be present in a cooperative contribution to a discourse. If the listener trusts that the speaker intends to be cooperative, they may look for ways in which the contribution is indeed informative. As such, the listener may ‘rescue’ the speaker’s seemingly redundant contribution by drawing an atypicality inference: For this individual, the mentioned event is newsworthy and the speaker intends to convey that the individual’s behavior is atypical for that individual (whose usual behavior is at odds with accepted norms for such situations).

Atypicality inferences are thus an example of non-trivial pragmatic reasoning. They require the listener to draw a conclusion (that the individual doesn’t typically do X) from a statement whose literal meaning provides evidence of the opposite (that the individual has done X). In this way, they depend on subtle reasoning that reflects broader expectations of speaker cooperativity alongside an awareness that what a speaker intends to convey may go beyond what has been explicitly said.

Typicality inferences

Unlike in atypicality inferences, typicality inferences do not appear to be pragmatic in nature. There is no apparent redundancy when a listener hears "Anahita thanked her boss. And hey, guess what, she was in tears [!]" Being in tears while thanking one’s boss is generally atypical and therefore aligns with listeners’ expectations to contribute the appropriate Quantity of information to the discourse. Upon learning this new information about Anahita, the listener may update their beliefs and think that for Anahita, crying is actually more typical than it is for others. As such, we do not consider typicality inferences to be pragmatics; but rather a straightforward belief update.

The EventTyp Corpus

Note that for all LLM generation and filtering steps, we use GPT-4o, specifically ‘gpt-4o-2024-08-06,’ with a temperature of 1. We selected this model specifically because it was the best-performing model that was within our budget.

Context Event Extraction

Source data. We source our context events from ATOMIC ([41, 16]), a commonsense reasoning dataset consisting of if-then event pairs. We limit events to those with up to two named Person entities (PersonX and PersonY).

We cannot directly use event relations from ATOMIC. We originally wanted to see if we could directly pull (Context, TypicFollowUp) and (Context, AtypicFollowUp) pairs from ATOMIC, but this does not work because:

  • After manually looking at some of the events, we determine that the proportion of TypicFollowUps that would meet our criteria (see § 3.3 and §B.3) is too low. For example, highly atypical events like “PersonX murders PersonY” are quite common, as well as directly entailed events like “PersonX takes PersonY’s advice” isAfter “PersonY gives advice to PersonX”. Although our LLM-generated followups still have errors, the filtering did not need to eliminate as much.

  • We wanted one TypicFollowUp and one AtypicFollowUp for every Context, but that does not follow the structure of ATOMIC, where one Context might appear any number of times, possibly only mapping to one follow-up type (TypicFollowUp or AtypicFollowUp) each of those times.

  • Aggregating all unique events from the isBefore/isAfter relation type, regardless of position, enabled us to start with the largest possible pool of Contexts from which to generate follow-ups.

So, regardless of whether an event was originally considered the ‘before’ or the ‘after’ event in a pair, we put all events into the same pool of Contexts which we later use to generate the TypicFollowUps and AtypicFollowUps.

Context Event Filtering

The ATOMIC dataset is noisy, so we use both rules-based and LLM-based methods to filter it. We replace all instances of PersonX with Mary and

PersonY with Linda, replacing all pronouns with she/her/hers. We perform LLM-filtering in two passes using GPT-4o: the easy pass and the hard pass. The easy pass consists of filtering out utterances with incomplete or otherwise malformed clauses, idioms, passive voice, implicit verbs, non-present tense, pronouns with no antecedents, qualitative statements, and statements explicitly about habituality. The hard pass attempts to remove items that are too vague, make absolutely no sense without context, or describe extreme situations like murder. The result is 12,978 Contexts.

Easy pass prompt

You are an expert annotator of natural
language. You will be presented with a
clause.
Your task is to determine if any of the
following is true:

- The clause is incomplete
- The clause is malformed
- The clause is abstract.
- The clause is idiomatic or not literal.
- The clause contains a passive.
- The clause contains an implicit verb.
- The clause does not take a simple present
  tense form.
- The clause contains a pronoun with no
  antecedent.
- The clause contains a qualitative or
  evaluative action.
- The clause contains a habitual or stative
  verb form, or its meaning can be
  interpreted as habitual/stative.

If any of the above are true, respond with
the tuple (True, [above item which most
clearly is true]). Otherwise, respond
with the tuple (False, None).

CLAUSE: Mary loses Linda arm
RESPONSE: (True, 'malformed')

CLAUSE: Mary loses her keys
RESPONSE: (False, None)

CLAUSE: Mary reinvents the wheel
RESPONSE: (True, 'idiomatic')

CLAUSE: Mary wants a car.
RESPONSE: (True, 'stative')

CLAUSE: Mary knows would die
RESPONSE: (True, 'malformed')

CLAUSE: Mary exposes oneself
RESPONSE: (True, 'pronoun')

CLAUSE: Mary sings to her
RESPONSE: (True, 'pronoun')

CLAUSE: Mary sings at the competition
RESPONSE: (False, None)

CLAUSE: Mary spends a day
RESPONSE: (True, 'incomplete')

CLAUSE: Mary buys some apples
RESPONSE: (False, None)

CLAUSE: Mary sings to me
RESPONSE: (True, 'pronoun')

CLAUSE: Mary exposes them
RESPONSE: (True, 'pronoun')

CLAUSE: Mary means nothing to Linda
RESPONSE: (True, 'abstract')

CLAUSE: Linda forgets her coat
RESPONSE: (True, 'implicit verb')

CLAUSE: Mary gets insulted by Linda
RESPONSE: (True, 'passive')

CLAUSE: Linda insults Mary
RESPONSE: (False, None)

CLAUSE: Mary insults him
RESPONSE: (True, 'pronoun')

CLAUSE: Mary is calm in stressful situations
RESPONSE: (True, 'habitual')

CLAUSE: Mary does well at the competition
RESPONSE: (True, 'evaluative')

CLAUSE: Mary competes against Linda
RESPONSE: (False, None)

CLAUSE: Mary gets on the bus
RESPONSE: (False, None)

CLAUSE: Mary remembers her coat
RESPONSE: (True, 'implicit verb')

CLAUSE: Mary wants to eat lunch
RESPONSE: (True, 'not simple present')

CLAUSE: Mary eats lunch
RESPONSE: (False, None)

CLAUSE: {}
RESPONSE:

caption: Listing 1: classify-easy

Hard pass prompt

You are an expert annotator of natural
language. You will be presented with a
clause.

Your task is to determine if any of the
following is true:

- The clause describes a vague action that
  would typically be seen with more
  context, e.g. with a modifier or with a
  more detailed predicate.- The clause
  appears to be grounded in a specific
  context that is not specified.
- The clause describes a situation which is
  extremely unusual in daily life. Perhaps
  the situation is common on fictional
  stories, but not real life.
- The clause describes an action which is
  abnormal or nonsensical without context.

If any of the above are true, respond with
the tuple (True, [above item which most
clearly is true]). Otherwise, respond
with the tuple (False, None).

CLAUSE: Mary loses her keys
RESPONSE: (False, None)

CLAUSE: Mary buys some apples
RESPONSE: (False, None)

CLAUSE: Mary leaves the apple
RESPONSE: (True, 'needs context')

CLAUSE: Linda kills Mary
RESPONSE: (True, 'unusual')

CLAUSE: Mary sings at the competition
RESPONSE: (False, None)

CLAUSE: Mary comes over
RESPONSE: (True, 'vague')

CLAUSE: Mary comes over to Linda's house
RESPONSE: (False, None)

CLAUSE: Mary smells the nail
RESPONSE: (False, 'needs context')

CLAUSE: Mary smells the flower
RESPONSE: (True, None)

CLAUSE: Mary tells Linda
RESPONSE: (True, 'vague')

CLAUSE: Mary steals Linda's child
RESPONSE: (True, 'unusual')

CLAUSE: Mary competes against Linda
RESPONSE: (False, None)

CLAUSE: Mary beats her swords into
ploughshares
RESPONSE: (True, 'abnormal')

CLAUSE: Linda hits the apple
RESPONSE: (True, 'abnormal')

CLAUSE: Mary eats lunch
RESPONSE: (False, None)
54 CLAUSE: Mary gets on the bus
55 RESPONSE: (False, None)
56
57 CLAUSE: Mary finds out her biological dad is
   not hers
58 RESPONSE: (True, 'unusual')
59
60 CLAUSE: {}
61 RESPONSE:

Listing 2: classify-hard

Typical Follow-ups

We prompt GPT-4o to generate up to three TypicFollowUps: a state of being, subaction, and next-action (see Table 4 for examples). This approach increases the diversity of our dataset compared to Kravtchenko and Demberg’s [25] corpus, which consists of mainly subaction or next-action tuples. Although some state-of-being utterances may not be considered ‘events,’ for simplicity, we use the term ‘event’ throughout. We allow the model to return None at its discretion (Listing 3).

TypicFollowUp CategoryExample followup to Mary thanks her boss
StateMary is at the office
SubactionMary means it
Next-actionMary’s boss says you’re welcome

Table 4. Example TypicFollowUp events for the three utterance categories.

1  You are an expert annotator of natural
   language. You will be presented with a
   clause, which is presumably about taking
   an action or being at a specific place
   or event.
2  Generate three potential next clauses: A
   state of being, a sub-action, and a
   subsequent action.
3
4  Criteria for all clauses:
5  - The clause must be about a natural
     assumption given the original clause
     which feels obvious and redundant.
6  - The clause must not be a direct entailment
     of the original clause
7  - The clause must not be a paraphrase of the
     original clause
8  - The clause can be about the same entity as
     that in the original, or about another
     entity associated with the original.
9  - The clause you generate should be brief.
     Do not add extra modifiers or details to
     it.
10
11 Criteria for the state of being clause:
12 - The state of being must not be a
     presupposition of the original clause.
13 - It must not be directly inferable from the
     original clause. In other words, there
     cannot be a logical entailment
     relationship.
14
15 Criteria for the sub-action clause:
16 - It must not be a direct entailment of the
     original clause.
17 - It must not be a paraphrase of the same
     action in the original clause.
18 - It might not be possible to generate a
     good sub-action clause. For example,
     when the original clause contains no
     action, or contains an action that is
     too short to contain a sub-action, it
     may not be possible.
19 Criteria for the subsequent action clause:
20 - It must be a very obvious assumption given
     the original clause, to the point that
     it seems redundant to explicitly mention
     in natural speech.
21 - It should be as general or vague as
     possible, with no details that would
     make it unique or important to mention
     in real life.
22 Your response format should be a three-item
   tuple of (state of being, sub-action,
   subsequent action). Each item in the
   tuple is optional. When there is no good
   clause for that category, respond with
   None in that index of the tuple. If
   there are no good clauses to generate at
   all, respond with (None, None, None).
CLAUSE: Mary goes to the restaurant
RESPONSE: ("Mary is hungry", None, "Mary
          eats")
CLAUSE: Mary goes to a picnic
RESPONSE: ("Mary wants to", "Mary brings a
          blanket", "Mary eats")
CLAUSE: Mary sings at the contest
RESPONSE: (None, "The crowd notices", "Mary
          receives applause")
CLAUSE: Linda buys candy at the store
RESPONSE: ("Linda has money on her", "Linda
          pays the cashier", "Linda does not throw
          the candy away")
CLAUSE: Linda attends a concert
RESPONSE: ("Linda knows the band", "Linda
          sings along", "Linda claps at the end")
CLAUSE: Mary attends the talk
RESPONSE: ("Mary knows who the speaker is",
          "Mary pays attention", "Mary does not
          regret it")
CLAUSE: Mary drives home after a long day of
         work
RESPONSE: (None, "Mary has both hands on the
          wheel", "Mary parks without trouble")
CLAUSE: Mary waves her hands in Linda's face
RESPONSE: ("Mary knows Linda", None, "Linda
          notices")
CLAUSE: Mary does the reading for next week'
         s math class
RESPONSE: ("Mary is enrolled in the class",
          "Mary absorbs some of it", "Mary attends
          the next class")
CLAUSE: {}
RESPONSE:

Listing 3: gen3clause

LLM-based Criteria Filtering Table 5 describes each prompt we use and links to our specific prompts in §H. Note that even if, during the prompting process, an LLM’s answer warrants rejection of a follow-up, we still prompt it with all 27 prompts. This is to a) allow us to go back and change individual prompts if we want to and b) remove pairs of answers when they contradict each other. Specifically, a logical LLM should never answer entailment (too likely, §H.2) prompts and Expectedness (not likely enough) (§H.3) prompts the same way. So, when this happens, we remov the LLMs’ answers to both of these sets of prompts.

Unnatural for dialogueWrong typicality
Prompt(s)DescriptionPrompt(s)Description
Malformed Context (2): Listing 22, Listing 23Has errors or is awkward/redundantAbstract (3): Listing 11, Listing 12, Listing 13Idiomatic, not concrete, or solely happens in one person’s head
Malformed event utterance (3): Listing 24, Listing 25, Listing 26Has errors or is incoherentEntailed/almost entailed (5): Direct (Listing 14; Death (Listing 15), Definition (Listing 16), Paraphrase (Listing 17), Plausibility (Listing 18)Event only false if the person dies; Event is definition of context; Event is paraphrase of context; Implausible that it didn’t happen
Not brief (Listing 27)Event utterance is too descriptiveNot sufficiently likely (3): Unexpected (Listing 19), Extra info (Listing 20), Non-default (Listing 21)Not RW-Typic; Adds info/detail; Interprets context strangely
Should be definite (Listing 28)‘the’ neededContext event not sufficiently frequent (Listing 30, Listing 31)
Narration (Listing 29)Context-Event pair reads like text, not dialogueTime-gap (Listing 32)Context and event utterances take place far apart in time
Passive (Listing 34)Context utterance in passive voiceTime-scope (Listing 33)Context and event utterances are not in the same time-scope
Not standalone (Listing 35)Context utterance needs more context to stand alone
Too vague (Listing 36)Underspecified, uninterpretable
No typicality (Listing 37)E.g. Event utterance is about an inherent trait

Table 5. 27 LLM-filtering prompts and their classes (Unnatural for dialogue vs. Wrong typicality)

Similarly, we had originally tried Explanation (Listing 38) and Result (Listing 39, Listing 40) prompts. However, we find that the LLM is especially bad at assessing causality, and answers the three prompts extremely inconsistently. Since almost every set of answers contradicts each other, the prompts have no effect in the end.

Generating typicality questions used for TypicFollowUp filtering For the prompt we used to filter out (Context, TypicFollowUp) pairs which had no typicality (Listing 37), we input a question about typicality to the LLM and ask it if the question makes sense (Listing 4). In order to do this, we first separately prompt the LLM to generate the typicality question in the form of: “When {Context}, how often {TypicFollowUp}?” We find that this question format forces a frequency interpretation unlike the “In general, when...how likely” format used in our later experiments, and this frequency interpretation makes the question sound infelicitous when the event has no typicality. So, that is why

we choose this question format.

You are an expert annotator of natural
    language data.
You will read STATEMENT_ONE and
    STATEMENT_TWO.
Your task is to produce a QUESTION in the
    format of "When [STATEMENT_ONE], how
    often [STATEMENT_TWO]?
When STATEMENT_ONE uses 'the' to modify its
    object, please switch to using 'a' in
    the first clause of your generated
    QUESTION.
But if STATEMENT_TWO uses 'the' to modify
    its object, please use 'the' in the
    second clause of the QUESTION.

STATEMENT_ONE: When Mary finds the stray dog
STATEMENT_TWO: Mary pets it
QUESTION: When Mary finds a stray dog, how
    often does she pet it?
STATEMENT_ONE: When Mary finds a stray dog
STATEMENT_TWO: Mary is an animal lover
QUESTION: When Mary finds a stray dog, how
    often is she an animal lover?
STATEMENT_ONE: Mary builds the house
STATEMENT_TWO: Mary takes a whole year
QUESTION: When Mary builds a house, how
    often does she take a whole year?
STATEMENT_ONE: Mary builds Linda's house
STATEMENT_TWO: Linda moves in
QUESTION: When Mary builds Linda's house,
    how often does Linda move in?
STATEMENT_ONE: Mary makes cookies
20 STATEMENT_TWO: Mary takes the cookies to the office
21 QUESTION: When Mary makes cookies, how often does she take the cookies to the office
22 STATEMENT_ONE: Mary makes friendship bracelets
23 STATEMENT_TWO: Mary uses more than one bead shape
24 QUESTION: When Mary makes friendship bracelets, how often does she use more than one bead shape?
25 STATEMENT_ONE: Mary makes friendship bracelets
26 STATEMENT_TWO: Mary enjoys crafting as a hobby
27 QUESTION: When Mary makes friendship bracelets, how often does she enjoy crafting as a hobby?
28 STATEMENT_ONE: Mary sees shoes for sale
29 STATEMENT_TWO: Mary buys them
30 QUESTION: When Mary sees shoes for sale, how often does she buy them?
31 STATEMENT_ONE: Mary sees shoes for sale
32 STATEMENT_TWO: Mary is interested in fashion
33 QUESTION: When Mary sees shoes for sale, how often is she interested in fashion?
34 STATEMENT_ONE: {}
35 STATEMENT_TWO: {}
36 QUESTION:

Listing 4: Prompt used to generate the “When...how often” typicality question for the LLM-filtering.

Atypical Follow-ups

Generation Below are the prompts (Listing 5 for states, Listing 6 for subactions, and Listing 7 for next-actions) used for AtypicFollowUp generation. Both the state and subaction prompts originally had errors where a few of the examples were not joined properly with the newline character, and this could have degraded LLM performance due to tokenization problems. However, since all generated AtypicFollowUps were manually filtered, this should not have caused any downstream problems.

1 You will be presented with two utterances, ORIGINAL and STATE. They appear in a sequence, where ORIGINAL is what a person does, and STATE describes a corresponding state of being.
2 Your task is to generate your version of a STATE utterance (STATE_YOURS).
3 STATE_YOURS should be temporary: that is, it cannot be something that usually lasts for weeks, months, or years.
4 Examples of some non-temporary states to avoid generating: Mary is a woman, Mary has a car, Mary has a disability.
5 The most notable exception to this rule is the state of knowing. For example, in ' Mary knows the band,' the concept of knowing a band can be applied to many other bands, so it can apply to multiple situations in a person's life.
6 In addition to being temporary in a person's life, make sure STATE_YOURS is more specific than STATE. It also must be as close to the original STATE length as possible. It is acceptable to make it just one word longer, one word shorter, or the same length.
7
8 ORIGINAL: Mary goes to the restaurant
9 STATE: Mary is hungry
10 STATE_YOURS: Mary is ravenous
11 ORIGINAL: Mary goes to a picnic
12 STATE: Mary wants to
13 NEXT_YOURS: Mary is starving
14 ORIGINAL: Mary sings at the contest
15 STATE: Mary sounds good
16 STATE_YOURS: Mary sounds magnificent
17 ORIGINAL: Linda buys candy at the store
18 STATE: Linda has money on her
19 STATE_YOURS: Linda has a craving
20 ORIGINAL: Linda attends a concert
21 STATE: Linda knows the band
22 STATE_YOURS: Linda knows every song
23 ORIGINAL: Mary attends the talk
24 STATE: Mary knows who the speaker is
25 STATE_YOURS: Mary knows the speaker's mentor
26 ORIGINAL: Mary drives home after a long day of work
27 STATE: Mary has her hands on the wheel
28 STATE_YOURS: Mary has plans to watch a movie
29 ORIGINAL: Mary does the reading for next week's math class
30 STATE: Mary is enrolled in the class
31 STATE_YOURS: Mary is bored with it
32 ORIGINAL: {}
33 STATE: {}
34 STATE_YOURS:

Listing 5: Prompt to generate the AtypicFollowUp state events.

1 You will be presented with two utterances, ORIGINAL and SUBACTION. They appear in a sequence, where ORIGINAL is what a person does, and SUBACTION describes a corresponding subaction.
2 Your task is to generate your version of a SUBACTION utterance (SUBACTION_YOURS). Make it more interesting than SUBACTION.
3 But try to keep it as close to the original SUBACTION length as possible. It is acceptable if you make it just one word longer, one word shorter, or the same length.
4
5 ORIGINAL: Mary goes to the restaurant
6 SUBACTION: Mary drives there
7 SUBACTION_YOURS: Mary runs late
8 ORIGINAL: Mary goes to a picnic
9 SUBACTION: Mary brings a blanket
10 SUBACTION_YOURS: Mary brings mayonnaise
11 ORIGINAL: Mary sings at the contest
12 SUBACTION: The crowd notices
13 SUBACTION_YOURS: The crowd goes wild
14 ORIGINAL: Linda buys candy at the store
15 SUBACTION: Linda pays the cashier
16 SUBACTION_YOURS: Linda uses coupons
17 ORIGINAL: Linda attends a concert
18 SUBACTION: Linda sings along
19 SUBACTION_YOURS: Linda screams along
20 ORIGINAL: Mary attends the talk
21 SUBACTION: Mary pays attention
22 SUBACTION_YOURS: Mary falls asleep
23 ORIGINAL: Mary drives home after a long day
   of work
24 SUBACTION: Mary keeps her hands on the wheel
25 SUBACTION_YOURS: Mary gets caught in traffic
26 ORIGINAL: Mary waves her hands in Linda's
   face
27 SUBACTION: Linda notices it
28 SUBACTION_YOURS: Linda ignores it
29 ORIGINAL: Mary does the reading for next
   week's math class
30 SUBACTION: Mary absorbs some of it
31 SUBACTION_YOURS: Mary absorbs nothing
32 ORIGINAL: {}
33 SUBACTION: {}
34 SUBACTION_YOURS:

Listing 6: Prompt to generate the AtypicFollowUp subaction events.

1   You will be presented with two utterances,
    ORIGINAL and NEXT. They appear in a
    sequence, where ORIGINAL is what a
    person does, and NEXT is what happens
    next.
2   Your task is to generate your version of a
    NEXT utterance (NEXT_YOURS). Make it
    more specific than NEXT.
3   But try to keep it as close to the original
    NEXT length as possible. It's best if
    you don't make it any longer, but it is
    acceptable if you make it just one word
    longer.
4
5   ORIGINAL: Mary goes to the restaurant
6   NEXT: Mary eats food
7   NEXT_YOURS: Mary eats pizza
8   ORIGINAL: Mary goes to a picnic
9   NEXT: Mary eats food
10  NEXT_YOURS: Mary eats two sandwiches
11  ORIGINAL: Mary sings at the contest
12  NEXT: The audience claps at the end
13  NEXT_YOURS: Mary gets third place
14  ORIGINAL: Linda buys candy at the store
15  NEXT: Linda pays the cashier
16  NEXT_YOURS: Linda gives some to Mary
17  ORIGINAL: Linda attends a concert
18  NEXT: Linda leaves when it ends
19  NEXT_YOURS: Linda leaves late
20  ORIGINAL: Mary attends the talk
21  NEXT: Mary does not regret it
22  NEXT_YOURS: Mary talks to the speaker
23  ORIGINAL: Mary drives home after a long day
    of work
24  NEXT: Mary parks without trouble
25  NEXT_YOURS: Mary heats up leftovers
26  ORIGINAL: Mary waves her hands in Linda's
    face
27  NEXT: Linda notices it
28  NEXT_YOURS: Linda is annoyed
29  ORIGINAL: Mary does the reading for next
    week's math class
30  NEXT: Mary attends the next math class
31  NEXT_YOURS: Mary barely understands it
32  ORIGINAL: {}
33  NEXT: {}
34  NEXT_YOURS:

Listing 7: Prompt to generate the AtypicFollowUp next-events.

Manual validation, correction, & removal We choose to manually validate our generated AtypicFollowUps in order to minimize error propagation (as the AtypicFollowUps were generated from already LLM-filtered Context-TypicFollowUp pairs). Since we have close to 4,000 generated events (as opposed to over 12,000 or TypicFollowUps), one of the paper authors had enough time to do the validation.

During the AtypicFollowUp validation, we find that some of the TypicFollowUps that had passed earlier filtering had errors. So, in those cases, we also correct the TypicFollowUps when possible.

But sometimes, it isn’t possible to correct the TypicFollowUp, because the Context it depends on was problematic (e.g. the filtering from §B.2 had not filtered it out). In those cases, we just remove the tuple entirely. We find some problematic cases during the later manual validation stage (see §D.1) and remove them retroactively. In our released data, we trace each generation and filtering stage and provide a list of which tuples got retroactively removed.

Data Sample

After the AtypicFollowUp validation and filtering, we are left with 4,113 tuples. Since we had originally generated up to 3 follow-ups per Context (state/subaction/next-action), there are some duplicate Contexts. We sample so that each Context is unique, oversampling states and subactions to try to alleviate imbalances in the original data, but the resulting sample still has many more next-actions than the other two categories. We also use GPT-4o to extract the main ‘action’ from each Context utterance and further deduplicate those, only keeping one Context per action. Nonetheless, this action extraction was not strict, so some Context in EventTyp are still about the same topic, e.g. “camp with friends” and “go camping” might both be kept. The result is 2,330 tuples, upon which we add the 24 from Kravtchenko and Demberg [25].

For our data collection to be complete, we needed responses from 540 participants who passed our attention checks. So, our policy was to recruit replacements for any participants who failed the attention checks. Out of the 676 possible participants, 634 attempted the main experiment. Of these attempts, 575 resulted in approval,9 39 resulted in a return, 19 resulted in rejection, and 1 timed out. Our pay averaged $21.50 over all rounds of the experiment.

Human and LLM Details

Human Participant Details

Below are more details about our human participants. All participants were US-based, had at least a 98% approval rating on Prolific, had at least 100 previous submissions, and stated that their first and primary language were both English. Our aim was to pay equivalent to $12.50/hr, but since we paid per task and not per hour, the estimation was not perfectly accurate, and we averaged out to paying more.

Table 6 shows our exact attention check questions for both the Commonsense Knowledge and the Event-Typicality Inference experiments. Before deciding on them, we informally piloted each one in order to ensure that any participant who was paying attention could answer them correctly. We format the questions to blend in with the rest of the items in their corresponding study. In each study, all participants see the exact same attention checks.

Accepted AnswerNorming versionMain version
1Likely/Very LikelyIn general, when someone puts on a t-shirt and skirt, how likely is it that they put clothes on?Bob and Sarah are having a conversation about their friend Sharon. Bob: “Sharon put on a t-shirt and skirt. She put clothes on.” In general, when Sharon puts on a t-shirt and skirt, how likely is it that she puts clothes on?
2Likely/Very LikelyIn general, when someone plays a game of chess, how likely is it that they play a game?Linda and Sabrina are having a conversation about their friend Grace. Linda: “Grace played a game of chess. She played a game.” In general, when Grace plays a game of chess, how likely is it that she plays a game?
3Likely/Very LikelyIn general, when someone enters a REM cycle, how likely is it that they are asleep?“Simon and Jared are having a conversation about their friend David. Simon: “David entered a REM cycle. He was asleep.” In general, when David enters a REM cycle, how likely is it that he is asleep?
4Unlikely/Very UnlikelyIn general, when someone goes to a grocery store, how likely is it that they give birth inside?George and Sally are having a conversation about their friend Jane. George: “Jane went to the grocery store. She gave birth inside.” In general, when Jane goes to a grocery store, how likely is it that she gives birth inside?
5Unlikely/Very UnlikelyIn general, when someone works a shift at the restaurant, how likely is it that they call an ambulance for a customer?Mildred and Lily are having a conversation about their friend Steve. Mildred: “Steve worked a shift at the restaurant. He called an ambulance for a customer.” In general, when Steve works a shift at the restaurant, how likely is it that he calls an ambulance for a customer?
6Unlikely/Very UnlikelyIn general, when someone sees a dog, how likely is it that it bites off their left hand?Jack and Stacy are having a conversation about their friend Joseph. Jack: “Joseph saw the dog. It bit off his left hand.” In general, when Joseph sees a dog, how likely is it that it bites off his left hand?

Table 6. Attention check questions. These qualify as ‘nonsensical items’ per Prolific’s attention check policy.

Commonsense Knowledge Experiment We recruited 354 participants in total, and we required 330 complete submissions (3×23 \times2 pilot batches of 24, 3×1083 \times108 larger batches of 44.) Each batch also included 6 attention checks (see §C.1), so we had 30 questions for the pilot batches and 50 for the rest.

340 were approved, 12 returned, 1 timed out, and 1 was rejected. Out of the 340 approved, 332 had usable submissions (i.e., passed the attention checks); however, due to a an error in our processing, we overwrote data from 2 participants and had to recruit 2 replacements. For this study, we had auto-approve enabled, and therefore were only able to reject one participant who failed attention checks but was flagged by Prolific as having completed the study too quickly. This means 9 people total failed the attention checks, but 8 were approved due to their responses being completed in a normal amount of time and therefore not being flagged by the Prolific system.

Event-Typicality Inference Experiment For this experiment, we only recruited participants who had passed a pragmatics pre-test (see §E.4 for details). Due to batching, each participant answered between 30 and 32 questions, consisting of about 24 questions from EventTypGold and 6 attention checks. All participants see the same attention checks. We randomized and divide the data so that each question is answered by 5 different participants and no participant sees any Context utterance more than once.

LLM Details

Table 7 describes which LLMs we choose to use, as well as their sizes and whether we used the base versions, instruct versions, or both. We do not change the model parameters. Since our inference is performed on so little data (N=4,708N = 4{,}708) for the Commonsenses Experiment and (N=2,700N = 2{,}700) for the Event-Typicality Inference experiment, it takes about 5 minutes for us to run Prompting inference on NVIDIA A100 GPUs and about 30 minutes for CoT-prompting. For Suprisal, it usually took between 5-30 minutes, as we did not optimize with vLLM and directly used the transformers library.

LLMBaseInstructSmallBig
GPT-5.4 ([34])x✓nano-2026-03-172026-03-05
Llama-3.1 ([10])✓✓8b70b
Olmo-3 ([32])✓✓7b32b
Qwen3 ([47])✓✓8b30b
gemma-3 ([9])✓✓12b27b

Table 7. LLMs used. For those where prompting was available, we tried prompting both with and without CoT (see §C.2.2. For details on model version selection, see §C.2.1. Since OpenAI has not made GPT-5.4’s parameter number publicly available, we just put the timestamp model IDs.

LLM version selection

Our goal is to keep each LLM configuration as consistent as possible across open-weight LLMs. This means our goal is to select models where both a base and instruction-tuned version exist, and for the instruction-tuned version, we wanted to avoid using Thinking so we could use the CoT prompting approach across models. Therefore, we made two decisions for Olmo-3 and Qwen3 specifically.

  • Olmo-3 ([32]): Olmo-3 has an Instruct model for its smaller version, but its larger model (32B) uses Thinking which cannot be switched off. So, for the larger model with prompting, we used Olmo-3.1-Instruct, whose base version is the same as for Olmo-3.

  • Qwen3 ([47]): For our instruction-tuned versions, we chose Qwen3-30B-A3B and Qwen3-8B, but we disabled thinking for both our CoT and non-CoT prompting configurations. This is because we wanted a fair comparison with other models with our own CoT prompt (see §C.2.2 and Listing 9 for an example).

Detailed Setup of Prompting

To create our prompts, we append the text “Choose one of the following options: ‘Very Unlikely’, ‘Unlikely’, ‘Likely’, ‘Very Likely’ to the end of each question in EventTypGold. We find this to be necessary in order to prevent unwanted bias toward ‘Likely’ that is introduced by the prompt (“...how likely is it ...”).

We sample 20 answers per prompt with temperature 1.0 for all model families except GPT-5.4, for which we sample 10. We reverse the response-option order for half of the samples to minimize order bias.

We consider with or without CoT for the prompting experiment. Here, we provide details for each setting, along with an example prompt and the LLM’s response.

  • Without CoT. For open-source models, we force the LLM to directly output the answer by constraining the output space to the four answer options. For GPT-5.4, we set ‘reasoning_effort’ to none to disable reasoning.10

  • With CoT. For open-source models, we force the LLM to generate reasoning about its response first and then give its final answer. We achieve this by using “structured output” generation from vLLM ([28]). For GPT-5.4, we set ‘reasoning_effort’ to medium.

[Prompt] Chedeline and Kathem are having a conversation about their friend Anahita.
Chedeline: "Anahita thanked her boss.
And hey, guess what, she was in tears!"
In general, when Anahita thanks her boss
, how likely is it that she is in tears
?\nChoose one of the following options:
Very Unlikely, Unlikely, Likely, Very
Likely.

[Response] "Unlikely"

Listing 8: An example prompt used for the prompting experiment without CoT and an example response (from Llama-3.1-70B).

[Prompt] Chedeline and Kathem are having a conversation about their friend Anahita.
Chedeline: "Anahita thanked her boss.
And hey, guess what, she meant it!" In
general, when Anahita thanks her boss,
how likely is it that she means it?
\nAnswer the question with the following
schema: {"type": "object", "properties":
{"reasoning": {"type": "string",
"description": "The step-by-step
reasoning process for the answer."},
"answer": {"type": "string", "enum": [
"Very Unlikely", "Unlikely", "Likely",
"Very Likely"]}}, "required": ["reasoning
", "answer"]}

[Response] "{\"reasoning\": \"The statement
is that Anahita thanked her boss and
actually meant it. The tone of the
conversation suggests surprise, implying
that it is not typical for Anahita to
genuinely thank her boss.\", \"answer\":
\"Very Unlikely\"}"

Listing 9: An example prompt used for the prompting experiment with CoT and an example response (from Llama-3.1-70B).

Commonsense Knowledge Experiment

Experimental Setup

Figure 8 shows an example of a likelihood rating question in the user interface format our human participants saw. We use the prompt in Listing 10 to generate the likelihood questions. Note that the actual raw output is a question in the format of:

Example of what human participants saw in our Commonsense Knowledge experiment

Figure 8. Example of what human participants saw in our Commonsense Knowledge experiment.

When {Context}, how likely {Follow-up}?

We then use rules-based methods to pre-pend ‘In general,’ to the question and lowercase the ‘When.’ For each resulting question, each human answer validates it and corrects an errors (e.g., pronoun ambiguity).

1  You are an expert annotator of natural
   language data.
2  You will read STATEMENT_ONE and
   STATEMENT_TWO.
3  Your task is to produce a QUESTION in the
   format of "When [STATEMENT_ONE], how
   likely is it that [STATEMENT_TWO]?
4  When STATEMENT_ONE uses 'the' to modify its
   object, please switch to using 'a' in
   the first clause of your generated
   QUESTION.
5  But if STATEMENT_TWO uses 'the' to modify
   its object, please use 'the' in the
   second clause of the QUESTION.
6  If only one person exists in the statements,
   use the word 'someone' instead of their
   name.
7  If two people exist, use the phrase 'one
   person' instead of the first name and '
   second person' for the second name.
8
9  STATEMENT_ONE: Mary finds the stray dog
10 STATEMENT_TWO: Mary pets it
11 QUESTION: When someone finds a stray dog,
   how likely is it that they pet it?
12 STATEMENT_ONE: Mary finds a stray dog
13 STATEMENT_TWO: Mary is happy
14 QUESTION: When someone finds a stray dog,
   how likely is it that they are happy?
15 STATEMENT_ONE: Mary builds the house
16 STATEMENT_TWO: Mary takes a whole year
17 QUESTION: When someone builds a house, how
   likely is it that they take a whole year
   ?
18 STATEMENT_ONE: Mary builds Linda's house
19 STATEMENT_TWO: Linda moves in
20 QUESTION: When one person builds a second
   person's house, how likely is it that
   the second person moves in?
21 STATEMENT_ONE: Mary makes cookies
22 STATEMENT_TWO: Mary takes the cookies to the
   office
23 QUESTION: When someone makes cookies, how
   likely is it that they take the cookies
   to the office?
24 STATEMENT_ONE: Mary makes friendship
   bracelets
25 STATEMENT_TWO: Mary uses more than one bead
   shape
26 QUESTION: When someone makes friendship
   bracelets, how likely is it that they
   use more than one bead shape?
27 STATEMENT_ONE: Mary cooks pasta for Linda
28 STATEMENT_TWO: Linda eats it
29 QUESTION: When one person cooks pasta for a
   second person, how likely is it that the
   second person eats it?
30 STATEMENT_ONE: {}
31 STATEMENT_TWO: {}
32 QUESTION:

Listing 10: Prompt used to generate the typicality questions used in the norming study.

Event-Typicality Inference Experiment

Figure 9 shows an example of a likelihood rating question from EventTypGold in the format that our human participants saw.

Example of what human participants saw during our Event-Typicality Inference experiment

Figure 9. Example of what human participants saw during our Event-Typicality Inference experiment. Each participant answered between 30 and 32 questions.

Data Sample

To create our 450-tuple set of experimental materials, we sampled from EventTypGold. To help narrow down the data, we decided to limit our experiments to context events that happen relatively quickly (take a full day or less to complete) and frequently in a person’s life (at least once a year). So, via manual labeling, we set aside tuples that did not meet these criteria. However, we still consider these tuples to be high quality and hope that followup experiments can include them all.

Dialogue Scenarios: Name selection

For both the dialogue names and the Mary/Linda name replacements, we choose to use names from non-English speaking countries that are rare in the United States. This is to fit with both our human and LLM experiments: we want to minimize the chance the LLMs have seen the exact sentences appear in their training data. We source names from a rare names list,11 dropping names from English-speaking countries and removing those which were in the top 10,000 names from a US census dataset.12 We balance names by gender and sample from 198 of them. Note that we use the name ‘Anahita’ in Figure 1, but the actual name EventTyp is ‘Anahito.’ Both names are valid spellings of a popular Tajik name; however, ‘Anohito’ can easily be mixed up with a Japanese pronoun. So, to prevent confusion, we use ‘Anahita.’

More explanation of the snarky label

When a speaker wants to communicate something negative about another person, they may communicate via inference so their on-record utterance is positive (“Hey, guess what, she meant it”) instead of saying something negative directly (“She usually doesn’t mean it when she says thank you.”). As an example of a non-snarky dialogue, consider “Anahita thanked her boss. And hey, guess what, she was in her boss’s office!”. The atypicality inference (=Anahita usually does not thank her boss while in her boss’s office) does not imply anything negative about Anahita – maybe her and her boss usually meet outside of that office, but this does not make either of them look bad in any way. Hence, we would label this utterance as not snarky.

Relationship with commonsense knowledge & irony In the context of our research questions, we can define snark as a specific way atypicality inferences can interact with commonsense knowledge to produce irony.

Commonsense knowledge: In the snark case, the TypicFollowUp event is not only generally typical, but is also an event that should happen according to some social convention or standard of competence.

Irony: Verbal irony can be defined as a speaker expressing something by using language that would usually convey the opposite [13], where the speaker expects the listener to recognize the “critical or mocking attitude” behind their words [46]. Sperber et al. [42] have connected this “critical or mocking attitude” to utterances which violate listeners’ expectations of informativity, e.g. in our case with a speaker mentioning a (Context, TypicFollowUp) event pair. In this paper, we call this “critical or mocking attitude” Snark. After making an atypicality inference, a listener would only interpret the speaker as being snarky if the speaker’s utterance was about someone fulfilling a social convention or meeting a minimal standard. e.g., the atypicality inference from “Anahita thanked her boss. And hey, guess what, she meant it!” conveys that Anahita is usually the opposite of thankful. In other words, if the atypicality inference can be interpreted as ironic, we label it as snarky.

Pragmatics Pre-test for Human Participants

Justification and Source data We choose to focus on participants who we know are capable of making pragmatic inferences. Therefore, before recruiting participants for this experiment, we conduct an 18-question pragmatics pre-test using data from PragMega, a human-written multiple-choice dataset which includes questions regarding multiple pragmatic phenomena [14, 7].

Floyd et al. [7] conclude that pragmatic reasoning can be broken into the understanding of 1) social conventions (SocCon), 2) intonation, and 3) world-knowledge based causal reasoning. Since the task of understanding atypicality inferences from text-only data requires both 1) and 3), we choose questions from those two categories: 12 SocCon (half implicatures, half implicit requests) and 6 commonsense-knowledge based coherence questions (half coherent, half incoherent). Half of our SocCon questions have literal answers as the correct answer, and the other half nonliteral. We release our pre-test questions to the public.13

Usage and survey setup Figure 10 and Figure 11 contain examples for an implicit request and coherence questions respectively. We set up the pre-test on Prolific and conduct three separate rounds, where the pay is $18.00 an hour averaged over the rounds. We give a passing score if a participant gets at least 5 out of 6 answers correct for each group of 6 questions (implicatures, requests, coherence) and finishes the test in 7 minutes or less. We do NOT inform participants that this is a pre-test, and we pay all participants regardless of their score. Out of 1,255 total participants, 1,196 are approved, 53 return, and 6 time out. Among the approved participants, 676 pass (56.52%) and are subsequently invited to the main experiment.

Example of an implicit request question shown in the pragmatics pre-test

Figure 10. Example of an implicit request question participants saw in the pragmatics pre-test. The expected answer here is d).

Example of a coherence question shown in the pragmatics pre-test

Figure 11. Example of a coherence question participants saw in the pragmatics pre-test. The expected answer here is ‘Incoherent.’

Ordinal Regressions

We use mixed-effects ordinal regressions to model the rating category (very unlikely to very likely) with fixed effects for Follow-up type (TypicFollowUp/AtypicFollowUp) and setting (NONE/Plain/HeyGW) and their interaction, along with random effects for Context utterance, and, in the human data case, participant. The mixed-effects ordinal regressions were built in R, using clmm from the library ordinal. For all regressions, fixed effects were centered and maximal random effect structure was included, as permitted by the data.

We predict opposite effects of Follow-up setting for TypicFollowUp (where an atypicality inference should yield lower ratings for Plain or HeyGW compared to NONE) and AtypicFollowUp (where a typicality inference should yield lower ratings for NONE). We test the Snark effect in a further Follow-up ×\times setting ×\times snark model, where the atypicality inference pattern is predicted to be stronger for snarky scenarios.

On human responses We collect 13,500 rating responses. Table 8 reports the regression output for the main model (Follow-up ×\times setting), and TaTable 9 reports the output for the larger model with snark (Follow-up×setting×snarkFollow\text{-}up \times setting \times snark). The table uses abbreviations for T(ypicality) and S(nark). All interactions in both tables are significant, inviting the critical subset comparisons listed in Table 10.

β\betazzpp
Follow-up2.8739.85<0.001
Plain.NONE1.2026.15<0.001
NONE.HeyGW-0.27-5.28<0.001
Follow-up:Plain.NONE-1.05-11.47<0.001
Follow-up:NONE.HeyGW1.9518.51<0.001

Table 8. Regression output for the main mixed effects model fit on human data (Follow-up ×\times setting).

SubsetSettingβzp
TypicFollowUpPlain.NONE0.6610.06<0.001
TypicFollowUpNONE.HeyGW0.648.77<0.001
AtypicFollowUpPlain.NONE1.8626.50<0.001
AtypicFollowUpNONE.HeyGW-1.34-17.98<0.001
TypicFollowUp-SnarkPlain.NONE0.657.06<0.001
TypicFollowUp-SnarkNONE.HeyGW0.817.17<0.001
TypicFollowUp-NoSnarkPlain.NONE0.657.08<0.001
TypicFollowUp-NoSnarkNONE.HeyGW0.545.64<0.001
AtypicFollowUp-SnarkPlain.NONE1.8717.01<0.001
AtypicFollowUp-SnarkNONE.HeyGW-1.24-11.13<0.001
AtypicFollowUp-NoSnarkPlain.NONE1.8520.51<0.001
AtypicFollowUp-NoSnarkNONE.HeyGW-1.43-14.15<0.001

Table 10. Regression output for the subset mixed effects models on human data.

βzp
Follow-up2.7740.57<0.001
Plain.NONE1.1826.50<0.001
NONE.HeyGW-0.24-4.72<0.001
Snark-0.22-3.46<0.001
Follow-up:Plain.NONE-1.02-11.88<0.001
Follow-up:NONE.HGW1.9522.69<0.001
Follow-up:Snark-0.87-6.83<0.001
Plain.NONE:Snark-0.005-0.06<0.001
NONE.HeyGW:Snark0.252.53<0.001
Follow-up:Plain.NONE:Snark-0.04-0.24<0.001
Follow-up:NONE.HeyGW:Snark0.150.92<0.001

Table 9. Regression output for the snark mixed effects model fit on human data, Follow-up×setting×snarkFollow\text{-}up \times setting \times snark.

Table 10 shows the subsequent analyses that test for a main effect of setting in each of the relevant subsets. Regular font indicates a typicality inference, and bold font indicates an atypicality inference.

On LLM responses In the same way as for the human data but without the participant ID random effect, we fit mixed effects models for each of the sets of LLM results. Like for humans, we fit models on the full LLM data. Table 11 summarizes the results. Only CoT-Prompting model configurations consistently follow human-like patterns, 4 out of the 5 making atypicality inferences, typicality inferences, and exhibiting the snark effect.

LLMSurprisalPromptingCoT-Prompting
AtypInfTypInfSnarkEffectAtypInfTypInfSnarkEffectAtypInfTypInfSnarkEffect
GPT-5.4–––x✓x✓✓✓
Llama-3.1-70B✓x✓✓✓✓✓✓✓
Olmo3-32B✓xxx✓x✓✓x
Qwen3-30Bx✓xx✓xx✓x
gemma-3-27bxxxx✓x✓✓✓
GPT-5.4-nano–––x✓xx✓x
Llama-3.1-8B✓xxx✓x✓✓✓
Olmo3-7Bx✓xx✓xx✓x
Qwen3-8B✓x✓x✓xx✓x
gemma-3-12bxxxx✓xx✓x

Table 11. Summary of mixed effects models fit on LLM output data. Atypicality and typicality inferences abbreviated to AtypInf and TypInf. Note that the snark effect is only present if atypicality inferences are made.

Additional Figures

As an example, Figure 12 shows the Snark effect for GPT-5.4 (with CoT), but full results are in Table 11.

Figure 13 shows a significant correlation (R2=0.235R^2 = 0.235, p<0.01p < 0.01) between higher commonsense knowledge and more event-typicality inferences, which is in line with our belief that event-typicality inferences rely on commonsense knowledge. To get the correlation values, we fit an ordinary least-squares linear model in R with the commonsense knowledge agreement as a predictor of InfScore (InfScore∼κ\sim\kappa).

Correlation between Commonsense Knowledge human-LLM agreement and the percentage of Event-Typicality Inference Experiment HeyGW setting tuples

Figure 13. Correlation (p<0.01p<0.01) between Commonsense Knowledge (CKnowledge) human-LLM agreement (κCombined\kappa_{\mathrm{Combined}}, §G) and the percentage of Event-Typicality Inference Experiment HeyGW setting tuples where the LLM made both an atypicality inference (on the TypicFollowUp) and a typicality inference (on the AtypicFollowUp).

Additional Discussion

Here, we further discuss LLMs’ oversensitivity to prompt wording instead of actual event typicality. Because we found that LLMs and humans do not share the same Commonsense Knowledge, we wondered whether flipping the labels of each event in EventTypGold to match the LLMs’ voted response typicality (flipping from TypicFollowUp to AtypicFollowUp and vice versa) would increase their number of “correct” inferences (i.e. typicality inferences for AtypicFollowUp events and atypicality inferences for TypicFollowUp events). However, Figure 14 and Figure 15 show that this does not work: the scores remain about the same whether we use the human knowledge labels or the LLM ones.

Figure 14. Percentage of the time a typicality inference was made on events voted as AtypicFollowUp by LLM (X-axis) vs. humans (Y-Axis)

Figure 15. Percentage of the time an atypicality inference was made on events voted as TypicFollowUp by LLM (X-axis) vs. humans (Y-Axis)

This is not surprising given the lack of consistency illustrated in Figure 7. If the LLMs are responding to each question based on how it’s asked (with dialogue context vs. without) and not based on any underlying knowledge about the events, then changing the event labels would not make a difference.

Metric definitions

We denote the dataset as D\mathcal{D} with ∣D∣=2,354|\mathcal{D}| = 2{,}354 for EventTyp and ∣D∣=450|\mathcal{D}| = 450 for EventTypGold. Let F∈{TypF,AtypF}\mathcal{F} \in\{\text{TypF}, \text{AtypF}\} be the Follow-up (abbreviated from TypicFollowUp and AtypicFollowUp in order to save space). We collect KK responses for each datapoint ii and Follow-up F\mathcal{F}. We use RiF[k]R_{i}^{\mathcal{F}}[k] to indicate the kk-th response (k=1,…,Kk = 1, \ldots, K) to the question from ii-th datapoint (i=1,…,∣D∣i = 1, \ldots, |\mathcal{D}|).

Each response RiF[k]R_{i}^{\mathcal{F}}[k] takes one of the integer values from {1,2,3,4}\{1, 2, 3, 4\}, which map to the labels ‘Very Unlikely’, ‘Unlikely’, ‘Likely’, and ‘Very Likely’, respectively. For example, RiAtypF[k]=3R_{i}^{\text{AtypF}}[k] = 3 means that the kk-th response to the ii-th datapoint with a Follow-up of AtypF is ‘Likely’. We use two different ways of aggregating responses, which are detailed below.

Average-response μiF∈[1,4]\mu_i^{\mathcal{F}} \in[1,4] is average of KK response values, i.e.,

μiF=1K∑k=1KRiF[k].\mu_i^F = \frac{1}{K}\sum_{k=1}^{K} R_i^F[k].

Voted-response viF∈{−1,1}v_i^F \in\{-1,1\} is result of majority-voting from KK responses, specifically,

viF={1if vote⁡iF>121if (vote⁡iF=12) and (μiF≥2.5)−1otherwisev_i^F = \begin{cases} 1 & \text{if } \operatorname{vote}_i^F > \frac{1}{2}\\ 1 & \text{if } \left(\operatorname{vote}_i^F = \frac{1}{2}\right) \text{ and } \left(\mu_i^F \ge2.5\right)\\ -1 & \text{otherwise} \end{cases}

where

vote⁡iF=1K∑k=1KI(RiF[k]∈{3,4}).\operatorname{vote}_i^F = \frac{1}{K}\sum_{k=1}^{K}\mathbb{I}\left(R_i^F[k] \in\{3,4\}\right).

In words, viFv_i^F becomes 1 (‘Likely’) if the majority of responses are positive (3 or 4), and −1 (‘Unlikely’) if the majority are negative (1 or 2). When a 50:50 tie occurs (i.e., vote⁡iF=12\operatorname{vote}_i^F = \frac{1}{2}), we break the tie using the average response. Let {viTypF}i=1∣D∣\{v_i^{\mathrm{TypF}}\}_{i=1}^{|\mathcal{D}|} be the set of all votes for TypicFollowUps and {viAtypF}i=1∣D∣\{v_i^{\mathrm{AtypF}}\}_{i=1}^{|\mathcal{D}|} be the set of all votes for AtypicFollowUps.

Agreement κ\kappa: We can now define the following three agreement scores, where κ\kappa is Cohen’s Kappa [5]. Note we abbreviate Combined to Comb here for space.

κTypF=κ({viTypF}i=1∣D∣),\kappa^{\mathrm{TypF}} = \kappa\left(\left\{v_i^{\mathrm{TypF}}\right\}_{i=1}^{|\mathcal{D}|}\right),
κAtypF=κ({viAtypF}i=1∣D∣),\kappa^{\mathrm{AtypF}} = \kappa\left(\left\{v_i^{\mathrm{AtypF}}\right\}_{i=1}^{|\mathcal{D}|}\right),
κComb=κ({1[viTypF=1∧viAtypF=−1]}i=1∣D∣).\kappa^{\mathrm{Comb}} = \kappa\left(\left\{\mathbb{1}\left[v_i^{\mathrm{TypF}}=1 \land v_i^{\mathrm{AtypF}}=-1\right]\right\}_{i=1}^{|\mathcal{D}|}\right).

Now, let S∈{‘NONE’, ‘Plain’, ‘HeyGW’}S \in\{\text{‘NONE’, ‘Plain’, ‘HeyGW’}\} be the Follow-up setting from our Event-Typicality Inference experiment (see Table 3). All the definitions from above stay the same, but we additionally use RiF,S[k]R_i^{F,S}[k] (e.g. RiAtypF,Plain[k]=3R_i^{\mathrm{AtypF,Plain}}[k] = 3 means the kk-th response to the ii-th datapoint with (AtypF, Plain)), along with μiF,S\mu_i^{F,S} and viF,Sv_i^{F,S}.

Consistency score CF∈[0,1]C^F \in[0,1], which is defined for each FF, is the proportion of datapoints from D\mathcal{D} where the voted-response in the Commonsense Knowledge experiment viFv_i^F matches the voted-response with NONE setting in the Event-Typicality Inference experiment viF,NONEv_i^{F,\mathrm{NONE}}. i.e.,

CTypF=1∣D∣∑i=1∣D∣I(viTypF=viTypF,NONE),C^{\mathrm{TypF}} = \frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\mathbb{I}\left(v_i^{\mathrm{TypF}}=v_i^{\mathrm{TypF,NONE}}\right),
CAtypF=1∣D∣∑i=1∣D∣I(viAtypF=viAtypF,NONE).C^{\mathrm{AtypF}} = \frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\mathbb{I}\left(v_i^{\mathrm{AtypF}}=v_i^{\mathrm{AtypF,NONE}}\right).

Inference score (InfScore) IiF,SMentioned∈[−3,3]I_i^{F,S_{\mathrm{Mentioned}}} \in[-3,3], which is defined for each datapoint ii with some FF, measures the change in average-response when a MENTIONED setting (Plain or HeyGW) is introduced, compared to the baseline setting without followup (NONE), i.e.,

IiF,SMentioned=μiF,NONE−μiF,SMentioned,I_i^{F,S_{\mathrm{Mentioned}}} = \mu_i^{F,\mathrm{NONE}}-\mu_i^{F,S_{\mathrm{Mentioned}}},

where SMentioned∈{‘Plain’, ‘HeyGW’}S_{\mathrm{Mentioned}} \in\{\text{‘Plain’, ‘HeyGW’}\}. A positive InfScore indicates an atypicality inference, a negative InfScore indicates a typicality inference, and zero indicates that no inference occurred.

All LLM-filtering prompts

Abstract

1 You are an expert annotator of natural language data. 2 You will read a QUESTION, a STATEMENT, and some EXAMPLES. 3 To the best of your knowledge, and according to the examples, answer the QUESTION with yes or no. 4 5 QUESTION: Is this statement about an abstract action? Definition of an abstract action: an action that may encompass a series of undefined, highly contextual, and potentially abstract steps. 6 STATEMENT: Mary loses hope 7 ANSWER: Yes 8 STATEMENT: Mary loses the bet 9 ANSWER: No 10 STATEMENT: Mary asserts her independence 11 ANSWER: Yes 12 STATEMENT: Mary gets Linda a job 13 ANSWER: Yes 14 STATEMENT: Mary gets Linda a glass of water 15 ANSWER: No 16 STATEMENT: Mary drives Linda crazy 17 ANSWER: Yes 18 STATEMENT: Mary drives home 19 ANSWER: No 20 STATEMENT: Mary applies herself 21 ANSWER: Yes 22 STATEMENT: Mary applies for a job 23 ANSWER: No 24 STATEMENT: {Context} 25 ANSWER:

Listing 11: Abstract-Direct: Applies to only Context utterances.

1 You are an expert annotator of natural language data. 2 You will read a QUESTION, a STATEMENT, and some EXAMPLES. 3 To the best of your knowledge, and according to the examples, answer the QUESTION with yes or no. 4

5 QUESTION: Does this statement contain any
  idioms?
6 STATEMENT: Mary reinvents the wheel
7 ANSWER: Yes
8 STATEMENT: Mary eats an apple
9 ANSWER: No
10 STATEMENT: {Context}
11 ANSWER:

Listing 12: Abstract-Idiom: Applies to only Context utterances.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, a STATEMENT, and
  some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
4
5 QUESTION: Does what is expressed in this
  statement happen solely in someone's
  head? As in, it's their thought or their
  feeling?
6 STATEMENT: Mary feels a sharp sting
7 ANSWER: Yes
8 STATEMENT: Mary realizes Linda's error
9 ANSWER: Yes
10 STATEMENT: Mary considers buying them
11 ANSWER: Yes
12 STATEMENT: Mary jumps
13 ANSWER: No
14 STATEMENT: {Context}
15 ANSWER:

Listing 13: Abstract-Imagine: Applies to only Context utterances.

Entailment or near-entailment

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
4
5 QUESTION: Does statement A entail statement
  B?
6 STATEMENT_A: Mary makes a reservation
7 STATEMENT_B: Mary provides a time
8 ANSWER: Yes
9 STATEMENT_A: Mary makes a reservation
10 STATEMENT_B: Mary plans to go
11 ANSWER: No
12 STATEMENT_A: Mary invites Linda to come to
  her party
13 STATEMENT_B: Mary is having a party
14 ANSWER: Yes
15 STATEMENT_A: Mary invites Linda to come to
  her party
16 STATEMENT_B: Linda accepts the invitation
17 ANSWER: No
18 STATEMENT_A: {Context}
19 STATEMENT_B: {Event}
20 ANSWER:

Listing 14: Entail-Direct: Applies to all Context-Event pairs.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
4
5 QUESTION: Is death or the apocalypse the
  only way you can imagine statement B
  being false? - Falling asleep \rightarrow
  Waking up, - Going somewhere \rightarrow
  Leaving, - Doing an action
  that always has some kind of end \rightarrow
  It ending
6 STATEMENT_A: Mary visits Linda's father
7 STATEMENT_B: Mary leaves
8 ANSWER: Yes
9 STATEMENT_A: Mary visits Linda's father
10 STATEMENT_B: Linda's father recognizes her
11 ANSWER: No
12 STATEMENT_A: Mary takes a hike
13 STATEMENT_B: Mary returns home
14 ANSWER: Yes
15 STATEMENT_A: Mary takes a hike
16 STATEMENT_B: Mary follows the trail
17 ANSWER: No
18 STATEMENT_A: Mary goes to sleep
19 STATEMENT_B: Mary wakes up
20 ANSWER: Yes
21 STATEMENT_A: Mary covers her head
22 STATEMENT_B: Mary uncovers her head later
23 ANSWER: Yes
24 STATEMENT_A: {Context}
25 STATEMENT_B: {Event}
26 ANSWER:

Listing 15: Entail-Death: Applies to all Context-Event pairs.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
4
5 QUESTION: Can the main point of statement B
  be seen as a part of the definition of
  an action in statement A? Note that if,
  according to its definition, statement A
  can happen without statement B
  occurring, your answer should be No.
6 STATEMENT_A: Mary plays chess
7 STATEMENT_B: Mary makes a move
8 ANSWER: Yes
9 STATEMENT_A: Mary plays chess
10 STATEMENT_B: Mary wins
11 ANSWER: No
12 STATEMENT_A: Mary buys groceries

13 STATEMENT_B: Mary selects food items 14 ANSWER: Yes 15 STATEMENT_A: Mary buys groceries 16 STATEMENT_B: Mary selects apples 17 ANSWER: No 18 STATEMENT_A: Mary follows Linda's steps 19 STATEMENT_B: Mary reaches the same destination as Linda 20 ANSWER: Yes 21 STATEMENT_A: {Context} 22 STATEMENT_B: {Event} 23 ANSWER:

Listing 16: Entail-Definition: Applies to all Context-Event pairs.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
5 QUESTION: Can statement B be seen as a
  paraphrase of statement A?
6 STATEMENT_A: Mary gets a ticket
7 STATEMENT_B: Mary takes the ticket
8 ANSWER: Yes
9 STATEMENT_A: Mary gets a ticket
10 STATEMENT_B: Mary goes to the event
11 ANSWER: No
12 STATEMENT_A: {Context}
13 STATEMENT_B: {Event}
14 ANSWER:

Listing 17: Entail-Para: Applies to all Context-Event pairs.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
5 QUESTION: Is it extremely implausible for
  statement B to be false given statement
  A? In other words, does it take mental
  gymnastics to imagine statement B being
  false?
6 STATEMENT_A: Mary invites Linda to join
7 STATEMENT_B: Mary has plans
8 ANSWER: Yes
9 STATEMENT_A: Mary invites Linda to join
10 STATEMENT_B: Linda accepts the invitation
11 ANSWER: No
12 STATEMENT_A: Mary makes chocolate chip
  cookies
13 STATEMENT_B: Mary mixes the ingredients
14 ANSWER: Yes
15 STATEMENT_A: Mary makes chocolate chip
  cookies
16 STATEMENT_B: Mary doubles the recipe
17 ANSWER: No
18 STATEMENT_A: Mary accuses Linda of cheating

19 STATEMENT_B: Linda responds 20 ANSWER: Yes 21 STATEMENT_A: Mary accuses Linda of cheating 22 STATEMENT_B: Linda denies it 23 ANSWER: No 24 STATEMENT_A: Mary watches the film 25 STATEMENT_B: The credits roll 26 ANSWER: Yes 27 STATEMENT_A: {Context} 28 STATEMENT_B: {Event} 29 ANSWER:

Listing 18: Entail-Plausible: Applies to all Context-Event pairs.

Not sufficiently likely

Note that the prompt may look strange for Listing 19. This is the result of iterative prompt engineering to get the best result. Previously, more intuitive prompts resulted in the LLM rejecting almost everything or almost nothing.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
5 QUESTION: Given statement A, how surprised
  are you that statement B is true? On a
  scale of 0 to 10, where 10 is the most
  surprised, if your surprisal is 4 or
  less, answer Yes. Otherwise, answer No.
6 STATEMENT_A: Mary gives Linda a small smile
7 STATEMENT_B: Linda acknowledges it
8 ANSWER: Yes
9 STATEMENT_A: Mary gives Linda a small smile
10 STATEMENT_B: Mary is in a joyful mood
11 ANSWER: No
12 STATEMENT_A: Mary picks berries
13 STATEMENT_B: Mary takes them home
14 ANSWER: Yes
15 STATEMENT_A: Mary picks berries
16 STATEMENT_B: Mary is in a forest
17 ANSWER: No
18 STATEMENT_A: Mary decides to try out for the
  team
19 STATEMENT_B: Mary plans to attend practice
20 ANSWER: Yes
21 STATEMENT_A: Mary decides to try out for the
  team
22 STATEMENT_B: Mary fills out an application
23 ANSWER: No
24 STATEMENT_A: Mary gets Linda a job
25 STATEMENT_B: Mary thanks Linda
26 ANSWER: Yes
27 STATEMENT_A: Mary gets Linda a job
28 STATEMENT_B: Mary loves Linda
29 ANSWER: No
30 STATEMENT_A: Mary catches the dog
31 STATEMENT_B: Mary makes sure it gets home
32 ANSWER: Yes
33 STATEMENT_A: Mary catches the dog
34 STATEMENT_B: Mary clings to the dog

35 ANSWER: No 36 STATEMENT_A: {Context} 37 STATEMENT_B: {Event} 38 ANSWER:

Listing 19: Expect-Direct: Applies to all Context-Event pairs.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.

5 QUESTION: Does statement B add unexpected
  information that is not contained in
  statement A? The most common way of
  adding unexpected information is the
  addition of extra entities or characters
  into the situation. In the prize
  example, you can see 'judges' being
  added, where they did not exist and were
  not automatically inferable from
  statement A. In the drink example, some
  vague person who needs an apology has
  been added.
6 STATEMENT_A: Mary wins the prize
7 STATEMENT_B: Mary thanks the judges
8 ANSWER: Yes
9 STATEMENT_A: Mary wins the prize
10 STATEMENT_B: Mary is happy
11 ANSWER: No
12 STATEMENT_A: Mary spills a drink
13 STATEMENT_B: Mary apologizes
14 ANSWER: Yes
15 STATEMENT_A: Mary spills a drink
16 STATEMENT_B: Mary cleans it up
17 ANSWER: No
18 STATEMENT_A: {Context}
19 STATEMENT_B: {Event}
20 ANSWER:

Listing 20: Expect-Extra: Applies to all Context-Event pairs.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.

5 QUESTION: Does statement B interpret
  statement A in a non-default way? The
  most common way to take a photo is to
  use a smartphone, in which case storing
  the photo and adjusting the focus should
  happen automatically. So, statement B
  seems to be interpreting statement A as
  being about a real camera, which is non-
  default.
6 STATEMENT_A: Mary takes a photo
7 STATEMENT_B: Mary stores the photo
8 ANSWER: Yes
9 STATEMENT_A: Mary takes a photo
10 STATEMENT_B: Mary adjusts the focus
11 ANSWER: Yes
12 STATEMENT_A: Mary takes a photo
13 STATEMENT_B: Mary uses flash
14 ANSWER: No
15 STATEMENT_A: {Context}
16 STATEMENT_B: {Event}
17 ANSWER:

Listing 21: Expect-Nondefault: Applies to all Context-Event pairs.

Malformed context

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, a STATEMENT, and
  some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.

5 QUESTION: Is this statement ungrammatical,
  misspelled, malformed, or otherwise
  nonsensical?
6 STATEMENT: Mary loses Linda arm
7 ANSWER: Yes
8 STATEMENT: Mary loses Linda's keys
9 ANSWER: No
10 STATEMENT: The mall shops back
11 ANSWER: Yes
12 STATEMENT: Mary buys many items
13 ANSWER: No
14 STATEMENT: Mary goes hopping at the mall
15 ANSWER: Yes
16 STATEMENT: Mary goes shopping at the mall
17 ANSWER: No
18 STATEMENT: {Context}
19 ANSWER:

Listing 22: Malformed-A-Base: Applies to all Context utterances.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, a STATEMENT, and
  some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.

5 QUESTION: Is this statement very awkward to
  say because it gives unnecessary
  information? For example, shoes always
  goes on feet, so it is unnecessary to
  mention feet if someone is putting shoes
  on. Similarly, lunch is always food, so
  it is unnecessary to mention that
  someone made food for lunch. People grab
  things with their hands, so it is
  unnecessary to mention the use of hands
  for grabbing something.
6 STATEMENT: Mary puts shoes on her feet
7 ANSWER: Yes
8 STATEMENT: Mary puts shoes on
9  ANSWER: No
10 STATEMENT: Mary makes food for lunch
11 ANSWER: Yes
12 STATEMENT: Mary makes lunch
13 ANSWER: No
14 STATEMENT: Mary grabs the cup with her hand
15 ANSWER: Yes
16 STATEMENT: Mary grabs the cup
17 ANSWER: No
18 STATEMENT: {Context}
19 ANSWER:

Listing 23: Malformed-A-Redundant: Applies to all Context utterances

Malformed event utterance

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Is anything in either statement
   ungrammatical, misspelled, malformed, or
   otherwise nonsensical?
6  STATEMENT_A: Mary loses Linda keys
7  STATEMENT_B: Linda is upset
8  ANSWER: Yes
9  STATEMENT_A: Mary loses Linda's keys
10 STATEMENT_B: Linda is upset
11 ANSWER: No
12 STATEMENT_A: Mary loses Linda's arm
13 STATEMENT_B: Linda is upset
14 ANSWER: Yes
15 STATEMENT_A: Mary goes hopping at the mall
16 STATEMENT_B: Mary buys many ittems
17 ANSWER: Yes
18 STATEMENT_A: Mary goes shopping at the mall
19 STATEMENT_B: Mary buys many items
20 ANSWER: No
21 STATEMENT_A: {Context}
22 STATEMENT_B: {Event}
23 ANSWER:

Listing 24: Malformed-AB-Base: Applies to all Context-Event pairs

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Are statements A and B both in
   simple present tense? Note that no other
   tenses are included, and present
   progressive is NOT simple present tense.
6  STATEMENT_A: Mary goes shopping at the mall
7  STATEMENT_B: Mary was excited
8  ANSWER: No
9  STATEMENT_A: Mary goes shopping at the mall
10 STATEMENT_B: Mary is excited
11 ANSWER: Yes
12 STATEMENT_A: Mary goes shopping at the mall
13 STATEMENT_B: Mary buys many items
14 ANSWER: Yes
15 STATEMENT_A: Mary went shopping at the mall
16 STATEMENT_B: Mary buys many items
17 ANSWER: No
18 STATEMENT_A: Mary went shopping at the mall
19 STATEMENT_B: Mary is buying many items
20 ANSWER: No
21 STATEMENT_A: {Context}
22 STATEMENT_B: {Event}
23 ANSWER:

Listing 25: Malformed-AB-Tense: Applies to all Context-Event pairs

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Put together, are statements A and
   B coherent? In other words, if
   statements A and B make sense and are
   related to each other, answer Yes.
   Otherwise, answer No.
6  STATEMENT_A: Mary goes shopping at the mall
7  STATEMENT_B: Mary buys many items
8  ANSWER: Yes
9  STATEMENT_A: Mary goes shopping at the mall
10 STATEMENT_B: The mall shops back
11 ANSWER: No
12 STATEMENT_A: Mary goes shopping at the mall
13 STATEMENT_B: The elf smiles back
14 ANSWER: No
15 STATEMENT_A: {Context}
16 STATEMENT_B: {Event}
17 ANSWER:

Listing 26: Malformed-AB-Coherent: Applies to all Context-Event pairs

Utterance not brief

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Could statement B be edited down
   to be more brief, stripped of
   descriptive modifiers, while still
   retaining its meaning? If minor
   modifiers or details can be deleted,
   answer Yes. Otherwise, answer No. Please
   be very careful. If you think the
   entirety of statement B can be deleted,
   answer No.
6 STATEMENT_A: Mary purchases the bike
7 STATEMENT_B: Mary inspects it briefly
8 ANSWER: Yes
9 STATEMENT_A: Mary purchases the bike
10 STATEMENT_B: Mary inspects it
11 ANSWER: No
12 STATEMENT_A: Mary watches television
13 STATEMENT_B: Mary turns it off later
14 ANSWER: Yes
15 STATEMENT_A: Mary watches television
16 STATEMENT_B: Mary turns it off
17 ANSWER: No
18 STATEMENT_A: Mary takes a shopping cart to
the store
19 STATEMENT_B: Mary fills the cart with
products
20 ANSWER: Yes
21 STATEMENT_A: Mary takes a shopping cart to
the store
22 STATEMENT_B: Mary fills the cart with cheese
23 ANSWER: No
24 STATEMENT_A: {Context}
25 STATEMENT_B: {Event}
26 ANSWER: STATEMENT_A: Context
27 STATEMENT_B: Event
28 ANSWER:

Listing 27: Brevity: Applies to all Context-Event pairs

Strange use of definites

1 You are an expert annotator of natural
language data.
2 You will read a QUESTION, STATEMENT_A,
STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
to the examples, answer the QUESTION
with yes or no.
5 QUESTION: If statement B has an object with
'a', 'an', or a possessive at preceding
it, can it be replaced with 'the' or one
of 'her/his/their/its' and still make
sense? If statement B already has an
object modified by 'the', answer No. If
statement B no article or possessive, or
no object, answer No Only answer yes if
statement B has 'a', 'an', or a
possessive modifying the object, AND it
can be replaced with the while still
making sense.
6 STATEMENT_A: Mary buys new sheets
7 STATEMENT_B: Mary selects a color
8 ANSWER: Yes
9 STATEMENT_A: Mary buys new sheets
10 STATEMENT_B: Mary selects the color
11 ANSWER: No
12 STATEMENT_A: Mary goes bowling with Linda
13 STATEMENT_B: Mary picks a bowling ball
14 ANSWER: Yes
15 STATEMENT_A: Mary goes bowling with Linda
16 STATEMENT_B: Mary picks her bowling ball
17 ANSWER: No
18 STATEMENT_A: Mary makes friendship bracelets
19 STATEMENT_B: Mary ties knots
20 ANSWER: Yes
21 STATEMENT_A: Mary makes friendship bracelets
22 STATEMENT_B: Mary ties the knots
23 ANSWER: No
24 STATEMENT_A: {Context}
25 STATEMENT_B: {Event}
26 ANSWER:

Listing 28: Definite: Applies to only subaction Events.

Narration

1 You are an expert annotator of natural
language data.
2 You will read a QUESTION, STATEMENT_A,
STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
to the examples, answer the QUESTION
with yes or no.
5 QUESTION: Paired together, do statements A
and B seem like they would be found in a
novel as opposed to a text message or
an online forum? Clues to look for: -
Verbs that are loaded, colorful, or
literary: 'tries' to clean up a spill is
loaded because it implies some failure
to complete the task. 'Rushes' instead
of 'goes' or 'clutches' instead of 'uses'
are more colorful/literary than their
alternatives. - Descriptions of minute
actions: e.g. 'Mary turns her head' -
Information, like an internal monologue,
that a regular friend would not be
privy too, but that an author of a story
might write.
6 STATEMENT_A: Mary buys new sheets
7 STATEMENT_B: Mary thinks about her bed
8 ANSWER: Yes
9 STATEMENT_A: Mary buys new sheets
10 STATEMENT_B: Mary washes them
11 ANSWER: No
12 STATEMENT_A: Mary spills a drink
13 STATEMENT_B: Mary tries to clean it up
14 ANSWER: Yes
15 STATEMENT_A: Mary spills a drink
16 STATEMENT_B: Mary cleans it up
17 ANSWER: No
18 STATEMENT_A: Mary hears a thud
19 STATEMENT_B: Mary turns her head
20 ANSWER: Yes
21 STATEMENT_A: Mary hears a thud
22 STATEMENT_B: Mary gets up
23 ANSWER: No
24 STATEMENT_A: Mary cuts Linda's hair short
25 STATEMENT_B: Linda looks in the mirror
26 ANSWER: Yes
27 STATEMENT_A: Mary cuts Linda's hair short
28 STATEMENT_B: There is a mirror
29 ANSWER: No
30 STATEMENT_A: Mary roasts marshmallows
31 STATEMENT_B: Mary clutches a stick
32 ANSWER: Yes
33 STATEMENT_A: Mary roasts marshmallows
34 STATEMENT_B: Mary uses a stick
35 ANSWER: No
36 STATEMENT_A: {Context}
37 STATEMENT_B: {Event}
38 ANSWER:

Listing 29: Narration: Applies to all Context-Event pairs

Context event not sufficiently frequent

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, a STATEMENT, and
   some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Does this statement describe a
   situation or activity which can be
   considered abnormal or hyperspecific?
   For example, fishing as a general
   activity is normal, but adding certain
   details to the statement can make it
   either abnormal or hyperspecific. Murder
   or other kinds of non-petty violence do
   happen in the world daily, but these
   are not considered a part of normal life
   .
6  STATEMENT: Mary goes fishing
7  ANSWER: No
8  STATEMENT: Mary goes fishing for treasure
9  ANSWER: Yes
10 STATEMENT: Mary brings four rods to the lake
   to fish
11 ANSWER: Yes
12 STATEMENT: Mary makes dinner for Linda
13 ANSWER: No
14 STATEMENT: Mary gives Linda a present
15 ANSWER: No
16 STATEMENT: Mary gives Linda poison
17 ANSWER: Yes
18 STATEMENT: Mary meets with her dad
19 ANSWER: No
20 STATEMENT: Mary meets with her dad's fortune
   teller
21 ANSWER: Yes
22 STATEMENT: Mary finds out her dad is not her
   biological dad
23 ANSWER: Yes
24 STATEMENT: {Context}
25 ANSWER:

Listing 30: Life-Daily-Life: Applies to all Context utterances.

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, a STATEMENT, and
   some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Is this statement about something
   that happens as a milestone in a person's
   lifetime? Note that if it can be a
   milestone, but it can also be a regular
   occurrence for certain people's jobs or
   hobbies, your answer should be no. An
example of this is winning a competition
: this can happen regularly for athletes
and other types of people. But
something like graduating high school,
by definition, can only happen once in a
lifetime. And giving birth can happen a
few times, but is always a significant
event.
6  STATEMENT: Mary attends her class
7  ANSWER: No
8  STATEMENT: Mary graduates high school
9  ANSWER: Yes
10 STATEMENT: Mary gives Linda a present
11 ANSWER: No
12 STATEMENT: Mary gives birth
13 ANSWER: Yes
14 STATEMENT: {Context}
15 ANSWER:

Listing 31: Life-Lifetime: Applies to all Context utterances.

Time-gap

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Is there a very wide time-gap or
   conceptual gap between statements A and
   B?
6  STATEMENT_A: Mary waters Linda's plants
7  STATEMENT_B: Linda's plants stay healthy
8  ANSWER: Yes
9  STATEMENT_A: Mary waters Linda's plants
10 STATEMENT_B: Linda thanks her
11 ANSWER: No
12 STATEMENT_A: Mary finds a wife
13 STATEMENT_B: Mary plans a wedding
14 ANSWER: Yes
15 STATEMENT_A: Mary gets engaged
16 STATEMENT_B: Mary plans a wedding
17 ANSWER: No
18 STATEMENT_A: Mary walks to the library
19 STATEMENT_B: Mary reads a book
20 ANSWER: Yes
21 STATEMENT_A: Mary walks to the library
22 STATEMENT_B: Mary checks out a book
23 ANSWER: No
24 STATEMENT_A: {Context}
25 STATEMENT_B: {Event}
   ANSWER:

Listing 32: Gap: Applies to all next-action Events.

Time-scope

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Can statement A be considered much
   wider in scope than statement B?
6  STATEMENT_A: Mary takes many photos
7  STATEMENT_B: Mary adjusts the lens
8  ANSWER: Yes
9  STATEMENT_A: Mary takes a photo
10 STATEMENT_B: Mary adjusts the lens
11 ANSWER: No
12 STATEMENT_A: Mary plays chess
13 STATEMENT_B: Mary plans her next move
14 ANSWER: Yes
15 STATEMENT_A: Mary moves her chess piece
16 STATEMENT_B: Mary plans her next move
17 ANSWER: No
18 STATEMENT_A: Mary sails along the coast
19 STATEMENT_B: Mary adjusts the sail
20 ANSWER: Yes
21 STATEMENT_A: Mary sails along the coast
22 STATEMENT_B: Mary documents it
23 ANSWER: No
24 STATEMENT_A: Mary spends the day at the
   beach
25 STATEMENT_B: Mary sets up a towel
26 ANSWER: Yes
27 STATEMENT_A: Mary spends the day at the
   beach
28 STATEMENT_B: Mary swims
29 ANSWER: No
30 STATEMENT_A: {Context}
31 STATEMENT_B: {Event}
32 ANSWER:

Listing 33: Scope: Applies to just subaction Events.

Passive

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, a STATEMENT, and
   some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Does this statement contain
   passive voice?
6  STATEMENT: Mary gets told off by Linda
7  ANSWER: Yes
8  STATEMENT: Linda tells Mary off
9  ANSWER: No
10 STATEMENT: {Context}
11 ANSWER:

Listing 34: Passive: Applies to all Context utterances.

Not standalone

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, a STATEMENT, and
   some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Can this statement stand alone
   without extra context? Answer Yes if the
   statement makes no references to a
   previous context and it makes sense
   without extra context.
6  STATEMENT: Mary leaves the shop
7  ANSWER: Yes
8  STATEMENT: Mary goes back to the shop
9  ANSWER: No
10 STATEMENT: Mary crashes into a car
11 ANSWER: Yes
12 STATEMENT: Mary crashes into another car
13 ANSWER: No
14 STATEMENT: Mary gives Linda money
15 ANSWER: Yes
16 STATEMENT: Mary gets Linda's money back
17 ANSWER: No
18 STATEMENT: Mary gets Linda's money
19 ANSWER: Yes
20 STATEMENT: Mary loses her keys
21 ANSWER: Yes
22 STATEMENT: Mary finds the keys at home
23 ANSWER: No
24 STATEMENT: {Context}
25 ANSWER:

Listing 35: Standalone: Applies to all Context utterances.

Too vague

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Put together, are statements A and
   B extremely underspecified? Note that
   if the statements are general, but not
   extremely underspecified, your answer
   should be No. Your answer should only be
   Yes if the generalness exceeds a
   threshold where a core part of the
   meaning of the text seems to be missing.
6  STATEMENT_A: Mary sees the look on Linda's
   face
7  STATEMENT_B: Mary reacts
8  ANSWER: Yes
9  STATEMENT_A: Mary sees the angry look on
   Linda's face
10 STATEMENT_B: Mary reacts
11 ANSWER: No
12 STATEMENT_A: Mary sees the look on Linda's
   face
13 STATEMENT_B: Mary calls it out
14 ANSWER: No
15 STATEMENT_A: Mary starts a club
16 STATEMENT_B: Mary is interested in a topic
17 ANSWER: Yes
18 STATEMENT_A: Mary starts a chess club
19 STATEMENT_B: Mary is interested in chess

20 ANSWER: No 21 STATEMENT_A: {Context} 22 STATEMENT_B: {Event} 23 ANSWER:

Listing 36: Vague: Applies to all Context-Event pairs

No typicality

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, a TEXT, and some
   EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.

4
5  QUESTION: Does the following question "When
   {}, how often is {}?" make sense
   grammatically and meaning-wise? For
   example, if statement B is about an
   inherent trait of someone, the question
   does not make sense and your answer
   should be No. Generally, answer No if it
   does not make sense to think about a
   situational relationship between
   statements A and B. Otherwise, answer
   Yes.
6  TEXT: When Mary finds a stray dog, how often
   does she pet it?
7  ANSWER: Yes
8  TEXT: When Mary finds a stray dog, how often
   is she an animal lover?
9  ANSWER: No
10 TEXT: When Mary builds a house, how often
   does she take a whole year?
11 ANSWER: Yes
12 TEXT: When Mary builds Linda's house, how
   often does Linda move in?
13 ANSWER: No
14 TEXT: When Mary makes friendship bracelets,
   how often does she use more than one
   bead shape?
15 ANSWER: Yes
16 TEXT: When Mary makes friendship bracelets,
   how often does she enjoy crafting as a
   hobby?
17 ANSWER: No
18 TEXT: When Mary sees shoes for sale, how
   often does she buy them?
19 ANSWER: Yes
20 TEXT: When Mary sees shoes for sale, how
   often is she interested in fashion?
21 ANSWER: No
22 TEXT: When {}, how often is {}?
23 ANSWER:

Listing 37: No-Typicality: Applies to all Context-Event pairs

Prompts that had no effect in the end

1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.

3 To the best of your knowledge, and according to the examples, answer the QUESTION with yes or no.

4 5 QUESTION: Is statement B the answer for why statement A happened? In other words, can you easily infer the word BECAUSE between statements A and B? "Context BECAUSE Event" Please consider your answer in terms of coherence relations. In this case, is BECAUSE more likely than AS A RESULT? If AS A RESULT is more likely than or equally likely to BECAUSE, answer No. If statement B is just background information about the scenario, answer No. 6 STATEMENT_A: Mary takes a shopping cart to the store 7 STATEMENT_B: Mary plans to buy multiple items 8 ANSWER: Yes 9 STATEMENT_A: Mary takes a shopping cart to the store 10 STATEMENT_B: Mary has a list 11 ANSWER: No 12 STATEMENT_A: Mary goes back to sleep 13 STATEMENT_B: Mary is tired 14 ANSWER: Yes 15 STATEMENT_A: Mary goes back to sleep 16 STATEMENT_B: Mary is on a bed 17 ANSWER: No 18 STATEMENT_A: Mary covers her head 19 STATEMENT_B: Mary feels cold 20 ANSWER: Yes 21 STATEMENT_A: Mary covers her head 22 STATEMENT_B: Mary is outside 23 ANSWER: No 24 STATEMENT_A: {Context} 25 STATEMENT_B: {Event} 26 ANSWER:


Listing 38: Explain: Applies to only state Events.

```text
1  You are an expert annotator of natural
   language data.
2  You will read a QUESTION, STATEMENT_A,
   STATEMENT_B, and some EXAMPLES.
3  To the best of your knowledge, and according
   to the examples, answer the QUESTION
   with yes or no.
4
5  QUESTION: Is statement B the direct result
   of statement A happening? In other words
   , is the text meaning equivalent to
   "Event, AND AS A RESULT Context"?
6  STATEMENT_A: Mary smells smoke
7  STATEMENT_B: Mary is concerned
8  ANSWER: Yes
9  STATEMENT_A: Mary smells smoke
10 STATEMENT_B: Mary has something in the oven
11 ANSWER: No
12 STATEMENT_A: Mary wins the prize
13 STATEMENT_B: Mary is overjoyed
14 ANSWER: Yes
15 STATEMENT_A: Mary wins the prize
16 STATEMENT_B: Mary is far ahead of everyone
   else
17 ANSWER: No
18 STATEMENT_A: {Context}
19 STATEMENT_B: {Event}
20 ANSWER:

Listing 39: Result-AB: Applies to only state Events.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, STATEMENT_A,
  STATEMENT_B, and some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
4
5 QUESTION: Is statement A the direct result
  of statement B happening? In other words
  , is the text meaning equivalent to "
  Context, AND AS A RESULT Event"?
6 STATEMENT_A: Mary is concerned
7 STATEMENT_B: Mary smells smoke
8 ANSWER: Yes
9 STATEMENT_A: Mary has something in the oven
10 STATEMENT_B: Mary smells smoke
11 ANSWER: No
12 STATEMENT_A: Mary is overjoyed
13 STATEMENT_B: Mary wins the prize
14 ANSWER: Yes
15 STATEMENT_A: Mary is far ahead of everyone
   else
16 STATEMENT_B: Mary wins the prize
17 ANSWER: No
18 STATEMENT_A: {Context}
19 STATEMENT_B: {Event}
20 ANSWER:

Listing 40: Result-BA: Applies to only state Events.

1 You are an expert annotator of natural
  language data.
2 You will read a QUESTION, a STATEMENT, and
  some EXAMPLES.
3 To the best of your knowledge, and according
  to the examples, answer the QUESTION
  with yes or no.
4
5 QUESTION: Is this a statement explicitly
  about somebody's habits?
6 STATEMENT: Mary plays football weekly
7 ANSWER: Yes
8 STATEMENT: Mary plays in the football game
9 ANSWER: No
10 STATEMENT: {Context}
11 ANSWER:

Listing 41: Habitual: Applies to all Context utterances.

References

  1. [1]Leon Bergen, Roger Levy, and Noah Goodman. 2016. Pragmatic reasoning through semantic inference. Semantics and Pragmatics, 9:20–1.DOI
  2. [2]Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, Ben He, Shanshan Jiang, and Bin Dong. 2024. ChatGPT is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3098–3110, Torino, Italia. ELRA and ICCL.arxiv.org/abs/2303.16421
  3. [3]Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023. Say what you mean! large language models speak too positively about negative commonsense knowledge. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9890–9908, Toronto, Canada. Association for Computational Linguistics.arxiv.org/abs/2305.05976
  4. [4]Gustavo Cilleruelo, Emily Allaway, Barry Haddow, and Alexandra Birch. 2025. Generics are puzzling. can language models find the missing piece? In Proceedings of the 31st International Conference on Computational Linguistics, pages 6571–6588, Abu Dhabi, UAE. Association for Computational Linguistics.
  5. [5]Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.DOI
  6. [6]Judith Degen, Robert D Hawkins, Caroline Graf, Elisa Kreiss, and Noah D Goodman. 2020. When redundancy is useful: A bayesian approach to “overinformative” referring expressions. Psychological review, 127(4):591.DOI
  7. [7]Sammy Floyd, Olessia Jouravlev, Moshe Poliak, Zachary Minerอฟ, Edward Gibson, and Evelina Fedorenko. 2025. Three distinct components of pragmatic language use: Social conventions, intonation, and world knowledge–based causal reasoning. Proceedings of the National Academy of Sciences, 122(50):e2424400122.
  8. [8]Michael Franke. 2011. Quantity implicatures, exhaustive interpretation, and rational conversation. Semantics and Pragmatics, 4(1):1–82.DOI
  9. [9]Gemma Team. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.
  10. [10]Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.arxiv.org/abs/2407.21783
  11. [11]HP Grice. 1975. Logic and conversation. Syntax and semantics, 3.DOI
  12. [12]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-rl incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081).
  13. [13]Henk Haverkate. 1990. A speech act analysis of irony. Journal of pragmatics, 14(1):77–109.DOI
  14. [14]Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4194–4213, Toronto, Canada. Association for Computational Linguistics.arxiv.org/abs/2212.06801
  15. [15]Jennifer Hu, Felix Sosa, and Tomer Ullman. 2025. Shades of zero: Distinguishing impossibility from inconceivability. Journal of Memory and Language, 143:104640.
  16. [16]Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (comet-) atomic 2020: On symbolic and neural commonsense knowledge graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6384–6392.arxiv.org/abs/2010.05953
  17. [17]Duygu Sezen Islakoglu and Jan-Christoph Kalo. 2025. ChronoSense: Exploring temporal understanding in large language models with time intervals of events. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 590–602, Vienna, Austria. Association for Computational Linguistics.
  18. [18]Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sriparna Saha, Adam Jatowt, and Sandipan Dandapat. 2023. Do language models have a common sense regarding time? revisiting temporal commonsense reasoning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6750–6774, Singapore. Association for Computational Linguistics.DOI
  19. [19]Shengwei Ji, Longfei Liu, Jizhong Xi, Xiaoxue Zhang, and Xinlu Li. 2024. Klr-kgc: Knowledge-guided llm reasoning for knowledge graph completion. Electronics, 13(24):5037.DOI
  20. [20]Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.arxiv.org/abs/2310.06825
  21. [21]Abhinav Joshi, Areeb Ahmad, and Ashutosh Modi. 2024. Cold: Causal reasoning in closed daily activities. Advances in Neural Information Processing Systems, 37:5145–5187.arxiv.org/abs/2411.19500
  22. [22]Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2023. Event knowledge in large language models: the gap between the impossible and the unlikely. Cognitive Science, 47(11):e13386.DOI
  23. [23]Jongho Kim and Seung-won Hwang. 2025. Counterfactual-consistency prompting for relative temporal understanding in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1210–1225, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2502.11425
  24. [24]Mahnaz Koupaee, Xueying Bai, Mudan Chen, Greg Durrett, Nathanael Chambers, and Niranjan Balasubramanian. 2025. Causal graph based event reasoning using semantic relation experts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26169–26199, Vienna, Austria. Association for Computational Linguistics.
  25. [25]Ekaterina Kravtchenko and Vera Demberg. 2015. Semantically underinformative utterances trigger pragmatic inferences. In Proceedings of the annual meeting of the cognitive science society, volume 37.
  26. [26]Ekaterina Kravtchenko and Vera Demberg. 2022. Informationally redundant utterances elicit pragmatic inferences. Cognition, 225:105159.
  27. [27]Charlotte Kurch, Margarita Ryzhova, and Vera Demberg. 2024. Large language models fail to derive atypicality inferences in a human-like manner. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 86–100, Bangkok, Thailand. Association for Computational Linguistics.
  28. [28]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.arxiv.org/abs/2309.06180
  29. [29]Michael Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, and Ellie Pavlick. 2026. Is this just fantasy? language model representations reflect human judgments of event plausibility. In International Conference on Learning Representations, volume 2026, pages 113814–113831.arxiv.org/abs/2507.12553
  30. [30]Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulić, and Nigel Collier. 2025. Aligning with logic: Measuring, evaluating and improving logical preference consistency in large language models. Preprint, arXiv:2410.02205.arxiv.org/abs/2410.02205
  31. [31]Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskája, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, and Barbara Plank. 2025. Pragmatics in the era of large language models: A survey on datasets, evaluation, opportunities and challenges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8679–8696, Vienna, Austria. Association for Computational Linguistics.arxiv.org/abs/2502.12378
  32. [32]Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groen-eveld, Faeze Brahman, Finbarr Timbers, Hamish Ivi-son, and 1 others. 2025. Olmo 3. arXiv preprint arXiv:2512.13961.
  33. [33]OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. GPT-4o system card. Preprint, arXiv:2410.21276.
  34. [34]OpenAI. 2026. GPT-5.4 Thinking System Card. Technical report, OpenAI.
  35. [35]Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2021. Did the cat drink the coffee? challenging transformers with generalized event knowledge. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 1–11, Online. Association for Computational Linguistics.
  36. [36]Susanne Raisig, Tinka Welke, Herbert Hagendorf, and Elke Van der Meer. 2009. Insights into knowledge representation: The influence of modal and perceptual variables on event knowledge retrieval from memory. Cognitive Science, 33(7):1252–1266.DOI
  37. [37]Alice Rees, Vilde Reksnes, and Hannah Rohde. 2026. Why are you telling me this? The availability and timing of triviality-driven inferences. Journal of Memory and Language, 148:104741.
  38. [38]Hannah Rohde and Paula Rubio-Fernandez. 2022. Color interpretation is guided by informativity expectations, not by world knowledge about colors. Journal of Memory and Language, 127:104371.
  39. [39]Margarita Ryzhova, Alexandra Mayn, and Vera Demberg. 2023. What inferences do people actually make upon encountering informationally redundant utterances? an individual differences study. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45.DOI
  40. [40]Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minnie Huang. 2024. EmoBench: Evaluating the emotional intelligence of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5986–6004, Bangkok, Thailand. Association for Computational Linguistics.
  41. [41]Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3027–3035.arxiv.org/abs/1811.00146
  42. [42]Dan Sperber, Deirdre Wilson, and Deirdre Wilson. 1986. Relevance: Communication and cognition, volume 142. Harvard University Press Cambridge, MA.
  43. [43]Armin Toroghi, Faeze Moradi Kalarde, and Scott Sanner. 2026. Commonwhy: A dataset for evaluating entity-based causal commonsense reasoning in large language models. Preprint, arXiv:2605.12918.arxiv.org/abs/2605.12918
  44. [44]Shaofei Wang, Yunan Liu, Xiaolan Tang, and Wenlong Chen. 2026. Collaborative enhancement of large and small models for question answering via dual knowledge transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 40:33630–33638.
  45. [46]Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743.DOI
  46. [47]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengeng Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.arxiv.org/abs/2505.09388
  47. [48]Akhila Yerukola, Saujas Vaduguru, Daniel Fried, and Maarten Sap. 2024. Is the pope catholic? yes, the pope is catholic, generative evaluation of non-literal intent resolution in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 265–275, Bangkok, Thailand. Association for Computational Linguistics.
  48. [49]Zirui Zhao, Wee Sun Lee, and David Hsu. 2023. Large language models as commonsense knowledge for large-scale task planning. Advances in neural information processing systems, 36:31967–31987.arxiv.org/abs/2305.14078

Paper details

vlnc:W_vyzztmym
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents