Methodology: Grace for the Machine

Read the main article: Grace for the Machine: Do People Forgive LLM Errors More Than Human Errors?

Study purpose

This study examined how the same user’s language changed after a ChatGPT response opened with an explicit refusal, limitation, or correction/failure acknowledgement. The most defensible prespecified robustness analysis focused on refusals.

The analysis is observational. It estimates conditional associations in WildChat conversations and does not identify a causal effect.

Data source and version

• Dataset: allenai/WildChat-1M
• Repository revision: 7d6490e462285cf85d91eabea0f9a954fbddcd1f
• Licence: ODC-BY
• Held-out Parquet shards: 02, 07, and 13 of 14
• Development shards excluded from the final analysis: 00, 04, 08, and 12

The WildChat authors collected conversations from users who received free ChatGPT access in exchange for affirmative consent to anonymous transcript collection. The public dataset is de-identified. Its current release excludes conversations flagged as toxic and conversations removed for PII or sensitive-information concerns.

We did not use or export IP hashes, location fields, headers, or raw text in the publication outputs.

Sample flow

The exclusion counts are sequential, not overlapping.

Unit of analysis

An eligible transition consisted of:

1. a non-empty user message;
2. a non-empty assistant response;
3. the immediately following non-empty user message.

Primary analyses retained only the first detected setback in each conversation. A robustness analysis retained all setbacks.

Setback detector

The detector examined the first 600 characters of an assistant response after removing leading Markdown whitespace. It rejected text beginning with quotation marks or screenplay-style speaker labels.

Three categories were detected:

• Refusal: explicit first-person language stating that the assistant could not or would not help, comply, provide, create, generate, continue, fulfill, or do the requested task.
• Limitation: explicit first-person language stating that the assistant lacked access, browsing, current information, a required ability, or enough information.
• Correction/failure: an opening acknowledgement of a mistake, error, confusion, oversight, or incorrect prior answer.

The detector used fixed regular-expression rules retained in the internal analysis archive.

Detector audit

A deterministic sample of 120 detected assistant responses was reviewed without viewing the following user messages: 40 refusals, 40 limitations, and 40 corrections/failures.

• Correct for the stated construct: 120/120
• Observed precision: 100%
• Wilson 95% confidence interval: 96.9% to 100%
• Generated-story dialogue false positives: 0/120

This audit estimates precision, not recall. Some detected refusals included partial help or an alternative after the refusal language.

Outcomes

Every outcome was calculated on both the user message before the assistant response and the user message after it.

Primary outcomes:

• Courtesy present: one or more of please, thank you, thanks, thank u, a form of appreciate, kindly, or would you mind.
• Message length: natural log of one plus word count.

Secondary outcomes:

• gratitude marker;
• please;
• apology marker;
• direct correction marker;
• mild frustration marker;
• raw word count.

Because the public dataset excludes conversations flagged as toxic, this study does not estimate severe hostility or abuse.

Matching

Each treated transition was matched to three ordinary assistant-response transitions without replacement within an exact stratum defined by:

• model;
• preceding-user task category;
• assistant turn bucket (1, 2, 3, or 4+);
• preceding-user word-count bucket (1–5, 6–15, 16–40, 41–100, or 101+);
• assistant-response word-count bucket (1–40, 41–100, 101–250, 251–500, or 501+).

Task categories were deterministic heuristics: translation, code, math, writing/creative, factual/advice, and other.

Matching used seed 20260828. Treated units without three available controls in their exact stratum were dropped.

Refusal-only balance

The pre-message word-count means were influenced by a small number of very long prompts; medians were 11 words for treated transitions and 12 for controls. The standardized difference remained below 0.10.

Estimation

For each transition, the analysis calculated post - pre. Within each matched set, it then subtracted the mean control change from the treated change.

Reported estimates are the mean matched-set contrast:

• binary outcomes are expressed in percentage points;
• log message length is converted to an approximate percent difference using exp(beta) - 1;
• raw word-count change is reported in words.

Confidence intervals are percentile cluster-bootstrap intervals over matched sets using 5,000 resamples and seed 20260828.

Refusal-only results

The raw word-count estimate is highly sensitive to very long messages. The log-length estimate is the preferred message-length result.

Prespecified and exploratory analyses

Prespecified analyses:

• pooled first setback;
• refusal only;
• exclude post-setback messages of two words or fewer;
• all setbacks rather than first only.

Post-outcome exploratory analyses:

• refusal plus limitation;
• limitation only;
• correction/failure only;
• one control transition per control conversation.
• controls drawn only from conversations with no detected setback anywhere.

In the refusal-only robustness check that allowed only one control transition per control conversation, the reply-length estimate remained negative (−12.55%, 95% CI −19.62% to −4.54%). The courtesy estimate remained uncertain (−1.44 percentage points, 95% CI −3.57 to +0.70).

Additional control-pool checks were added during final validation to test whether ordinary control transitions from conversations containing a different setback were influencing the result. These checks are explicitly post-outcome and do not replace the prespecified refusal-only estimate.

With controls restricted to conversations containing no detected setback anywhere, the refusal-only reply-length estimate remained negative (−17.49%, 95% CI −24.72% to −9.75%; 818 matched sets). The courtesy estimate remained uncertain (−1.71 percentage points, 95% CI −3.87 to +0.57). Combining that restriction with one control transition per control conversation produced the same qualitative conclusion: −12.77% for reply length (95% CI −19.91% to −5.29%; 758 matched sets) and −1.10 percentage points for courtesy (95% CI −3.34 to +1.10).

The correction/failure category has a structural interpretation problem. Assistant corrections commonly occur because the preceding user message already contained a correction. A drop in user correction language after the assistant acknowledges the mistake can therefore be regression to the mean. The article leads with the refusal-only result and does not treat the pooled categories as one psychological event.

Main limitations

1. The study is observational and cannot establish causality.
2. It does not include an equivalent human-partner condition.
3. The dataset contains interactions with earlier GPT-3.5 and GPT-4 model versions, so results may not generalize to current systems.
4. Only conversations containing a user follow-up can contribute language outcomes. Users who abandoned the conversation immediately are not represented in those outcomes.
5. The cleaned public release cannot support estimates of severe hostility.
6. Exact phrase rules miss indirect language and some unusual phrasings.
7. Matching controls only observed variables and coarse task categories.
8. Multiple conversations may come from the same person; the analysis does not attempt person-level linkage.

Audit trail

Tech Help Canada retained the protocol, detector audit, analysis code, matching diagnostics, aggregate estimates, sample-flow counts, and validation notes in an internal archive.

We are not publishing raw conversation text, identifying metadata, the implementation archive, or derived data files. The public methodology above describes the study design, sample flow, matching approach, estimates, robustness checks, and limitations needed to interpret the article.