betaFeedback

WorksComputer Science

Conditional Endorsement: Simulating How Adults with Different Skills Evaluate Canada's National AI Strategy

Abstract

Canada's 2026 national AI strategy, AI for All, commits to free AI training for everyone and to expanding business AI adoption, yet it is difficult to anticipate whether these policies will reach adults with uneven skills and circumstances. We ask how adults with diverse skills might rate these policies on support, effectiveness, and personal benefit, how their skills and circumstances shape these evaluations, and what implementation conditions they raise. We built 3,200 generative agents from a weighted sample of the Programme for the International Assessment of Adult Competencies (PIAAC) Cycle 2 respondents, collected their ratings and explanations for 20 policies, and analyzed dialogues between agents with contrasting ratings. The agents supported nearly every policy but were far less confident that they would improve adult AI literacy or benefit them personally. This gap widened as digital technology use declined: agents modeled on adults who had never used a digital device still endorsed the policies but rarely expected to benefit. Agents traced the gap to participation costs, hard-to-exercise rights, and limited community control, and their dialogues specified these conditions rather than resolving disagreement. We describe this pattern as conditional endorsement and argue that AI literacy policy should be designed and evaluated for adults with the fewest digital skills.

Introduction

In June 2026, the Government of Canada released its national artificial intelligence (AI) strategy, AI for All [32], that commits to free AI training for everyone in Canada and to raising business AI adoption to 60% by 2034 [72]. Only 12.2% of Canadian businesses reported using AI in mid-2025 [62], and almost half of workers who used AI tools at work had received no formal training [17]. For employers, workforce developers, and policymakers, the success of such commitments depends on whether they reach adults with very different skills, jobs, and circumstances. However, before implementation, it is difficult to know how these adults would receive a policy. In this paper, we treat adults’ skills and circumstances as drivers of how they receive AI policy, and use generative agents grounded in Canadian adult-skills data to simulate how diverse adults evaluate the strategy’s policies.

The problem is hard because the adults whom AI literacy policy most needs to reach are often the least visible in the processes that shape it. Research on the digital divide shows that inequalities in skills and outcomes persist beyond access to devices [28, 70]. However, national AI strategies tend to articulate public skills and inclusion through broad commitments, while rarely outlining specified mechanisms [18, 56]. The strategy’s public consultation relied on self-selected participation, and critics argued that the strategy was introduced “without any public debate” [14].

Large language model (LLM)-based simulations of survey respondents can approximate aggregate opinions [3, 53]. However, concerns remain about misalignment with respondents’ views [6, 58] and the flattening of identity groups [71]. The Programme for the International Assessment of Adult Competencies (PIAAC)1 provides a basis for grounding these simulations in observed variation: it records foundational skills and circumstances for a nationally representative sample of adults [48]. We use data from its Canadian respondents to construct generative agents as structured, albeit imperfect, proxies for this variation. These agents rate and explain their support for the strategy’s policies.

policies, their perceived effectiveness, and their anticipated personal benefit. We ask the following research questions:

RQ1: How do PIAAC-grounded agents evaluate the policies in the national AI strategy?

RQ2: How do the underlying personas’ skills and circumstances shape these evaluations?

RQ3: What implementation conditions do the agents raise individually and in simulated dialogues?

To answer these questions, we constructed 3,200 personas from a weighted PIAAC Cycle 2 sample. Agents rated 20 policies from AI for All, yielding over 63,000 valid ratings on each of three dimensions. We then simulated 1,533 dialogues between agents with contrasting ratings. We compared ratings across persona characteristics and analyzed the responses and dialogues with inductive concept analysis [35]. For RQ1, 99.5% of support ratings fell in the upper half of the scale, compared with 56.4% of benefit and 35.9% of effectiveness ratings. For RQ2, expected benefit fell with digital technology use: 63.4% of benefit ratings from personas with higher use fell in the upper half, compared with 47% for lower use and 12.3% for personas who had never used a digital device. For RQ3, the agents attributed the gap to participation costs, rights that are difficult to exercise, and control over resources and institutions; and their dialogues, instead of resolving disagreements, elaborated these conditions. We describe this pattern as conditional endorsement: support for AI literacy policy does not imply it will reach adults with varying levels of digital skills equally without active measures. In the following sections, we review related work (§ 2), describe our research design (§ 3), present our findings (§ 4), and discuss their implications and limitations (§ 5).

Literature Review

Our literature review brings together three bodies of work that frame how to evaluate AI literacy policy before it reaches the public: research on AI literacy and adoption, national AI strategies as instruments of governance, and the use of survey-grounded generative agents to simulate policy evaluation.

AI Literacy as a Condition of Equitable AI Adoption

As AI systems become embedded in work, public services, and everyday communication, scholars have increasingly treated AI literacy as a foundational capacity rather than a specialized technical skill. Long and Magerko define AI literacy as a set of competencies that enables individuals to critically evaluate AI technologies, communicate and collaborate effectively with AI, and use AI as a tool online, at home, and in the workplace [40]. Similarly, Ng and colleagues’ review organizes AI literacy around knowing and understanding AI, using and applying it, evaluating and creating with it, and attending to its ethical implications [46]. Earlier efforts to define what learners should know about AI [64] have since been extended to higher and adult education [36]. These conceptualizations shift attention from whether people can operate an AI tool to whether they can judge when, how, and on whose terms to use it. Such judgment also depends on knowing the limits of one’s own skill: people who trust automated systems too readily tend to stop checking their output [51], and recent AI literacy measures treat self-efficacy as a competency in its own right [7].

However, the capacity to acquire such competencies is unevenly distributed. Research on the digital divide has long shown that inequalities extend beyond physical access to devices and connectivity, encompassing differences in skills, usage, and the outcomes people derive from technology [28, 69, 70], and that digital disadvantage tends to compound existing social disadvantage [57]. AI adoption risks reproducing these layered inequalities: people with lower foundational skills, limited digital experience, or precarious employment may face both fewer opportunities to learn about AI and greater exposure to its disruptions. While LLMs are affecting a large share of work tasks across occupations [16], the Organization for Economic Co-operation and Development (OECD) has emphasized that training and worker involvement shape whether AI adoption improves job quality [47]. Adoption also depends on trust. Global studies of public attitudes, most recently spanning 47 countries, have found that people’s willingness to rely on AI is closely tied to their confidence in the institutions that develop, deploy, and regulate it [23, 24]. Hence, AI literacy and responsible adoption are not only matters of individual learning, but also of the institutional conditions under which learning, access, and trust become possible.

National AI Strategies as Instruments of Literacy and Adoption Governance

Governments have increasingly articulated these institutional conditions through national AI strategies. Since the first national AI strategies appeared in 2017, including the Pan-Canadian Artificial Intelligence Strategy, scholars have examined these strategies as governance documents that set priorities, allocate resources, and signal which publics the state expects to benefit from AI [18, 56]. Policy documents frame AI through competing narratives of research capacity, economic competitiveness, and societal challenge [65], and while ethical guidelines converge on principles such as transparency, justice, and responsibility, they diverge on how to implement them [33].

In Canada, a review of 84 federal and provincial AI governance initiatives found that most focus on industry, research, and public administration, while few address AI education and training or social and workforce development [4]. Similarly, Das and colleagues’ analysis of Canada’s federal AI Register suggested that, while the register aims to make government AI use visible, it provides insufficient information for the public to contest that [12]. However, Canada’s national AI strategy, released in June 2026, places public capacity more centrally than earlier strategies [32]. The strategy is organized around six pillars, including protecting Canadians and safeguarding democracy, empowering Canadians, and powering AI adoption. It commits to free AI training for everyone in Canada and to up to 90,000 AI-related jobs and work placements for young Canadians by 2031 [32]. The strategy also reports a consultation involving more than 11,000 Canadians. Overall, these commitments position AI literacy and adoption as national policy goals that depend on reaching adults across very different occupational, linguistic, and social circumstances.

While a strategy document describes intended actions and measurable targets, it remains unclear how the people it aims to reach might receive them. Public consultations provide valuable input, but they rely on self-selected participation and are typically conducted before policies are specified at the program level. Consequently, we know comparatively little about how adults with different foundational skills, digital experiences, and life circumstances might evaluate specific AI literacy and adoption policies, and what implementation conditions they would consider necessary.

Survey-Grounded Generative Agents as Simulated Policy Evaluators

Answering this question requires both data that describe adults’ skills and circumstances and a method that can elicit evaluations of specific policies from those profiles. The OECD’s PIAAC provides the former. Its second cycle assessed literacy, numeracy, and adaptive problem solving among adults aged 16–65, alongside detailed background information on education, employment, technology use, learning, and well-being [48]. While Canadians scored above the average of participating OECD countries across the three domains [61], these national averages conceal substantial variation among adults, which AI literacy policy must address. Since PIAAC links skills and skill use to employment, learning, and social participation [48], it allows skills to be examined as outcomes that a given policy seeks to change and as drivers of how individuals might receive that policy.

Recent advances in LLMs have made it possible to simulate how people with particular backgrounds might respond to questions or situations, prompting debate about whether such simulations can complement, or even substitute for, human participants in social science research [2, 13, 26].

Park and colleagues introduced generative agents, i.e., LLM-based agents that maintain memory, reflection, and planning to produce believable individual and social behavior [52]. Subsequent work has grounded such agents in real individuals’ data: agents built from qualitative interviews with 1,052 people replicated participants’ General Social Survey responses 85% as accurately as participants replicated their own answers two weeks later [53]. Related studies have used conditioned LLMs as “silicon samples” of survey populations [3], replicated classic human-subject experiments [1], and simulated economic decision-makers [29]. Ghaffarzadegan and colleagues describe this direction as generative agent-based modeling (GABM), in which LLM-powered agents replace hand-coded decision rules, allowing researchers to simulate how heterogeneous agents reason and interact [21]; similar designs have simulated social media platforms populated by persona-conditioned agents [63]. Moreover, multi-agent exchanges, in which agents respond to and critique one another, have been used to surface and refine reasoning that a single agent might not produce [15], and networks of LLM agents have been used to simulate opinion dynamics [9].

Generative agents have also been applied to social and policy questions, including macroeconomic activity [37], universal basic income in a society of more than 10,000 agents [55], vaccine-hesitancy interventions among census-grounded agents [30], team formation at a simulated job fair [38], and diplomatic positions on international events [54]. These studies mostly model behavior and its aggregate outcomes rather than how people with different profiles evaluate a specific policy, and even census-grounded agents sometimes acted inconsistently with their assigned demographics [30].

However, scholars have also raised various concerns about treating LLM outputs as substitutes for human respondents. LLMs’ opinions can be misaligned with those of many demographic groups [58], synthetic survey responses can be less varied and less stable than human responses [5], and LLMs prompted to represent identity groups can misportray and flatten them [71]. Persona variables may also explain only a limited share of the variation in subjective judgments [31]. In interaction, LLMs tend to agree with their interlocutors [59] and to converge toward consensus [9], which may suppress the disagreement that deliberation is meant to surface. These critiques suggest that GABM should be treated not as a replacement for public consultation, but as an exploratory instrument whose outputs require careful interpretation. Such interpretation can draw on established practice in computationally assisted qualitative analysis, where machine learning helps researchers explore large text corpora while interpretation remains with the researcher [44, 45]. Grounding personas in survey data, eliciting ratings with explanations, and analyzing the resulting texts with LLM-assisted tools such as LloOM [35] extend this practice to agent outputs, making their interpretation systematic and scalable.

Research Design

Our research design combines GABM [21, 52] with mixed-methods analyses of single-agent responses and simulated dialogues. In this section, we describe the inputs, single-agent evaluation, dyadic discussions, and AI-assisted qualitative analysis. We treat the agents’ responses not as estimates of Canadian public opinion, but as simulated, survey-grounded accounts of how people with particular backgrounds, skills, and circumstances might reason about AI policy. Figure 1 summarizes the research design, and Appendix § 8 provides further procedural details.

Figure 1. An overview of the research design. Personas built from PIAAC responses evaluated the 20 policies identified in AI for All, and agents with contrasting ratings on the same policy, selected by the rating distance DD, then discussed it in dyads. We analyzed the rating distributions, induced concepts from responses and transcripts with LLooM, and used an XLM-RoBERTa model trained on single-agent responses to estimate rating changes within each dialogue.

Policies and Challenges Identification in National AI Strategy

We conducted a structured qualitative analysis of the Canada’s National Artificial Intelligence Strategy: AI for All document [32]. We coded passages that specified government actions as policies and passages describing risks, barriers, and constraints as landscape challenges. After consolidation, this process yielded 20 policies and nine landscape challenges, such as public trust and AI literacy, unequal access, and privacy and online harms (Appendix § 8.§ 1). We used an LLM to paraphrase each policy and challenge into a self-contained description.

PIAAC-grounded Persona Construction

Before constructing agents from PIAAC respondents, we examined the PIAAC data directly. Appendix § 7 compares Canadian adults’ ratings of their own assessment performance with their measured proficiency and finds that confidence tracks proficiency only loosely: at equal proficiency, men and adults with a bachelor’s degree or higher were more likely to rate their performance above what their scores supported. Because how adults perceive their skills can differ from what PIAAC measures, the personas below express skills relative to the weighted sample rather than through respondents’ own self-assessments.

We drew a weighted sample of 3,200 records from the PIAAC Cycle 2 public-use files (PUF) [48] using the final full-sample weight (Appendix § 8.§ 2)3. Following prior work that conditions LLMs on survey respondents’ characteristics [3, 53], we constructed a synthetic persona from each sampled respondent’s selected PIAAC characteristics, including demographic background, language, education, employment, household and economic context, digital technology use, learning and training, personality, social trust, and health. Thus, the personas embodied skills in terms of education, skill use at home and at work, and digital technology use, expressed as lower, typical, or higher levels relative to the weighted sample (Appendix § 8.§ 3). These personas are synthetic representations derived from PIAAC survey responses. Each persona was represented by one generative agent, instantiated as a separate instance of GPT-5-mini. We also assigned each persona a randomly selected subset of the landscape challenges, which served as context for considering how broader implementation barriers might shape the persona’s assessment.

Single-Agent Policy Evaluation

We presented each agent with the policy announcements and asked it to respond from its persona’s perspective, considering the persona’s background, skills, circumstances, and likely priorities. For each policy, the agent provided (1) a brief first reaction; (2) three six-point ratings, each with a brief justification, of support for the policy, the policy’s perceived effectiveness, and the likelihood that it would directly benefit the persona; and (3) one to three sentences suggesting changes that might make the policy more effective for someone in the persona’s circumstances, given the assigned challenges. The effectiveness rating asked how effective the policy would be in improving AI literacy among Canadian adults, and the benefit rating asked how likely the policy was to benefit the persona directly. We generated prompts in English or French based on the persona’s PIAAC questionnaire language, and instructed agents not to add personal details absent from their descriptions (Appendix B.4).

Dyadic Discussions

For each policy, we distinguished lower-rating agents, who rated a policy 1–3 on support, effectiveness, and benefit, from higher-rating agents, who rated it 4–6 on all three. A contrasting dyad paired one agent from each group for that particular policy. Only 9 of the 20 policies had agents in both groups, yielding 15,222 possible dyads. For each dyad, we calculated the L1 distance across the three ratings:

D=∣Slow−Shigh∣+∣Elow−Ehigh∣+∣Blow−Bhigh∣,(1)D = \lvert S_{\mathrm{low}} - S_{\mathrm{high}} \rvert+ \lvert E_{\mathrm{low}} - E_{\mathrm{high}} \rvert+ \lvert B_{\mathrm{low}} - B_{\mathrm{high}} \rvert, \tag*{(1)}

where SS, EE, and BB denote support, effectiveness, and benefit ratings. After excluding mixed-language pairs, we selected approximately 10% of the dyads (n=1,533n = 1{,}533) using a coverage-first procedure: up to 10 high-distance dyads per eligible policy, followed by selecting the remaining dyads based on the highest DD across policies (Appendix B.5).

Then, we asked the two agents in each dyad to discuss the policy on which they had previously provided contrasting ratings. The conversation proceeded in alternating turns, with each agent allowed up to five turns, responding to the other agent’s most recent comment, and instructed to contest, qualify, connect, or build on the other speaker’s arguments. To characterize how conversations opened, we checked whether the second speaker’s first turn began with an explicit agreement phrase (e.g., “I agree” or “You’re right”) in English or French.

To examine whether the initial differences changed during the conversation, we fine-tuned a multilingual XLM-RoBERTa model [11] on the single-agent responses and their ratings, with one classification head per rating dimension, and evaluated it on held-out response–rating pairs (Appendix B.6). We used the model to predict support, effectiveness, and benefit ratings for each conversational turn and compared each agent’s predicted ratings in the first and final turns. We compared these estimates with a TF-IDF logistic-regression classifier trained on the same data (Appendix B.7). Based on changes between each agent’s first and final turns, we classified dyads as converging, maintaining their initial disagreement, or becoming more polarized.

Qualitative Analysis

We conducted an automated qualitative analysis of the single-agent responses and the dyad conversations using LIoMoM [35], which generates concepts inductively from text. For the single-agent responses, we ran LIoMoM separately for each policy on the first reactions, rating justifications, suggested changes. For the dyads, we analyzed each transcript as a single document, identifying disagreements, shared positions, and recommendations introduced or developed in later turns (Appendix § 8.8). We used the concepts as probes for interpretation, and not as prevalence estimates. We attended to the explanations and transcript snippets the concepts pointed to, noting how agents invoked their personas’ skills and circumstances, and organized the findings into themes that triangulate the ratings, single-agent explanations, and conversations. In the findings, we cite single-agent responses by persona ID and conversational turns in dyads by speaker ID, turn, and partner. Appendix § 8.9 describes our reproducibility measures.

Findings

We organize the findings around five themes. The first shows that the agents’ near-universal support for the 20 policies did not translate into expectations of literacy reach or personal benefit. The second shows how the personas’ skills and circumstances, particularly their digital technology use and employment, shaped who expected to benefit. The third and fourth trace the conditions that the agents attached to their support: participation costs, and rights and control that people could actually exercise. The fifth examines what the dyadic conversations added to these conditions and what they cannot show. Within each theme, we triangulate across the ratings, explanations, and conversations, and we note evidence that complicates the pattern.

Near-Universal Support Alongside Divided Effectiveness and Benefit

Support, direct-benefit, and effectiveness ratings had markedly different distributions (Figure 10–Figure 12 in Appendix § 9.§ 1). Of the valid ratings, 99.5% of support ratings fell in the upper half of the scale (4–6), compared with 56.4% of direct-benefit and 35.9% of effectiveness ratings. Every policy had more than 95% of its valid support ratings at 4–6. Within the responses with all three ratings present, 63.6% combined upper-half support with a lower-half effectiveness rating, and 43.1% combined upper-half support with a lower-half direct-benefit rating (Figure 2).

Figure 2. Joint distribution of support with literacy effectiveness (left) and direct benefit (right) among the 63,875 responses with all three ratings. Cell values are percentages of these responses (values below 0.05% omitted), and outlined cells mark upper-half support combined with a lower-half rating.

Figure 10. PIAAC-persona-based agents’ support for different AI policies. Bars show the percentage of valid ratings at each scale point; blue bars mark the upper half (4–6).

Figure 12. PIAAC-persona-based agents’ perceived direct benefit of different AI policies. Blue bars mark the upper half (4–6).

The rating prompts explain much of this divergence. “Effectiveness” asked whether a policy would improve AI literacy among Canadian adults, not whether it would achieve all of its stated objectives, and “benefit" asked about direct benefit to the persona. For example, the consumer privacy reform policy received upper-half support from every valid response and upper-half benefit ratings from 98.5%, yet only 10.7% rated it in the upper half for improving adult AI literacy (Table 23 in Appendix § 9.§ 2). Upper-half support exceeded 99% for an AI mission in health care, health-data research platforms, and AI adoption in federal public services, but at most 0.6% rated their literacy effectiveness at 4–6. Conversely, effectiveness was highest where a policy had an explicit educational component: all valid ratings for free introductory AI education were 4 or 5, whereas 91.6% of ratings for financing and commercialization were 1 or 2. Thus, these proposals could be valued as privacy, health, or economic initiatives while being seen as weak literacy interventions.

PolicySupport (%)Benefit (%)Effectiveness (%)
All 20 policies99.556.435.9
Consumer privacy reform100.098.510.7
Online safety legislation100.087.15.3
AI safety evaluation and certification100.080.462.1
Identifying AI systems and generated content100.095.294.9
Free introductory AI education100.088.3100.0
AI tools for postsecondary students99.96.768.5
AI-related employment and training100.054.983.4
Indigenous leadership in AI100.00.447.3
AI support for Canadian creators99.846.651.3
English and French in government AI100.045.517.1
Accessibility and inclusion in AI100.084.481.1
Support for small-business AI adoption99.231.315.3
AI mission focused on healthcare100.089.70.6
Health-data platforms for research and AI99.846.90.3
AI adoption in federal public services99.291.60.4
Canadian-controlled AI infrastructure99.953.51.0
AI research and talent recruitment95.314.50.2
Financing and commercialization for Canadian AI companies97.517.30.1
International cooperation on sovereign AI99.930.60.4
Support for open-source AI100.064.278.0

Table 23. Share of valid ratings in the upper half of the scale (4–6), by policy and dimension.

In turn, direct benefit separated public endorsement from personal proximity. For the Indigenous leadership in AI policy, every valid support rating was 4–6, whereas only 0.4% of direct-benefit ratings were rated in the upper half. For the AI tools for post-secondary students policy, the figures were 99.9% and 6.7%. The agents located this gap in their own eligibility rather than in the policy’s value:

“I welcome a policy that centers Indigenous leadership—supporting languages and cultural expression feels important and right to me even though I’m not Indigenous. [...] I’m not Indigenous and I’m out of the labor force, so this would probably not change my day-to-day life even if I appreciate its goals." (PIAAC-5597)

A response about post-secondary tools similarly stated, “I’m not a student or an educator now, so I’m unlikely to get direct personal benefit from this program" (PIAAC-11312-3). These low benefit scores reflect the item’s individual framing and the policy’s target population, not a judgment that Indigenous leadership or student access is unimportant.

Skills and Circumstances Shaped Who Expected to Benefit

Because each agent reasoned from a PIAAC-derived profile, we could examine how skills and circumstances operated as drivers of its evaluations. Support barely varied across persona characteristics, but expected benefit and literacy effectiveness did (Table 1).

CharacteristicLevelSupport (%)Effect. (%)Benefit (%)
Digital use at homeNever used a device90.413.212.3
Digital use at homeLower99.431.047.0
Digital use at homeTypical99.737.557.7
Digital use at homeHigher99.839.163.4
Employment statusOut of the labour force98.732.445.3
Employment statusUnemployed99.337.052.6
Employment statusEmployed99.736.959.2
EducationBelow upper secondary98.529.541.8
EducationUpper secondary499.435.152.2
EducationTertiary99.938.663.5

Table 1. Share of valid ratings in the upper half of the scale (4–6) across all 20 policies, by persona characteristic. Levels of digital technology use at home are relative to the weighted sample.

Digital Technology Use as a Lens on Benefit

Digital technology use showed the steepest gradient. Across the 20 policies, 63.4% of benefit ratings from personas with higher digital technology use at home fell in the upper half, compared with 47.0% from those with lower use. The 48 personas who had never used a digital device rated benefit in the upper half only 12.3% of the time and effectiveness only 13.2%, although 90.4% of their support ratings were 4–6. Personas with lower digital use expected less benefit than those with higher use for 19 of the 20 policies, with the largest gaps for the ones about AI support for Canadian creators (15.9% vs. 70.1%) and support for open-source AI (36.8% vs. 83.5%). The agents’ explanations show this gradient from both sides. Among benefit rationales for accessibility and inclusion standards, LLooM surfaced a Daily Digital Use concept, in which agents expected some benefit because they already relied on AI-enabled products:

“I use digital tools every day, so accessible AI would likely make some products I rely on easier and safer” (PIAAC-7748).

Agents with low digital use reasoned in the opposite direction about public-service automation:

“I’m wary of this—I don’t trust new tech in government and worry it will make services harder for people like me who aren’t online. [...] It’s somewhat unlikely to directly help me since I don’t use digital devices and depend on non-digital ways to deal with government offices.” (PIAAC-1754)

Similarly, an agent based on the persona of a self-employed craft worker (PIAAC-2950-2) “with low digital use for transactions and no workplace training” expected no benefit from accessibility standards “unless there’s local, practical help”. This pattern complicates the strategy’s premise. The adults whom AI literacy policy most needs to reach were the least likely to expect it to reach them, because its delivery appeared to assume the digital engagement they lacked. The one exception was free introductory AI education, the policy designed for beginners: personas with lower digital use rated its benefit in the upper half more often than those with higher use (94.8% vs. 82.9%).

Employment, Migration, and Health as Frames for Evaluation

Employment and education produced smaller but consistent gradients. Personas out of the labor force rated benefit in the upper half 45.3% of the time, compared with 59.2% for employed personas, and personas without upper secondary education did so 41.8% of the time, compared with 63.5% for those with tertiary education. Employment, migration, and health also shaped which features agents valued. Unemployed PIAAC respondents’ persona-based agents welcomed faster public-service processing because they “depend[ed] on services” and could receive “benefits or approvals sooner,” while worrying about losing human help when decisions affected them (PIAAC-6882-3, PIAAC-10565). An agent based on the persona of a first-generation immigrant “with limited digital skills” doubted that accessibility standards would help without “plain-language resources, translations, and local outreach” (PIAAC-5825), and an agent whose “poor health and limited digital comfort” could leave them “left out of new tech quickly” made a similar demand (PIAAC-8440). Thus, the policy was evaluated through different stakes: speed of access for those who rely on services, usability for those already online, and the risk of exclusion for those with limited digital skills.

Participation Required Time, Devices, and Support

Agents treated formal eligibility for funding, training, or tools as only the first condition of participation.

Access Beyond Formal Availability

In the support for small-business AI adoption policy, one agent wrote that “money alone won’t fix” lack of expertise, fragmented data, vendor lock-in, and weak worker training (PIAAC-1258). Another worried that the funds might reach better-connected firms while frontline workers experienced job changes without training (PIAAC-1409). The LLooM concepts of Paid Training Time, Partnerships with Colleges, and Childcare and Support Services reflect the same constraints: a place in a course, a loan, or an AI account was not sufficient if attendance cost wages, childcare, travel, or device access, echoing prior scholarship that locates digital inequality in skills and usage [57, 69]. In a post-secondary-tools dyad, a lower-rating agent challenged the assumption that an AI tool could serve as its own entry point:

PIAAC-10138-3: “Use the AI agents themselves to help bridge the gap: have them run plain-language, step-by-step on-boarding [...] tied to local supports like libraries, device-lending and community mentors so people get guided practice, not just a login."

PIAAC-1799-2: “[...] for people like me who’ve never used a device those digital on-boarding bots come too late. Before an AI can help, there must be local, in-person supports—device lending, short hands-on sessions at libraries or community centers, and a phone helpline—plus printed step-by-step guides I can keep."

PIAAC-10138-3: “[...] I’d add two practical ideas: recruit and train student ‘digital ambassadors’ to run short pop-up sessions at libraries, community centers and campuses [...], and pre-configure loaned devices with an offline step-by-step tutorial [...] so newcomers can try things without needing data or a complex sign-up."

Thus, the higher-rating agent revised its own proposal, moving the AI agent from the entry point to a step that follows in-person support.

4.3.2   Participation Costs as Program Design Requirements. Recommendations made these costs part of program design. One employment response proposed “short accredited, hands-on modules delivered by local colleges or unions," paid time off, and “portable credentials" (PIAAC-11035-5), and a self-employed agent asked for “flexible, low-cost micro-credentials and remote short courses" (PIAAC-10001). A dyad developed a related chain of supports, from paid training blocks and childcare or travel help to community “on-ramp" programs with device loans (PIAAC-3906-3 and PIAAC-10008-4, turn 0). By the final turn, the discussion had reached an issue absent from a simple enrollment target: training stipends might interfere with social assistance. The partner proposed a written assessment of benefit eligibility, so participants would not be “forced to choose between retraining and basic needs" (PIAAC-10008-4, turn 4). This progression locates exclusion in the interaction among programs, not only in individual willingness to learn. Small-business related dyads similarly translated adoption funding into service obligations, such as on-site coaches, vendor setup visits, a human helpline, and low-risk pilot tasks (PIAAC-11278, turn 0, paired with PIAAC-10216-5), although they generated many plausible features without weighing cost or choosing among them.

Rights and Control Had to Be Exercisable

Though the policies that were committed to Privacy, oversight, and community-leadership drew support, agents questioned whether people could actually use them, who would govern resources, and which institutions the proposed safeguards would constrain.

4.4.1   Consent and Oversight as Supported, Contestable Processes. Across policies, the issue was less a generalized distrust of AI than uncertainty about who could understand, contest, or enforce the promised protections. One agent worried about “who can access the data and whether privacy promises will really hold" (PIAAC-4809-2), and another feared that “real people could lose control over important decisions without clear ways to appeal" (PIAAC-9305). In a dyad focused on health data, one agent made the non-digital route concrete:

“A non-digital opt-out/control must be real: paper forms at clinics, a staffed phone line, and community-center assistance in multiple languages. Funded in-person navigators (at hospitals and public health offices) to explain uses and help people opt out or consent would make this usable for older, non-tech-savvy patients." (PIAAC-5595, turn 0, paired with PIAAC-1799-2)

Subsequent turns added independent patient representation, record correction, revocable consent, and mailed or telephone confirmation. Thus, consent was treated not as a one-time notice but as a process that had to remain understandable, reversible, and supported by someone accountable, echoing critiques of notice-and-consent regimes [60]. The public-service dyad followed a comparable path. The lower-rating speaker asked who decides which automated decisions are “meaningful" (PIAAC-1799-2, turn 0, paired with PIAAC-1532), and the partner proposed a threshold based on consequences for legal status, benefits, or essential services, along with documented human review, paper or in-person options, and pilots with published error and exclusion rates. The promise of human oversight thereby became a sequence of decisions about who qualifies for review, how a person learns what happened, and what remedy follows an error.

One low-rating persona (PIAAC-1799-2) recurred across several of these dyads, which complicates any claim that the dialogues independently converged on non-digital routes. However, single-agent suggestions raised the same routes without a partner, calling for “local help (phone or in-person) for people not comfortable with digital systems" (PIAAC-2952-3) and for “non-digital service options so people with low digital skills or language barriers aren’t left behind" (PIAAC-2711-2).

Community Control over Funds and Knowledge

In the creator-policy responses, one agent worried that the program could miss Indigenous and racialized creators or neglect consent (PIAAC-3067-2), and another proposed reporting “who receives funding and how datasets are used" (PIAAC-3127). Indigenous leadership responses asked for funding that is “multi-year and controlled by communities" (PIAAC-1562-2) and safeguards for “Indigenous data sovereignty and IP" (PIAAC-3349). These claims concern not only entry into a program but authority to govern knowledge, data, and the terms of participation, consistent with Indigenous data governance principles [8, 20]. The creator dyad extended this authority to withdrawal, moving from in-person help and template contracts to consent that could expire, with the partner’s final turn proposing:

PIAAC-1799-3: “Add this for people like me who can’t read long letters or go online: require a mailed, plain-language, signed certificate of deletion plus a phone call from a local liaison to confirm action. Let people name a trusted proxy (family member, community rep) to act and receive confirmations on their behalf."

PIAAC-5205: "[...] two practical things: require an alternative non-written confirmation (a recorded phone call or short audio certificate sent on request) for people who can’t read, and give creators a simple time-limited consent option (or automatic expiry/auto-delete after a set period) so they don’t feel locked in forever."

Safeguards as Constraints on Enforcers and Dominant Firms

Other concerns targeted the institutions responsible for a policy. An online-safety respondent asked for “transparency and appeal mechanisms so enforcement doesn’t unintentionally silence legitimate speech" (PIAAC-8328). In a dyad on international technology cooperation, a lower-rating agent argued that a country already had “the money and contracts to shape standards and win procurement" (PIAAC-1799-2, turn 0, paired with PIAAC-6187-3), and the partner translated this concern into procurement criteria for interoperability, independent testing, and public progress reporting. These proposals would change the obligations of governments, vendors, and platforms rather than individual AI literacy. The dyad later wandered into detailed procedures for home visits, illustrating how the instruction to keep adding something new could pull a conversation away from the policy’s core mechanism.

Dialogue Specified Delivery Conditions but Did Not Resolve Disagreement

Although every dyad paired an agent who rated the policy 1–3 on all three dimensions with one who rated it 4–6, the contrast rarely produced an adversarial opening. The second speaker began with an explicit agreement phrase in 91.6% conversations. In one small-business policy-related dyad, the first agent’s prior ratings were 3, 2, and 1, whereas the partner’s were 5, 4, and 5. Their first turns nonetheless began as follows:

PIAAC-11278: “I welcome the money—but as someone who’s never used a digital device, I worry loans and tools alone won’t get small businesses to adopt AI. For many older owners or those in poor health, the barrier is knowing where to start and having patient, hands-on help."

PIAAC-10216-5: “I agree—money and generic tools won’t be enough for many small firms. From my experience, practical, on-site help makes the difference: sending trained ‘AI coaches’ to a shop or office, running short hands-on workshops at local libraries or community colleges."

Across topics, the recurring conversational move was acknowledgment followed by specification (Figure 3). In an exchange about the small-business focused policy, “money and generic tools won’t be enough" led to requests for on-site coaches, vendor follow-up, and a person who could troubleshoot. In dyads regarding health data, a broad privacy concern generated recommendations for paper forms, staffed lines, independent patient representatives, and revocation confirmation. In dyads focused on public services, a question about “meaningful" oversight generated criteria for human review, measurable pilots, appeals, and remediation. These chains converge with the single-agent suggestions reported above, which raised the same mechanisms without a partner.

Acknowledgment followed by specification in three dyads

Figure 3. Acknowledgment followed by specification in three dyads. Each column runs from the lower-rating speaker’s opening concern to mechanisms the two speakers added in later turns.

We did not ask agents to report a new rating in the middle of their conversations, so opinion changes were inferred from the first and last turns, and the estimates diverged by method. One model classified support as unchanged for 98.4% of the speaker trajectories and another for 86.2% (Appendix § 8.7). As the agents were instructed to “address the other speaker’s most recent point" and to “add a new perspective instead of repeating [their] entire position," the setup likely encouraged elaboration over conflict. Overall, the conversations document how agents turned a policy’s abstract promise into candidate delivery conditions with little evidence of persuasion. Our analysis highlights why endorsement for the policies in the national AI strategy alone is an incomplete measure of policy acceptance: a public aim could be supported while interrogating and doubting who would reach the service, who could govern its use, and what recourse would exist when its safeguards failed.

Implications and Future Directions

Our findings suggest that PIAAC profiles-grounded agents evaluated Canadian AI policies along at least three distinct axes: whether a policy’s public aim deserved support, whether it would reach adults as an AI literacy intervention, and whether it would reach the persona personally. Near-universal support coexisted with much lower effectiveness and benefit ratings, a gap that widened as the personas’ digital technology use, employment, and education declined, and that the agents located in the conditions of delivery rather than in the policies’ value. Below, we discuss our contributions, implications, and limitations.

Conditional Endorsement as a Lens on AI Policy Evaluation

Based on the synthetic policy evaluations by PIAAC-grounded agents, we propose an AI policy evaluation lens, which we dub conditional endorsement: support for a policy’s public aim that is contingent on how the policy is delivered, who can exercise it, and who governs it. Therefore, a high support rating is not a reliable verdict on whether a policy will work for a given person, but an endorsement qualified by three delivery conditions (Figure 4): participation, i.e., the time, devices, and people that attendance requires; exercisability, i.e., whether rights and oversight can be used through ordinary channels; and control, i.e., whether affected communities govern resources and safeguards constrain enforcers and vendors. These conditions bind unevenly: because they presuppose time, devices, and digital fluency, they weigh most on adults with the fewest digital skills, for whom expected benefit become minimal while support can still stay high. Hence, conditional endorsement becomes a framework for interpreting where, and for whom, an endorsed policy’s promise may fail to translate into reach and benefit.

Diagram showing conditional endorsement: a policy promise leads to participation, exercisability, and control, which lead to reach and benefit; skills and circumstances moderate reach and benefit, and dyadic dialogue specifies delivery conditions.

Figure 4. Conditional endorsement. Agents endorsed a policy’s public aim but qualified whether it would reach adults, or the persona, through three delivery conditions. The personas’ skills and circumstances moderated expected reach and benefit, and dyadic dialogue mainly specified the delivery conditions.

Our study also makes a methodological contribution. By pairing survey-grounded personas with separate ratings for support, effectiveness, and benefit, the design makes visible a gap that a single approval measure would obscure, and shows how that gap is distributed across adults’ skills. Moreover, the dyadic conversations functioned less as a persuasion experiment [2, 9] than as a device for specification, turning abstract concerns into delivery requirements.

Implications for AI Literacy Policy and Practice

First, policymakers and evaluators should not treat high support as evidence of effective or equitable AI literacy policy. A privacy reform can be valued while being seen as a weak literacy intervention, and a policy for Indigenous communities can be endorsed by individuals who do not expect to benefit. Consultation instruments should measure endorsement, reach, and personal relevance separately, report them together, and break them down by skill level. As public reactions questioned whether the national AI strategy would benefit ordinary Canadians [14], those distinctions also need to be contextualized in the public socioeconomic conditions and sociotechnical practices.

Second, AI literacy programs should be designed first for adults with little or no digital experience. Free introductory AI education was the only policy that personas with lower digital use expected to benefit from more often than those with higher use, which suggests that explicitly beginner-oriented programs can reverse the gradient that other policies reproduced. Such programs should offer in-person, telephone, and print entry points rather than treating an AI tool as own onboarding. Programs should also treat participation costs as design requirements rather than as individual barriers. Almost half of Canadian workers who used AI tools at work had received no formal training [17], so training cannot be assumed to follow from access to tools. Employers can contribute paid training time and on-site coaching, and service providers can offer device lending, childcare or travel support, and in-person help. Designers should also check that training stipends do not jeopardize social assistance.

Designing for adults with the fewest digital skills addresses reach, but using AI well also requires knowing when to check its output. A supplementary analysis of Canadian PIAAC Cycle 2 respondents (Appendix § 7) suggests that the adults most likely to overestimate their own skills are not those with the fewest digital skills. Among adults who completed PIAAC’s self-assessment item, men with a bachelor’s degree were the most likely to rate their performance above what their measured proficiency supported (about 14.6%), whereas women aged 55 and over with lower secondary schooling or less were the least likely (5.3%; Table 5). Because over-confidence of this kind can discourage people from verifying AI outputs [51], beginner-oriented programs could be complemented by verification and calibration content aimed at confident, credentialed adults, who are unlikely to see themselves as the audience for introductory AI training. This analysis uses self-assessment on the PIAAC assessment as a proxy and does not measure over-trust in AI directly.

PersonaEstimateSE
All adults10.20.4
Men 25–34, bachelor’s degree14.61.3
Men 55 and over, bachelor’s degree14.51.3
Men 24 or less, upper secondary14.51.7
Women 25–34, bachelor’s degree9.70.8
Women 24 or less, upper secondary9.61.2
Women 55 and over, lower secondary or less5.30.7

Table 5. Predicted share over-confident relative to measured skill for persona profiles (%), averaged over the observed distribution of every other characteristic.

Third, privacy and oversight commitments must be exercisable by people with limited digital fluency. This requires paper and telephone channels, plain-language and multilingual notices, revocable consent, explicit thresholds for human review, and published error and exclusion rates, so that oversight can be contested rather than merely promised. Prior work cautions that human oversight requirements can legitimize flawed systems without offering meaningful protection [25], and that transparency instruments can provide visibility without contestability [12, 42].

Fourth, community leadership and institutional safeguards require control rather than consultation alone, as Indigenous data governance principles and international guidance on AI ethics emphasize [8, 20, 66]. Community-led AI initiatives should provide multi-year funding controlled by communities and protect data sovereignty, whereas procurement and online-safety rules should constrain dominant firms and enforcers through open standards, independent testing, and appeal mechanisms.

Limitations and Future Work

This study has several limitations. First, the personas are synthetic. While simulations of this kind can help policymakers anticipate where delivery may break down, they cannot replace consultation with Indigenous communities, displaced workers, or people without digital access, whose own accounts remain necessary to validate and prioritize the conditions the agents raised. The generative agents’ responses should be interpreted as Canadian public’s survey-grounded simulated reasoning. This caveat applies with particular force to the skill gradients: because the persona descriptions stated characteristics such as digital technology use explicitly, the gradients may partly reflect the model’s assumptions about these labels. The gradients are also descriptive, rely on skill use rather than assessed proficiency, and rest in places on small groups, such as the personas who had never used a digital device. Appendix § 7 draws on PIAAC’s assessed proficiency directly, but only to examine how adults rate their own skills, not how they evaluate the policies. Second, the rating items were narrowly framed: effectiveness concerned adult AI literacy and benefit concerned the individual persona, which shaped the size of the observed gaps. Third, the dialogue corpus is uneven, with three policies accounting for more than 90% of conversations and a few low-rating personas recurring across many pairings. Fourth, the dialogues’ cooperative openings are consistent with LLMs’ documented sycophancy [59] and tendency toward consensus [9], and the instruction to add new perspectives likely further encouraged agreement and elaboration, so the conversations may understate genuine disagreement. Finally, the LLooM concepts were generated automatically and served only as probes for interpretation.

Future work should compare persona-grounded evaluations with human responses to the same policies, including from adults with low digital skills, validate inferred opinion change against manually annotated dialogue turns, and test whether less cooperative dialogue settings produce genuine disagreement. Extending the design to other models (e.g., open source) and technical capacities (e.g., retrieval augmented generation on community town hall transcripts) and jurisdictions (e.g., provincial) would show whether the concept of conditional endorsement generalizes.

References

Self-Assessed Skill and Over-Confidence

This appendix reports a complementary analysis of the PIAAC Cycle 2 data that asks which Canadian adults rate their skill above what their measured proficiency supports. Critical evaluation of AI output is a core AI literacy competency [40], self-efficacy is part of AI literacy in its own right [7], and people who trust an automated system too readily tend to stop checking its output [51]. PIAAC does not ask about AI, but it asks respondents to judge their performance on the assessment they have just completed, alongside a direct measure of that performance. We use over-confidence about one’s own assessment performance as a proxy for the risk of over-trusting AI, not as a measure of it, and the analysis is descriptive throughout. The design adapts that of Khan and colleagues [34], who studied the gap between subjective and objective financial knowledge among older Canadians with nested survey-weighted regressions and predicted profiles for recognizable groups. We keep the nested models and the profiles but do not use their difference score, which correlates mechanically with the objective measure through regression to the mean [22].

Analytic Sample

The analysis uses the Canadian sample of the PIAAC Cycle 2 public-use file [49], 11,697 respondents in all. Rather than fixing one analytic sample up front, we start from the full sample and add exclusions only as each model requires them. Dropping respondents routed to the doorstep interview, who never saw the self-assessment question, and those never administered the post-assessment Effort module leaves 10,455 adults (Table 2). Model 1 drops those missing any of its covariates, leaving 9,883 (Table 6), and Model 2 drops respondents without work-context data or a scored occupation, leaving 6,825 (Table 7). The adults who never reached the self-assessment item sit well below the retained sample on proficiency (Table 13), so the results describe adults who reached the item, not the full population.

Stepnn remainingExcluded
All records11,697
Not a doorstep interview11,60790
Completed all three cognitive domains11,6070
Administered the Effort module10,4551,152
Answered the self-assessment question10,4550
Analytic sample10,4551,242

Table 2. Derivation of the self-efficacy analytic sample.

Stepnn remainingExcluded
All records10,455
ICT use at home reported10,35699
Conscientiousness scale derived10,32234
Openness scale derived10,222100
Emotional stability scale derived10,21012
Extraversion scale derived10,19614
Agreeableness scale derived10,17422
Self-rated health recorded10,1713
Immigrant generation recorded10,11556
Engagement with the assessment reported9,883232
Age band recorded9,8830
Gender recorded9,8830
Education recorded (EDCAT6_TC1)9,8830
Analytic sample9,883572

Table 6. Model 1 exclusion ledger, with covariate completeness layered on the analytic sample.

Stepnn remainingExcluded
All records9,883
ICT use at work reported7,4582,425
Occupation in a scored ISCO major group6,840618
Task discretion reported6,8382
Learning at work reported6,82513
Analytic sample6,8253,058

Table 7. Model 2 exclusion ledger, layered on Model 1’s retained sample so each row shows only what the new rule drops.

GroupEstimateSE
Dropped233.33.0
Retained273.90.7

Table 13. Measured proficiency of respondents dropped by Effort-module non-administration versus those retained. The sample is positively selected on ability.

Measures and Models

Estimation. Every quantity in this appendix is estimated separately on each of the ten plausible values and pooled by Rubin’s rules, with standard errors from the 80 Fay-adjusted replicate weights (VEMETHODN = 4, VEFA FYFAC = 0.3, c≈0.02551c \approx0.02551 for Canada in Cycle 2).

Self-assessment and proficiency. The background questionnaire asks each respondent to place the share of assessment items they answered correctly in one of five ordered bands (EffortP001S, from 20% or less to 81–100%). The public-use file carries no per-item correctness, so we examine relative calibration: whether a respondent rates their performance higher than their measured proficiency would lead one to expect. Because the item refers to the assessment as a whole, the benchmark is a General Cognitive Proficiency Index (GCPI), the mean of the literacy, numeracy, and adaptive problem solving scores, built and standardized once per plausible value and never averaged into a single score.

Nest­ed models. Rather than differencing the self-estimate against proficiency, we model the self-assessment band as a function of GCPI and covariates with an ordered (proportional-odds) logit. It is fit by maximum likelihood on each plausible value and each replicate weight, so standard errors come from the replicate spread and the between-plausible-value variance rather than the likelihood’s Hessian; a survey-weighted ordered logit (survey::svyolr [41]) reproduces the coefficients (Table 16). Model 0 uses GCPI alone. Model 1, the primary specification, adds assessment effort (EffortP002S) as a calibration control, self-rated health, ICT use at home, the five BFI-2 personality traits, age, gender, education, and immigrant generation. Model 2 restricts to employed adults with a scored occupation and adds occupational AI exposure, ICT use at work, task discretion, and learning at work. Coefficients are log-odds per standard deviation with two-sided Wald pp-values on the design-based standard errors, and no pair of continuous predictors is collinear (Table 15).

PredictorVIFMost correlated with∣r∣|r|
GCPI1.12Assessment effort0.19
ICT use at home1.06Extraversion0.15
Conscientiousness1.35Emotional stability0.35
Openness1.08Extraversion0.17
Emotional stability1.26Conscientiousness0.35
Extraversion1.24Conscientiousness0.34
Agreeableness1.20Conscientiousness0.33
Assessment effort1.07GCPI0.19

Table 15. Variance-inflation factors and the largest absolute weighted correlation for each continuous predictor. No pair is collinear.

TermCustom est.Custom SEsvyolr est.svyolr SE
GCPI1.2710.0651.2710.063
ICT use at home0.0530.0390.0530.039
Conscientiousness0.0520.0370.0520.037
Openness0.0790.0380.0790.038
Emotional stability0.0650.0400.0650.039
Extraversion0.1330.0410.1330.041
Agreeableness−0.0700.034−0.0700.034
Assessment effort0.8300.0510.8300.050
Health: very good−0.1380.119−0.1380.119
Health: good−0.3530.129−0.3530.128
Health: fair−0.2120.174−0.2120.174
Health: poor−0.3240.261−0.3240.259
Age 24 or less0.3830.1250.3830.119
Age 25–34−0.0590.106−0.0590.105
Age 45–54−0.1620.094−0.1620.093
Age 55+−0.0890.118−0.0890.117
Female−0.7520.072−0.7520.071
Lower secondary or less−0.1120.140−0.1120.140
Post-secondary, non-tertiary−0.0560.101−0.0560.099
Tertiary, professional0.0880.1020.0880.102
Tertiary, bachelor0.4990.1280.4990.128
Tertiary, master/research0.5360.1510.5360.151
1st-generation immigrant0.0590.1090.0590.108
2nd-generation immigrant−0.0440.133−0.0440.132
One foreign-born parent0.1070.1540.1070.154

Table 16. Cross-check of the custom replicate engine against survey::svyolr on one plausible value. Point estimates match and standard errors agree closely.

Occupational AI exposure. We use the AI Occupational Exposure index of Felten and colleagues [19], which scores 774 US occupations of the 2010 Standard Occupational Classification (SOC) on how closely their required abilities relate to ten common AI applications. The Canadian public-use file reports occupation only as the ISCO-08 major group (ISCO1C), so each score is carried from SOC to ISCO-08 unit groups through the crosswalk of Hardy and colleagues [27] and averaged up to the nine major groups, with a SOC occupation that maps to several unit groups contributing an equal share to each. Scores carried through the US Bureau of Labor Statistics crosswalk [67] differ by at most 0.005 on any major group. Armed forces occupations have no score, and the major-group score is standardized within the Model 2 sample.

Self-Assessment and Measured Proficiency

Most adults rate themselves favorably: 69% place themselves in the top two bands, claiming at least three-fifths of the assessment correct (Figure 5). Confidence tracks ability, with a one-standard-deviation increase in GCPI raising the log-odds of a higher band by 1.38 in Model 0 (Table 3), but the relationship is loose. Mean GCPI rises across the bands, yet adults of similar proficiency place themselves across several adjacent bands and the interquartile ranges overlap heavily (Figure 6).

TermEstimateSEpp-value
GCPI1.3830.057<0.001

p-value: two-sided Wald test (estimate / SE) against the standard normal.

Table 3. Model 0 regression output, an ordered logit of self-assessment on GCPI alone (log-odds per SD).

Weighted distribution of self-assessed performance bands

Figure 5. Weighted distribution of self-assessed performance bands, with 95% confidence intervals.

Pooled GCPI proficiency by self-assessed performance band

Figure 6. Pooled GCPI proficiency by self-assessed performance band, with 95% confidence intervals (points) and the 25th–75th percentile range (shaded).

In Model 1, proficiency remains dominant (1.28 log-odds per SD; Table 4). The clearest association beyond proficiency is assessment effort (0.83), and women rate themselves well below men at equal proficiency (−0.76). ICT use at home, conscientiousness, openness, and emotional stability are near zero, and only extraversion reaches conventional significance, modestly. Net of proficiency, self-assessment moves with engagement and gender rather than digital exposure or personality. The GCPI slope is similar at each band threshold (Table 17), and the fitted model reproduces the observed band distribution to within about a percentage point (Table 18). Among employed adults, none of the work-context terms in Model 2 adds much (Table 8): occupational AI exposure is small and, if anything, negative (−0.095, SE 0.065), though exposure enters only at the major-group level.

TermEstimateSEpp-value
GCPI1.2780.062<0.001
ICT use at home0.0570.0410.164
Conscientiousness0.0550.0380.145
Openness0.0850.0390.028
Emotional stability0.0640.0410.116
Extraversion0.1250.0410.003
Agreeableness−0.0650.0350.064
Assessment effort0.8310.049<0.001
Health: very good−0.1410.1170.229
Health: good−0.3610.1300.005
Health: fair−0.2400.1780.176
Health: poor−0.3530.2460.152
Age 24 or less0.3740.1300.004
Age 25–34−0.0570.1060.594
Age 45–54−0.1570.0990.114
Age 55+−0.1020.1210.403
Female−0.7560.069<0.001
Lower secondary or less−0.0860.1450.554
Post-secondary, non-tertiary−0.0380.0990.701
Tertiary, professional0.1000.0960.300
Tertiary, bachelor0.4920.126<0.001
Tertiary, master/research0.5310.152<0.001
1st-generation immigrant0.0640.1040.540
2nd-generation immigrant−0.0420.1360.755
One foreign-born parent0.1250.1530.414

Table 4. Model 1 regression output with all ordered-logit coefficients, in log-odds per SD for continuous predictors and versus the omitted reference for factors (age 35–44, upper secondary education, non-immigrant, male, excellent health). p-value: two-sided Wald test (estimate / SE) against the standard normal.

TermEstimateSEpp-value
GCPI1.3520.074<0.001
ICT use at home0.0280.0540.602
Conscientiousness0.0320.0530.546
Openness0.0300.0510.551
Emotional stability0.1080.0590.064
Extraversion0.1390.0520.007
Agreeableness−0.053-0.0530.0480.267
Assessment effort0.8660.059<0.001
Health: very good−0.056-0.0560.1330.677
Health: good−0.202-0.2020.1580.201
Health: fair−0.158-0.1580.2060.442
Health: poor0.0330.3090.916
Age 24 or less0.2450.1610.129
Age 25–34−0.102-0.1020.1240.408
Age 45–54−0.202-0.2020.1090.064
Age 55+−0.139-0.1390.1600.386
Female−0.721-0.7210.074<0.001
Lower secondary or less−0.219-0.2190.2050.287
Post-secondary, non-tertiary−0.043-0.0430.1410.762
Tertiary, professional0.0200.1180.863
Tertiary, bachelor0.5230.147<0.001
Tertiary, master/research0.5210.1660.002
1st-generation immigrant0.0270.1290.832
2nd-generation immigrant−0.064-0.0640.1750.716
One foreign-born parent0.1570.1760.370
ICT use at work0.0560.0630.374
Occupational AI exposure−0.095-0.0950.0650.142
Task discretion−0.007-0.0070.0370.847
Learning at work−0.020-0.0200.0610.741

Table 8. Model 2 regression output, adding the work-context terms to Model 1 on the employed subsample (log-odds per SD). Factor references as in Model 1. pp-value: two-sided Wald test (estimate / SE) against the standard normal.

ThresholdEstimateSE
Band ≥2\ge21.2490.091
Band ≥3\ge31.2930.069
Band ≥4\ge41.4000.066
Band ≥5\ge51.4700.081

Table 17. The GCPI slope estimated separately at each band threshold, a check on the proportional-odds assumption. The slopes stay close, so the single ordered-logit summary is fair.

BandObservedSEPredictedSE
13.90.33.40.3
27.30.46.70.4
319.70.718.80.7
434.60.935.50.9
534.50.835.60.9

Table 18. Predicted versus observed weighted share (%) in each self-assessment band, Model 1.

Who Over-Rates Their Skill

To identify who over-rates their skill, we do not use each adult’s Model 1 residual, for two reasons. Because Model 1 adjusts for gender, age, and education, those characteristics shift each adult’s benchmark rather than their measured over-confidence, so profiling the residual on them largely restates the model. The five-band scale also caps the residual, leaving the most room to overshoot to adults the model expects to rate themselves low. Instead, the benchmark is proficiency alone, modeled as an ordered logit of the self-assessment band on a natural spline of GCPI, and the comparison is made on the model’s latent scale through the surrogate residual of Liu and Zhang [39], which has no ceiling. An adult counts as over-confident when their residual falls in the top tenth of its distribution under the model; since a reported band pins the latent value only to an interval, each adult receives the exact probability of falling in that tail. Roughly one adult in ten is over-confident by design, and the share is flat across proficiency, between 9.8% and 10.7% in every GCPI quintile (Table 9).

GCPI quintileEstimateSE
Quintile 110.61.4
Quintile 210.71.3
Quintile 39.80.8
Quintile 410.10.5
Quintile 59.90.3

Table 9. Share of adults over-confident relative to their measured skill, by GCPI quintile (%).

A weighted fractional logit [50] then relates each adult’s over-confidence probability to proficiency and the Model 1 covariates (Table 11). Men are over-confident more often than women (12.1% against 8.3%; Table 10), and education raises over-confidence even with proficiency held fixed: a bachelor’s degree adds 0.40 log-odds while each standard deviation of proficiency subtracts 0.22. Adults under 25 over-rate more than those aged 35 to 44 (0.38) once other characteristics are held fixed. Assessment effort is the strongest predictor, from 4.3% among adults reporting the least effort to 15.6% among those reporting the most, which likely reflects respondents reading their effort as success, so we treat it as a control rather than a target.

CharacteristicGroupEstimateSE
GenderMen12.10.6
Women8.30.4
Age24 or less9.90.9
25–349.30.7
35–4410.70.7
45–5410.31.2
55 and over10.80.7
EducationLower secondary or less8.11.0
Upper secondary9.50.9
Post-secondary, non-tertiary9.20.7
Tertiary, professional11.00.8
Bachelor’s11.60.8
Master’s or research11.61.0
Immigrant generationNon-immigrant9.50.5
One foreign-born parent10.21.0
2nd-generation immigrant8.71.1
1st-generation immigrant12.10.8
Self-rated healthExcellent12.20.9
Very good10.70.7
Good8.80.6
Fair10.41.3
Poor5.71.3
Assessment effort (1–10)1–64.30.5
7–87.80.4
9–1015.60.8

Table 10. Share of adults over-confident relative to their measured skill, by characteristic (%).

TermEstimateSEpp-value
GCPI−0.2240.052<0.001
ICT use at home0.0260.0410.522
Conscientiousness0.0600.0380.113
Openness0.0030.0340.933
Emotional stability0.0050.0470.909
Extraversion0.0470.0400.249
Agreeableness−0.0260.0390.500
Assessment effort0.6220.053<0.001
Health: very good−0.1280.1110.246
Health: good−0.3150.1230.011
Health: fair−0.0460.1550.767
Health: poor−0.4630.2620.076
Age 24 or less0.3820.1310.003
Age 25–34−0.0050.1110.962
Age 45–54−0.0620.1200.605
Age 55+−0.0180.1040.862
Female−0.4810.081<0.001
Lower secondary or less−0.2690.1540.080
Post-secondary, non-tertiary0.0330.1460.819
Tertiary, professional0.1840.1350.172
Tertiary, bachelor0.3980.1310.002
Tertiary, master/research0.3880.1330.004
1st-generation immigrant0.0830.0900.353
2nd-generation immigrant−0.0640.1280.618
One foreign-born parent0.0530.1420.706

Table 11. Persona model, a fractional logit of the over-confidence probability on measured proficiency and the Model 1 covariates, in log-odds per SD for continuous predictors and versus the Model 1 reference levels for factors.

Following Khan and colleagues [34], we fix the characteristics a program can target (age, gender, and education) and average the predicted share over everything else (Table 5). Men with a bachelor’s degree are over-confident at about 14.6% whether aged 25 to 34 or 55 and over, as are men under 25 with upper secondary schooling, while women aged 55 and over with lower secondary schooling or less are the least likely, at 5.3%. This reverses the education gradient Khan and colleagues found for financial knowledge, where the over-confident profile was the older man with a high school education or less, so groups to target for AI calibration cannot be read off from financial literacy work.

Robustness and Limitations

The Model 1 proficiency coefficient is similar whether the benchmark is GCPI or literacy, numeracy, or adaptive problem solving alone (Table 12), and fitting Model 1 as a linear regression of the band score reproduces the ordered logit’s signs and significance (Table 19).

Proficiency benchmarkEstimateSE
Composite (GCPI)1.2780.062
Literacy1.1070.062
Numeracy1.1980.056
Adaptive problem solving1.1080.054

Table 12. The Model 1 proficiency coefficient under each benchmark, the composite GCPI versus each domain alone in its place.

TermOrdered logitLinear
EstimateSEEstimateSE
ICT use at home0.0570.0410.0260.016
Conscientiousness0.0550.0380.0210.015
Openness0.0850.0390.0280.015
Emotional stability0.0640.0410.0120.015
Extraversion0.1250.0410.0520.016
Agreeableness−0.0650.035−0.0250.013
Assessment effort0.8310.0490.3440.021
Health: very good−0.1410.117−0.0370.040
Health: good−0.3610.130−0.1160.050
Health: fair−0.2400.178−0.0850.063
Health: poor−0.3530.246−0.1770.107
Age 24 or less0.3740.1300.1770.050
Age 25–34−0.0570.106−0.0080.038
Age 45–54−0.1570.099−0.0540.039
Age 55+−0.1020.121−0.0520.046
Female−0.7560.069−0.2830.025
Lower secondary or less−0.0860.145−0.0550.061
Post-secondary, non-tertiary−0.0380.0990.0090.038
Tertiary, professional0.1000.0960.0700.037
Tertiary, bachelor0.4920.1260.1750.046
Tertiary, master/research0.5310.1520.2040.053
1st-generation immigrant0.0640.1040.0100.037
2nd-generation immigrant−0.0420.136−0.0060.053
One foreign-born parent0.1250.1530.0530.055

Table 19. Model 1 as an ordered logit (log-odds) and as a linear regression of the band score (bands), side by side.

Four limitations apply. First, plausible values are generated for proficiency as an outcome, and using them as a predictor attenuates the proficiency–confidence slope, most for the roughly one-third of the sample whose score in a given domain is imputed (Table 14). Second, occupational AI exposure is available only at the ISCO major group, whose occupations span as much as 4.2 standard deviations of exposure, and the crosswalk assumes that US exposure carries over to Canadian jobs. Third, PIAAC measures neither AI use nor verification behavior, so the link from miscalibration to AI over-trust is a theoretically motivated proxy. Fourth, the self-assessment item was not reached by doorstep respondents, non-completers, and adults routed through the easier adaptive paths, so the sample is positively selected on ability (Table 13).

DomainEstimateSE
Literacy0.6540.010
Numeracy0.6780.008
Adaptive problem solving0.6520.009

Table 14. Share of the analytic sample administered each cognitive domain directly, rather than by imputation from the background questionnaire.

Supplementary Tables

pp-value: two-sided Wald test (estimate / SE) against the standard normal.

Methodological Details

Identifying Policies and Landscape Challenges

We identified passages in Canada’s National Artificial Intelligence Strategy: AI for All that described government actions, including regulatory reforms, investments, programs, institutional changes, partnerships, and measurable targets. We treated a passage as a policy when the document specified an action that the government had taken, had committed to take, or planned to pursue; general statements about AI’s potential benefits were not coded as policies. We preserved distinctions among existing initiatives, newly announced measures, and future commitments, and recorded their page locations. We then iteratively consolidated overlapping actions into broader policy areas, while retaining named programs as separate items where appropriate.

We also analyzed the document for references to risks, harms, barriers, gaps, dependencies, vulnerabilities, and resource constraints. We grouped these challenges into recurring areas: public trust and AI literacy, employment disruption, privacy and online harms, unequal access, business adoption, foreign technological dependence, infrastructure and environmental constraints, fragmented data, and difficulties in retaining talent and commercializing Canadian AI research. We used an LLM to paraphrase the 20 policies and nine challenges into self-contained descriptions and stored them in separate indexed corpora. Tables 20 and 21 list the resulting English descriptions; we also prepared French translations of both.

PolicyDescription as presented to agents
Consumer privacy reformThe Government of Canada announces plans to modernize consumer privacy legislation. The proposed changes would recognize privacy as a fundamental right, strengthen people’s control over their personal information, and protect children’s information from exploitation. The government also plans to address inappropriate uses of personal data, including surveillance pricing.
Online safety legislationThe Government of Canada announces plans to introduce online safety laws addressing harms associated with digital technologies and AI. Its stated priorities include protecting children, providing legal tools to address deepfakes, making chatbot interactions safer, and holding those responsible for online harms accountable. It also commits to protecting elections against AI-enabled misinformation and foreign interference.
AI safety evaluation and certificationThe Government of Canada announces plans to create a Canada Trusted AI Certification program intended to help people identify trustworthy AI products. It also commits CAD 50 million to expand the Canadian AI Safety Institute’s work on emerging risks, technical research, and transparent evaluations of AI models.
Identifying AI systems and generated contentThe Government of Canada announces that it will work on measures to make AI use more transparent, including capabilities such as watermarking AI-generated content. The stated aim is to help people recognize when they are interacting with an AI system or encountering AI-generated material.

Table 20. The 20 policies identified in AI for All.

PolicyDescription as presented to agents
Free introductory AI educationThe Government of Canada announces plans for a National AI Literacy Initiative offering free introductory AI education accessible to all Canadians. Public libraries and community organizations would help deliver learning opportunities, including in rural, remote, and northern communities. The initiative would cover practical AI use as well as recognizing bias, misinformation, privacy risks, and unsafe uses.
AI tools for postsecondary studentsThe Government of Canada announces a commitment to give all postsecondary students access to trusted AI agents. It also plans to bring AI literacy content to one million students entering postsecondary education and provide classroom AI learning resources and training to more than 3,000 educators.
AI-related employment and trainingThe Government of Canada announces a plan to support up to 90,000 AI-related employment and work-placement opportunities by 2031 through student employment, skills, and Mitacs programs. It also plans to assess and expand employer-led AI training for workers in the middle of their careers, including people in skilled trades, working with employers, unions, colleges, and other training providers.
Indigenous leadership in AIThe Government of Canada announces plans to support Indigenous-led AI initiatives that strengthen Indigenous languages and cultural expression. It commits to working with Indigenous communities on their ability to shape how AI is developed and used in Indigenous contexts, including applications involving language, land management, and cultural heritage.
AI support for Canadian creatorsThe Government of Canada announces plans to establish a CAD 50 million Creative Technology Program. The program is intended to support Canadian creators in using AI on their own terms, as part of a wider commitment to represent Canadian voices, languages, and cultural knowledge in AI development and use.
English and French in government AIThe Government of Canada announces a commitment that AI systems deployed within government will perform equally well in English and French. It also plans to support the adoption of AI tools that protect and promote French, including its use in Quebec and Francophone minority communities elsewhere in Canada.
Accessibility and inclusion in AIThe Government of Canada announces plans to promote a national standard on accessible and equitable AI and to address accessibility barriers in AI systems. It also commits to applying Gender-Based Analysis Plus across AI policy, skills development, innovation, and governance, including attention to groups that experience disproportionate harms or barriers.
Support for small-business AI adoptionThe Government of Canada announces plans to use the Business Development Bank of Canada’s CAD 500 million LIFT program to help small and medium-sized businesses finance AI adoption. It also commits CAD 500 million to expand the Regional Artificial Intelligence Initiative, alongside tools and advice to help businesses identify uses for AI. The government sets a target for 60 percent of Canadian businesses to have adopted AI by 2034.

Table 1 (PDF p. 32).

PolicyDescription as presented to agents
AI mission focused on healthcareThe Government of Canada announces plans for an AI Missions Program, with CAD 200 million committed to its first mission on improving health outcomes. The mission would bring together researchers, Canadian AI companies, governments, and healthcare practitioners. The government’s stated goals include improving access to primary care, reducing emergency-room waits, preventing avoidable visits, and reducing administrative work for physicians.
Health-data platforms for research and AIThe Government of Canada announces plans to invest CAD 100 million in a Health Sector Data Space with the Canadian Institute for Health Information, linking secure, standardized health datasets for research and performance measurement. It also commits CAD 100 million to expand VITAL into five additional provinces, supporting the use of hospital clinical data for health research and AI development. The strategy commits to strong privacy protections for this data infrastructure.
AI adoption in federal public servicesThe Government of Canada announces plans to accelerate the procurement and deployment of AI in federal public services through the Office of Digital Transformation. Its stated aims include faster application processing and reduced administrative work. The government commits to transparency, privacy, and accountability standards, with human oversight of meaningful decisions and tasks, and plans a Prime Minister’s Innovation Fellows Program to recruit technical expertise.
Canadian-controlled AI infrastructureThe Government of Canada announces plans to expand Canadian-controlled AI infrastructure, including a public supercomputer by 2031, domestic cloud services, and data centres. It will prioritize building key capabilities in Canada, while also partnering with trusted allies or purchasing existing solutions where appropriate. The strategy connects new data-centre development with clean-energy expansion, environmental standards, and benefits for local communities.
AI research and talent recruitmentThe Government of Canada announces plans to expand the Canada CIFAR AI Chairs program from 130 to nearly 200 researchers and strengthen Canada’s national AI institutes. It also plans to expand the Global Talent Stream to accelerate the arrival and onboarding of highly skilled AI workers and align permanent-residency measures with retaining recruited talent.
Financing and commercialization for Canadian AI companiesThe Government of Canada announces plans for a CAD 500 million Canadian Tech Growth Fund to support the expansion of Canadian AI companies, including possible government equity investments. It also commits an additional CAD 700 million through the Compute Access Fund and CAD 130 million for commercialization programs at national AI institutes. Federal purchasing under the Buy Canadian policy would help provide customers for domestic firms.
International cooperation on sovereign AIThe Government of Canada announces plans to expand the Sovereign Technology Alliance launched with Germany. The alliance would coordinate work on AI models, digital infrastructure, research, safety evaluations, and procurement opportunities among participating countries. The government describes the alliance as a way to increase technological choice and reduce dependence on dominant technology ecosystems.

Table 24.

PolicyDescription as presented to agents
Support for open-source AIThe Government of Canada announces plans to work with other countries, researchers, civil society, industry, and AI institutes to sustain open-source AI development in the public interest. It also plans an inventory and shared library of AI tools to support adoption by researchers, small and medium-sized businesses, nonprofit organizations, and public-interest innovators.

Table 25.

ChallengeDescription as presented to agents
Low public trust and AI literacyMany Canadians have limited AI understanding, training, or confidence. The strategy reports that 24% of Canadians have received AI training.
Employment disruptionWorkers may need to reskill as jobs change, entry-level roles contract, and some female-dominated sectors face particular exposure.
Privacy, discrimination, and online harmsRisks include misuse of personal information, algorithmic discrimination, deepfake sexual violence, fraud, misinformation, and threats to democratic institutions.
Unequal access and disproportionate harmsIndigenous peoples, racialized communities, and people with disabilities may face distinct barriers or harms, alongside cultural and linguistic representation concerns.
Business adoption barriersBusinesses face costs, limited expertise, and uncertainty about useful applications. The strategy reports that 78% of non-adopting firms do not see how AI benefits their goods or services.
Dependence on foreign technologyReliance on foreign cloud providers, chips, infrastructure, and models raises concerns about control, foreign legal regimes, vendor lock-in, and technological choice.
Infrastructure and environmental constraintsAI expansion requires substantial computing, electricity, and potentially water, while available power can constrain infrastructure growth.
Fragmented and inaccessible dataUseful datasets remain isolated across institutions and require common standards, secure infrastructure, and privacy protections.
Talent retention and commercializationCanada faces competition for researchers, limited growth capital, difficulties translating research into products, and relocation of firms abroad.

Table 21. The nine landscape challenges identified in AI for All.

PIAAC Data Preparation

We combined the raw PIAAC Cycle 2 public-use survey data with the international codebook to produce a readable Excel dataset. We mapped categorical response codes to their corresponding labels and did not impute missing-value codes. Using the final full-sample weight, we then drew a weighted sample of 3,200 records.

Persona Construction

Each persona drew on the sampled respondent’s demographic background, language, education, employment, occupation, household and economic context, digital technology use, learning and training experiences, personality, social trust, health, and calibration measures. We converted categorical variables into labeled descriptions. For some numeric variables, such as years of education (YRSQUALC2), we retained exact values. We converted behavioral, workplace, and personality measures into lower, typical, or higher categories using the weighted sample’s 25th and 75th percentiles. These measures included digital technology use at home (ICTHOMEC2), reading and numeracy at home (READHOMEC2_T1, NUMHOMEC2), learning at work (LEARNATWORKC2), and openness (OPEM, OPEM_CURI). We organized the variables into thematic sections, namely demography, education, employment, digital skills, learning and training, personality, and social and health characteristics. Each persona received a stable identifier and was stored in a JSON file (Figure 7).

Example persona constructed from PIAAC responses

Figure 7. Example persona constructed from PIAAC responses (PIAAC-1754, quoted in Section ??). Fields absent from a respondent’s record, such as income, are omitted.

PIAAC-1754

Demography: age: 55 plus; gender: Male; immigration background: Non 1st or 2nd generation immigrants; questionnaire language: ENG; language alignment: Test language same as native language.

Education: educational attainment: ISCED 2.

Employment: employment status: Out of the labour force; occupation: No paid work for past 5 years.

Household and economic context: household: not living with spouse or partner.

Core skills: reading use at home: lower relative to the sample; numeracy use at home: lower relative to the sample.

Digital skills: ever used a digital device: No.

Learning and training: formal education in the last 12 months: Did not participate in FE; non-formal education in the last 12 months: Did not participate in NFE.

Personality: agreeableness: lower relative to the sample; conscientiousness: lower relative to the sample; emotional stability: lower relative to the sample; extraversion: lower relative to the sample; open-mindedness: lower relative to the sample; intellectual curiosity: lower relative to the sample; anxiety (weak general-affect context): typical relative to the sample.

Social and health: social trust (0–10): You can’t be too careful; health: Good.

We assigned each persona a randomly selected subset of the nine landscape challenges (Figure 8): low public trust and AI literacy; employment disruption; privacy, discrimination, and online harms; unequal access and disproportionate harms; business adoption barriers; dependence on foreign technology; infrastructure and environmental constraints; fragmented and inaccessible data; and talent retention and commercialization. The number of assigned challenges ranged from zero to nine. We provided these challenges to the agent alongside the persona description and the policy prompt as context that structured, rather than determined, the agent’s response.

Examples of the landscape challenges assigned to different agents

Figure 8. Examples of the landscape challenges assigned to different agents.

AgentAssigned landscape challenges
PIAAC-4362Low public trust and AI literacy
PIAAC-1754Low public trust and AI literacy; Unequal access and disproportionate harms; Infrastructure and environmental constraints; Talent retention and commercialization; Privacy, discrimination, and online harms
PIAAC-10565Fragmented and inaccessible data; Privacy, discrimination, and online harms; Dependence on foreign technology; Unequal access and disproportionate harms; Infrastructure and environmental constraints; Talent retention and commercialization; Low public trust and AI literacy; Employment disruption

Prompt Design for Single-Agent Evaluation

For each rating, the agent provided a brief justification and selected a value from 1 to 6. Lower values indicated opposition, low effectiveness, or limited benefit, and higher values indicated support, effectiveness, or substantial benefit. Because the rating questions appeared in a randomized order, we stored the realized order so that each response could be mapped back to the correct rating dimension.

The agents were given the web search skill, which allowed them to support externally verifiable factual claims with sources. The prompt also instructed each agent to report its information effort as none, one source, or several sources. This field was intended to reflect whether a person with the persona’s background, skills, and temperament would realistically seek information about a government policy before forming an opinion. The agents were also instructed to avoid personal details that are not in their PIAAC-derived descriptions. For example, a persona with lower reported digital technology use might reasonably express concern about access to AI training. However, the agent should not turn that concern into an invented fact, such as claiming that the person had never used a computer or had been excluded from a particular service.

Dyad Selection

For each policy, the number of possible dyads was the product of the numbers of lower- and higher-rating agents. The 11 policies without possible dyads lacked either agents with consistently low ratings or agents with consistently high ratings. The distance DD identified the maximum rating distance and the total number of possible contrasting dyads for each policy. After filtering out mixed-language pairs, we ranked the dyads by DD, i.e., the extent of their quantified disagreement.

In the first step of the coverage-first procedure, we selected up to 10 high-distance dyads per eligible policy in a round-robin sequence, to ensure that policies with relatively few possible pairs were represented. In the second step, we filled the remaining sample with the dyads that had the highest DD across all policies. The 1,533 conversations covered nine of the 20 policies, each with five turns per speaker (15,330 agent turns in total). We verified for every conversation that one member had rated all three dimensions 1–3 and the other had rated all three 4–6 in the single-agent responses. The corpus is uneven: support for Canadian creators (933 conversations), small-business AI adoption (384), and postsecondary AI tools (131) account for 1,448 conversations (94.5%).

Rating Prediction Model

We paired each single-agent response with its support, perceived-effectiveness, and perceived-benefit ratings to form a labeled training dataset. We evaluated the fine-tuned XLM-RoBERTa model [11] on held-out response–rating pairs using exact-match accuracy, macro-F1, mean absolute error, and quadratic weighted kappa [10]. Because the ratings are ordinal, mean absolute error and weighted kappa distinguished small, adjacent-rating errors from larger disagreements. We also compared predicted and observed rating distributions using the Wasserstein distance [68]. On the held-out pairs, the model achieved exact-match accuracy of 0.92, 0.86, and 0.87 for support, effectiveness, and benefit, respectively; quadratic weighted kappa of 0.84, 0.90, and 0.93; and mean absolute error of 0.08, 0.14, and 0.13. For comparison, a TF-IDF logistic-regression classifier trained on the same split reached accuracy of 0.84, 0.80, and 0.83.

Dyad Corpus and Inferred Opinion Change

Since some policies had more representation in our dyad sample, counts of conversational themes are not equally sampled across policy domains, and a few low-rating personas recur in many pairings.

Because no conversation asked agents to report a new numerical rating, we inferred change from each speaker’s first and last turns (3,066 speaker trajectories; Table 22). The TF-IDF logistic-regression classifier classified support as unchanged for 98.4% of trajectories, whereas the fine-tuned XLM-RoBERTa model did so for 86.2%. An LLM annotation of a 415-conversation subset (830 trajectories), which coded a single overall stance rather than the three ratings, instead marked 576 trajectories as higher and only 235 as unchanged. Both full-corpus models inferred more upward than downward movement on literacy effectiveness and direct benefit, but the sizes of these changes varied considerably, and we evaluated the models on held-out single-agent responses rather than on manually rated dialogue turns. Both models also assigned every first conversational turn an upper-half support rating, including turns from agents selected for lower-half support, and therefore did not recover the designed contrast.

MethodSupportEffectivenessBenefit
SameHigherLowerHigherLowerHigherLower
TF-IDF logistic regression98.40.90.69.21.348.712.0
XLM-RoBERTa86.212.90.830.64.836.28.3

Table 22. Inferred opinion change between first and last turns (percentages of speaker trajectories).

LLooM Analysis

Following social computing research guidelines [43], we did not calculate inter-rater reliability for the automatically generated concepts and instead used them as prompts for interpretation. Concepts that emerged for the single-agent responses included access, training, trust, privacy, safety, equity, and institutional capacity. The qualitative analysis aimed to identify recurring arguments, concerns, implementation conditions, and interaction patterns that the rating analysis could not capture.

For the single-agent responses, the analyzed texts captured how agents interpreted each policy, why they supported or opposed it, how effective they expected it to be, and whether they believed it would benefit them directly. For the dyads, we identified sources of disagreement, persistent concerns, shared positions, and qualifications, and the analysis also examined how subsequent turns accepted, challenged, qualified, or extended earlier points, and identified counterarguments and other concepts related to apparent movement toward or away from the other agent’s position. In doing so, it treated implementation measures and proposed solutions as products of interaction, rather than as isolated statements.

Visualization and Reproducibility

We combined the conversation-level annotations with the original transcripts to produce a publicly accessible, interactive HTML viewer5 (see Figure 9). The viewer allows users to select a policy and a dyad, move through the conversation turn by turn, and inspect the position associated with each response, with supportive and opposing positions displayed in distinct colors. All randomized steps, including weighted sampling, challenge assignment, and response sampling, used fixed random seeds. We retained intermediate files (persona descriptions, policy responses, parsed workbooks, concept tables, sampled responses, annotations, and analysis manifests) to support traceability from the final findings back to the underlying model outputs.

Dyad opinion viewer

Figure 9. Dyad opinion viewer: one response at a time, but both agents remain visible.

Findings Details

Additional Figures

Here are the figures that offer additional (but not mandatory) details about the findings.

Figure 11. PIAAC-persona-based agents’ perceived effectiveness of different AI policies for improving adult AI literacy. Blue bars mark the upper half (4–6).

Additional Tables

Here are the tables that offer additional (but not mandatory) details about the findings.

References

  1. [1]Gati V. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. In Proceedings of the 40th International Conference on Machine Learning (PMLR, Vol. 202).arxiv.org/abs/2208.10264
  2. [2]Jacy Reese Anthis et al. 2025. Position: LLM Social Simulations Are a Promising Research Method. In Proceedings of the 42nd International Conference on Machine Learning (PMLR, Vol. 267).arxiv.org/abs/2504.02234
  3. [3]Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis 31, 3 (2023), 337–351.DOI
  4. [4]Blair Attard-Frost, Ana Brandusescu, and Kelly Lyons. 2024. The governance of artificial intelligence in Canada: Findings and opportunities from a review of 84 AI governance initiatives. Government Information Quarterly 41, 2 (2024), 101929.
  5. [5]James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. 2024. Synthetic replacements for human survey data? The perils of large language models. Political Analysis 32, 4 (2024), 401–416.
  6. [6]Abel Brodeur, David Valenta, Alexandru Marcoci, Juan P Aparicio, Derek Mikola, Bruno Barbarioli, Rohan Alexander, Lachlan Deer, Tom Stafford, Lars Vilhuber, et al. 2025. Comparing human-only, AI-assisted, and AI-led teams on assessing research reproducibility in quantitative social science. Technical Report. 14R Discussion Paper Series.DOI
  7. [7]Astrid Carolus, Martin J Koch, Samantha Straka, Marc Erich Latoschik, and Carolin Wienrich. 2023. MAILS-Meta AI literacy scale: Development and testing of an AI literacy questionnaire based on well-founded competency models and psychological change-and meta-competencies. Computers in Human Behavior: Artificial Humans 1, 2 (2023), 100014.arxiv.org/abs/2302.09319
  8. [8]Stephanie Russo Carroll, Ibrahima Garba, Oscar L. Figueroa-Rodríguez, et al. 2020. The CARE Principles for Indigenous Data Governance. Data Science Journal 19, 1 (2020), 43. doi:10.5334/dsj-2020-043DOI
  9. [9]Yun-Shiuan Chuang, Agam Goyal, Nikunj Harlalka, Siddharth Suresh, Robert Hawkins, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. 2024. Simulating Opinion Dynamics with Networks of LLM-based Agents. In Findings of the Association for Computational Linguistics: NAACL 2024. 3326–3346.arxiv.org/abs/2311.09618
  10. [10]Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin 70, 4 (1968), 213–220.pubmed.ncbi.nlm.nih.gov/19673146
  11. [11]Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 8440–8451.arxiv.org/abs/1911.02116
  12. [12]Dipto Das, Christelle Tessono, Syed Ishtiaque Ahmed, and Shion Guha. 2026. Bureaucratic silences: What the Canadian AI register reveals, omits, and obscures. In The 2026 ACM Conference on Fairness, Accountability, and Transparency. 4003–4026.
  13. [13]Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can AI language models replace human participants? Trends in Cognitive Sciences 27, 7 (2023), 597–600.DOI
  14. [14]Isabelle Docto. 2026. “No one asked for this”: Canadians aren’t happy about Carney’s new AI plan. Daily Hive, https://dailyhive.com/canada/carney-ai-plan-reactions-canada. June 5, 2026.
  15. [15]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of the 41st International Conference on Machine Learning.arxiv.org/abs/2305.14325
  16. [16]Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2024. GPTs are GPTs: Labor market impact potential of LLMs. Science 384, 6702 (2024), 1306–1308.DOI
  17. [17]Environics Institute for Survey Research. 2025. The use of artificial intelligence in the workplace: an update. https://environicsinstitute.org/the-use-of-artificial-intelligence-in-the-workplace-an-update/. [Accessed: Oct 7, 2026].
  18. [18]Samar Fatima, Kevin C Desouza, and Gregory S Dawson. 2020. National strategic artificial intelligence plans: A multi-dimensional analysis. Economic Analysis and Policy 67 (2020), 178–194.DOI
  19. [19]Edward Felten, Manuel Raj, and Robert Seamans. 2021. Occupational, industry, and geographic exposure to artificial intelligence: A novel dataset and its potential uses. Strategic management journal 42, 12 (2021), 2195–2217.DOI
  20. [20]First Nations Information Governance Centre. 2014. Ownership, Control, Access and Possession (OCAP): The Path to First Nations Information Governance. First Nations Information Governance Centre, Ottawa.
  21. [21]Navid Ghaffarzadegan, Aritra Majumdar, Ross Williams, and Niyousha Hosseinichimeh. 2024. Generative agent-based modeling: an introduction and tutorial. System Dynamics Review 40, 1 (2024), e1761.DOI
  22. [22]Gilles E Gignac and Marcin Zajenkowski. 2020. The Dunning-Kruger effect is (mostly) a statistical artefact: Valid approaches to testing the hypothesis with individual differences data. Intelligence 80 (2020), 101449.DOI
  23. [23]Nicole Gillespie, Steven Lockey, Caitlin Curtis, Javad Pool, and Ali Akbari. 2023. Trust in Artificial Intelligence: A Global Study. Technical Report. The University of Queensland and KPMG Australia.DOI
  24. [24]Nicole Gillespie, Steven Lockey, Tabi Ward, Alexandria Macdade, and Gerard Hassed. 2025. Trust, attitudes and use of artificial intelligence: A global study 2025. Technical Report. The University of Melbourne and KPMG. doi:10.26188/28822919DOI
  25. [25]Ben Green. 2022. The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review 45 (2022), 105681.DOI
  26. [26]Igor Grossman, Matthew Feinberg, Dawn C. Parker, Nicholas A. Christakis, Philip E. Tetlock, and William A. Cunningham. 2023. AI and the transformation of social science research. Science 380, 6650 (2023), 1108–1109.
  27. [27]Wojciech Hardy, Roma Keister, and Piotr Lewandowski. 2018. Educational upgrading, structural change and the task composition of jobs in Europe. Economics of Transition 26, 2 (2018), 201–231.DOI
  28. [28]Eszter Hargittai. 2002. Second-Level Digital Divide: Differences in People’s Online Skills. First Monday 7, 4 (2002).
  29. [29]John J. Horton. 2023. Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? Working Paper 31122. National Bureau of Economic Research.
  30. [30]Abe Bohan Hou, Hongru Du, Yichen Wang, Jingyu Zhang, Zixiao Wang, Paul Pu Liang, Daniel Khashabi, Lauren Gardner, and Tianxing He. 2025. Can a society of generative agents simulate human behavior and inform public health policy? A case study on vaccine hesitancy. arXiv preprint arXiv:2503.09639 (2025).arxiv.org/abs/2503.09639
  31. [31]Tiancheng Hu and Nigel Collier. 2024. Quantifying the Persona Effect in LLM Simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, and Vivek Srikuamr (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 10289–10307. doi:10.18653/v1/2024.acl-long.554DOI
  32. [32]Innovation, Science and Economic Development Canada. 2026. Canada’s National Artificial Intelligence Strategy: AI for All. https://ised-isde.canada.ca/site/ised/en/node/1036. [Accessed: July 4, 2026].
  33. [33]Anna Jobin, Marcello Ienca, and Effy Vayena. 2019. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1, 9 (2019), 389–399.DOI
  34. [34]Mohammad Nuruzzaman Khan, David W Rothwell, Katrina Cherney, and Tamara Sussman. 2017. Understanding the financial knowledge gap: A new dimension of inequality in later life. Journal of gerontological social work 60, 6-7 (2017), 487–503.
  35. [35]Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. 2024. Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LooM. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). ACM. doi:10.1145/3613904.3642830DOI
  36. [36]Matthias Carl Laupichler, Alexandra Aster, Jana Schirch, and Tobias Raupach. 2022. Artificial intelligence literacy in higher and adult education: A scoping literature review. Computers and education: Artificial intelligence 3 (2022), 100101.DOI
  37. [37]Nian Li, Chen Gao, Mingyu Li, Yong Li, and Qingmin Liao. 2024. Econagent: large language model-empowered agents for simulating macroeconomic activities. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15523–15536.arxiv.org/abs/2310.10436
  38. [38]Yuan Li, Lichao Sun, and Yixuan Zhang. 2025. MetaAgents: Large language model based agents for decision-making on teaming. Proceedings of the ACM on Human-Computer Interaction 9, 2 (2025), 1–27.DOI
  39. [39]Dungang Liu and Heping Zhang. 2018. Residuals and diagnostics for ordinal regression models: A surrogate approach. J. Amer. Statist. Assoc. 113, 522 (2018), 845–854.
  40. [40]Duri Long and Brian Magerko. 2020. What is AI Literacy? Competencies and Design Considerations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). ACM. doi:10.1145/3313831.3376727DOI
  41. [41]Thomas Lumley et al. 2004. Analysis of complex survey samples. J. Stat Softw 9, 1 (2004), 1–19.DOI
  42. [42]Henrietta Lyons, Eduardo Velloso, and Tim Miller. 2021. Conceptualising contestability: Perspectives on contesting algorithmic decisions. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–25.DOI
  43. [43]Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019). doi:10.1145/3359174DOI
  44. [44]Michael Muller, Shion Guha, Eric PS Baumer, David Mimno, and N Sadat Shami. 2016. Machine learning and grounded theory method: convergence, divergence, and combination. In Proceedings of the 2016 ACM international conference on supporting group work. 3–8.
  45. [45]Laura K Nelson. 2020. Computational grounded theory: A methodological framework. Sociological methods & research 49, 1 (2020), 3–42.
  46. [46]Davy Tsz Kit Ng, Jac Ka Lok Leung, Samuel Kai Wah Chu, and Maggie Shen Qiao. 2021. Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence 2 (2021), 100041.DOI
  47. [47]OECD. 2023. OECD Employment Outlook 2023: Artificial Intelligence and the Labour Market. OECD Publishing, Paris. doi:10.1787/08785bba-enDOI
  48. [48]OECD. 2024. Do Adults Have the Skills They Need to Thrive in a Changing World? Survey of Adult Skills 2023. OECD Publishing, Paris. doi:10.1787/b263dc5d-enDOI
  49. [49]OECD. 2025. Survey of Adult Skills 2023 Technical Report. OECD Publishing, Paris. doi:10.1787/80d9f692-enDOI
  50. [50]Leslie E Papke and Jeffrey M Wooldridge. 1996. Econometric methods for fractional response variables with an application to 401 (k) plan participation rates. Journal of applied econometrics 11, 6 (1996), 619–632.
  51. [51]Raja Parasuraman and Dietrich H Manzey. 2010. Complacency and bias in human use of automation: An attentional integration. Human factors 52, 3 (2010), 381–410.
  52. [52]Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). ACM. doi:10.1145/3586183.3606763DOI
  53. [53]Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. 2024. Generative Agent Simulations of 1,000 People. arXiv preprint arXiv:2411.10109 (2024).
  54. [54]Jianxiang Peng, Ling Shi, Xinwei Wu, Hanwen Zhang, Fujiang Liu, Haocheng Lyu, and Deyi Xiong. 2025. DiplomacyAgent: Do LLMs Balance Interests and Ethical Principles in International Events? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 13732–13750.DOI
  55. [55]Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, et al. 2025. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society. (2025).arxiv.org/abs/2502.08691
  56. [56]Roxana Radu. 2021. Steering the governance of artificial intelligence: national strategies in perspective. Policy and Society 40, 2 (06 2021), 178–193. doi:10.1080/14494035.2021.1929728DOI
  57. [57]Laura Robinson, Shelia R. Cotten, Hiroshi Ono, Anabel Quan-Haase, Gustavo Mesch, Wenhong Chen, Jeremy Schulz, Timothy M. Hale, and Michael J. Stern. 2015. Digital inequalities and why they matter. Information, Communication & Society 18, 5 (2015), 569–582.DOI
  58. [58]Shibani Santurkar, Esin Durmus, Faisal Ladakh, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose Opinions Do Language Models Reflect?. In Proceedings of the 40th International Conference on Machine Learning (PMLR, Vol. 202).arxiv.org/abs/2303.17548
  59. [59]Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, et al. 2024. Towards Understanding Sycophancy in Language Models. In The Twelfth International Conference on Learning Representations (ICLR).arxiv.org/abs/2310.13548
  60. [60]Daniel J. Solove. 2013. Privacy Self-Management and the Consent Dilemma. Harvard Law Review 126, 7 (2013), 1880–1903.
  61. [61]Statistics Canada. 2024. Literacy, numeracy and adaptive problem-solving skills of Canadians: Results from the 2022 Programme for the International Assessment of Adult Competencies. https://www.statcan.gc.ca/o1/en/app/7502-literacy-numeracy-and-adaptive-problem-solving-skills-canadians-results-2022-programme.
  62. [62]Statistics Canada. 2025. Analysis on artificial intelligence use by businesses in Canada, second quarter of 2025. https://www150.statcan.gc.ca/n1/pub/11-621-m/11-621-m2025008-eng.htm. [Accessed: Oct 7, 2026].
  63. [63]Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. 2023. Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv preprint arXiv:2310.05984 (2023).arxiv.org/abs/2310.05984
  64. [64]David Touretzky, Christina Gardner-McCune, Fred Martin, and Deborah Seehorn. 2019. Envisioning AI for K-12: What Should Every Child Know about AI?. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9795–9799.DOI
  65. [65]Inga Unlicane, William Knight, Toni Leach, Bernd Carsten Stahl, and Winter-Gladys Wanjiku. 2021. Framing governance for a contested emerging technology: insights from AI policy. Policy and Society 40, 2 (2021), 158–177.
  66. [66]UNESCO. 2021. Recommendation on the Ethics of Artificial Intelligence. United Nations Educational, Scientific and Cultural Organization, Paris.
  67. [67]U.S. Bureau of Labor Statistics. 2015. 2010 SOC to ISCO-08 Crosswalk. https://www.bls.gov/soc/soccrosswalks.htm.
  68. [68]SS Vallender. 1974. Calculation of the Wasserstein distance between probability distributions on the line. Theory of Probability & Its Applications 18, 4 (1974), 784–786.DOI
  69. [69]Alexander J. A. M. van Deursen and Jan A. G. M. van Dijk. 2014. The digital divide shifts to differences in usage. New Media & Society 16, 3 (2014), 507–526.
  70. [70]Jan van Dijk. 2020. The Digital Divide. Polity Press, Cambridge, UK.
  71. [71]Angelina Wang, Jamie Morgenstern, and John P Dickerson. 2025. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence 7, 3 (2025), 400–411.
  72. [72]Lexie Wie. 2026. Canada launches AI strategy to advance digital sovereignty, adoption. IAPP https://iapp.org/news/a/canada-launches-ai-strategy-to-advance-digital-sovereignty-adoption. [Accessed: Oct 7, 2026].

Paper details

vlnc:W_swqff5yt
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents