Reading Time Modeling with Pixel-Based Language Models

Introduction

Reading coordinates a stratification of low- and high-level cognitive processes [7] across multiple modalities [14]. Naturalistic reading data, for example, indicate that a considerable proportion of words is processed without a fixation; they are possibly recognized in the parafovea [36, 33] with only partial visual information [34]. Such a rapid

pre-attentive visual processing suggests the importance of a bottom-up sensory prediction [17, 44], especially how graded visual presentation affects reading time and skipping [10] via top-down prediction [6].

In computational psycholinguistics, neural language models (LMs) have been ubiquitous in estimating information-theoretic values, such as surprisal [11, 20], to instantiate linguistic theories. Despite their successful interdisciplinary bridge, recent studies have also identified drawbacks of ordinal LMs defined over a fixed set of token IDs (token-based language models; TLMs) as tools for cognitive modeling. In particular, their tokenizers and vocabularies, predefined regardless of their cognitive plausibility [13], potentially limit feasible experimental designs, e.g., estimating highly incremental probabilities across more fine-grained units, or implicitly bias experimental results [23, 26, 8, 27, 16]. For example, Nair and Resnik [23] reported that a tokenizer or (sub)word definition easily biases the surprisal estimate — $\log p(\mathrm{word}|\mathrm{context})$. Another example is that a naive accumulation of subword probabilities results in their misallocation to the following word, e.g., “He give me”, where the load of “give” can be dislocated to “me,” if the alternative “gives” is also split into the common prefix “give” and “s” [26, 8]. Although some problems have quick solutions, tokenizer selection is inherently the source of biases in computational psycholinguistic experiments with TLMs. More fundamentally, TLM also limits the possibility of experimental design; for example, the word-to-ID conversion makes it impossible to study the visual and orthographic effects in reading.

In this paper, to handle such tokenization bottlenecks spread in modern TLM-based cognitive modeling, we examine the use of tokenization-free, pixel-based language models (PLMs) [1, 35, 21] in computational psycholinguistics. PLMs encode text as pixel-based images, potentially without loss of visual fidelity to what humans can indeed see, enabling flexible research designs, for example, to explore how text layout, font, or orthographic complexity drives reading [1]. This work, as a case study, specifically explores the psychometric predictive power of PLMs and parafoveal effects in reading, positing that early-stage visual processing may already begin before fixating on the exact word, and from an expectation-based reading perspective, visual word identifiability will be a strong predictor of human gaze duration and scan path. We compute word predictability and entropy with PLMs that rely on orthographic information and a blurred future context, intended to reflect parafoveal sight. These estimates are used to model reading time and word skipping behavior during naturalistic reading.

Our experiments address the following questions:

• Q1: Do PLMs without a tokenizer predict reading time and word skipping better than or complementary to TLM surprisal, which relies on a tokenizer?

• Q2 : Does PLM-estimated visual word-prediction difficulty offer a general predictor for cognitive modeling across different languages and writing systems?

Our experimental results show that the visual word prediction difficulty estimated by PLMs offers complementary predictive power to conventional token-based surprisal. This advantage is observed across languages, particularly in those with scripts other than the Latin alphabet, demonstrating that PLM is a useful infrastructure for estimating word predictability and modeling human reading across diverse languages.

Related Work

Reading Time Modeling with Token-based Language Models

Language models (LMs) have been widely used as tools to implement and quantitatively test (neuro and psycho)linguistic theories [28] (Caucheteux et al. 2023). Given that LMs are basically next-word prediction machines, their use is typically relevant to testing an expectation-based account of human sentence processing through estimating information-theoretic measures such as surprisal [11, 20, 12]. Curiously, a misalignment between human reading measures and LM-estimated predictive measures has been indicated by recent studies [18, 37, 25], raising several possible explanations. A conceptually intuitive implication is that prediction is not a complete representation of human sentence processing [41], which features structural modularistic underpinnings [4], and thus is not purely probabilistic. But practically, another possibility rests on the biases of the proxies, i.e., LMs, in estimating predictive measures [24]. Existing studies [23, 8, 26, 27], for example, demonstrate that the tokenizer substantially affects LM-based cognitive modeling. Besides, language prediction based on token identity prevents any bottom-up visual and orthographic decoding processes. This study explores a more neutral, realistic design for text encoding in LM-based cognitive modeling, i.e., simply inputting the text image as we read texts with our eyes.

Pixel-Based Language Models

Fortunately, there are practical techniques for processing language using text images as input. Pixel-based language model (PLM) [35], using a visual Transformer (ViT) as its backbone, is a representative model that seamlessly models language in a tokenizer-free paradigm, regardless of vocabulary and character-inventory conconstraints ([21]). Such PLMs have been adopted, for example, to enable a flexible addition of a new language or to process low-resource languages, such as historical languages without transcriptions ([1]). Beneath their powerful performance in the visual domain, probing analyses indicate that PLMs also learn linguistic knowledge, such as semantics and syntax ([45]) and exhibit cross-script generalizability through multilingual pre-training ([15]). In contrast to these studies, we examine the use of PLMs for cognitive modeling.

Methods

This section briefly introduces PLMs, which model text sequences in the visual modality without relying on a tokenizer or a discrete vocabulary. We then explain how to compute next-word predictability with PLMs.

Pixel-Based Language Models

Given a token sequence $w = [w_1, w_2, \ldots, w_n]$, a rendering function converts the sequence into an image $x \in\mathbb{R}^{H \times W \times C}$, where $C$, $H$, and $W$ denote the number of channels (e.g., RGB), height, and width, respectively. The rendered image is divided into $N = \frac{HW}{P^2}$ non-overlapping patches $X = [x_1, \ldots, x_N] \in\mathbb{R}^{N \times P^2 \times C}$, where $P$ is the height/width of each patch. Each patch $x_i \in\mathbb{R}^{P^2 \times C}$ is flattened and projected into a $d$-dimensional embedding $h_i \in\mathbb{R}^d$ with an embedding layer of a PLM:

where $W_E \in\mathbb{R}^{P^2 C \times d}$ and $b_E \in\mathbb{R}^d$ are learnable parameters, and $e_i^{\mathrm{pos}} \in\mathbb{R}^d$ denotes the positional embedding of the $i$-th patch. $H = [h_1, \ldots, h_N]$ denotes the sequence of patch embeddings.

During training, a subset of patches in $H$ is randomly masked. Let indices for masked patches be $\mathcal{I} \subset[1, \ldots, N]$. The masked input $H_{\mathrm{masked}}$ is processed by a PLM $\mathcal{F}$ to reconstruct the original, unmasked version of the patches:¹

where $\hat{X} = \{\hat{x}_1, \ldots, \hat{x}_N\} \in\mathbb{R}^{N \times P^2 \times C}$. Then the reconstruction loss $\mathcal{L}$ is calculated over the indices $\mathcal{I}$ where the input was masked, and basically, the following objective is minimized:

Visual Word Prediction Difficulty (VWPD) in Context

PLMs can quantify the prediction difficulty (VWPD) of the next word in the pixel space. Given a sentence $w = [w_1, \ldots, w_n]$ and its image patches $X = [x_1, \ldots, x_N]$, suppose a target token $w_t$ spans from $s_t$-th to $e_t$-th patches $[x_{s_t}, \cdots, x_{e_t}] \subset\mathbf{X}$ in the rendered image ($1 \le s_t \le e_t \le N$). A token may span over several patches, and a straightforward way to compute the next-word prediction difficulty is:

where the target patches $[x_{s_t}, \cdots, x_{e_t}]$ are masked one-by-one to enforce the prediction (black masking parts in Figure 2). A function $s(\cdot)$ quantifies the reconstruction error of the actual visual token (also see Figure 3 for examples of predictions). For example, one can compute the instance-wise MSE in the pixel space:

where $F_k$ returns the $k$-th patch of the output, $\circ$ denotes the concatenation of patches, and $m(\cdot)$ is a masking function. A high prediction error of patches $\vec{s}(x_k|\mathbf{X}_{<k})$ is aggregated into the prediction difficulty of the word in Eq. (4); a high value of $\vec{I}(w_t|w_{<t})$ is expected to be linked with longer reading time. Another reconstruction error is also examined in § 4.

Off-the-shelf PLMs, however, are typically implemented with a bidirectional encoder or encoder-decoder architecture, and fully omitting future context $\mathbf{X}_{>t}$ from the input may incur a severe training-inference gap. Reflecting our vision, the future context is not fully invisible, but parafoveal sight still catches some superficial cues, e.g., approximate word length or sentence length. Thus, we propose the simulation of parafoveal sight by feeding blurred future context information $\mathbf{X}_{>t}$ with a blurring function $g(\cdot)$ (see Figure 2).² This blurring design is ablated in the main experiments. Now, the prediction difficulty of a word $w_t$ is redefined as follows:

where the prediction difficulty $\overleftrightarrow{I}(w_t|w_{\backslash t})$ is conditioned by bi-directional context (with partial blurring to the future context). Through our experiments, we use this version (Eq. (7)) of visual word predictability.

SSIM Reconstruction Measure

For the reconstruction error term, $\vec{s}(\cdot)$ in Eq. (5) or $\overleftrightarrow{s}(\cdot)$ in Eq. (8), another formulation beyond instance-wise MSE can also be examined. For example, structural similarity (SSIM) [47] is widely used in the computer vision field. Given that the SSIM is a similarity measure, one minus the SSIM can be adopted for estimating VWPD:

where $\mu_{\tilde{x}}$ and $\mu_x$ denote the mean pixel intensities of the reconstructed patch $\tilde{x}_k$ and ground-truth patch $x_k$, respectively. $\sigma_{\tilde{x}}^2$ and $\sigma_x^2$ denote their variances, $\sigma_{\tilde{x}x}$ denotes the covariance between them, and $C_1$ and $C_2$ are small constants for numerical stability. One can apply this formula in Eq. (5) or Eq. (8) instead of instance-wise MSE, and we preliminarily examine this variant in § 5.1.

Visual Prediction Uncertainty in Context

We also examined the entropy of next-word prediction in the pixel space, which quantifies the uncertainty in predicting black (stroke) or white (no stroke) rather than noisy gray predictions. Each dimension in a reconstructed pixel $\hat{p}_k$ at position $k$ is first normalized with a sigmoid function (0 for white pixel prediction and 1 for black pixel prediction):

The entropy measure is computed as follows:

where $\hat{p}_{k,l,m,c} \in[0,1]$ denotes $(l,m)$-th element in $c$-th channel of $k$-th patch prediction. A high uncertainty value may parallel with the general difficulty of anticipating the next word, expected to be linked with longer reading time. We also examine the entropy measures from PLMs and TLMs for word skipping modeling.

Experimental Settings

Language Models

We examine PLMs as well as TLMs with a comparable model size. All model details are in Table 2 in Appendix § 8.

Lang.	Unidirectional LMs	Model size (#parameters)	Bidirectional LMs	Model size (#parameters)
Deu	dbmdz/german-gpt2	510MB ($\sim$137M)	google-bert/bert-base-german-cased	439MB ($\sim$109M)
Eng	openai-community/gpt2	548MB ($\sim$137M)	google-bert/bert-base-cased	436MB ($\sim$108M)
Esp	DeepESP/gpt2-spanish	548MB ($\sim$124M)	dccuchile/bert-base-spanish-wwm-cased	440MB ($\sim$110M)
Far	gpt2-persian	1.31GB ($\sim$328M)	HooshvareLab/bert-base-parsbert-uncased	654MB ($\sim$163M)
Kor	skt/kogpt2-base-v2	513MB ($\sim$125M)	kykim/bert-kor-base	476MB ($\sim$118M)
Simp (zh)	uer/gpt2-chinese-cluecorpussmall	421MB ($\sim$102M)	google-bert/bert-base-chinese	412MB ($\sim$102M)
Trad (zh)	ckiplab/gpt2-base-chinese	421MB ($\sim$102M)	ckiplab/bert-base-chinese	409MB ($\sim$102M)
Tur	ytu-ce-cosmos/turkish-gpt2	498MB ($\sim$124M)	dbmdz/bert-base-turkish-cased	445MB ($\sim$111M)
All	mGPT	3.45GB ($\sim$1.4B)	Pixel-m4 (including finetuned ones)	449MB ($\sim$112M)

PLM Settings. For English experiments, we evaluate pixel-m4. For Chinese reading, we also use the off-the-shelf pixel-m4 model (Kesen et al., [15]), which has been pretrained on Chinese script. For other languages, we finetune pixel-m4 on 100,000 examples from the allenai/c4 dataset (Raffel et al., [32]) for 150,000 steps with a batch size of 32. The finetuned models reach a loss below 0.020, comparable to the official checkpoint on the pre-training data. For the blurring function $g(\cdot)$ in Eq. (8), we use Gaussian Blurring with a kernel size of 5 and a sigma of 3 (see Appendix § 9.§ 2). We also examine a no-blurring condition, in which the future context is fully visible, similarly to BERT (Devlin et al., [5]).

TLM Settings. As baselines, we use monolingual GPT2 (Radford et al., [31]), monolingual BERT (Devlin et al., [5]), and multilingual mGPT (Shliazhko et al., [38]). Comparisons between TLMs and PLMs with parafoveal information clarify the general advantage of orthographically grounded prediction measures compared to discrete token prediction without any access to future word information. BERT is also compared, given the bi-directional nature of the PLM encoder. We use the surprisal (Hale, [11]; Levy, [20]) and entropy (Hale, [12]) computed by TLMs as predictors. Specifically, for the GPT and mGPT models, the surprisal of a word is calculated as:

For the BERT models, the surprisal is computed with a bidirectional context:

caption: We use multilingual and multi-script eye-movement-while-reading data across 459 readers.

:::

Entropy is calculated as the expected value of surprisal over the model’s tokenizer vocabulary (Pimentel et al., [30]).

Human Reading Data

We model reading time and word skipping behavior of a total of 459 subjects from different eye-movement-while-reading corpora, covering German (Deu), English (Eng), Spanish (Esp), Farsi (Far), Korean (Kor), Simplified Chinese (Simp), Traditional Chinese (Trad), and Turkish (Tur). Data are from MECO Wave 1 (Siegelman et al., [39]) or Wave 2 (Siegelman et al., [40]), except for Persian reading corpus (Tekbudak et al., [46]). Data statistics and details are presented in Table 1. We take the following reading measures into consideration: First Fixation Duration (FFD), First-pass Reading Time (FPT), Total Reading Time (TRT), First-pass Skipping (FSKP), and Overall Skipping (SKP). For durative measures, any zero fixation duration and zero reading time are excluded. We adopt two word skipping measures: (i) first-pass skipping — whether an interest area is skipped before a later interest area is fixated (1=yes, 0=no); and (ii) Overall skipping — whether an interest area is not fixated throughout the current trial (1=yes, 0=no).

Regression Models

Predictors of Interest. We extract visual prediction difficulty (Eq. (7)) and visual entropy (Eq. (12)) from PLMs, as well as the widely used token-level surprisal and entropy from TLMs. For each regression model, we only use one predictor of interest, denoted as $x$. The regression formula is as follows:

where $l$ is word length, and $f$ is zipf word frequency, $\operatorname{idx}_n$ is the word index in sentence. The regression formula for the baseline is as follows:

Here, $y$ is the reading measures, i.e., FFD, FPT, TRT, FSKP, or SKP. For reading measures (FFD, FPT, and TRT), we applied log transformation, and skipping measures (FSKP and SKP) are converted with binomial family. We conduct our main experiments with generalized additive mixed models (GAMM) (Sóskuthy, [43]; Wood et al. [50]), using mgcv packages (Wood, [49]) and setting all effects as linear. For main experiments, we use the following designs of $x$:

• pixel: VWPD from PLM, without blurring the future context.

• pixel_uni: VWPD from PLM, with blurring the future context.

• gpt: surprisal from the respective monolingual GPT2.

• mgpt: surprisal from multilingual GPT.

• bert: bi-directional surprisal from the respective monolingual BERT.

• pixel_uni + gpt: pixel_uni and gpt are both included, without interaction.

• pixel_uni * gpt: Interaction between pixel_uni and gpt with their main effects.

Evaluation Metrics

Following existing studies (Goodkind and Bicknell, [9]), we use the difference in loglikelihood between two nested models with and without the predictor of interest $x_n$ from TLMs or PLMs. In our case, the joint contribution of the predictor and its

subject-specific random slope is measured. We refer to this metric as psychometric predictive power (PPP). To test robustness, we rerun each formula 100 times using a crossed random-effects bootstrap with replacement to obtain a distribution of the log-likelihood difference. Similarly to Goodkind and Bicknell [9], we fit regression models to explain the data of interest, but in Appendix § 11, we also adopt 10-fold cross-validation, as in (Wilcox et al., [48]), yielding consistent results. When reporting a single PPP value for each language with several datasets (Table 1), PPPs for different datasets in the target language are averaged.

Experimental Results

Preliminary Experiments: Comparing Two VWPD Measures

In § 3.2, we introduced MSE and SSIM as an image reconstruction error in Eq. (8). As preliminary experiments, we compare MSE-based and SSIM-based image reconstruction errors as general indicators of next-word prediction difficulty. Figure 4 shows that different PPPs are obtained with different error formulations, and there was a clear empirical trend that SSIM is slightly better or comparable to MSE, except for the Chinese (Simplified and Traditional) language. This might reflect a special visual processing of logograms or the complexity of Chinese characters. In our main experiments, we show SSIM results across all languages, except for MSE for the Chinese (Simplified and Traditional) data. Full SSIM and MSE comparison is shown in Appendix § 12.

Comparing PLMs and TLMs

We evaluate two versions of PLMs with and without blurring for the future context, and three TLMs (monolingual GPT2, BERT, and mGPT).

Fixation and Reading Time Figure 5 shows the results for FFD, FPT, and TRT modeling. Visual word predictability (VWPD) is used for PLMs, and surprisal is used for TLMs. PPPs for PLMs are somewhat worse than TLMs, but when these two predictors are jointly used, the PPP substantially increases, indicating their complementary advantage in reading time modeling.

Word Skipping Figure 6 shows the results for FSKP and SKP; whether the word is skipped or not during reading is predicted. In contrast to gaze duration measures, PLMs yield comparable or better predictive power, especially for the Farsi (Far), Korean (Kor), and Chinese (Simp and Trad) languages; interestingly, these languages do not adopt the Latin alphabet. In these languages, in other words, if a word can easily be predicted visually from the surrounding context, it is likely to be skipped during reading. This idea is also compatible with entropy measures (how the next word is uncertain), and Figure 7 shows the comparison of entropy computed by PLMs and TLMs. The results generally reproduce our observations with surprisal, i.e., token-level and visual entropy are complementary. We report the cross-validation analysis in appendix D, which substantiates the tendencies we have pointed out above.

Comparing Addition and Interaction

Comparing the PPP of the additive model (pixel_uni + gpt) and the interactive model (pixel_uni $\ast$ gpt), we sometimes observe an advantage of the interaction term, especially in the first-pass measures (FFD and FSKP) in German (Deu), Spanish (Esp), and Turkish (Tur). These have somewhat longer word length and richer morphology than those of other languages [39]. Thus, a strong interaction may imply a more general hierarchical engagement of prediction in reading, spanning from shallow visual perception to morphological or richer linguistic processing. In the following section, we look into how this interaction shapes first-pass reading behavior. The detailed breakdown of our main results into individual dataset is in Figure 13 and Figure 14.

Analysis: Characteristics of VWPD

Interaction with TLM Surprisal

Given the complementary advantage of predictors from PLM and TLM, we exploratorily extend the analysis, specifically on how PLM and TLM predictors interact. We focus on early-stage reading measures, such as first fixation duration (FFD), which typically aligns with early lexical access [19], first-pass reading time (FPT), as well as first-pass refixation duration (FPT minus FFD; FPREF), which indicates the “failure to achieve lexical access” [22]. For simplicity, we only consider the fixed effect of PLM surprisal (namely VWPD) and TLM surprisal (we use GPT2 surprisal in each language). We split the data into those with relatively high surprisal (+1 SD) and low surprisal (−1 SD) and analyze how the VWPD effect differs between the two subsets.

Figure 8 shows the interaction between VWPD and GPT2 surprisal. Interestingly, for FFD, we found that high surprisal (+1 SD in the figure) can reverse (e.g., in Deu and Tur) the VWPD–FFD relationship to be negative. In contrast to FFD, FPREF results indicate a positive mediation of VWPD, as high surprisal (+1 SD in the figure) leads to a positive VWPD-FPREF relationship. That is, words with a high surprisal (lexical difficulty) tend to weaken the VWPD effect for first fixation but strengthen it for refixation. One possible interpretation of the neutralized or negative interaction between VWPD and surprisal in FFD prediction is that lexical prediction (token-level surprisal) may involve pre-activation of visual representations [2]; if more effort is devoted to predicting the next word, early-stage visual processing can be skipped. The positive interaction in FPREF aligns with the finding that orthographic confusion is significant only for unpredictable words [51], as the positive interaction was only observed when people need multiple fixations (FPREF), potentially when lexical access initially failed.

Gaussian Blurring Effects

We also analyze how Gaussian blurring of the future context affects PPP. The improvement in predictive power (See Figure 9) with the blurring treatment is not substantial. Still, the advantage is slightly pronounced for SKP in Farsi (Far), Korean (Kor), and Traditional Chinese (Trad), where non-Latin alphabets are used. The advantage of blurring in Trad has also been suggested by the finding that reading Traditional Chinese characters narrows the perceptual span [29], supporting an information-processing account of visual perception during reading, as assumed in the OB1-Reader model [42]. Nevertheless, our Gaussian Blurring is our tentative implementation, and more cognitively motivated designs should be explored in future studies.

Conclusion

We demonstrate that pixel-based LMs can estimate expectation-based visual word-processing measures that complement those estimated by token-based LMs, which are especially relevant to pre-fixation processing on partial input. This study is the first to introduce a pixel-based language model into computational cognitive modeling. The results demonstrate, at least partial cognitive plausibility of PLM, especially its alignment with early-stage language processing, such as first-pass skipping.

Limitations

First, a pixel-based language model has various rendering strategies [21]. We only use the most cross-lingually generalizable setting, namely bigram rendering, which is the foundation of the only multilingual pixel-based model so far. We agree that calculating visual word prediction metrics is highly sensitive to the text rendering strategy. We encourage future work to explore different rendering strategies and analyze their effect on the cognitive plausibility of pixel-based language modeling. Second, for pixel blurring, we have not conducted an intensive hyperparameter analysis regarding the effects of kernel size and sigma on model performance. We also only tried Gaussian Blurring, without exploring other image corruption methods such as Gaussian Noise. Future work could also compare more cognitively motivated parafoveal blurring and noise. Besides, the human parafovea exhibits graded visual acuity, which is not accounted for in this study. We hope that the current study initiates the primary work and establishes the foundational settings, and that future work will continue to explore parafoveal vision modeling by manipulating the gradedness of visual acuity.

Ethical Considerations

This study uses open-source eye-movement-while-reading data collected from IRB-approved studies. No personally identifiable information is used in our experiment. Our study aims to gain knowledge about human cognition. We do not curate any system to replace humans in terms of reading.

AI Writing/Coding Assistance Policy

AI writing assistance systems like Grammarly and ChatGPT are occasionally used to correct grammatical and spelling errors and to adjust phrasing. Our use of writing assistants complies with the AI writing assistance policy described in ARR guidelines under category (a), assistance purely with the language of the paper.

Acknowledgments

This work was supported by JSPS KAKENHI Grant Numbers 23K16938 and 26H02511.

Models

In Table 2, we present the details of models used in this study.

PLM details

Conditions

Figure 10 shows all four conditions explored in this study. The first option is whether to mask all the patches related to the target word (whole masking) or only the target patch (partial masking) when computing patch predictability. The second option is whether to apply the noisy blurring to the future context, potentially simulating the parafoveal information, or not.

Gaussian Blurring

We apply the following noise function to the original image $X$:

As a cognitively implausible baseline, we also examine a no-blurring condition in which the right-side context is visible.

Others

To adapt Persian to the bigram renderer, we leave the space between two characters, even if they are connected visually. However, the space does not change the pre-space word into an unconnected form if it is originally in a connected form. To gather VWPD from Persian, we did not include the white-area. For the results in the current paper, we used the checkpoint after the 33,500th step results of the PSR dataset.

Detailed Results

Interactive Effect of PLM prediction difficulty to TLM surprisal

Figure 11 shows a more detailed comparison between the regression model with GPT surprisal and that with GPT surprisal and the interaction between GPT surprisal and PLM VWPD.

Effects of Whole Word Masking

Compared with blurring, the effect of masking is more nuanced across languages. We notice an advantage of whole-word masking in word skipping in Traditional Chinese, but a disadvantage in Korean. This potentially indicates a language-dependent sub-word processing strategy during naturalistic reading.

Main Results by Dataset

Below are the detailed main results for each dataset:

Cross-validation

In Figure 15, we reran the inter-measure comparison results in Figure 4 through 10-fold cross-validation. In Figure 16, Figure 17 and Figure 18, we reproduce the results in Figure 5, Figure 6 and Figure 7, and Table 4.

The cross-validation results do not contradict the observation that the MECO Chinese data show a stronger advantage of MSE-based VWPD. A revisit of the 10-fold cross-validation result supports the identical tendencies we have captured in § 5. In the first place, an integration, especially an interaction, between token and visual prediction improves reading-time and skipping modeling. Besides, visual prediction accounts for skipping better than token prediction in certain scripts, notably Persian (Far), Korean, and Chinese scripts.

Comparing MSE and SSIM across datasets

We report the performance of MSE and SSIM in terms of their advantage against the GPT2 model in Table 3.

VWPD

Measure

Esp (es)

Esp (ch)

Deu (po)

Deu (zu)

Eng (uk)

Eng (en)

Kor

Far (PSR)

Simp

Trad

Tur

pixel_uni vs. gpt

SSIM

FFD

-23.0***

-28.3***

-32.7***

-48.4***

-68.9***

-30.6***

-5.6*

-126.6***

-27.5***

-24.3***

-4.1*

FPT

-37.3***

-74.3***

-60.1***

-84.1***

-100.4***

-49.9***

-44.8***

-321.2***

-63.8***

-24.0***

-13.2***

TRT

-20.8***

-66.7***

-214.0***

-235.6***

-194.3***

-94.6***

-68.1***

-129.1***

-90.6***

-30.9***

-15.5***

FSKP

+26.4***

-6.9**

+16.4***

+28.9***

+41.8***

-6.1*

+49.2***

+177.0***

-23.1***

-16.1***

-6.3***

SKP

-4.3*

-22.1***

-8.7***

-55.4***

-6.2

-28.0***

+107.6***

+214.9***

-45.6***

-21.1***

+7.8***

MSE

FFD

-20.3***

-21.0***

-25.9***

-30.1***

-75.6***

-34.3***

-31.4***

-140.0***

-13.9***

-0.9

-11.2***

FPT

-33.7***

-63.9***

-73.6***

-94.2***

-126.2***

-72.4***

-73.0***

-408.7***

-49.9***

-1.5

-12.4***

TRT

-23.8***

-70.3***

-250.2***

-257.0***

-268.6***

-128.4***

-123.7***

-249.6***

-52.9***

+39.2***

-22.6***

FSKP

+22.6***

-4.6*

+10.3**

+7.3*

+16.1***

-0.3

-14.2***

-109.4***

+14.2***

+37.9***

-14.2***

SKP

-9.1***

-21.7***

-14.5***

-63.9***

-68.1***

-35.8***

-40.5***

-70.7***

+30.0***

+53.0***

-8.1***

pixel_uni × gpt vs. gpt

SSIM

FFD

+26.6***

+13.7***

+31.7***

+37.0***

+18.1***

+12.1***

+21.7***

+103.8***

+11.1***

+4.0***

-6.0

FPT

+20.2***

+28.7***

+55.9***

+78.9***

+76.5***

+53.0***

+36.2***

+181.7***

+37.0***

+34.7***

-17.1***

TRT

+43.1***

+26.6***

+74.4***

+65.9***

+100.3***

+54.2***

+59.8***

+228.4***

+19.3***

+7.3***

-4.4*

FSKP

+134.7***

+49.0***

+76.1***

+143.9***

+65.1***

+56.2***

+114.5***

+328.7***

+16.2***

+23.7***

+16.1***

SKP

+42.5***

+16.9***

+43.0***

+19.1***

+85.4***

+42.0***

+167.4***

+337.6***

+19.2***

+23.7***

+15.7***

MSE

FFD

+30.6***

+21.8***

+34.8***

+45.0***

+16.1***

+8.7***

-0.7

+56.2***

+33.4***

+28.9***

-11.1***

FPT

+23.1***

+37.1***

+57.2***

+84.9***

+64.0***

+40.0***

+18.4***

+57.7***

+66.3***

+59.9***

-14.8***

TRT

+41.2***

+21.4***

+64.6***

+73.6***

+51.8***

+31.1***

+23.3***

+79.2***

+78.4***

+82.7***

-10.8***

FSKP

+132.7***

+54.0***

+66.1***

+112.0***

+36.8***

+56.1***

+44.2***

+56.2***

+61.0***

+72.2***

+5.6*

SKP

+37.1***

+19.5***

+41.9***

+27.5***

+35.4***

+46.6***

+31.1***

+63.0***

+119.1***

+97.4***

+2.9

VWPDMeasure	Esp (es)	Esp (ch)	Deu (po)	Deu (zu)	Eng (uk)	Eng (en)
pixel_uni vs. gpt
SSIM FFD	-0.000033	-0.000153	+0.000029	+0.000045^	-0.000421^	-0.000188^
SSIM FPT	-0.000374	-0.001012**	-0.000885^	-0.000158	-0.001189^	-0.000768***
SSIM TRT	-0.000125	-0.000871***	-0.004425***	-0.002344***	-0.002263***	-0.001323***
SSIM FSKP	+0.000098	+0.000007	-0.000392***	-0.000599***	+0.000323*	+0.000003
SSIM SKP	-0.000009	-0.000087	-0.000167^	-0.000562***	-0.000131	-0.000313**
MSE FFD	-0.000036	-0.000146^	+0.000089	+0.000148	-0.000500*	-0.000215*
MSE FPT	-0.000376	-0.001025**	-0.001090*	-0.001178*	-0.001984**	-0.001001**
MSE TRT	-0.000134	-0.000860***	-0.005057***	-0.002882***	-0.003238***	-0.001759***
MSE FSKP	+0.000123	+0.000002	-0.000342**	-0.001444***	+0.000232^	+0.000027
MSE SKP	-0.000004	-0.000005	-0.000246^	-0.000756**	-0.000857***	-0.000382**
pixel_uni × gpt vs. gpt
SSIM FFD	-0.000008	+0.000003	+0.000429*	+0.000408^	-0.000064	-0.000207^
SSIM FPT	+0.000040	+0.000230*	+0.000511	+0.001832***	+0.001246***	+0.000365***
SSIM TRT	-0.000155	+0.000232**	+0.000863*	+0.000743*	+0.000505	+0.000352*
SSIM FSKP	+0.001094***	+0.000184	+0.000259*	+0.001035***	+0.000131	+0.000318**
SSIM SKP	+0.000186**	-0.000164*	-0.000454*	-0.000338***	+0.000578**	+0.000175*
MSE FFD	-0.000006	+0.000027	+0.000397**	+0.000283	-0.000072	-0.000219^
MSE FPT	-0.000005	+0.000108	+0.000677	+0.001291**	+0.000839***	+0.000222**
MSE TRT	-0.000179	+0.000171^	+0.000861*	+0.000642*	+0.000021	+0.000042
MSE FSKP	+0.001180***	+0.000225	+0.000162	+0.000210	+0.000030	+0.000331**
MSE SKP	+0.000169**	-0.000063	-0.000309^	-0.000299**	+0.000060	+0.000258**

VWPDMeasure	Kor	Far (PSR)	Simp	Trad	Tur
pixel_uni vs. gpt
SSIM FFD	+0.000710^	+0.000173	-0.000287	+0.000074	+0.000127
SSIM FPT	-0.001143^	-0.000866	-0.000944**	+0.000049	+0.000777^
SSIM TRT	-0.002521**	+0.003256***	-0.000795	+0.000122	-0.000284
SSIM FSKP	+0.001036^	+0.012088***	-0.000023	-0.000052	+0.000854*
SSIM SKP	+0.003796***	+0.013201***	-0.000286*	-0.000024	+0.001453*
MSE FFD	-0.000061	+0.000217	+0.000329	+0.000978*	+0.000007
MSE FPT	-0.003004**	-0.003013***	-0.000419	+0.000917^	-0.000087
MSE TRT	-0.005010***	+0.000656*	+0.000167	+0.001579**	-0.000645*
MSE FSKP	+0.000079	+0.002944***	+0.000431*	+0.000877***	+0.000464*
MSE SKP	+0.000351	+0.003176***	+0.001179***	+0.001274**	+0.000643^
pixel_uni × gpt vs. gpt
SSIM FFD	+0.000555	+0.001022***	-0.000181^	-0.000408	+0.000556
SSIM FPT	+0.001384^	+0.003938***	+0.000578	+0.000228	+0.000777*
SSIM TRT	+0.001847**	+0.004196**	+0.000071	+0.000149	+0.000076
SSIM FSKP	+0.001260*	+0.009560***	-0.000053	+0.000180	+0.003422**
SSIM SKP	+0.003754***	+0.010907***	-0.000003	+0.000073	+0.001872*
MSE FFD	-0.000212^	+0.000435*	+0.000687*	+0.000520	+0.000395
MSE FPT	+0.000341	+0.000923**	+0.001424**	+0.000952	-0.000039
MSE TRT	+0.000694**	+0.000879*	+0.001578*	+0.001596*	-0.000039
MSE FSKP	+0.000048	+0.001996***	+0.000391	+0.000988***	+0.000332*
MSE SKP	+0.000702***	+0.002047***	+0.001822***	+0.001355***	+0.001367*