WorksComputer Science
EyeQ: Exploring Corneal Reflection for Mobile Context Sensing
Abstract
What if a single face-facing camera could sense your surroundings—through your eyes? We introduce EyeQ, which explores corneal reflection as a modality for context-aware mobile interaction. Traditional vision-based context sensing relies on world-facing cameras that demand explicit aiming or dual-camera setups; EyeQ instead shows that an off-the-shelf VLM can read scene context from corneal reflections captured with a smartphone camera. To recover usable detail from mobile corneal crops, EyeQ pairs on-device iris-guided capture with server-side AI enhancement, then leverages VLMs to extract scene types, names, and objects from warped, unfamiliar eye-image sequences. In an N=10 study, we characterize how capture conditions shape reflection fidelity, evaluate what VLMs detect, misclassify, and hallucinate, and demonstrate interaction opportunities, including detecting a steering wheel to distinguish drivers from passengers. This work reframes the eye in HCI—from a target for gaze estimation to a passive mirror of one's surroundings—and examines the privacy tensions it raises.

Figure 1. We introduce EyeQ, which passively extracts environmental context by capturing the user’s cornea with a smartphone camera. EyeQ employs a multi-stage pipeline—including eye detection, adaptive focus, auto-cropping, AI-based image restoration, and VLM analysis—to infer scene type, name, and visible objects from corneal reflection time-series. In this office scene, EyeQ identifies two monitors, a keyboard, desk, and even a cat image displayed on screen.
Introduction
Context-aware mobile systems—from voice assistants that adapt to their surroundings [30, 39] to AR interfaces that recognize nearby objects [11, 29]—depend on knowing what is around the user, yet acquiring this information remains cumbersome. Traditional vision-based context sensing systems require a world-facing camera to observe the scene and often a face-facing camera for gaze or attention [27, 30, 39], demanding active device aiming, raising social acceptability and privacy concerns [1, 6], and excluding many everyday configurations where the only useful camera is oriented toward the user (e.g., phones held at a natural downward angle, laptops, webcams, propped-up tablets). A rich signal, however, is already visible to any single face-facing camera: the human cornea acts as a convex mirror, reflecting a wide-angle view of the surrounding scene [47] and effectively turning the user’s eye into a passive scene camera.
Corneal reflections—the specular reflections of the environment visible on the corneal surface—have been studied in computer vision [51, 55], where prior work has recovered environment maps and 3D scene geometry from eye images [2, 47, 49], and in security as a side-channel threat [4, 5]. However, they remain largely unexplored as a sensing modality in Human-Computer Interaction (HCI). A key barrier is that existing pipelines assume controlled capture (e.g., DSLR optics, studio lighting) and often require substantial processing time, limiting their use on everyday devices. Within HCI, eye reflections have been used for near-device interaction, such as sensing grip posture and finger pose from screen glints [42, 80] or detecting input around reflective eyewear [16], but not yet for broader scene understanding. Most relevant to our work, hEYEbrid [27] uses a multi-camera wearable for calibration-free gaze estimation, but it treats the corneal image as a geometric cue for gaze direction rather than a source of scene information. We instead explore whether a vision-language model (VLM) can read scene context from low-resolution corneal reflections captured with a smartphone camera, while also confronting the privacy implications of this intimate signal.
To study this, we built EyeQ, an end-to-end pipeline that extracts scene information—scene type (i.e., indoor vs. outdoor), scene name (e.g., kitchen), and object inventories—from corneal reflections captured by a smartphone camera over time. The core challenge is that a corneal reflection occupies only a tiny fraction of the face image, yielding eye crops that are often too small, blurry, and dark for direct analysis. EyeQ addresses this through a split architecture. On-device, it keeps each frame sharp, well-exposed, and centered on the cornea through four stages: gyroscope-based motion gating, MediaPipe Iris [71] for steering auto-exposure and auto-focus to the eye, 48-megapixel (MP) photo capture (rear camera as a stand-in1), and iris-guided corneal cropping. Server-side, EyeQ recovers faint scene detail from the raw crop in five stages: Real-ESRGAN [75] 4× super-resolution, CLAHE local contrast enhancement, NAFNet [10] denoising, unsharp masking, and saturation boosting. Both raw and enhanced versions are batched over time and sent to three VLMs—Google Gemini 3.1 Pro [69], Gemini 3 Flash [67], and Gemini 3.1 Flash Lite [68]—for progressive scene analysis. To our knowledge, this is the first work to infer scene content from corneal reflections captured with smartphone-grade optics, and the first to apply VLMs to these low-resolution signals.
To evaluate feasibility, we conducted an within-subjects study that varied capture conditions (e.g., motion, phone-to-eye distance, angle, and lighting) and examined physiological factors (e.g., eye color) across indoor and outdoor scenarios, measuring their impact on image quality and VLM scene recognition. Results show that VLMs can reliably classify indoor vs. outdoor scenes (96.4%) and detect objects from corneal reflections, including monitors (87.5% of office trials), windows (100% in kitchen), trees (40% outdoors), and even a cat image on a screen. Most false positives were misclassifications of real optical evidence rather than hallucinations, suggesting VLMs see more in the cornea than they correctly name. To close that gap, we asked whether corneal reflections could be combined with other context available to a phone, such as GPS, time, motion, or the rest of the frame around the eye. Providing a minimal scene prior (e.g., “the user is in a kitchen”) improved detection across all three scenes and halved false positives. We complement the study with five demonstrations of interactions this signal can support, from warning a phone-distracted pedestrian about an approaching vehicle to pausing a voice assistant when someone else is nearby. Finally, we discuss how future advances could extend this capability, alongside the ethical tensions of corneal reflection, from participants’ reactions to seeing their own eyes, to our decision not to fine-tune models or release our 10,635-image dataset.
In summary, our contributions include:
EyeQ, a pipeline that infers scene information from corneal reflections captured with smartphone-grade optics, combining on-device iris-guided capture with server-side AI enhancement and VLM scene analysis.
Findings from an study characterizing what off-the-shelf VLMs detect, misclassify, and hallucinate, how capture conditions shape reflection quality, and how participants perceive comfort and privacy implications—establishing a zero-shot benchmark.
Proof-by-demonstration scenarios illustrating interaction opportunities and risks this capability raises.
Related Work
Corneal reflections have been studied across computer vision, privacy, and HCI. We build on this interdisciplinary foundation to examine the potential and risks of analyzing eye reflections, as well as how EyeQ extends this space by enabling such techniques with smartphone-grade optics and VLMs.
Scene Understanding from Eye Reflections
The human cornea acts as a convex mirror, providing a natural reflective surface for environment capture. Seminal work by Nishino & Nayar modeled the eye as a catadioptric imaging system, recovering wide-angle environment maps from single eye images [47, 49]. They showed that corneal reflections encode both panoramic views of the surroundings and retinal images aligned with gaze, laying the foundation for using eyes as scene cameras. Later studies refined this mapping with more accurate, aspherical eye models [45].
A complementary thread treats the cornea as a light probe for estimating scene illumination. Tsumura et al. estimated the directions of multiple light sources from eye reflections [70]. Nishino & Nayar then extracted dense spherical illumination maps for face relighting [48], and Wang et al. separated specular reflections from iris texture to isolate light sources across varied eye colors [73]. Corneal illumination estimates have also supported downstream tasks, including face [46] and place [56] recognition.
Subsequent work improved corneal image quality and extended it to 3D reconstruction. Nitschke & Nakazawa applied super-resolution to eye videos by aggregating reflections over time [50]. Nishino & Nayar modeled stereo geometry between both eyes for triangulating reflected scene points [47, 49], with related work recovering scene geometry from corneal reflections of known screen patterns [53, 54, 64]. More recently, Liang et al. recovered lighting, shape, and appearance from a single unconstrained eye image via inverse rendering [33], while Alzayer et al. jointly estimated 3D eye poses and a NeRF from short face videos, treating each corneal image as a novel view for scene reconstruction [2].
While we are inspired by this body of work, it typically relies on DSLR-quality images under controlled conditions (e.g., tripod-mounted cameras, studio lighting). Instead, we explore whether corneal reflection cues can be captured by a handheld smartphone and analyzed by VLMs for context-aware HCI applications.
Gaze and Attention via Corneal Reflections
Corneal reflections are foundational to model-based gaze tracking, though the term refers to a different signal than in our work. Classic pupil-center corneal reflection (PCCR) methods use glints—point reflections from known infrared sources—as geometric markers, estimating gaze from their position relative to the pupil [19, 20]. EyeQ instead analyzes reflected scene content, not glint geometry.
A closer line of work estimates gaze by analyzing scene imagery visible on the corneal surface. Nakazawa et al. estimated a 3D point of gaze using coded IR patterns [43], while Nitschke et al. estimated point-of-gaze directly from the corneal image [52]. Related approaches leverage aspherical eye models for geometric gaze estimation [41], estimate point-of-regard from corneal images [65], and enable calibration-free view estimation on mobile head-worn cameras [44]. Notably, Takemura et al. found the object of focus from its corneal reflection [66], while HiFiGaze [26] shows that screen reflections visible on the cornea can refine mobile gaze estimation.
HCI Systems Using Corneal Reflections
Several HCI systems have leveraged corneal reflections for context-aware interaction [55]. For example, ReflecTouch [80] and ReflecTrace [42] analyze finger occlusions in the corneal reflection of the screen to infer grip posture and mid-air hover gestures, respectively. Most relevant to our work, hEYEbrid [27] mounted two eye-facing cameras—one infrared, one for corneal imaging—on a glasses frame for calibration-free gaze estimation, using corneal reflection as a geometric cue for gaze direction. Beyond personal devices, Nagamatsu et al. demonstrated gaze-responsive public displays using corneal reflections without calibration [40]. Recent work has also curated datasets pairing corneal reflections with scene views [13, 56], though neither is publicly available due to privacy concerns.
HCI researchers have also explored external reflective surfaces beyond the cornea. GlassHands [16] demonstrated around-device input by analyzing reflections of the phone in eyewear such as sunglasses and visors, an approach later adapted to use corneal reflections directly [63]. Other work has captured reflections on physical mirrors [31, 37, 81] and device-mounted mirrors [76] to extend interaction areas and sense occluded objects.
Prior work has explored corneal reflections primarily for near-device interaction and gaze estimation; we instead investigate them as a source of environmental context, leveraging VLMs to extract scene information from low-resolution corneal imagery. Our approach also requires only a single smartphone camera, simplifying the hardware demands of prior eye-driven context systems (e.g., 3C-hEYEbrid [27], WorldGaze [39] and GazePointAR [30], all of which require both eye- and world-facing cameras) and removing the need to aim the device at the scene.
Privacy Concerns around Reflections
A growing body of work has shown that reflections in everyday objects can inadvertently leak sensitive visual information. High-resolution images of corneal reflections can expose bystanders, screen contents, or entire environments [4, 5, 24]. Prior work has also reconstructed typed input from reflections in sunglasses [60], recovered screen contents via teapots, spoons, plastic bottles, and eyeglasses [5], exploited multi-bounce reflections to reconstruct obscured input [78], and even identified websites browsed during video calls using eyeglass glints [34]. Recent advances in computer vision push these threats further, with neural rendering techniques extracting 3D scene geometry from eye reflections [2]. These studies collectively demonstrate that visual side-channels pose a credible privacy risk in both public and private settings. As corneal reflection sensing becomes feasible with smartphone-grade optics, we examine both its potential and privacy risks.
EyeQ Implementation
To explore corneal reflection sensing with smartphone-grade optics, EyeQ pairs on-device capture with server-side enhancement in a single pipeline Figure 2. This split architecture is needed because the cornea occupies a tiny fraction of the face image, yet vision interpretation demands more detail than the raw crop provides. EyeQ captures iris-targeted 48 MP photos, streams each corneal crop to a GPU server for AI-based restoration, and sends raw–enhanced pairs to VLMs for analysis over time.

Figure 2. EyeQ pipeline overview. An iOS client captures iris-targeted 48 MP photos, streams corneal crops to a server for AI-based enhancement, and sends raw–enhanced pairs to three VLMs in parallel for progressive scene analysis.
Mobile Capture Client
We developed an iOS client targeting an iPhone 16 Pro’s 48 MP rear sensor (8064×6048, 24 mm, ). We chose the rear sensor to test corneal sensing at the upper end of current smartphone optics. The front camera on our device captures at 12 MP, which our piloting found insufficient for corneal analysis. To give participants a sense that they were still using a smartphone, we pair the rear camera with a mirrored display so they have a screen to look at (see Figure 3 inset). We drew inspiration from how people already use external preview displays or a foldable phone’s outer screen to take selfies with the rear camera. Moreover, front cameras are catching up quickly [59]—the iPhone 17 Pro now has an 18 MP front sensor and 50 MP front sensors have begun shipping on Android phones [17, 18]—hence the rear camera serves as a reasonable stand-in. The client handles three tasks: real-time iris localization, high-resolution photo capture, and corneal cropping and transmission.

Figure 3. Study apparatus. The three scenarios (office, kitchen, campus), their ground-truth objects, and the rear camera phone-holding posture with a magnetic mirrored display (bottom-left inset).
Real-time iris localization. Because the camera’s default auto-exposure and auto-focus (AE/AF) optimize for the face, we first have to steer capture toward the cornea. We run Google’s MediaPipe Iris [58, 71] on each live preview frame (1920×1080 at 60 FPS), choosing it over classical eye detectors (e.g., Haar cascades) that lose reliability at the upward viewing angles typical of handheld phone use. MediaPipe Iris detects 10 iris landmarks—five per eye, comprising a center point and four cardinal boundary points—from which we select the larger iris and compute its centroid. Before each capture, EyeQ locks the camera’s AE/AF point of interest onto the detected iris center, ensuring the photo is optimized for the corneal region. EyeQ also continuously estimates phone-to-eye distance using a pinhole camera model and the known human iris diameter (≈11.7 mm) [6, 7, 21, 62], warning the user when the phone is closer than the lens’s minimum focus distance (~15 cm).
High-resolution photo capture. Maximizing pixel density on the cornea is critical. We empirically evaluated five iOS camera configurations—48 MP ProRAW, 48 MP HEIC, 24 MP Deep Fusion, 12 MP burst mode, and 2–5× digital zoom—finding 48 MP HEIC at 1× zoom with quality prioritization and flash disabled offered the best trade-off. ProRAW forgoes Apple’s computational photography pipeline (i.e., Deep Fusion and Smart HDR), which proved critical for corneal detail; burst mode sacrifices resolution; and digital zoom amplified noise and blur. Before each capture, EyeQ waits for the iris-steered AE/AF to stabilize, and a gyroscope-based motion gate suppresses capture during rapid hand movement (rotation rate rad/s). This loop repeats as fast as the hardware allows—a “lucky imaging” [14, 28] strategy at Hz, capturing many frames in hopes of obtaining at least one sharp corneal image.
Corneal crop and transmission. After each 48 MP capture, EyeQ runs a second MediaPipe Iris pass on the embedded preview thumbnail to quickly obtain landmark positions. These normalized coordinates are mapped to the full-resolution image with aspect-ratio correction. EyeQ then crops a square region around the larger iris with 15 px padding from the full photo. Because the 48 MP sensor stores pixels in Display P3 with 10-bit depth, we convert the crop to 8-bit sRGB and encode it as a lossless PNG to preserve fine specular detail. Each crop is streamed to the server over a persistent WebSocket connection.
Server-Side Enhancement
Even at 48 MP, the corneal crop typically spans only 150–300 px—which is often too small for reliable VLM interpretation. The backend receives each corneal image and applies fast, multi-stage AI restoration to recover faint scene detail.
Quality gating. Not all incoming crops are worth processing, as motion blur, blinks, and inaccurate iris detections produce unusable frames. We compute a sharpness score from Sobel gradient variances over the center 50% of the crop and reject frames below an empirically set threshold ().
Enhancement stages. Accepted crops pass through five stages, each with parameters that adapt to the input quality:
Real-ESRGAN 4× [75] super-resolution upscales the crop to 600–1200 px. We chose this over face-restoration models (e.g., GFPGAN [74]), which impose facial geometry priors that distort scene reflections, and over diffusion-based approaches, which are slower and risk hallucinating detail.
Contrast-Limited Adaptive Histogram Equalization (CLAHE) on the L channel of LAB color space boosts local contrast, pulling detail from dark iris regions (clip limit scales with input brightness: 1.0–1.5).
NAFNet [10] denoising suppresses noise amplified by CLAHE while preserving fine edges; we found it retained specular detail better than Gaussian smoothing or bilateral filter.
Unsharp Masking in float32 (to avoid clipping corneal highlights near 255) re-sharpens structural edges such as reflection contours (strength 1.0–1.5 based on input sharpness).
Saturation Boost in HSV space increases color separation (factor ranges 1.05–1.15 based on input contrast).
All thresholds were empirically tuned. Because these enhancement models are trained on everyday photos rather than corneal reflections, they can reveal detail not apparent in the raw crop but may also introduce artifacts. EyeQ therefore stores both raw and enhanced versions for VLM analysis. Per-component latencies are reported below in Table 1.
| iOS Capture | Server-Side AI Enhancement | ||
| MediaPipe Iris [71] | 0.008 ± 0.005 | Real-ESRGAN 4× [75] | 0.10 ± 0.05 |
| Lock Auto-Focus/Exposure | 0.39 ± 0.23 | CLAHE | 0.003 ± 0.001 |
| 48MP Photo Capture | 1.29 ± 0.22 | NAFNet [10] | 0.31 ± 0.05 |
| Corneal Crop | 0.05 ± 0.01 | Unsharp Mask | 0.02 ± 0.006 |
| Image Stream (iOS → Server) | 0.10 ± 0.12 | Saturation Boost | 0.008 ± 0.002 |
| VLM Scene Understanding + LLM Judge | |||
| Gemini 3.1 Pro [69] | 31.88 ± 24.24 | Gemini 3 Flash [67] | 10.61 ± 2.05 |
| Gemini 3.1 Flash Lite [68] | 8.48 ± 1.42 | LLM Judge (GPT-4.1 [57]) | 1.06 ± 0.24 |
Table 1. Per-component latency (, in seconds). Server-side measurements on an NVIDIA RTX 4080.
Scene Analysis
Once corneal crops have been captured and enhanced, EyeQ sends them to three VLMs in parallel—Google Gemini 3.1 Pro [69], Gemini 3 Flash [67], and Gemini 3.1 Flash Lite [68]—all with the highest available thinking budget, each with a different performance–latency trade-off. Images are grouped into windows and sent as raw–enhanced pairs so each VLM can leverage whichever is clearer. Each VLM progressively builds a separate scene understanding, dispatching a new window once the previous call returns and at least four new image pairs are available: every call receives the current master scene state and returns a refined version, accumulating confirmed objects (clearly identifiable) and tentative objects (educated guesses) while pruning false positives. The output follows a fixed JSON schema containing scene_type (i.e., indoor vs. outdoor), scene_name (e.g., office, kitchen), confirmed_objects and tentative_objects lists, and a brief scene_description. Responses are used as they arrive, so faster models can support interaction sooner (see § 6) while slower ones contribute more reliable scene state, a tiering strategy inspired by WorldScribe [9].
Because VLM outputs use open-ended vocabulary (e.g., “table lamp” vs. “desk lamp”), EyeQ applies an OpenAI GPT-4.1 [57] LLM judge to normalize detected objects and resolve synonyms, making downstream matching more robust. The full VLM and LLM judge prompts are provided in Appendices B and C.
User Study
To assess technical feasibility, we conducted an within-subjects study using EyeQ as the capture instrument. The primary goal was to characterize how capture conditions (e.g. lighting, phone-to-eye distance, angle, motion) and physiological factors (e.g. eye color) affect image quality and VLM scene analysis across indoor and outdoor scenarios. We also gauged comfort and privacy perceptions around corneal sensing before and after participants saw their own eye images. Each session comprised a pre-study questionnaire, a tutorial, 16 capture trials, and a post-study debrief.
Participants
We recruited 10 adults (; ) through social media posts and snowball sampling. The sample included five female and five male; five had dark-colored eyes and five had light-colored eyes. Two wore contacts. See Table 2. We excluded participants who wore glasses, as frames occlude the cornea and lenses add competing reflections. This was a confound to control for, not a limit of the approach, since prior work has used eyewear reflections [16]. We also excluded those with visual, auditory, or mobility impairments. Demographics were collected via a pre-study screening questionnaire. We aimed to balance eye-color representation across the global distribution (79% brown, 8–10% blue, 5% hazel, 2% green [12]), though this limited our sample size, as light-eyed participants were harder to recruit.
| PID | Gender | Age | Eye color | Contacts? | Consent to image inclusion? |
| P1 | Male | 26 | Brown | No | No |
| P2 | Female | 30 | Brown | No | Yes |
| P3 | Male | 24 | Dark brown | No | No |
| P4 | Male | 22 | Dark brown | No | No |
| P5 | Female | 36 | Green | Yes | Yes |
| P6 | Male | 22 | Green | No | Yes |
| P7 | Female | 20 | Hazel | No | Yes |
| P8 | Male | 30 | Green | No | Yes |
| P9 | Female | 31 | Brown | Yes | Yes |
| P10 | Female | 27 | Blue | No | No |
Table 2. Participant demographics ()
Apparatus
We gave participants an iPhone 16 Pro running EyeQ, using its rear camera to capture 48 MP photos and a magnetically mounted mirrored display to preserve the form factor of an ordinary phone (Figure 3). Phone-to-eye distance was measured with a flexible tape measure and phone angle with a digital protractor. For tripod trials, the phone was fixed using a gooseneck holder; for handheld trials, both EyeQ and the researcher monitored posture drift, warning participants of substantial deviation. Lighting was measured as incident illuminance on the face using a light-meter app. Unlike prior corneal reflection work that uses controlled studio lighting, we relied only on typical ambient lighting (e.g., ceiling fixtures), and scenes were populated with everyday objects.
Procedure
Each session consisted of four phases: (1) pre-study attitude questions (~10 min), (2) tutorial (~10 min), (3) capture trials (~90 min), and (4) post-study debrief (~10 min), for a total of about two hours.
Pre-study questionnaire. Before introducing capture tasks, we asked four five-point Likert questions probing: (1) comfort with visible camera use in public, (2) prior awareness that eye reflections reveal surroundings, (3) comfort with a hypothetical “eye-zoom” capability on everyday phones, and (4) comfort if eye data were kept on-device. Participants were asked to explain each rating.
Tutorial. We demonstrated the phone-holding orientation used in the experiment, then ran a short practice trial where participants looked at objects on an office desk (scenario 1) for ~1 minute.
Controlled capture trials. Each participant completed 16 trials across three scenarios, varying four two-level factors: (1) Motion: tripod vs. handheld; (2) Distance: ~20 cm vs. ~40 cm phone-to-eye distance; (3) Angle: ~30° vs. ~60° below eye level; and (4) Lighting: high (floor lamp added) vs. low (lamps off, ceiling dimmed to 20%). Distance and angle values were informed by prior work on phone-holding posture [3, 32, 79] and refined through piloting. The 40 cm, 60° condition approximates typical phone use; the closer levels are less common but remain within that reported range, and also preview what a higher-resolution sensor might one day capture at the farther distance.
To keep the session manageable, not all factors were crossed in every scenario. The Office (10 trials) fully crossed motion, distance, and angle (2 × 2 × 2 = 8) at typical office illumination ( lux; ), plus two lighting-isolation trials ( lux, ; , ) at a fixed posture (20 cm, 30°, tripod). The Kitchen (5 trials; lux, ) crossed distance and angle (2 × 2 = 4) at handheld, plus one trial with EyeQ propped on a counter stand, simulating propped-up tablets and laptop webcams. Campus (1 trial; lux, ) had no factorial control, under naturally varying ambient conditions. The scenarios progressively relaxed control: the office controlled sunlight (blinds closed) and body motion (seated); the kitchen introduced standing and walking; campus added uncontrolled lighting, sustained walking, and squinting in bright sun.
Each trial followed a fixed script: the researcher set the target distance and angle, then ran capture for 2 minutes (office), 3 minutes (kitchen), or 5 minutes (campus). Participants could look around freely but were asked to move a little more slowly than typical and pause briefly when fixating, as AE/AF and 48 MP capture require settled time on current hardware. Breaks were offered between trials.
Scenes and tasks. Each scenario contained 12 ground-truth objects against which we scored VLM outputs. The Office included: desk, monitor, secondary monitor, image of a cat, mouse, keyboard, notebook, pen cup, desk lamp, coffee mug, eyeglasses, and headphones. The Kitchen included: fridge, sink, faucet, coffee machine, coffee cup, bowl, fork, cutting board, pot, spatula, window, and a person (the researcher, periodically entering the scene). Here, we designed five short naturalistic tasks—opening the fridge, chopping on the cutting board, inspecting the sink, checking whether a cup is clean, and reading coffee-machine labels—so participants naturally varied proximity (e.g., standing upright vs. bending to inspect lower objects) and gaze direction. Campus included: fountain, bench, trees, bushes, grass, fire hydrant, people, path, lampposts, trash can, road sign, and buildings. We included the campus scenario to probe whether current hardware can extract any scene information in a fully unconstrained setting.
Post-study debrief interview. After the capture trials, we conducted a semi-structured interview mirroring the pre-study questions. We first asked about comfort with an app inferring surroundings from eye reflections, then showed participants their own corneal images and asked whether seeing the data changed their comfort level. We also showed a rear-camera photo of the same scene for comparison, asked whether the corneal images were clearer or blurrier than expected, what applications they envisioned, and their likelihood of adopting a future version of EyeQ.
Measures and Analysis
Because the study focuses on data collection rather than real-time interaction (which we demonstrate separately in § 6), only the raw corneal crops were saved during the trials; AI enhancement and VLM analysis were performed post-hoc. The raw–enhanced image pairs were then batched into 10-second sliding windows, simulating EyeQ’s progressive analysis, and run through all three VLMs (Gemini 3.1 Pro, Gemini 3 Flash, Gemini 3.1 Flash Lite) independently, each three times, to compare model accuracy across capture conditions. The LLM judge scored each trial’s output against the scenario’s 12-object ground truth, classifying detections as true positives (TPs), false positives (FPs), or false negatives (FNs), from which we computed precision, recall, and F1 score. Two researchers verified the judge’s scoring, reviewing outputs for incorrectly assigned TPs, FPs, and FNs. We also analyzed Likert ratings and open-ended responses on comfort and privacy perceptions. The judge prompt and ground-truth definitions are in Appendix § 11.
The same two researchers then took a closer look at false positives, as an aggregate FP count does not distinguish whether the VLM invented something that was never there or saw something and named it wrong. They independently reviewed every FP against the corresponding raw and enhanced crops, labeling each as a misclassification (optical evidence for a real object was present but mislabeled—e.g., a bright patch from an illuminated window labeled “monitor”) or a hallucination (no corresponding optical evidence in the image), resolving disagreements through discussion.
Finally, because corneal reflections captured on a smartphone are low-resolution, we tested whether fusing them with other context sources could reduce interpretation errors. We prepended a single sentence naming the scene to the VLM prompt (e.g., “The user is in a kitchen”), which we call a “minimal scene prior,” and re-ran the full evaluation. This is not an artificial boost: it simulates what a phone can already assemble from sensors it has (excluding a world-facing camera), such as GPS, time, motion, activity state, ambient light, audio, or the rest of the frame around the eye crop.
Ethics and Privacy
An eye crop may not constitute personally identifiable information today (since this is not a retinal scan), but advancing AI could expose more from these images. Given this uncertainty, we took a conservative approach. Our IRB approved the study after an extended discussion reflecting these novel considerations; the approved protocol does not permit dataset release or use for model training. Participants could opt out of having their eye images appear in this paper; many did, so we also show corneal images from the research team. Recruitment was challenging, as several prospective participants declined once we explained that the study requires close-up eye imaging, citing concerns about data misuse. We discuss these concerns and potential mitigation strategies in § 7.2.
Results
We analyzed 10,635 corneal images across 10 participants (see Figure 6; additional examples in Appendix § 9). We report five analyses: overall feasibility, capture condition effects, object-level detection and error patterns, a minimal scene-prior ablation, and participant perceptions. We evaluated three Gemini models (3.1 Pro, 3 Flash, and 3.1 Flash Lite; all high thinking level), running each three times to account for variability. A Friedman test showed no significant effect of model on F1 score (, ). That said, Flash achieved the highest F1 and most detected objects, so we report Flash results below; per-model breakdowns are in Table 3.
| Scene | Model | No scene context: F1 | P | R | TP | FP | FN | With scene context: F1 | P | R | TP | FP | FN | |
| Office | Pro | .223 ± .171 | .438 ± .309 | .157 ± .136 | 1.9 | 2.1 | 10.1 | .295 ± .132 | .629 ± .202 | .203 ± .117 | 2.4 | 1.5 | 9.6 | 80 |
| Flash | .295 ± .153 | .480 ± .212 | .220 ± .133 | 2.6 | 2.7 | 9.4 | .385 ± .122 | .649 ± .165 | .281 ± .112 | 3.4 | 1.8 | 8.6 | 80 | |
| Flash Lite | .294 ± .130 | .723 ± .256 | .191 ± .096 | 2.3 | 0.9 | 9.7 | .327 ± .121 | .791 ± .211 | .214 ± .097 | 2.6 | 0.8 | 9.4 | 80 | |
| Kitchen | Pro | .171 ± .118 | .404 ± .279 | .117 ± .088 | 1.4 | 2.4 | 10.6 | .221 ± .128 | .575 ± .277 | .147 ± .099 | 1.8 | 1.4 | 10.2 | 50 |
| Flash | .157 ± .073 | .302 ± .185 | .113 ± .060 | 1.4 | 3.7 | 10.6 | .316 ± .131 | .732 ± .216 | .212 ± .106 | 2.5 | 1.0 | 9.5 | 50 | |
| Flash Lite | .085 ± .107 | .178 ± .229 | .058 ± .077 | 0.7 | 2.8 | 11.3 | .202 ± .140 | .534 ± .312 | .133 ± .109 | 1.6 | 1.3 | 10.4 | 50 | |
| Campus | Pro | .080 ± .089 | .208 ± .233 | .050 ± .055 | 0.6 | 1.8 | 11.4 | .278 ± .080 | .860 ± .172 | .175 ± .069 | 2.1 | 0.6 | 9.9 | 10 |
| Flash | .078 ± .106 | .148 ± .193 | .058 ± .084 | 0.7 | 3.4 | 11.3 | .347 ± .115 | .697 ± .152 | .242 ± .108 | 2.9 | 1.3 | 9.1 | 10 | |
| Flash Lite | .059 ± .060 | .107 ± .113 | .042 ± .042 | 0.5 | 3.5 | 11.5 | .222 ± .096 | .502 ± .187 | .158 ± .102 | 1.9 | 2.2 | 10.1 | 10 |
Table 3. VLM scene recognition performance by scene and model, without and with minimal scene prior. F1-score (F1), Precision (P), and Recall (R) reported as . TP = avg true positive count, FP = avg false positive count (misclassifications and hallucinations), FN = avg false negative count (missed), N = number of trials. Each scene contains 12 ground-truth objects.

Figure 6. Participants’ and researchers’ raw–enhanced corneal pairs from darker (top left) to lighter (bottom right) eye colors
Can VLMs Understand Corneal Reflections?
Though likely never trained on corneal imagery, VLMs still extracted meaningful scene information from this low-resolution signal—albeit with limitations. All results below are zero-shot, with no fine-tuning or domain adaptation.
VLMs classified scene type reliably but struggled with scene names. Flash correctly classified scene type (indoor vs. outdoor) in 96.4% of trials (135/140; lighting trials analyzed separately). However, predicting specific scene names proved harder: office was labeled correctly in 71% of cases, but kitchen was mislabeled “office” in 62% of trials, and campus received a mix of indoor (e.g., “room”) and outdoor (e.g., “backyard,” “garden”) guesses (see Figure 4).
Figure 4. Scene name confusion matrix (Flash, without minimal scene prior). Indoor vs. outdoor classification was reliable (96.4%), but specific scene names were harder: office was correctly identified in 71% of cases, kitchen was mislabeled “office” in 62%, and campus received varied guesses (e.g., “backyard,” “garden”).
VLMs detected real objects across all three scenes. A Friedman test revealed a significant effect of scene on F1 (, ), with office performing best (, ), followed by kitchen (, ) and campus (, ), all pairwise comparisons significant (Office–Kitchen: ; Office–Campus: ; Kitchen–Campus: ). Still, EyeQ detected real objects in every scene. In the office (), Flash detected the monitor in 87.5% of trials, the desk in 81.2%, and the keyboard in 37.5%. In the kitchen (), the window was detected in 100.0% of trials and a person in 18.0%. Even outdoors on a campus (), trees were identified in 40.0% of trials, with buildings, people, and a road sign each detected at least once. These detections skew toward larger objects, and the small items in our ground truth (e.g., pen cup, fork, eyeglasses) went undetected, lowering F1. For now, this suggests corneal sensing is best suited for applications needing coarse context and large, high-contrast objects.
Best trials. The strongest individual trial achieved (P1, brown eyes, office, Gemini Pro): 7 true positives with zero false positives, correctly identifying the monitor, secondary monitor, a cat image displayed on the monitor, the desk, coffee mug, headphones, and keyboard. The best kitchen trial reached (P1, brown eyes, Pro), detecting 5 objects: the faucet, sink, bowl, coffee cup, and window. The best campus trial reached (P2, brown eyes, Flash), detecting trees, a building, and a road sign.
What Affects Capture Quality and Analysis?
We examine how capture conditions affect performance. We varied phone distance, angle, motion, and lighting as primary conditions, and additionally analyzed physiological factors (eye color, contact lenses) across participants.
Phone-to-eye distance. Closer capture yielded significantly higher F1 score (, ): 20 cm (, ) outperformed 40 cm (, ).
Phone-to-eye angle. Phone angle alone (30° vs. 60° below eye level) did not significantly affect F1 (, ).
Distance × angle interaction. A Friedman test over the four conditions was significant (, ; Figure 5, left), with 20 cm/30° yielding the highest () and 40 cm/60° the lowest (; , ). Post-hoc analyses indicate distance drove this interaction: 20 cm outperformed 40 cm at both 30° (, ) and 60° (, ), while angle was non-significant at either 20 cm (, ) or 40 cm (, ).
Figure 5. Capture conditions with significant effects on F1. Left: the distance × angle interaction, where 20 cm outperformed 40 cm at both angles. Right: handheld outperformed tripod. Error bars indicate standard deviation. (*: )
Motion. Handheld capture (, ) significantly outperformed tripod (, ; , ; Figure 5, right). Handheld capture allows natural head and arm movement, varying the reflected scene region across frames and giving the VLM more coverage of the surrounding objects.
Lighting. A Friedman test showed no significant effect of lighting (, ). Descriptively, medium ambient lighting () outperformed high () and low ().
Eye color. A Mann–Whitney U test showed no significant effect of eye color (, ), though dark eyes (, ) descriptively outperformed light eyes (, ). Dark-eyed participants also achieved higher ceilings across all three scenes (across all models): office (P1, brown, Pro) vs. .632 (P7, hazel, Flash); kitchen .556 (P1, brown, Pro) vs. .353 (P6, green, Pro); campus .316 (P2, brown, Flash) vs. .143 (P8, green, Pro), with 14 of 15 top-F1 trials coming from dark-eyed participants. This is optically intuitive: darker irises provide a higher-contrast background against which the specular corneal reflection is more visible.
Contact lenses. With only one contact-lens wearer per eye-color group, we report descriptive results only. Both fell below their group means: P9 (brown) vs. .272; P5 (green) .105 vs. .222.
What Do VLMs Detect, Misclassify, and Hallucinate?
Here, we examine what Flash actually detects at the individual object level, and what it gets wrong.
What is detectable. Detection is driven by object size and contrast (Figure 7a). In the office (), the monitor was detected in 87.5% of trials, the desk in 81.2%, and the keyboard in 37.5%. Even a cat image on the secondary monitor was identified in 11.2% of trials. Other office objects appeared less frequently: secondary monitor and desk lamp (each ~12%), coffee mug and headphones (each 6.2%), and mouse and notebook (each 5.0%). In the kitchen (), the window was detected in 100.0% of trials, followed by a person (18.0%), coffee cup (12.0%), and faucet, sink, and bowl (each 2.0%). On campus (), trees were detected in 40% of trials, with buildings, people, and a road sign each at 10.0%.
Figure 7. Per-object rates by scene, without and with a minimal scene prior (Flash): (a) true positive rate against the 12 ground-truth objects, (b) top-5 misclassification rate, and (c) top-5 hallucination rate. Additional context improved detection of objects the VLM could already see but misidentified, and wrong-scene errors (e.g., “monitor” and “desk” in the kitchen) largely vanished.
During our review of the judge’s scoring, we found that Flash also correctly identified objects outside our 12-object ground truth lists—e.g., office chair (33.8% of trials) and laptop (18.8%) in the office; wall art (16.0%), bookshelf (12.0%), and dish rack (2.0%) in the kitchen; and sky (20.0%) and sun (10.0%) outdoors—suggesting VLMs can extract richer information than our scoring rubric captures.
What is missed. Many ground-truth objects were never detected: pen cup and eyeglasses (0/80 office trials); fridge, coffee machine, fork, cutting board, pot, and spatula (0/50 kitchen trials); and fountain, bench, grass, bushes, path, lamppost, and trash can (0/10 campus trials). Moreover, some had high miss rates, such as the secondary monitor (88.8% missed), desk lamp (87.5%), headphones (93.8%), and coffee mug (93.8%) in the office. For small objects (e.g., eyeglasses, fork), the limitation may be optical, yet others (e.g., headphones, coffee mug, fountain) were visually discernible to researchers, suggesting the bottleneck is partly VLM interpretation of this unfamiliar image domain (see §7.1).
Most false positives are misclassifications, not hallucinations. An aggregate FP count does not separate what the optics revealed from what the VLM inferred, so we manually reviewed all 433 false-positive occurrences. The majority—293 (67.7%)—were misclassifications, and this held consistently across scenes: 66.2% in the office (141/213), 69.4% in the kitchen (129/186), and 67.6% on campus (23/34). This distinction matters for feasibility. A hallucination means the VLM reported something the cornea never captured. A misclassification means the cornea captured it and the VLM saw it, but named it wrong—which fine-tuning, better priors, or higher-resolution capture could address. The corneal signal may therefore carry more than the F1 scores alone convey.
What is misclassified. Misclassification patterns were scene-specific (Figure 7b). In the kitchen (), Flash projected office objects onto the scene: “monitor” appeared in 86.0% of trials and “desk” in 82.0%. Our audit traced both to real optical evidence—“monitor” typically corresponded to the bright interior of the open fridge or an illuminated window (43 occurrences), and “desk” to the brown cutting board and countertop (41). In the office (), “window” was the dominant misclassification (87.5% of trials, 70 occurrences), triggered by the bright monitor screen, while “ceiling light” and “light source” (39 combined) corresponded to the desk lamp. On campus (), “window” (80.0%) corresponded to building façades and “desk” (50.0%) to a wooden bench. The common thread is that at this resolution many pixel patches are visually ambiguous—for example, a bright rectangular region could equally be a monitor, window, open fridge, or desk lamp.
What is hallucinated. The remaining 140 false positives (32.3%) had no identifiable optical evidence (Figure 7c). In the kitchen, the projection of office objects extended to “keyboard” (23 occurrences), “lamp” (9), “mouse” (9), and “laptop” (7). In the office, “person” was the most frequent hallucination (27 occurrences) despite no person being present, followed by “blinds” (16) and “trees” (8). On campus, hallucinations were sparse (11 occurrences) and led by “monitor” (4). Notably, both the misclassifications and hallucinations in the kitchen were predominantly office objects, suggesting a cascade: an office label applied to the reflection of a kitchen object shifts the scene guess, biasing future VLM calls to report office objects that were never there. We discuss this further in §7.1.
Can Minimal Scene Context Reduce Errors?
Many objects were optically visible in the corneal reflection, yet VLMs failed to name them correctly. We therefore asked whether adding context from other sensors could close that gap. Phones already have access to other context sources, such as GPS, time, motion, and ambient light. To simulate fusing these with corneal reflections, we prepended a single short sentence naming the scene (e.g., “The user is in a kitchen”) and re-ran the full evaluation (§4.4).
Scene context substantially improved performance. Flash F1 rose from .230 to .358 overall, with precision nearly doubling (.393 → .682)—suggesting that even a minimal scene prior was enough to reduce wrong guesses. True positive count increased from 286 to 426 (+49.0%), while false positives dropped from 433 to 210 (−51.5%). The improvement was most dramatic where performance had been worst: campus F1 rose from .078 to .347, with building detection jumping from 10.0% to 100.0%, trees from 40.0% to 80.0%, and previously invisible objects now detected (path: 0.0% → 40.0%, people: 10.0% → 30.0%, lamppost: 0.0% → 20.0%, bench: 0.0% → 10.0%). Kitchen F1 improved from .157 to .316, with faucet rising from 2.0% to 56.0%, sink from 2.0% to 34.0%, bowl from 2.0% to 28.0%, and fridge from 0% to 16.0% of trials. Office F1 improved from .295 to .385, with the monitor reaching 100.0% detection, the desk reaching 98.8%, and the keyboard rising from 37.5% to 62.5% (Figure 7). Best-case performance also improved: the strongest kitchen trial with context detected 7 objects (, 1 false positive)—including the fridge and cutting board, both undetectable without context—compared to 5 objects without. The campus ceiling rose from 3 to 6 detected objects (), now including path, bench, and person. This suggests corneal reflections become more powerful when fused with other context a phone already collects.
Scene context reduced both misclassification and hallucination. Re-auditing false positives with scene prior, misclassifications fell from 293 to 150 (−48.8%) and hallucinations from 140 to 60 (−57.1%). In the kitchen scenario, office-object errors largely vanished: “monitor” dropped from 86.0% of trials to 2% and “desk” from 82.0% to 0%, while the hallucinated “keyboard” fell from 46.0% to 0%. The remaining kitchen errors stayed within the correct scene: misclassifications such as “small appliance” and “overhead light,” and “microwave” (8 occurrences) as the top hallucination. Errors shifted from impossible to plausible. That said, the model also began using more generic labels (e.g., “small appliance”), despite prompt instructions to be specific.
Scene context reduced the influence of capture conditions. With context, the distance () and motion () effects were no longer significant. This suggests a minimal scene prior reduced sensitivity to suboptimal capture.
What scene context did not fix. Despite improvements, pen cup, eyeglasses, fork, pot, spatula, fountain, grass, fire hydrant, bushes, and trash can remained at 0.0% detection with or without context. These objects are too small or too low-contrast to produce a resolvable reflection. This suggests the added scene context was not simply forcing VLMs to make plausible guesses, since if that were the case, obvious kitchen and outdoor objects like the fork and grass would have appeared. A scene prior appears to help VLMs resolve visual ambiguity in low-resolution reflection patches, largely without biasing them to invent what the cornea never captured.
Participant Perceptions
We collected 5-point Likert ratings (1: very uncomfortable, 5: very comfortable) and open-ended responses before and after the study, probing comfort and privacy concerns around corneal reflection.
Privacy perceptions. Before the study, participants were slightly uncomfortable with corneal reflection (, ), with concerns about surveillance (P1, P7) and not having autonomy over what is captured (P9, P10). Comfort increased in a hypothetical where data stayed on-device (, ), though some remained skeptical: P9 worried “what if other people still found ways to have my corneal reflection?” and P1 asked “what if my phone gets hacked?” The main post-study concern was stealthiness—that corneal capture is invisible to bystanders, unlike a visible camera. P1 felt “less sure if a person is capturing or not,” and P7 noted “AI is getting better . . . so they might find out everything about me.”
Reactions to seeing their own corneal images. After the capture trials but before seeing their own eye images, participants were moderately comfortable (, ), with six participants assuming the images would not reveal much. After seeing their own corneal reflections, comfort dipped slightly (, ) but reactions diverged. Five participants (P1, P2, P3, P6, P7) found images clearer than expected, which often decreased comfort: P1 dropped from 4 to 2 (“it revealed more than I thought… a bit scary”), and P3 noted that seeing “specific components, like a handle of a cup” made them “less confident with sharing.” Five others (P4, P5, P8, P9, P10) found images blurrier, which sometimes increased comfort: P5 rose from 4 to 5, and P4 noted “they still have relatively low quality, which I actually like.” Despite these concerns, interest in trying a future version remained high (, ). Six participants saw value in a camera that can capture surroundings during ordinary phone use, when neither front nor rear camera faces the world.
Corneal vs. RGB photo capture. When comparing corneal reflection to a rear-camera view (, ; 1: prefer rear camera, 5: prefer corneal image), participants surfaced tension between convenience, control, and presence. Some valued that corneal capture lets users stay present in the moment rather than experiencing life through a phone screen—P9 noted “if I’m hiking or in a safari, looking at the world is quite important,” and P6 imagined “a report of moments in your life” without being on the phone. Others preferred the intentionality of a rear camera: P6, who saw merit in both, valued “the feeling of controlling the camera myself,” and P7 wanted “more choice over what is being captured.” Participants envisioned various applications, including pedestrian safety (P1, P9), screen-free photo capture (P6, P9), accessibility for blind or low-vision users (P9), attention monitoring (P2), and grocery assistance (P5, P7). We implemented two of these—pedestrian safety and attention monitoring—as demonstration scenarios (§ 6).
Proof by Demonstration
To explore the real-world potential of corneal reflections as a context source, we built five interaction scenarios on EyeQ’s live pipeline (Figure 8). We selected these scenarios because each requires knowing something about the surroundings, yet asking users to aim a camera would be impractical. In many, no world-facing view is available at all: a phone held below eye level points at rear camera at the ground, and a phone mounted in a car points it at the dashboard. In others, we used the phone as a proxy for laptops and webcams, which have no world-facing camera.

Figure 8. Proof-by-demonstration scenarios. Each column shows a real-world scenario (top) and the corresponding best raw and AI-enhanced corneal images (bottom). All five interactions were successfully triggered by objects detected in live corneal reflections. (*driver detection required a scene-context prior)
The scenarios are: (1) a Social Presence Guard that detects nearby people to pause a voice assistant, (2) an Attention Tracker that detects screen drift to auto-play or pause video, (3) a Pedestrian Safety Alert that warns phone-distracted users of approaching vehicles, (4) a Courtesy Switch that recognizes a presentation surface to activate Do Not Disturb, and (5) a Driver Detector that identifies a steering
wheel to restrict phone use. In all of these, corneal reflection is the only reasonable view of the scene these devices have.
All five triggered correctly: EyeQ detected the relevant objects from live corneal reflections and fired the intended action. VLMs identified “person” to pause the assistant; “phone” to play and “window” to pause video; “vehicle” and “road” to warn the pedestrian; “whiteboard” and “screen” to trigger Do Not Disturb; and “steering wheel” and “dashboard” to infer driving. Four required no scene prior; only driver detection needed one (i.e., “the user is in a car”). Detection latency varied, requiring 2–4 VLM retries (∼23 seconds) before a stable scene state emerged (see supplementary video).
More interactions become possible once a device with only a user-facing camera can sense what is in front of the user. On a phone, a user reading a grocery list could have the aisle recognized, surfacing relevant coupons without lifting and pointing the rear camera. Beyond phones, the Social Presence Guard could run on an office webcam, detecting someone approaching from behind the screen and dimming the display to prevent shoulder surfing. The Courtesy Switch could run on a student’s laptop in a classroom. On smartglasses, an inward-facing camera could supply both attention tracking and scene context, reducing the sensing hardware such devices require. While challenges remain, corneal-reflection sensing can already support meaningful context-driven interactions.
Discussion
As an initial exploration of corneal-reflection sensing with smartphone-grade optics, our primary goal was to establish whether scene understanding from this low-resolution signal is feasible, and what remains open. The results are encouraging: off-the-shelf VLMs classified indoor vs. outdoor at 96.4% and detected real objects across all three scenes, with the VLM more often misclassifying what the cornea captured than hallucinating what it did not. We discuss where corneal sensing is headed, the privacy questions it raises, and the limitations of this work.
Where Corneal Sensing Is Headed
We begin by enumerating what corneal sensing can already achieve, before turning to its limitations and trajectory.
What the cornea already reveals. Our results suggest corneal sensing can support coarse context tasks—distinguishing indoors from outdoors, detecting bigger objects like a monitor, desk, faucet, or steering wheel, and sensing nearby people—while collapsing the two-camera requirement of prior systems [27, 39] into one. More fundamentally, these findings reframe the eye in HCI: beyond an input channel for gaze and attention [23, 36], the cornea can act as a passive scene camera. Yet current performance is almost certainly a lower bound. VLMs see far more than they name: 67.7% of false positives were misclassifications rather than hallucinations, one sentence of scene prior raised true positives 49% while halving false positives, and Flash produced 346 correct detections outside our ground truth. Several paths could raise this floor, such as using the full RGB face frame as scene context before cropping, video or burst super-resolution across consecutive corneal frames [2, 50, 77] when front cameras catch up, and fine-tuning image restoration models and VLMs for this domain they presumably never encountered. Smartglasses, with eye-facing cameras at close range [8], could also use corneal reflections given that closer capture improves performance. Moreover, with the phone propped ∼115 cm from the eye, EyeQ still detected the window in every kitchen trial, suggesting future smartphone cameras could better support handheld use. Longer-term, the gaze coupling is intriguing; since the pupil is darker than the iris, the reflection center is roughly aligned with gaze [66], potentially enabling joint scene and attention inference.
Challenges of continuous VLM inference. To build scene understanding over time, EyeQ chains VLM calls across successive batches, with each call receiving the previous state as JSON rather than the images behind it. This was among our most difficult challenges, and two key problems still remain. First, error cascading: a misclassified pixel patch (e.g., a window reflection in the kitchen labeled “monitor”) biases the scene guess (e.g., toward “office”) which increases wrong-scene predictions (e.g., “desk,” “keyboard”) and biases future calls. Because only the JSON state carries forward, a wrong label cannot be re-checked against the pixels that produced it. Second, the pipeline lacks object persistence: if a monitor leaves and re-enters the frame, EyeQ cannot tell whether it is the same one or a new detection. More broadly, chained inference must balance being too destructive (overwriting correct detections) and too trusting (accumulating errors)—a tension that affects any live VLM system, but amplified by corneal imagery’s low resolution and per-frame confidence. Stricter frame gating before inference, or passing some images alongside textual state, could reduce this cascading. This may also partly explain the gap between average and best-case performance, where the strongest office trial reached F1=.737 against a scene mean of .295. Relatedly, in manually auditing the corneal images, two researchers could often identify objects the VLM did not name; see Figure 6 and Appendix § 9 for examples.
Hard limits. Even with video-rate capture, better cameras, and fine-tuned models, some constraints will remain. Optically, the cornea is a convex mirror roughly 7.8 mm in radius [38], compressing a wide-angle view of the scene into a few millimeters where small objects will only take up a few pixels. Many objects at 0% detection (e.g., pen cup, fork, spatula) were small or low-contrast. Biologically, eye color, contact lenses, and eyelash occlusion introduce per-frame variance that temporal aggregation cannot resolve (e.g., 14/15 top-F1 trials came from dark-eyed participants). Practically, the eye has to be in frame, which users will not consciously maintain. Direct sunlight can also wash out reflections. EyeQ is ultimately bounded by the cornea’s ability to reflect the scene. But corneal sensing need not match a world-facing camera, as even coarse context already enables useful interactions.
Privacy as Both Risk and Opportunity
Corneal-reflection sensing raises privacy questions that differ from those of conventional cameras. We discuss three dimensions: stealthiness of capture, potential platform-level mitigations, and tension around improving this technology.
Stealthiness and consent. Unlike a rear-facing camera, corneal capture raises a two-sided transparency problem: users cannot easily verify what is inferred from their eyes, and bystanders cannot tell they are being captured at all. For users, future systems could incorporate a “what does my eye reveal?” preview and a visible capture indicator, following the transparency patterns modern OSes use for camera and location access. For bystanders, the problem is harder: corneal data is a side effect of any face-facing photo and video, making capture difficult to signal—though corneal reflections could be removed entirely (see below). Despite these concerns, participants rated interest in a future version at 4.1/5, and proposed applications we went on to prototype (pedestrian safety, attention monitoring), suggesting the interaction value is compelling enough to try to navigate the privacy tension.
Platform-level mitigations. Rather than treating corneal reflections purely as a threat, we see an opportunity for proactive platform-level protection. Since smartphone cameras already apply computational photography to each image [22, 77], adding corneal-region inpainting—analogous to red-eye reduction [15]—could mitigate this side channel before images reach applications. This is urgent, as selfies are already high-resolution, likely containing corneal data that could be analyzed retroactively. In fact, in 2019 a stalker located a Japanese pop idol by enlarging corneal reflections in her public selfies, identifying her local train station [35]. Fortunately, on-device protection is increasingly feasible, as our restoration models are lightweight and on-device VLM inference is rapidly maturing [72], ensuring corneal data need never leave the phone. Our participants’ comfort ratings support this, rising from to 3.9 in a hypothetical where eye data remains on-device.
The fine-tuning paradox. An obvious next step is training VLMs on corneal imagery, yet despite collecting over 10,000 images, we chose not to given privacy concerns. Our IRB did not permit dataset or model release, and many participants opted out of image inclusion. This tension is inherent: advancing corneal sensing requires data that ethical practices restrict. It cuts deeper—should we want this capability to improve? Better corneal understanding enables compelling interactions from a single camera, but also more potent surveillance [4, 5]. We believe the benefits warrant continued research, provided privacy is handled carefully. One path is synthetic corneal data, rendering scenes onto eye models [13, 25] to avoid real-eye data collection. Until such datasets exist, our results show what off-the-shelf VLMs achieve with no domain adaptation—itself a meaningful zero-shot benchmark for future work.
Limitations and Future Work
As the first exploration of VLM-based corneal scene understanding, our study prioritized breadth—testing multiple scenes, conditions, and models—over depth in any single dimension. Our sample () is small; a larger study is needed to draw more robust conclusions about physiological factors such as eye color effects. EyeQ also captures with the rear camera as a stand-in for higher-resolution front sensors (§3.1), and we tested only three scenes—environments such as moving vehicles or crowded public spaces may behave differently. Finally, we excluded glasses wearers, a substantial fraction of the population, and evaluated only the Gemini model family. A formal comparison of human and VLM performance on the same crops would help separate what the cornea carries from what current models can name.
Conclusion
In this work, we explored whether corneal reflections captured with smartphone-grade optics can serve as a source of scene context, and whether off-the-shelf VLMs can read them. We built EyeQ, an end-to-end pipeline pairing iris-targeted capture with server-side restoration and progressive VLM analysis. Across 10,635 corneal images from ten participants, VLMs classified indoor vs. outdoor reliably and detected objects in every scene, and most errors were misclassifications of real optical evidence rather than hallucinations. Because we did not fine-tune, these results serve as a zero-shot benchmark for future work. We hope this work motivates the HCI and AI communities to explore this untapped sensing channel—with the privacy care this intimate signal demands.
Full Eye Collage

Figure 9. A researcher’s (dark brown eyes) corneal reflection images across all study trials
VLM Scene Analysis Prompt
BASE_PROMPT = """
You are analyzing corneal reflection images - photographs of a human eye where the curved corneal surface acts as a convex
↪ mirror, capturing a wide-angle reflection of the surrounding environment. Think of it like a security mirror or a chrome
↪ ball reflection, but smaller and on an eye.
Visual characteristics to expect:
- The reflection appears in the pupil/iris region
- Images are warped due to the cornea's curved geometry
- Details may be low-resolution, noisy, or slightly blurred
- Lighting sources often appear as bright spots
- Shapes are distorted but preserve relative spatial relationships
The corneal reflection image was captured by a smartphone camera. You will likely see the smartphone (e.g., a dark rectangle
↪ with camera lenses) and the user's hand/arm or a tripod device holding it. Ignore the smartphone - it is the capture
↪ device, not part of the scene. Focus on describing everything else in the reflection.
You are receiving multiple image pairs. Each pair consists of:
- A RAW corneal crop (lower but decent resolution, unprocessed - preserves true highlight and edge detail)
- Its AI-ENHANCED version (4x resolution via Real-ESRGAN, improved contrast - may reveal more detail but could have minor
↪ artifacts)
Both images in a pair show the same reflection from the same moment. Use whichever (or both) provides clearer detail for each
↪ object. The raw version is more trustworthy for object details, while the enhanced version may reveal more dark or
↪ low-contrast objects/areas.
This is a multi-call pipeline: your JSON response will be passed as context to the next VLM call processing the next batch of
↪ images. Errors, hallucinations, or noise in your output will propagate and mislead future calls. Accuracy matters more
↪ than completeness - a short, correct list is far better than a long, noisy one. Image quality varies significantly between
↪ batches - many frames will be blurry or uninformative, with occasional clear frames that reveal real detail. Do not force
↪ observations from poor-quality images.
Within each batch, image quality will vary. Focus your analysis on the clearest pairs. Blurry or motion-affected pairs should
↪ be disregarded rather than interpreted - forcing meaning from noise is worse than skipping a frame.
Object naming rules:
- Be concise and specific (e.g., "giraffe", not "tall figure with four legs and a tail, which could be an elephant or giraffe")
- Before adding an object, check if it already exists under a synonym or different name (e.g., "sneakers" and "running shoes"
↪ are the same object). Keep the more specific name. Do NOT add duplicates.
- Partially visible or blurry objects can be identified through educated guesses using surrounding context. Pure guesses from
↪ ambiguous blobs with no contextual support should be omitted entirely.
- Do not let any examples in these instructions bias your observations. These examples are purely illustrative. Identify
↪ objects based solely on visual evidence in the corneal reflection.
"""
FIRST_CALL_CONTEXT = """
This is the FIRST batch of corneal reflection images. No previous scene analysis exists yet.
STEP 1 - OBSERVE: Examine these images and describe what you can see in the reflections. Identify the scene type, a specific
↪ scene name, and any distinct objects.
STEP 2 - RECORD: Build the initial scene understanding. For each object, assign a confidence level:
- "confirmed": Clearly identifiable - you can see its shape, color, or distinctive features
- "tentative": Partially visible, blurry, or obscured, but context and educated reasoning suggest what it is
If the images are too blurry to identify anything with reasonable confidence, it is acceptable to output a minimal response
↪ with just scene_type and scene_name. Do not pad the object lists with guesses from ambiguous blobs."""
SUBSEQUENT_CALL_CONTEXT = """
STEP 1 - OBSERVE: Examine these new images and describe what you can see in the reflections on their own - the scene type, a
↪ specific scene name, any objects, and spatial layout. Do this BEFORE looking at the previous scene understanding below.
STEP 2 - RECONCILE: Now, merge your fresh observations with the existing scene understanding.
CURRENT SCENE UNDERSTANDING (built from {window_count} previous batch(es)):
{master_json}
Merging rules:
- If your independent observation AGREES with the current understanding, reinforce it. Promote "tentative" objects to
↪ "confirmed" when you can clearly see their shape, color, or distinctive features in the current images - not simply
↪ because they appeared in a previous batch.
- If your independent observation CONTRADICTS the scene_type or scene_name, and the current images provide clear visual
↪ evidence for the correction, update it. Remove objects that no longer make sense in the corrected scene. Do not override
↪ an established scene label based on ambiguous or blurry images alone.
- ADD new objects with appropriate confidence. "confirmed" = clearly identifiable from the image evidence alone (distinct
↪ shape, color, or features). "tentative" = educated guess from partial, blurry, or obscured views where context suggests
↪ what it is. Check for duplicates and synonyms before adding.
- "tentative" objects may be removed if they seem inconsistent with accumulated evidence. "confirmed" objects should rarely
↪ be removed.
- If this batch of images is too blurry or low-quality to confidently identify anything new, return the previous understanding
↪ UNCHANGED. Do not add objects from ambiguous blobs just to contribute something.
IMPORTANT - confirmation bias: You are susceptible to anchoring on what is already in the master. An object appearing in the
↪ previous master does NOT make it more likely to be real. For each tentative object you are about to keep or promote, ask
↪ yourself: if this object were NOT already in the master, would the current images alone give me enough reason to add it?
↪ If no, do not promote it. Tentative objects should only be promoted to confirmed when you can independently identify them
↪ in the current batch - not because they have "been in the list for a while."
CRITICAL: The corneal reflection only captures a PORTION of the surrounding scene at any given time. As the eye moves,
↪ different objects enter and leave the reflection. The ABSENCE of an object in the current batch does NOT mean it was wrong
↪ - it simply may not be in the reflection right now. NEVER remove an object just because you don't see it in these specific
↪ images."""
LLM Judge Prompt
JUDGE_PROMPT = """
You are evaluating a Vision Language Model's scene analysis accuracy based on corneal reflections.
## Ground Truth
- scene_type: "{gt_scene_type}"
- scene_name: "{gt_scene_name}" (also acceptable: {gt_aliases})
- objects present in the scene: {gt_objects}
## Explicit Object Aliases (ALWAYS accept these matches)
{object_aliases}
## VLM Response
- scene_type: "{vlm_scene_type}"
- scene_name: "{vlm_scene_name}"
- confirmed_objects: {vlm_confirmed}
- tentative_objects: {vlm_tentative}
## Your Task
1. Check if scene_type matches (indoor/outdoor)
2. Check if scene_name is semantically equivalent to ground truth or any alias
3. For EACH VLM-detected object (both confirmed and tentative):
- FIRST check explicit aliases above - if VLM output matches any alias, ACCEPT it
- If NOT in explicit aliases, use your best judgment:
- Accept clear synonyms at similar specificity (e.g., "display" -> "monitor")
- Accept functional equivalents only if specific enough (e.g., "computer screen" -> "monitor")
- Reject if the term could match multiple ground truth objects (e.g., "light" when both "desk lamp" and "ceiling light"
↪ exist)
- Reject overly vague/generic terms: "device", "furniture", "electronics", "object", "thing", "bright rectangle", "dark
↪ shape"
- When genuinely uncertain, reject - we reward specificity
4. Identify which ground truth objects were NOT detected by the VLM (in either confirmed or tentative)
IMPORTANT: Evaluate confirmed and tentative objects TOGETHER as a single pool of detected objects. Both count equally for
↪ matching purposes.
Respond with ONLY valid JSON in this exact format:
{
"""
"scene_type_correct": <true or false>,
"scene_name_correct": <true or false>,
"true_positives": {{"<vlm_object>": "<matched_gt_object>", ...}},
"false_positives": ["<vlm_object_not_in_gt>", ...],
"false_negatives": ["<gt_object_not_detected>", ...]
}}
"""
{
"office": {
"scene_type": "indoor",
"scene_name": "office",
"scene_name_aliases": ["work desk", "home office", "workspace", "study", "desk area", "workstation", "computer desk"],
"objects": [
"monitor",
"secondary monitor",
"mouse",
"keyboard",
"notebook",
"pen cup",
"desk lamp",
"coffee mug",
"eyeglasses",
"headphones",
"cat image",
"desk"
],
"object_aliases": {
"monitor": ["display", "screen", "computer screen", "computer monitor", "primary monitor", "main monitor"],
"secondary monitor": ["second monitor", "second display", "second screen", "dual monitor", "side monitor"],
"mouse": ["computer mouse"],
"keyboard": ["computer keyboard"],
"notebook": ["stack of books", "books", "book", "notepad"],
"pen cup": ["pencil cup", "pen holder", "pencil holder", "pen container", "stationery holder", "cup of pens"],
"desk lamp": ["table lamp", "lamp", "desk light"],
"coffee mug": ["mug", "cup", "coffee cup"],
"eyeglasses": ["glasses", "spectacles", "reading glasses"],
"headphones": ["headset", "earphones", "over-ear headphones"],
"cat image": ["cat", "kitten"],
"desk": ["table", "work desk", "computer desk", "workstation"]
}
},
"kitchen": {
"scene_type": "indoor",
"scene_name": "kitchen",
"scene_name_aliases": ["kitchenette", "cooking area", "kitchen counter", "kitchen area"],
"objects": [
"fridge",
"sink",
"faucet",
"coffee machine",
"coffee cup",
"bowl",
"fork",
"cutting board",
"pot",
"spatula",
"person",
"window"
],
"object_aliases": {
"fridge": ["refrigerator", "freezer"],
"sink": ["kitchen sink"],
"faucet": ["kitchen faucet", "tap", "water tap"],
"coffee machine": ["coffee maker", "espresso machine", "coffee brewer"],
"coffee cup": ["mug", "cup", "coffee mug"],
"bowl": ["mixing bowl", "cereal bowl", "white bowl"],
"fork": ["utensil", "silverware"],
"cutting board": ["chopping board"],
"pot": ["cooking pot", "saucepan", "pan"],
"spatula": ["cooking utensil", "flipper"],
"person": ["human", "someone", "figure", "man", "woman", "people"],
"window": ["kitchen window", "windows"]
}
},
"outdoor": {
"scene_type": "outdoor",
"scene_name": "campus",
"scene_name_aliases": ["university quad", "quad", "campus quad", "courtyard", "university courtyard", "campus
↪ courtyard", "plaza", "college campus"],
"objects": [
"fountain",
"bench",
"trees",
"grass",
"fire hydrant",
"bushes",
"people",
"path",
"lamppost",
"trash can",
"road sign",
"buildings"
],
"object_aliases": {
"fountain": ["water fountain"],
"bench": ["stone bench", "wooden bench", "seat"],
"trees": ["tree", "large tree", "bare tree"],
"grass": ["lawn", "green grass", "grassy area", "turf"],
"fire hydrant": ["hydrant"],
"bushes": ["bush", "shrubs", "shrub"],
"people": ["person", "pedestrians", "pedestrian", "human", "humans", "figures", "passersby", "woman", "man"],
"path": ["walkway", "sidewalk", "pathway", "walking path", "footpath", "trail", "pavement"],
"lamppost": ["lampposts", "street light", "street lamp", "light pole", "street lights", "lamp post", "light post"],
"trash can": ["garbage can", "waste bin", "bin"],
"road sign": ["sign", "street sign", "signage"],
"buildings": ["building", "structure", "structures"]
}
}
}References
- [1]Noora Alsakar, Yasmeen Abdrabou, Simone Stumpf, and Mohamed Khamis. 2023. Investigating Privacy Perceptions and Subjective Acceptance of Eye Tracking on Handheld Mobile Devices. Proc. ACM Hum.-Comput. Interact. 7, ETRA, Article 164 (May 2023), 16 pages. doi:10.1145/3591133DOI
- [2]Hadi Alzayer, Kevin Zhang, Brandon Feng, Christopher A. Metzler, and Jia-Bin Huang. 2024. Seeing the World through Your Eyes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).arxiv.org/abs/2306.09348
- [3]Yuliya Bababekova, Mark Rosenfield, Jennifer E Hue, and Rae R Huang. 2011. Font size and viewing distance of handheld smart phones. Optometry and Vision Science 88, 7 (2011), 795–797.DOI
- [4]Michael Backes, Tongo Chen, Markus Duermuth, Hendrik P.A. Lensch, and Martin Welk. 2009. Tempest in a Teapot: Compromising Reflections Revisited. In 2009 30th IEEE Symposium on Security and Privacy. 315–327. doi:10.1109/SP.2009.20DOI
- [5]Michael Backes, Markus Duermuth, and Dominique Unruh. 2008. Compromising Reflections-or-How to Read LCD Monitors around the Corner. In Proceedings of the 2008 IEEE Symposium on Security and Privacy (SP ’08). IEEE Computer Society, USA, 158–169. doi:10.1109/SP.2008.25DOI
- [6]Martin Baumeister, Evdoxia Terzi, Yurdagül Ekici, and Thomas Kohnen. 2004. Comparison of manual and automated methods to determine horizontal corneal diameter. Journal of Cataract & Refractive Surgery 30, 2 (2004), 374–380.pubmed.ncbi.nlm.nih.gov/15030827
- [7]Jan PG Bergmanson and Jesus G Martinez. 2017. Size does matter: what is the corneo-limbal diameter? Clinical and Experimental Optometry 100, 5 (2017), 522–528.DOI
- [8]Andreas Bulling and Kai Kunze. 2016. Eyewear computers for human-computer interaction. Interactions 23, 3 (April 2016), 70–73. doi:10.1145/2912886DOI
- [9]Ruei-Che Chang, Yuxuan Liu, and Anhong Guo. 2024. WorldScribe: Towards Context-Aware Live Visual Descriptions. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST ’24). Association for Computing Machinery, New York, NY, USA, Article 140, 18 pages. doi:10.1145/3654777.3676375
- [10]Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. 2022. Simple baselines for image restoration. In European conference on computer vision. Springer, 17–33.arxiv.org/abs/2204.04676
- [11]Mustafa Doga Dogan, Eric J. Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colaco, Johnny Lee, Mar Gonzalez-Franco, and David Kim. 2024. Augmented Object Intelligence with XR-Objects. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST ’24). Association for Computing Machinery, New York, NY, USA, Article 19, 15 pages. doi:10.1145/3654777.3676379DOI
- [12]Saeed Dorgalaleh, Karim Naghipoor, Ahmad Barahouie, Farzad Dastaviz, and Morteza Oladnabi. 2020. Molecular and biochemical mechanisms of human iris color: A comprehensive review. Journal of Cellular Physiology 235, 12 (2020), 8972–8982.DOI
- [13]Mengqi Du, Yue Zhang, Jianhua Zhang, and Honghai Liu. 2024. CRED: A Corneal Reflection and Environment Dataset. In 2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD). 2246–2251. doi:10.1109/CSCWD61410.2024.10580597
- [14]David L Fried. 1978. Probability of getting a lucky short-exposure image through turbulence. Journal of the Optical Society of America 68, 12 (1978), 1651–1658.DOI
- [15]Matthew Gaubatz and Robert Ulichney. 2002. Automatic red-eye detection and correction. In Proceedings. International Conference on Image Processing, Vol. 1. IEEE, I–I.DOI
- [16]Jens Grubert, Eyel Ofek, Michel Pahud, Matthias Kranz, and Dieter Schmalstieg. 2016. GlassHands: Interaction Around Unmodified Mobile Devices Using Sunglasses. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS ’16). Association for Computing Machinery, New York, NY, USA, 215–224. doi:10.1145/2992154.2992162
- [17]GSMArena. 2024. Motorola Edge 50 Pro. https://www.gsmarena.com/motorola_edge_50_pro-12909.php
- [18]GSMArena. 2025. Oppo Find X9 Pro. https://www.gsmarena.com/oppo_find_x9_pro-14094.php
- [19]E.D. Gustrin and M. Eizenman. 2006. General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Transactions on Biomedical Engineering 53, 6 (2006), 1124–1133. doi:10.1109/TBME.2005.863952
- [20]Dan Witzner Hansen and Qiang Ji. 2009. In the eye of the beholder: A survey of models for eyes and gaze. IEEE transactions on pattern analysis and machine intelligence 32, 3 (2009), 478–500.
- [21]Hassan Hashemi, Mehdi Khabazkhoob, Mohammad Hassan Emamian, Mohammad Shariati, Abbasali Yekta, and Akbar Fotouhi. 2015. White-to-white corneal diameter distribution in an adult population. Journal of current ophthalmology 27, 1–2 (2015), 21–24.
- [22]Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35, 6, Article 192 (Dec. 2016), 12 pages. doi:10.1145/2980179.2980254
- [23]Robert JK Jacob and Keith S Karn. 2003. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In The mind’s eye. Elsevier, 573–605.DOI
- [24]Rohit Jenkins and Christie Kerr. 2013. Identifiable images of bystanders extracted from corneal reflections. PloS one 8, 12 (2013), e83325.DOI
- [25]Joohwan Kim, Michael Stengel, Alexander Majerick, Shalini De Mello, David Dunn, Samuli Laine, Morgan McGuire, and David Luebke. 2019. NVGaze: An Anatomically-Informed Dataset for Low-Latency, Near-Eye Gaze Estimation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3290605.3300780DOI
- [26]Taejun Kim, Miam Mollon, Riku Arakawa, and Chris Harrison. 2026. HiFiGaze: Improving Eye Tracking Accuracy Using Screen Content Knowledge. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26). Association for Computing Machinery, New York, NY, USA, Article 674, 12 pages. doi:10.1145/3772318.3791339DOI
- [27]Christian Lander, Markus Lothchefeld, and Antonio Krüger. 2018. hEYEBbrid: A hybrid approach for mobile calibration-free gaze estimation. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 149 (Jan. 2018), 29 pages. doi:10.1145/3161166
- [28]Nicholas M Law, Craig D Mackay, and John E Baldwin. 2006. Lucky imaging: high angular resolution imaging in the visible from the ground. Astronomy & Astrophysics 446, 2 (2006), 739–745.
- [29]Geonsun Lee, Min Xia, Nels Numan, Xun Qian, David Li, Yanhe Chen, Achin Kulshrestha, Ishan Chatterjee, Jinda Zhang, Dinesh Manocha, David Kim, and Ruofei Du. 2025. Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agents. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Association for Computing Machinery, New York, NY, USA, Article 119, 22 pages. doi:10.1145/3746059.3747748DOI
- [30]Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S. Rodriguez, and Jon E. Froehlich. 2024. GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 408, 20 pages. doi:10.1145/3613904.3642230DOI
- [31]Jongseok Lee, Hyunseong Kim, Masaaki Fukumoto, and Byungjoo Lee. 2017. Reflector: Distance-Independent, Private Pointing on a Reflective Screen. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 351–364. doi:10.1145/3126594.3126655
- [32]Soojeong Lee, Hwayeong Kang, and Gwen Lau. 2025. Hand flexion angle while using a smartphone. Ergonomics 58, 2 (2015), 220–226.
- [33]Shijun Liang, Haofei Wang, and Feng Lu. 2024. EyeIR: Single Eye Image Inverse Rendering In the Wild. In ACM SIGGRAPH 2024 Conference Papers (Denver, CO, USA) (SIGGRAPH ’24). Association for Computing Machinery, New York, NY, USA, Article 42, 11 pages. doi:10.1145/3641519.3657506
- [34]Yan Long, Chen Yan, Shilin Xiao, Shivan Prasad, Wenyuan Xu, and Kevin Fu. 2023. Private eye: On the limits of textual screen peeking via eyeglass reflections in video conferencing. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 3432–3449.DOI
- [35]Kate Lyons. 2019. Japanese assault suspect ‘tracked down pop star via eye reflection in selfie’. The Guardian. https://www.theguardian.com/world/2019/oct/11/japanese-assault-suspect-tracked-down-pop-star-via-eye-reflection-in-selfie-
- [36]Päivi Majaranta and Andreas Bulling. 2014. Eye tracking and eye-based human–computer interaction. In Advances in physiological computing. Springer, 39–65.DOI
- [37]Diego Martinez Plasencia, Florent Berthaut, Abhijit Karnik, and Sriram Subramanian. 2014. Through the combining glass. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 341–350. doi:10.1145/2642918.2647351
- [38]KP Mashige. 2013. A review of corneal diameter, curvature and thickness values and influencing factors. African Vision and Eye Health 72, 4 (2013), 185–194.DOI
- [39]Sven Mayer, Gierad Laput, and Chris Harrison. 2020. Enhancing Mobile Voice Assistants with WorldGaze. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–10. doi:10.1145/3313831.3376479
- [40]Takashi Nagamatsu, Kaoru Fukuda, and Michiya Yamamoto. 2014. Development of Corneal Reflection-based Gaze Tracking System for Public Use. In Proceedings of The International Symposium on Pervasive Displays (Copenhagen, Denmark) (PerDis ’14). Association for Computing Machinery, New York, NY, USA, 194–195. doi:10.1145/2611009.2614391DOI
- [41]Takashi Nagamatsu, Yukina Iwamoto, Junzo Kamahara, Naoki Tanaka, and Michiya Yamamoto. 2010. Gaze estimation method based on an aspherical model of the cornea: surface of revolution about the optical axis of the eye. In Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications. 255–258.DOI
- [42]Yudai Nakamura, Kaori Ikematsu, Naoto Takayanagi, Kunihiro Kato, Toshiya Isomoto, and Yuta Sugiura. 2026. ReflectTrace: Touchless Hover Interaction on Commodity Smartphones via Corneal Reflection. In Proceedings of the 31st International Conference on Intelligent User Interfaces (IUI ’26). Association for Computing Machinery, New York, NY, USA, 1855–1866. doi:10.1145/3742413.3789138DOI
- [43]Atsushi Nakazawa and Christian Nitschke. 2012. Point of gaze estimation through corneal surface reflection in an active illumination environment. In European Conference on Computer Vision. Springer, 159–172.DOI
- [44]Atsushi Nakazawa, Christian Nitschke, and Tōyoki Nishida. 2015. Non-calibrated and real-time human view estimation using a mobile corneal imaging camera. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE, 1–6.DOI
- [45]Atsushi Nakazawa, Christian Nitschke, and Tōyoki Nishida. 2016. Registration of eye reflection and scene images using an aspherical eye model. Journal of the Optical Society of America A 33, 11 (2016), 2264–2276.DOI
- [46]K. Nishino, P.N. Belhumeur, and S.K. Nayar. 2005. Using eye reflections for face recognition under varying illumination. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, Vol. 1. 519–526 Vol. 1. doi:10.1109/ICCV.2005.243
- [47]Ko Nishino and S.K. Nayar. 2004. The world in an eye. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 1. I–I. doi:10.1109/CVPR.2004.1315066DOI
- [48]Ko Nishino and Shree K. Nayar. 2004. Eyes for relighting. ACM Trans. Graph. 23, 3 (Aug. 2004), 704–711. doi:10.1145/1015706.1015783DOI
- [49]Ko Nishino and Shree K Nayar. 2006. Corneal imaging system: Environment from eyes. International Journal of Computer Vision 70, 1 (2006), 23–40.DOI
- [50]Christian Nitschke and Atsushi Nakazawa. 2012. Super-Resolution from Corneal Images.. In BMVC. 1–12.DOI
- [51]Christian Nitschke and Atsushi Nakazawa. 2018. Corneal Imaging. The Wiley Handbook of Human Computer Interaction 1 (2018), 445–514.DOI
- [52]Christian Nitschke, Atsushi Nakazawa, and Tōyoki Nishida. 2013. I see what you see: point of gaze estimation from corneal images. In 2013 2nd IAPR Asian Conference on Pattern Recognition. IEEE, 298–304.DOI
- [53]Christian Nitschke, Atsushi Nakazawa, and Haruo Takemura. 2009. Display-camera calibration from eye reflections. In 2009 IEEE 12th International Conference on Computer Vision Theory and Applications. IEEE, 1226–1233.
- [54]Christian Nitschke, Atsushi Nakazawa, and Haruo Takemura. 2011. Display-camera calibration using eye reflections and geometry constraints. Computer Vision and Image Understanding 115, 6 (2011), 835–853.DOI
- [55]Christian Nitschke, Atsushi Nakazawa, and Haruo Takemura. 2013. Corneal Imaging Revisited: An Overview of Corneal Reflection Analysis and Applications. IPSJ Transactions on Computer Vision and Applications 5 (2013), 1–18. doi:10.2197/ipsjtcva.5.1DOI
- [56]Yuki Ohshima, Kyosuke Maeda, Yusuke Edamto, and Atsushi Nakazawa. 2021. Visual Place Recognition from Eye Reflection. IEEE Access 9 (2021), 57364–57371. doi:10.1109/ACCESS.2021.3071406DOI
- [57]OpenAI. 2025. Introducing GPT-4.1 in the API. OpenAI Blog. https://openai.com/index/gpt-4-1/
- [58]Minso Park. 2023. mediapipe-prebuilt. https://github.com/miinso/mediapipe-prebuilt.
- [59]Ethan Qi. 2021. Smartphone Primary Camera Resolution Continues to Improve Amid Growing Pressure on BoM Cost. Counterpoint. https://counterpointresearch.com/en/insights/smartphone-primary-camera-resolution-improve Rahul Raguram, Andrew M. White, Dibyendusekhar Goswami, Fabian Monrose, and Jan-Michael Frahm. 2011. iSpy: automatic reconstruction of typed input from compromising reflections. In Proceedings of the 18th ACM Conference on Computer and Communications Security (Chicago, Illinois, USA) (CCS ’11). Association for Computing Machinery, New York, NY, USA, 527–536. doi:10.1145/2046707.2046769
- [62]Florian Rüfer, Anke Schröder, and Carl Erb. 2005. White-to-white corneal diameter: normal values in healthy humans obtained with the Orbscan II topography system. Cornea 24, 3 (2005), 259–261.pubmed.ncbi.nlm.nih.gov/15778595
- [63]Daniel Schneider and Jens Grubert. 2017. Towards Around-Device Interaction using Corneal Imaging. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (Brighton, United Kingdom) (ISS ’17). Association for Computing Machinery, New York, NY, USA, 287–293. doi:10.1145/3132272.3134277DOI
- [64]Dirk Schneiders, Xingdou Fu, and Kwan-Yee K. Wong. 2010. Reconstruction of display and eyes from a single image. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 1442–1449. doi:10.1109/CVPR.2010.5559799
- [65]Kentaro Takemura, Shunki Kimura, and Sara Suda. 2014. Estimating point-of-regard using corneal surface image. In Proceedings of the Symposium on Eye Tracking Research and Applications. 251–254.DOI
- [66]Kentaro Takemura, Tomohisa Yamakawa, Jun Takamatsu, and Tsukasa Ogasawara. 2014. Estimation of a focused object using a corneal surface image for eye-based interaction. Journal of eye movement research 7, 3 (2014), 13.DOI
- [67]The Gemini Team. 2025. Gemini 3 Flash: frontier intelligence built for speed. Google Research Blog. https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/
- [68]The Gemini Team. 2026. Gemini 3.1 Flash-Lite: Built for intelligence at scale. Google Research Blog. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-flash-lite/
- [69]The Gemini Team. 2026. Gemini 3.1 Pro: A smarter model for your most complex tasks. Google Research Blog. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/
- [70]Norimichi Tsumura, Minh Nguyen Dang, Takao Makino, and Yoichi Miyake. 2003. Estimating the directions to light sources using images of eye for reconstructing 3D human face. In Color and Imaging Conference, Vol. 11. Society of Imaging Science and Technology, 77–81.DOI
- [71]Andrey Vakunov and Dmitry Lagun. 2020. MediaPipe Iris: Real-time Iris Tracking & Depth Estimation. Google Research Blog. https://research.google/blog/mediapipe-iris-real-time-iris-tracking-depth-estimation/
- [72]Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Ling Li, Cem Koc, Nate True, Albert Antony, Gokula Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. 2025. FastVLM: Efficient Vision Encoding for Vision Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
- [73]Huiqiong Wang, Stephen Lin, Xiaopei Liu, and Sing Bing Kang. 2005. Separating reflections in human iris images for illumination estimation. In Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, Vol. 2. IEEE, 1691–1698.DOI
- [74]Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. 2021. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9168–9178.arxiv.org/abs/2101.04061
- [75]Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. 2021. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision. 1905–1914.arxiv.org/abs/2107.10833
- [76]Pui Chung Wong, Hongbo Fu, and Kening Zhu. 2016. Back-Mirror: back-of-device one-handed interaction on smartphones. In SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications (Macau) (SA ’16). Association for Computing Machinery, New York, NY, USA, Article 13, 2 pages. doi:10.1145/2999508.2999512DOI
- [77]Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Daniel Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. 2019. Hand-held multi-frame super-resolution. ACM Trans. Graph. 38, 4, Article 28 (July 2019), 18 pages. doi:10.1145/3306346.3323024
- [78]Yi Xu, Jared Heinly, Andrew M. White, Fabian Monrose, and Jan-Michael Frahm. 2013. Seeing double: reconstructing obscured typed input from repeated compromising reflections. In Proceedings of the 2013 ACM SIGSAC Conference on Computer & Communications Security (Berlin, Germany) (CCS ’13). Association for Computing Machinery, New York, NY, USA, 1063–1074. doi:10.1145/2508859.2516709DOI
- [79]Michitaka Yoshimura, Momoko Kitazawa, Yasuhiro Maeda, Masaru Mimura, Kazuo Tsubota, and Taishiro Kishimoto. 2017. Smartphone viewing distance and sleep: an experimental study utilizing motion capture technology. Nature and science of sleep (2017), 59–65.DOI
- [80]Xiang Zhang, Kaori Ikematsu, Kunihiro Kato, and Yuta Sugiura. 2022. Reflect-Touch: Detecting Grasp Posture of Smartphone Using Corneal Reflection Images. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 289, 8 pages. doi:10.1145/3491102.3517440DOI
- [81]Qiushi Zhou, Brandon Victor Siem, Beier Li, Jorge Goncalves, and Eduardo Velloso. 2024. Reflected Reality: Augmented Reality through the Mirror. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 4, Article 202 (Jan. 2024), 28 pages. doi:10.1145/3631431DOI


