JH
José Hernández-Orallo
cs.AIcs.LGcs.CLcs.CYstat.MLArtificial Intelligencecs.DBcs.ITmath.ITBenchmarking
On Valency
published · living versionsW_p8cd8pn6·v1 · currentpublished
A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations
with Peter Romero, Haotian Li, Hugo Save, Luning Sun +6
1 version
W_xaqzq29k·v1published
Preprints & journals
68 papers in the corpus · 2010–2026Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs2610.29507v1 · Auss Abbood, Fernando Mart'inez-Plumed, Jos'e Hern'andez-Orallo et al.2026 · 0 citationsarXivon Valency
Multi-agent AI systems outperform human teams in creativity2605.17885v1 · Tiancheng Hu, Yixuan Jiang, Haotian Li et al.2026 · 0 citationsarXiv
General scales unlock AI evaluation with explanatory and predictive power.41922702 · Zhou, Lexin, Pacchiardi, Lorenzo, Martínez-Plumed, Fernando et al.2026 · 7 citationsNature. 2026;652(8108):58-67
From Human-Level AI Tales to AI Leveling Human Scales2602.18911v2 · Peter Romero, Fernando Mart'inez-Plumed, Zachary R. Tidler et al.2026 · 0 citationsarXiv
AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games2602.17594v1 · Lance Ying, Ryan Truong, Prafull Sharma et al.2026 · 1 citationarXiv
The science and practice of proportionality in AI risk evaluations.41712728 · Mougan, Carlos, Morlock, Lauritz, Aguirre, Jair et al.2026 · 4 citationsScience (New York, N.Y.). 2026;391(6787):769-771
Seven simple steps for log analysis in AI systems2604.09563v2 · Magda Dubois, Ekin Zorer, Maia Hamin et al.2026 · 1 citationarXiv
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores2601.13885v2 · Esma Balk\ir, Alice Pernthaller, Marco Basaldella et al.2026 · 0 citationsarXiv
Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies2601.11699v4 · Miles Brundage, Noemi Dreksler, Aidan Homewood et al.2026 · 1 citationarXiv
Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia2512.03318v1 · Chandler Smith, Marwa Abdulhai, Manfred Diaz et al.2025 · 0 citationsarXiv
Psychometric Personality Shaping Modulates Capabilities and Safety in Language Models2509.16332v1 · Stephen Fitz, Peter Romero, Steven Basart et al.2025 · 0 citationsarXiv
Beyond the high score: Prosocial ability profiles of multi-agent populations2509.14485v1 · Marko Tesic, Yue Zhao, Joel Z. Leibo et al.2025 · 0 citationsarXiv
Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agents2506.08800v2 · Irene Testini, Jos'e Hern'andez-Orallo, Lorenzo Pacchiardi2025 · 1 citationarXiv
Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics.40418851 · Sun, Luning, Gibbons, Christopher, Hernández-Orallo, José et al.2025 · 4 citationsJournal of medical Internet research. 2025;27:e70901
Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach2505.18882v5 · Yuchen Wu, Edward Sun, Kaijie Zhu et al.2025 · 1 citationarXiv
Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships2505.16899v1 · Kerem Oktar, Katherine M. Collins, Jose Hernandez-Orallo et al.2025 · 0 citationsarXiv
Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models2505.10583v2 · Diogo Freitas, Brigt H\aavardstun, Cesar Ferri et al.2025 · 0 citationsarXiv
Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AI2503.21668v2 · Danaja Rutar, Alva Markelius, Konstantinos Voudouris et al.2025 · 0 citationsarXiv
The Animal-AI Environment: A virtual laboratory for comparative cognition and artificial intelligence research.40021555 · Voudouris, Konstantinos, Slater, Ben, Cheke, Lucy G et al.2025 · 6 citationsBehavior research methods. 2025;57(4):107
Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culture2502.15620v2 · John Burden, Marko Tesi'c, Lorenzo Pacchiardi et al.2025 · 4 citationsarXiv
PredictaBoard: Benchmarking LLM Score Predictability2502.14445v2 · Lorenzo Pacchiardi, Konstantinos Voudouris, Ben Slater et al.2025 · 3 citationsarXiv
What should an AI assessor optimise for?2502.00365v1 · Daniel Romero-Alvarado, Fernando Mart'inez-Plumed, Jos'e Hern'andez-Orallo2025 · 0 citationsarXiv
Open Problems in Machine Unlearning for AI Safety2501.04952v1 · Fazl Barez, Tingchen Fu, Ameya Prabhu et al.2025 · 0 citationsarXiv
Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers2410.11672v1 · Lorenzo Pacchiardi, Marko Tesic, Lucy G. Cheke et al.2024 · 0 citationsarXiv
Larger and more instructable language models become less reliable.39322679 · Zhou, Lexin, Schellaert, Wout, Martínez-Plumed, Fernando et al.2024 · 200 citationsNature. 2024;634(8032):61-68
100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances2409.03563v1 · Lorenzo Pacchiardi, Lucy G. Cheke, Jos'e Hern'andez-Orallo2024 · 1 citationarXiv
Conversational Complexity for Assessing Risk in Large Language Models2409.01247v3 · John Burden, Manuel Cebrian, Jose Hernandez-Orallo2024 · 3 citationsarXiv
Foundational Challenges in Assuring Alignment and Safety of Large Language Models2404.09932v2 · Usman Anwar, Abulhair Saparov, Javier Rando et al.2024 · 19 citationsarXiv
Learning Alternative Ways of Performing a Task2404.02579v1 · David Nieves, Mar'ia Jos'e Ram'irez-Quintana, Carlos Monserrat et al.2024 · 6 citationsExpert Systems With Applications, volume 148, 2020, 113263
Evaluating General-Purpose AI with Psychometrics2310.16379v2 · Xiting Wang, Liming Jiang, Jose Hernandez-Orallo et al.2023 · 6 citationsarXiv
An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI2310.14455v3 · Ross Gruetzemacher, Alan Chan, Kevin Frazier et al.2023 · 2 citationsarXiv
Predictable Artificial Intelligence2310.06167v3 · Lexin Zhou, Pablo A. Moreno-Casares, Fernando Mart'inez-Plumed et al.2023 · 9 citationsarXiv
Inferring Capabilities from Task Performance with Bayesian Triangulation2309.11975v2 · John Burden, Konstantinos Voudouris, Ryan Burnell et al.2023 · 1 citationarXiv
Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs2304.11164v1 · Anthony G Cohn, Jose Hernandez-Orallo2023 · 9 citationsarXiv
Rethink reporting of evaluation results in AI.37053341 · Burnell, Ryan, Schellaert, Wout, Burden, John et al.2023 · 103 citationsScience (New York, N.Y.). 2023;380(6641):136-138
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models2206.04615v3 · Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao et al.2022 · 569 citationsTransactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5Bvosj
Direct Human-AI Comparison in the Animal-AI Environment.35686061 · Voudouris, Konstantinos, Crosby, Matthew, Beyret, Benjamin et al.2023 · 22 citationsFrontiers in psychology. 2022;13:711821
General intelligence disentangled via a generality metric for natural and artificial intelligence.34819537 · Hernández-Orallo, José, Loe, Bao Sheng, Cheke, Lucy et al.2022 · 29 citationsScientific reports. 2021;11(1):22822
Compute and Energy Consumption Trends in Deep Learning Inference2109.05472v2 · Radosvet Desislavov, Fernando Mart'inez-Plumed, Jos'e Hern'andez-Orallo2021 · 275 citations"Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning" Sustainable Computing: Informatics and Systems (2023). Volume 38, April 2023, 100857
Conditional Teaching Size2107.07038v1 · Manuel Garcia-Piqueras, Jos'e Hern'andez-Orallo2021 · 1 citationarXiv
Automating Data Science: Prospects and Challenges2105.05699v2 · Tijl De Bie, Luc De Raedt, Jos'e Hern'andez-Orallo et al.2021 · 49 citationsCommunications of the ACM 65(3) 76-87 (2022)
Evaluating the Apperception Engine2007.05367v1 · Richard Evans, Jose Hernandez-Orallo, Johannes Welbl et al.2020 · 3 citationsarXiv
Making sense of sensory input1910.02227v2 · Richard Evans, Jose Hernandez-Orallo, Johannes Welbl et al.2019 · 58 citationsarXiv
Fairness and Missing Values1905.12728v1 · Fernando Mart'inez-Plumed, Cesar Ferri, David Nieves et al.2019 · 9 citationsarXiv
Analysing Results from AI Benchmarks: Key Indicators and How to Obtain Them1811.08186v2 · Fernando Mart'inez-Plumed, Jos'e Hern'andez-Orallo2018 · 24 citationsIEEE Transactions on Games, 2018
General-purpose Declarative Inductive Programming with Domain-Specific Background Knowledge for Data Wrangling Automation1809.10054v1 · Lidia Contreras-Ochando, C'esar Ferri, Jos'e Hern'andez-Orallo et al.2018 · 3 citationsarXiv
A multidisciplinary task-based perspective for evaluating the impact of AI autonomy and generality on the future of work1807.02416v1 · Enrique Fern'andez-Mac'ias, Emilia G'omez, Jos'e Hern'andez-Orallo et al.2018 · 8 citationsarXiv
Assessing the impact of machine intelligence on human behaviour: an interdisciplinary endeavour1806.03192v1 · Emilia G'omez, Carlos Castillo, Vicky Charisi et al.2018 · 9 citationsarXiv
Between Progress and Potential Impact of AI: the Neglected Dimensions1806.00610v2 · Fernando Mart'inez-Plumed, Shahar Avin, Miles Brundage et al.2018 · 1 citationarXiv
Finite Biased Teaching with Infinite Concept Classes1804.07121v1 · Jose Hernandez-Orallo, Jan Arne Telle2018 · 0 citationsarXiv
Career total: 304 works. 68 are in this corpus.Showing the 50 most recent.
Profile built from the corpus for this byline.
Author records are still filling in while the Hub is in alpha. If this is your page, you'll be able to claim it soon. Spot a mistake? Tell us.