Publications

Media Coverage

• June 20th, 2025: Our paper “LEXam: Benchmarking Legal Reasoning on 340 Law Exams” is covered on JURios
• April 29th, 2025: Our benchmark “INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge” - International Conference on Learning Representations” was used in the Qwen3 release
• March 10th, 2025: Our paper “SwiLTra-Bench: The Swiss Legal Translation Benchmark” is covered on Jack Clark’s blog Import AI
• June 23rd, 2024: Our paper “Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models” was discussed in Echo der Zeit by the Swiss National Radio and Television (SRF)

Open Source Contributions

• lighteval (adding datasets and support for custom models)
• datatrove (adding support for benchmarking and synthetic data)
• physics-intern (adding support for large open source models)
• finephrase (wrote whole repo for extensive synthetic data experiments)

Publications

For a complete, up-to-date list, please find me on Google Scholar.

• David Louapre, Joel Niklaus, Lewis Tunstall, “physics-intern: an autonomous agentic framework for physics research” – Pre-Print
  Article Code
• Joel Niklaus, Atsuki Yamaguchi, Michal Štefánik, Guilherme Penedo, Hynek Kydlíček, Elie Bakouch, Lewis Tunstall, Edward Emanuel Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, Thomas Wolf, “How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data” – Pre-Print
  Article Code 🤗 Dataset Slides
• Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, Rico Sennrich, “Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization” - Annual Meeting of the Association for Computational Linguistics (ACL) 2026
  Article
• Sina Ahmadi, Micha David Hess, Elena Álvarez-Mellado, Alessia Battisti, Cui Ding, Anne Göhring, Yingqiang Gao, Zifan Jiang, Andrianos Michail, Peshmerge Morad, Joel Niklaus, Maria Christina Panagiotopoulou, Stefano Perrella, Juri Opitz, Anastassia Shaitarova, Rico Sennrich, “ConLoan: A Contrastive Multilingual Dataset for Evaluating Loanwords” - Annual Meeting of the Association for Computational Linguistics (ACL) 2025
  Article
• Yu Fan, Jingwei Ni, Jakob Merane, Etienne Salimbeni, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, Joel Niklaus, “LEXam: Benchmarking Legal Reasoning on 340 Law Exams” - International Conference on Learning Representations (ICLR) 2026
  Article Code 🤗 Dataset Slides
• David Hall, Ahmed Ahmed, Christopher Chou, Abhinav Garg, Rohith Kuditipudi, Will Held, Nikil Ravi, Herumb Shandilya, Jason Wang, Jason Bolton, Siddharth Karamcheti, Suhas Kotha, Tony Lee, Nelson Liu, Joel Niklaus, Ashwin Ramaswami, Kamyar Salahi, Kaiyue Wen, Chi Heem Wong, Sherry Yang, Ivan Zhou, Percy Liang, “Introducing Marin: An Open Lab for Building Foundation Models” - Marin Community
  Article
• Daniel Kettiger, Andreas Lienhard, Thomas Myrach, Matthias Stürmer, Magda Chodup, Joel Niklaus “Open Justice vs. Privacy: Ergebnisse und Empfehlungen” - Richterzeitung
  Article
• Joel Niklaus, Jakob Merane, Luka Nenadic, Sina Ahmadi, Yingqiang Gao, Cyrill A. H. Chevalley, Claude Humbel, Christophe Gösken, Lorenzo Tanzi, Thomas Lüthi, Stefan Palombo, Spencer Poff, Boling Yang, Nan Wu, Matthew Guillod, Robin Mamié, Daniel Brunner, Julio Pereyra, Niko Grupen, “SwiLTra-Bench: The Swiss Legal Translation Benchmark” - Annual Meeting of the Association for Computational Linguistics (ACL) 2025
  Article Code 🤗 Dataset Slides
• Ronja Stern, Ken Kawamura, Matthias Stürmer, Ilias Chalkidis, Joel Niklaus, “From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudence” - Annual Meeting of the Association for Computational Linguistics (ACL) 2025
  Article 🤗 Dataset
• Luca Rolshoven, Vishvaksenan Rasiah, Srinanda Brügger Bose, Matthias Stürmer, Joel Niklaus, “Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerland” - International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2025 Findings
  Article 🤗 Dataset
• Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut, “INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge” - International Conference on Learning Representations (ICLR) 2025
  Article 🤗 Dataset
• Joel Niklaus, Lucia Zheng, Arya D. McCarthy, Christopher Hahn, Brian M. Rosen, Peter Henderson, Daniel E. Ho, Garrett Honke, Percy Liang, Christopher Manning, “LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain” - Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) Findings 2025
  Article Code 🤗 Dataset Slides Poster
• Dor Bernsohn, Gil Semo, Yaron Vazana, Gila Hayat, Ben Hagag, Joel Niklaus, Rohit Saha, Kyryl Truskovskyi, “LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text” - Conference of the European Chapter of the Association for Computational Linguistics (EACL) 2024 oral
  Article Code 🤗 Dataset
• Joel Niklaus, Magda Chodup, Thomas Lüthi, Daniel Kettiger, “Re-Identifizierung in Gerichtsurteilen mit Simap Daten” - Pre-Print
  Article Code
• Santosh T.Y.S.S, Nina Baumgartner, Matthias Stürmer, Matthias Grabmair, Joel Niklaus, “Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset” - Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023, Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING) 2024 Poster
  Article 🤗 Dataset Slides
• Joel Niklaus, Robin Mamié, Matthias Stürmer, Daniel Brunner, Marcel Gygli, “Automatic Anonymization of Swiss Federal Supreme Court Rulings” - Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023
  Article Slides 🤗 Models
• Ramona Christen, Anastassia Shaitarova, Matthias Stürmer, Joel Niklaus, “Resolving Legalese: A Multilingual Exploration of Negation Scope Resolution in Legal Documents” - Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023, Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING) 2024 oral
  Article Code 🤗 Dataset 🤗 Models Slides Video
• Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li, “LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models” - NeurIPS Datasets and Benchmarks 2023
  Article Code 🤗 Dataset Poster
• Alex Nyffenegger, Matthias Stürmer, Joel Niklaus, “Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models” - Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) Findings 2024
  Article Code 🤗 Dataset Slides
• Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E. Ho, Joel Niklaus, “SCALE: Scaling up the Complexity for Advanced Language Model Evaluation” – Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023, Data-centric Machine Learning Workshop (DMLR) @ International Conference on Learning Representations (ICLR) 2023
  Article 🤗 Dataset 🤗 Models Poster Slides
• Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E. Ho, “MultiLegalPile: A 689GB Multilingual Legal Corpus” – Data-centric Machine Learning Research Workshop (DMLR) @ International Conference on Machine Learning (ICML) 2023, Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023, Annual Meeting of the Association for Computational Linguistics (ACL) 2024 *oral- and *Outstanding Paper Award- (top 2% of accepted papers)
  Article Dataset Code Pretraining Code 🤗 Dataset 🤗 Models Slides Poster Video
• Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, Ilias Chalkidis, “LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain” – Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023 & EMNLP Findings 2023
  Article Code 🤗 Dataset Slides Video
• Tobias Brugger, Matthias Stürmer, Joel Niklaus, “MultiLegalSBD: A Multilingual Legal Sentence Boundary Detection Dataset” – International Conference on AI and Law (ICAIL) 2023
  Article Code 🤗 Dataset 🤗 Models Slides
• Joel Niklaus, Daniele Giofré, “BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?” – Efficient Natural Language and Speech Processing Workshop (ENLSP) @ Neural Information Processing Systems (NeurIPS) 2022 & Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) @ Annual Meeting of the Association for Computational Linguistics (ACL) 2023
  Article Poster 🤗 Models Video
• Gil Semo, Dor Bernsohn, Ben Hagag, Gila Hayat, Joel Niklaus, “ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US” – Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2022
  Article Code 🤗 Dataset
• Joel Niklaus, Matthias Stürmer, Ilias Chalkidis, “An Empirical Study on Cross-X Transfer for Legal Judgment Prediction” – Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL) and International Joint Conference on Natural Language Processing (IJCNLP) 2022
  Article Code Slides Video
• Joel Niklaus, Ilias Chalkidis, Matthias Stürmer, “Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark” – Natural Legal Language Processing Workshop (NLLP) @ International Conference on Empirical Methods in Natural Language Processing (EMNLP) 2021
  Article Code 🤗 Dataset Slides Video
• Christine Krebs, Michael Falkner, Joel Niklaus, Luca Persello, Stefan Klöppel, Tobias Nef, Prabitha Urwyler, “Application of Eye Tracking in Puzzle Games for Adjunct Cognitive Markers: Pilot Observational Study in Older Adults” – JMIR serious games 9
  Article
• Joel Niklaus, Michele Alberti, Rolf Ingold, Markus Stolze, Thomas Koller, “Challenging Human Supremacy: Evaluating Monte Carlo Tree Search and Deep Learning for the Trick Taking Card Game Jass” – Reinforcement Learning in Games (RLG) @ Association for the Advancement of Artificial Intelligence (AAAI) 2020 & International Conference on Artificial Intelligence and Soft Computing (ICAISC) 2020
  Article Code Poster
• Joel Niklaus, Michele Alberti, Vinay Pondenkandath, Rolf Ingold, Marcus Liwicki, “Survey of Artificial Intelligence for Card Games and Its Application to the Swiss Game Jass” – Swiss Conference on Data Science (SDS) 2019
  Article
• Zhongliang Zhao, Jose Carrera, Joel Niklaus, Torsten Braun, “Machine Learning-Based Real-Time Indoor Landmark Localization” – International Conference on Wired/Wireless Internet Communication (WWIC) 2018
  Article

Service

Workshop Organizer

• AI for Law @ ICML 2026
• Data Mining for Legal Domain @ IEEE ICDM 2025

Area Chair

• ARR February 2025

Reviewer

• NeurIPS 2026
• ARR January 2026
• ICLR 2026
• NLLP Workshop 2025
• ARR December 2024
• ARR October 2024
• ARR August 2024
• NLLP Workshop 2024
• NeurIPS 2024
• ARR June 2024
• ARR April 2024
• ARR February 2024
• ARR December 2023
• NLLP Workshop 2023
• ARR August 2023
• MLLD Workshop 2023
• ARR April 2022
• ARR February 2022

Memberships

• ACM
• ACL
• ELRA