AI Act Compliance Benchmark
The first benchmark evaluating whether AI systems satisfy EU AI Act requirements: risk classification, documentation, transparency, and human oversight at model and system level.
800+
Planned tasks
34
Regulatory articles
4
Compliance pillars
Q4 2026
Release
Leaderboard coming Q4 2026
First results will be published alongside the initial release. Contact the research team to participate in the pilot evaluation.
Task categories
Weighted contribution to the overall score
Evaluating whether systems accurately self-assess prohibited, high-risk, or limited-risk status
Completeness and accuracy of required documentation artifacts (Art. 11)
User notifications, limitations disclosure, and AI-generated content marking
Human-in-the-loop mechanisms and meaningful override capabilities
About
KAROKAN-ACT will provide the first systematic evaluation framework for EU AI Act compliance, enabling labs and deployers to assess their systems against binding regulatory requirements. The benchmark covers both the technical AI system layer and the organizational governance layer, reflecting the dual obligations under the AI Act for providers and deployers of high-risk AI systems.
Methodology
Tasks are organized along four regulatory pillars defined by the AI Act: risk classification accuracy, technical documentation completeness, transparency and explainability, and human oversight mechanisms. Each task is authored against specific articles of the regulation and reviewed by legal experts specializing in EU technology law. The scoring rubric maps directly to compliance evidence criteria expected by national market surveillance authorities.
Get involved
Access the dataset, submit a model for evaluation, or collaborate with our research team.
Contact research →Other benchmarks
The European AI Productivity Index
Assesses whether frontier AI models can perform economically valuable professional tasks in European contexts — EU law, multi-country taxation, industrial standards, and cross-border regulatory analysis.
European Multilingual Evaluation
The first rigorous benchmark evaluating LLM quality beyond English, across all 24 official EU languages in professional and institutional contexts.