Prompt Engineering & Evaluation Lead
Posted 12d ago
Posted by Karokan
karokan.ai
€80–140/hr
0 of 3 steps completed · 0%
Submit your resume or equivalent profile to start. We review background, communication quality, and evidence of relevant execution experience before moving forward.
ⓘ Application steps are reused across roles — you never upload your resume or take the same interview twice.
Role Overview
Design and run end-to-end evaluation pipelines for deployed LLM products, covering instruction tuning quality, regression testing, and performance benchmarking.
Karokan is working with AI labs and deployment teams across Europe to contract high-judgment specialists who can improve model quality through reliable human input. This role is designed for experts who can translate nuance into structured outputs, work with clear evaluation rules, and contribute directly to the performance of production AI systems.
You will operate in a fully remote setting, collaborate asynchronously, and help shape datasets, reviews, and decision layers that support the next generation of AI products. The work is intentionally rigorous: accuracy, consistency, and written reasoning matter more than generic throughput.
Key Responsibilities
- Build automated evaluation harnesses for deployed LLM endpoints
- Author evaluation datasets covering edge cases and regressions
- Define and track quality metrics across model iterations
Ideal Qualifications
- Strong background in prompt engineering or LLM product development
- Experience building evaluation frameworks or benchmarking pipelines
