LLM Evaluation & Benchmarking
Institutionalise 'every claim measured': build golden datasets, automated evaluation harnesses, LLM-as-judge with calibration checks, regression gates in CI, and honest reporting — including designing evals that catch the failure a stakeholder will find first.
What this course covers.
5 modules, 34 named skill atoms. Expand any module to see them.
1Eval Strategy6 skill atoms
2Metrics that Mean Something7 skill atoms
3LLM-as-Judge7 skill atoms
4Regression Gates7 skill atoms
5Failure-First Design7 skill atoms
AI-311 in the role journeys.
This course appears in 6 of our 45 role journeys. Here is what a learner takes immediately before and after it in each.
Roles this course serves
This course is authored to band B4.
Every course we run is written to one rung of the CASI ladder, so a plan can be assembled to take a team from where they are to where they need to be.
What do B1–B6 mean?The CASI Capability Ladder — click to expand
Every course targets a band on the CASI Capability Ladder — our six-band proficiency scale, anchored to open standards (O*NET, ESCO, NICE, NIST AI RMF, Bloom's). A band tells you how deep a course goes, and what evidence proves it.
A note on B6. Courses in this catalog target B1–B5. B6 is not taught — it is recognised, through a portfolio and a panel, once someone is setting direction for others. Every journey here is built to land a learner at B5.
Other AI courses at this level.
Departmental AI Adoption & Automation Design
Transformers from First Principles
Fine-Tuning with Hugging Face
Distributed Training Foundations
Synthetic Data Generation & Evaluation
Vector Databases & Hybrid Search
Run AI-311 for your team.
This course runs at several lengths depending on how deep you need to go and how much of it your people already have. Tell us who is being trained and we will scope it.