Agentic Evals System Design using DeepEval, Ragas, Langsmith, and Trulens
@aiengineeringinsiderChapter 1: Agent Evaluation Fundamentals
Core concepts, the four evaluation families, deterministic vs. model-based grading, statistical significance, Wilson intervals, pass@k vs. pass^k, and setting up the live local provider.
Chapter 2: Agent Architecture Evaluation
Evaluating Planner-Executor-Verifier vs. ReAct loops, feedback-carrying retries, verifier confusion matrices, and attributing pipeline regressions.
Chapter 3: Tool-Calling Evals
Tool selection F1, schema enforcement, argument accuracy, ungrounded argument extraction, ordering constraints, and error recovery against e-commerce databases.
Chapter 4: Agent Trajectory Evals
Multi-step episode evaluation, milestone graphs, efficiency vs. path optimality, cycle and loop detection, and diagnosing IT helpdesk agent state machines.
Chapter 5: LLM-as-a-Judge
Designing industrial rubrics, multi-judge consensus, Cohen's kappa calibration against human benchmarks, hard safety floors, and mitigating verbosity and position biases.
Chapter 6: Multi-Agent Evals
Supervisor-worker architectures, inter-agent handoff success (first-attempt vs. eventual), worker duplicate effort detection, and synthesis claim provenance.
Chapter 7: RAG Agent Evals
Evaluating retrieval-augmented generation: ranking metrics (Recall@k, NDCG@k), claim-level citation audit, active vs. obsolete document discrimination, and groundedness scoring.
Chapter 8: Safety & Security Evals
Complete mediation via tool gateways, ledger verification, continuous red teaming, adversarial payload mutations, benign twin discrimination, and indirect injection defenses.
Chapter 9: Robustness & Reliability Evals
Chaos engineering for AI agents: probabilistic fault injection, transient HTTP errors, provider outages, exponential backoff with full jitter, circuit breakers, and load amplification.
Chapter 10: Production Agent Evals
Building the enterprise evaluation platform: test suites, OpenTelemetry span export, baseline regression stores, paired McNemar CI/CD release gates, and automated canary deployments.
Chapter 11: Bonus Chapter — Evaluating an Agentic Chatbot using DeepEval, Ragas, LangSmith, and TruLens
The full capstone: connecting an interactive customer support chatbot built on LangGraph to DeepEval unit tests, Ragas retrieval metrics, LangSmith run-tree traces, and TruLens groundedness guardrails.