As Wolters Kluwer Legal & Regulatory executes its North Star to become the Intelligent Orchestration Platform for legal and regulatory work, the quality of AI‑generated outputs – particularly for research, analysis, and reasoning – becomes mission‑critical. As AI systems increasingly surface legal answers, interpretations, and recommendations at the point of work, trust depends on correctness, grounding, traceability, and consistency of those outputs. The Director of AI Quality & Safety ensures that AI‑driven research and decision support remain reliable, auditable, and compliant as agentic automation scales, reinforcing governed execution and making quality and trust durable differentiators at the orchestration layer.
The Director of AI Quality & Safety is accountable for establishing and operationalizing a comprehensive quality and safety framework across AI-enabled products, content systems, and agentic workflows within WK Legal & Regulatory. The role ensures that AI systems are reliable, auditable, compliant, and aligned with defined quality standards, while reducing production defects and AI-related risks.
This position sits at the intersection of product, engineering, data science, and regulatory compliance, with a mandate to define measurable standards and enforce quality and product standards across the AI lifecycle.
Key Responsibilities
1. AI Quality & Evaluation Frameworks
1. Design and implement standardized evaluation frameworks for AI models and agentic systems (e.g., LLMs, RAG pipelines, autonomous agents)
2. Define, build, and improve evaluation frameworks with SMEs for output correctness and factuality, task completion accuracy, robustness and edge-case handling
3. Establish benchmark datasets and continuous evaluation pipelines (offline + online)
4. Drive adoption of evaluation tooling and methodologies across product teams
2. Software Quality & Reliability (AI Systems)
5. Extend the rigor of traditional software QA practices to the outputs of AI-driven systems (probabilistic outputs, non-determinism)
6. Define SLAs/SLOs specific to AI performance (e.g., hallucination rate, response reliability, latency under load)
7. Partner with Engineering to integrate quality gates into CI/CD pipelines
8. Coordinate root-cause analysis for AI-related production issues and track implementation of systemic fixes
3. Content Correctness & Validation
9. Establish frameworks for validating legal and regulatory content generated or transformed by AI systems
10. Collaborate with editorial and domain experts to define “ground truth” and validation protocols
11. Implement human-in-the-loop and automated validation mechanisms where appropriate
12. Ensure traceability between AI outputs and authoritative sources
4. Safety, Compliance & Governance
13. Define and enforce AI safety standards aligned with regulatory requirements (e.g., EU AI Act, data protection laws) and internal WK risk and compliance policies
14. Implement formal standards and controls alongside internal teams for bias detection and mitigation, harmful or unsafe output prevention, and data privacy and secure handling
15. Ensure auditability of AI systems (logging, explainability, decision traceability)
16. Act as primary liaison with Risk, Legal, and Compliance on AI-related matters
5. Measurement, Benchmarking & Reporting
17. Define quality and safety standards and KPIs that apply across the AI lifecycle - including model and data selection, prompt and workflow design, and deployment - working in partnership with Product and Engineering
18. Build dashboards and reporting mechanisms for executive visibility
19. Track and benchmark performance over time and across product lines
20. Track development of, and evaluate products against, external industry benchmarks and work with recognized benchmarking bodies to represent WK interests
21. Drive continuous improvement loops based on measurable outcomes
22. Drive QA & KPIs awareness in LR businesses and provide comms support with key findings & insights that can be used for external comms & thought leadership
23. Partner with Sales, Marketing, and Customer Support on external benchmark communication and AI‑related incident response messaging
6. Governance & Operating Model
24. Define operating model for AI quality & safety across CPO and DXG
25. Introduce review boards, approval processes, and escalation mechanisms
26. Provide guidance and enablement to product teams on quality and safety best practices
27. Build and lead a small, high-impact team as the function scales
Success Criteria
28. Clear, standardized quality and safety metrics are defined and consistently enforced across all AI-enabled products
29. AI system behavior is measurable, benchmarked, and governed through repeatable frameworks
30. Significant reduction in production defects and AI-related incidents
31. High confidence in content correctness and traceability for legal/regulatory use cases
32. Full auditability and compliance alignment for AI systems across jurisdictions
Required Qualifications
33. 10+ years in product quality, AI/ML systems, or related domains, with at least 3–5 years in AI-focused roles
34. Demonstrated experience designing evaluation frameworks for AI/ML systems (e.g., LLM evaluation, model validation)
35. Strong understanding of modern AI architectures (LLMs, RAG, agents), software quality engineering principles, and data and content validation workflows
36. Experience with regulatory or compliance-heavy environments (preferred: legal, financial, healthcare)
37. Proven ability to operate cross-functionally at senior levels
Preferred Qualifications
38. Familiarity with emerging AI governance standards and regulations (e.g., EU AI Act)
39. Experience implementing human-in-the-loop systems at scale
40. Background in experimentation platforms, benchmarking, or observability for AI systems
41. Advanced degree in Computer Science, Data Science, Law, or related field
Key Competencies
42. Systems thinking (ability to unify software, AI, and content quality domains)
43. Analytical rigor and metric-driven decision making
44. Risk awareness and regulatory sensitivity
45. Influence without authority in a matrixed organization
46. Pragmatic execution with high standards for quality
Questions?
Reach out to Silvie Roelans (Talent Acquisition Consultant) on
Our Interview Practices