#MLOps.

You are an AI evaluation researcher. Design a rigorous evaluation framework for an LLM-powered product: [describe the product, e.g., "an AI customer support agent"]. Framework sections: 1) Evaluation Taxonomy — categorize what needs to be evaluated: Task Performance, Safety, Robustness, User Experience, Cost Efficiency, 2) For each category: specific metrics, measurement methodology (human eval vs automated vs hybrid), and scoring rubric, 3) Golden Dataset Design — how to build a ground truth evaluation set of [N] examples covering diverse scenarios including adversarial cases, 4) Regression Testing Protocol — how to ensure new model versions don't break existing capabilities, 5) Latency and Cost SLAs — acceptable p50/p95/p99 latency and cost per call, 6) Red-Teaming Plan — the 10 most important adversarial prompts to test for this product, 7) Human Eval Interface Design — what annotators see and how to ensure inter-rater reliability. Also recommend an open-source evaluation framework (Evals, RAGAS, LangSmith, etc.) suited for this use case.

You are a Staff Machine Learning Engineer. Design a production ML system for the following problem: [describe the business problem, e.g., "predict customer churn 30 days in advance"]. Deliverables: 1) Problem Formulation — reframe the business problem as an ML problem (classification/regression/ranking/generation?), define the prediction target precisely, 2) Data Requirements — what data is needed, where it comes from, what quality issues to expect, 3) Feature Engineering Plan — 10 candidate features with rationale; identify target leakage risks, 4) Model Selection — evaluate 3 candidate algorithms; recommend one with justification, 5) Training Infrastructure — compute requirements, training frequency, retraining triggers, 6) Evaluation Framework — the right metric for this problem (not just accuracy), offline vs online evaluation, a baseline to beat, 7) Deployment Architecture — batch vs real-time serving, A/B test design for model rollout, 8) Monitoring Plan — data drift, model drift, business metric correlation, 9) Failure Modes — what goes wrong when the model is confidently wrong?