AI Evaluation Rubric Auditor | $60-$85/Hour
We are looking for experienced AI evaluation professionals to audit the quality and accuracy of LLM evaluation tasks.
In this role, you will review work completed by AI Evaluation Reviewers, assess their decisions, and ensure that prompts and grading rubrics are fair, accurate, and aligned with evaluation guidelines. You will also provide actionable feedback to reviewers to improve the overall quality and consistency of AI evaluation.
Responsibilities
- Audit complete AI evaluation tasks, including prompts, grading rubrics, and model attempts.
- Assess whether identified model failures are genuinely justified.
- Verify whether the required information was available to the AI agent.
- Identify inaccurate, ambiguous, or unfair grading criteria.
- Determine whether valid model responses could incorrectly receive failing grades.
- Check...
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis.