Evaluate autonomous agents remotely using production-grade software skills and precise technical feedback. Outlier offers freelance work with flexible scheduling.
This opportunity supports projects training large language models to function as proactive, multi-step agents. Contributors assess complex system behaviors through rubrics, trace debugging, and stress testing. Experience with backend engineering, AI automation, or systems integration supports the work.
Key Responsibilities:

Develop rubrics for evaluating agent outputs
Inspect and debug traces from agent interactions
Test agents against complex workflow scenarios
Identify subtle failures in system behavior
Requirements:

Two or more years of relevant experience
Production-grade software development with modular separation
Practical experience with live system interactions
Clear, detailed feedback on complex technical behaviors
Experience with multi-stage coordination tasks preferred
Use agent evaluation, systems integration, and technical analysis skills to improve proactive AI systems through Outlier projects.

#J-18808-Ljbffr

Autonomous Agent Evaluation Engineer