AI Technical Consultant, LLM Safety Evaluation, Digital Impact Division, Digital Guardrails
Organizational Context
UNICEF operates in over 190 countries and territories, dedicated to saving children's lives, defending their rights, and helping them fulfill their potential from early childhood through adolescence. We are committed, passionate, and proud of our work, viewing the promotion of every child's rights not just as a job, but as a calling. UNICEF offers diverse opportunities for professional and personal development.
Job Purpose
The AI Technical Consultant will be responsible for preparing, cleaning, and structuring test prompt sets for Large Language Models (LLMs). This role involves conducting extensive testing of these prompts across various LLM providers and configurations, and curating the most informative model responses for expert review. The consultant will evaluate LLM performance under different safety/content-filter configurations, documenting their impact on model behavior. Key tasks include capturing model responses and relevant test metadata, troubleshooting API, model, safety-filter, or platform issues, and developing/maintaining code to interact with commercial and open-source LLM APIs for evaluation purposes. The consultant will also conduct targeted reviews of AI safety research to identify relevant prompts, failure modes, and emerging risks, organizing outputs for expert review and supporting rating, ranking, and feedback collection.
Responsibilities
The AI Technical Consultant will prepare, clean, and structure agreed test prompt sets for LLM evaluation. This includes conducting extensive testing of these prompts across multiple LLM providers and configurations, and curating the most informative model responses for expert review. The role involves running evaluations under different approved safety/content-filter configurations and documenting how these settings affect model behaviour. The consultant will capture model responses and relevant test metadata, and troubleshoot API, model, safety-filter, or platform issues. Development and maintenance of code to call commercial LLM APIs, and where required, deployment and interaction with open-source models for evaluation, are also key responsibilities. The consultant will conduct targeted reviews of state-of-the-art child-safety and AI-safety research to identify relevant prompts, failure modes, and emerging risks. Outputs will be organized into a simple format for expert review, with support for simple rating, ranking, and reviewer comments. Capture of review results in a structured, exportable format, and iteration of the prototype based on expert feedback are expected. Participation in communication with reviewers, collection of feedback on model behaviour and the review process, and refinement of the testing/review infrastructure will be undertaken. Support for re-testing when experts identify new failure modes, ambiguities, or edge cases, and help in translating expert feedback into new or refined test cases are also part of the role. Maintaining a clean, versioned prompt-response dataset is crucial.
Work Experience
Minimum of five years of relevant professional experience in AI/ML, software engineering, data science, model evaluation, or a closely related technical field.
Skills
Evaluating LLM outputs (qualitative assessment), Prompt testing, AI safety testing, AI red teaming, Python programming, LLM API integration, Designing and executing structured tests, Understanding of LLM safety controls, Working with CSV, JSON, JSONL, Troubleshooting model/API issues, Deploying open-source LLMs, Communicating technical findings