Skill diminta
Python
Deskripsi
Job Description
- Monitor and evaluate GenAI performance metrics through AI dashboards.
- Identify anomalies, accuracy gaps, or biases in system responses.
- Conduct regular reviews of model quality and user experience outcomes.
- Perform Human-in-the-Loop (HITL) testing by recommending test scenarios and use cases to validate GenAI responses.
- Collaborate with cross-functional teams to ensure human evaluation is effectively integrated into the AI development process.
- Analyze GenAI outputs and recommend reinforcement learning strategies.
- Provide actionable recommendations for prompt tuning, model refinement, and Retrieval-Augmented Generation (RAG) optimization.
- Review sentiment analysis results from user feedback (positive, negative, and neutral).
- Collaborate with cross-functional teams to generate insights that support continuous improvement of GenAI systems.
- Ensure QA processes comply with company standards, AI ethics guidelines, and regulatory requirements.
- Prepare documentation for QA testing protocols, evaluation results, and improvement
Requirements
- Bachelor's degree in Computer Science, Data Science, Engineering, or a related field (Master's degree is a plus).
- Experience evaluating GenAI systems using frameworks such as OpenAI Evals, RAGAS, and custom evaluation/scoring methods.
Strong understanding of cloud-based AI platforms (e.g., OpenAI, Gemini, Llama) and Generative AI concepts, including RAG pipelines, embeddings, and LLM evaluation.
- Proficient in query analysis, data validation, and AI evaluation metrics such as accuracy, precision, recall, BLEU, and ROUGE.
- Knowledge of or experience with feedback loops, reinforcement learning, and basic Python programming for AI quality assurance.
- Detail-oriented with strong problem-solving skills and the ability to design and execute AI system testing and evaluation plans.
Experienced in sentiment and text analysis of user feedback, with a background in AI Quality Assurance (QA) or AI Ethics & Governance projects.