AI - QA & Prompt Analyst
Role Overview
We are looking for an AI QA & Prompt Analyst to ensure the quality, reliability, and performance of our AI models and systems. You will design test strategies for LLM-based applications and work closely with our AI engineering teams.
What You’ll Do
- Design and execute test plans for LLM-based applications and AI systems
- Develop and optimize prompts for various AI use cases
- Identify and document AI model limitations, biases, and failure modes
- Create automated testing frameworks for AI quality assurance
- Collaborate with AI engineers to improve model performance
Must Have Skills
- 3+ years of QA experience with at least 1 year in AI/ML testing
- Strong understanding of LLMs and prompt engineering principles
- Experience with Python for test automation
- Familiarity with AI evaluation metrics and methodologies
- Excellent analytical and problem-solving skills
Nice-to-Have
- Experience with MLflow, LangChain, or similar frameworks
- Knowledge of RAG architectures and vector databases
- Background in NLP or data science
