About the role
Our client is a Barcelona-based technology company that builds digital solutions for large corporations around the world.
It operates at the intersection of business and technology, using agile methodologies and multidisciplinary teams that cover the full product development lifecycle.
We are looking for an AI QA Engineer to ensure the quality and reliability of code written by people as well as code generated by AI systems.
The role combines traditional QA with advanced evaluation of AI models, including designing testing frameworks, validating automatically generated code and defining performance metrics for models.
You will work closely with developers and ML teams to uphold quality and accuracy standards in every release.
Responsibilities
- Define and run test plans for human-written and AI-generated code.
- Design regression testing strategies for AI-assisted development environments.
- Work with ML teams to evaluate metrics such as precision, recall and output quality.
- Monitor automated pipelines that include AI validation steps.
- Curate benchmarking datasets for model training (SFT/DPO).
- Define and maintain quality thresholds for the output of automated systems.
- Promote a data-driven QA culture and the continuous improvement of AI models.
Requirements
- 3+ years of experience as a QA Engineer on web or mobile applications.
- Solid knowledge of testing and automation frameworks (Cypress, Playwright).
- Experience with CI/CD pipelines and test integration.
- Ability to analyse metrics and performance trends.
- Experience in API testing (Postman, Charles Proxy).
- Interest in LLM evaluation and AI monitoring.
Nice to have
- Knowledge of LLMOps tools (Weights & Biases, LangSmith, PromptLayer).
- Experience defining custom metrics for AI output.
- Basic scripting skills (Python or JavaScript).
- Familiarity with prompt engineering and AI-assisted test generation.
What we offer
- Private health insurance
- Meal allowance
- Hybrid model (remote and office in Barcelona)
- Flexible hours
- 24 days of holiday
- Referral bonus
- Office perks