Jobbeschreibung
Senior Machine Learning Engineer (IntelliJ AI)
In this role, you will:
* Design and drive evaluation frameworks for AI features, including metrics, experiments, and agent trace analysis.
* Diagnose model performance issues (e.g. prompt drift, context mismatches, and latency/quality trade-offs) and translate findings into actionable improvements.
* Experiment with contexts and lightweight models to continuously develop our ML system.
* Act as the ML liaison for product teams across JetBrains, adapting and scaling AI capabilities in JetBrains IDEs to their needs.
* Build and maintain small helper models (e.g. re-rankers, classifiers, embedding models) to support domain-specific tasks.
* Collaborate with colleagues in ML, product, engineering, and analytic teams to deliver improvements and monitor their impact in production.
* Stay up to date with research in the fields of LLMs, agents, and evaluation, bringing best practices into our workflows.
* Mentor junior engineers and help shape team culture, processes, and tooling around experimentation and evaluation.
Voraussetzungen
We’d be happy to have you on our team if you:
* Have 5+ years of experience as an ML Engineer, with a solid background in production-grade ML systems (especially LLMs and agent architectures).
* Have experience with LLM evaluation methods and frameworks.
* Can design and run end-to-end experiments – hypotheses, metrics, data collection (including traces/logs), analysis, and decision-making.
* Are skilled in context-aware pipelines or conversational/agent systems.
* Have strong Python programming skills.
* Bring hands-on experience in fine-tuning or training smaller models (e.g. domain-specific fine-tuning and lightweight customizations).
* Communicate clearly and effectively across teams, translating ML/AI insights into product features.
* Have prior mentorship experience with ML/evaluation engineers.
* Thrive in a cross-functional, fast-moving environment, taking ownership, iterating quickly, and delivering results.
We’d be especially thrilled if you have:
* Familiarity with agent-based systems and orchestrating multi-step reasoning agents.
* Experience with the Kotlin programming language.
Wir bieten Ihnen