Head of Evaluations

At Google DeepMind, we value diversity of experience, knowledge, backgrounds and perspectives and harness these qualities to create extraordinary impact. We are committed to equal employment opportunity regardless of sex, race, religion or belief, ethnic or national origin, disability, age, citizenship, marital, domestic or civil partnership status, sexual orientation, gender identity, pregnancy, or related condition (including breastfeeding) or any other basis as protected by applicable law. If you have a disability or additional need that requires accommodation, please do not hesitate to let us know.

Snapshot 

As Head of Evaluations in the Responsible Development and Innovation (ReDI) team, you’ll be responsible for driving our approach to evaluations of Google DeepMind’s most groundbreaking models and overseeing and expanding the evaluation portfolio ahead of new model launches.   

You will work with teams at Google DeepMind, and internal and external partners, to ensure that our work is conducted in line with responsibility and safety best practices, helping Google DeepMind to progress towards its mission.

About us

Artificial Intelligence could be one of humanity’s most useful inventions. At Google DeepMind, we’re a team of scientists, engineers, machine learning experts and more, working together to advance the state of the art in artificial intelligence. We use our technologies for widespread public benefit and scientific discovery, and collaborate with others on critical challenges, ensuring safety and responsibility are the highest priority.

The role

As Head of Evaluations in the ReDI team, you’ll oversee a team of specialists and be a critical part of the ReDI leadership team, using your expertise to deliver impactful work through direct collaboration on groundbreaking research projects and to help develop the broader governance ecosystem at Google DeepMind. You’ll be a critical input to informing Google DeepMind and Google leadership about the responsibility and safety of our models and to model development teams as they build new models. This role will work with teams from across Google DeepMind and Google, as well as external partners. 

Key responsibilities

• Lead and manage a team of specialists, fostering a collaborative and high-performing environment, providing mentorship and guidance to team members. Actively invest in their professional development through regular feedback, coaching, and opportunities for growth.
• Build and manage a roadmap of evaluations development that examine responsibility and safety questions for Google DeepMind’s most groundbreaking models across internal assurance evaluations, external evaluations and policy-aligned evaluations for model teams.
• Proactively engage with industry-wide developments in this space to promote best practices in Google DeepMind.
• Identify areas where new testing approaches are required, working with Google DeepMind research and engineering teams to: 
• Drive the development of new evaluations for upcoming model releases; 
• Oversee automated evaluations that support model development evaluations and assurance; and
• Engage external organisations to provide insights on the responsibility and safety aspects of Google DeepMind models. 
• Work closely with other areas of ReDI to develop a deep understanding of key policy areas which guide and inform the evaluations being run. 
• Continuously identify ways to scale evaluations work and drive efficiencies, including identifying right sized approaches to manage evaluations for new modalities and capabilities through to more established models. 
• Coordinate across relevant teams across the organization, such as those working on autonomy and cybersecurity, to create a comprehensive overview of the safety profile of Google DeepMind models. 
• Proactively improve the responsibility and safety of Google DeepMind’s models by sharing and communicating insights from evaluations with model development teams, senior stakeholders and decision makers and broader Google teams.
• Work closely with Google DeepMind and Google teams to ensure external commitments are upheld. 

About you

In order to set you up for success as a Head of Evaluations at Google DeepMind, we look for the following skills and experience: 

• Demonstrated prior experience designing and implementing audits or evaluations of cutting edge AI systems. 
• Deep understanding of AI technologies and machine learning principles. 
• Expertise in data science, statistics, algorithmic auditing, or other relevant fields
• Demonstrable expertise in identifying solutions to scale evaluations work, driving efficiencies to manage new modalities scaled through to established models
• Demonstrated experience in managing a high performing interdisciplinary team in a fast paced environment.
• Demonstrated ability to lead cross-functional teams, foster collaboration, and influence outcomes.
• Experience working with ethics and safety topics associated with AI development in a technology company such as child safety, privacy, representational harms and discrimination, misinformation, or other areas of content or model risks.
• Proven ability to engage with and influence a range of internal stakeholders from researchers and engineers through to senior leadership and external partners, from academia through to suppliers. 
• Excellent communication skills, both written and verbal, with the ability to effectively communicate complex technical concepts to a wide range of audiences.

Preferred Experience:

• Master's degree or PhD (or equivalent experience) in a relevant field, such as philosophy, ethics, computer science, or public policy
• Prior experience participating in or leading red teaming exercises for AI models.  
• Familiarity with cybersecurity principles and practices relevant to AI model safety and security.
• Product management expertise or other similar experience.

Deadline to apply: EOD Wednesday 7th August 2024. 

The US base salary range for this full-time position is between $168000 - $266000 + bonus + equity + benefits.