Senior QA Analyst, AI & Machine Learning Systems

<p><strong>Description</strong></p>
<p>Keeper is hiring a talented Senior QA Analyst to join our AI & Threat Analytics team. This is a 100% remote position from select locations, with an opportunity to work a hybrid schedule for candidates based in the El Dorado Hills, CA or Chicago, IL metro areas.</p>
<p>Keeper’s cybersecurity software is trusted by millions of people and thousands of organizations globally. Keeper is published in 23 languages and sold in over 150 countries. Join one of the fastest-growing cybersecurity companies and help ensure the quality, safety, and reliability of Keeper’s next-generation AI features and threat analytics capabilities.</p>
<p><strong>About Keeper</strong></p>
<p>Keeper Security is transforming cybersecurity for organizations globally with zero-trust privileged access management built with end-to-end encryption. Keeper’s cybersecurity solutions are FedRAMP and StateRAMP Authorized, SOC 2 compliant, FIPS 140-2 validated, as well as ISO 27001, 27017 and 27018 certified. Keeper deploys in minutes, not months, and seamlessly integrates with any tech stack to prevent breaches, reduce help desk costs and ensure compliance. Trusted by millions of individuals and thousands of organizations, Keeper is the leader for password, passkey and secrets management, privileged access, secure remote access and encrypted messaging. Learn how our zero-trust and zero-knowledge solutions defend against cyber threats at <a href="http://keepersecurity.com">KeeperSecurity.com</a>.</p>
<p><strong>About the Role</strong></p>
<p>The Senior QA Analyst, AI & Threat Analytics will provide technical leadership for testing A.I and Machine Learning driven features across Keeper’s platform. This role focuses on validating the behavior, reliability, and quality of AI systems, including LLM-based summarization, classification, autofill, and behavior analysis pipelines. You will design and execute test strategies for nondeterministic systems, develop Python-based automation and evaluation tools, and validate model outputs, guardrails, and data flows. This role does not involve training models, but works closely with ML engineers, security teams, and product engineering to ensure AI features meet Keeper’s quality, safety, and performance standards in production environments.</p>
<p><strong>Responsibilities</strong></p>
<ul>
<li>Design and execute test plans for A.I and Machine Learning driven features including classification, summarization, anomaly detection, and autofill systems</li>
<li>Develop Python-based automation and evaluation scripts to validate model performance, reliability, and output quality</li>
<li>Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk, and behavior under edge cases</li>
<li>Validate data pipelines, input/output transformations, and model integration points across Keeper’s platform</li>
<li>Conduct regression testing on ML models and AI workflows to ensure reliability across releases</li>
<li>Test enterprise-scale behavior analysis features driven by data from ARAM, PAM, and session recordings</li>
<li>Validate security, privacy, and zero-trust requirements for AI model interactions and data handling</li>
<li>Collaborate with ML engineers and cross-functional teams to identify defects and improve model behavior</li>
<li>Analyze logs, metrics, and model outputs to detect anomalies, performance issues, or silent failures</li>
</ul>
<p><strong>Requirements</strong></p>
<ul>
<li>3+ years of experience in QA, test automation, or evaluation of complex software systems</li>
<li>Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs, or data pipelines</li>
<li>Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection, or behavior analysis</li>
<li>Experience testing LLM-integrated features or AI-driven workflows</li>
<li>Familiarity with ML-related tooling such as Pytest, Jupyter Notebooks, Hugging Face libraries, or similar</li>
<li>Experience with API testing, JSON validation, and test data generation</li>
<li>Ability to test nondeterministic systems and evaluate model quality using structured methodologies</li>
<li>Strong analytical, troubleshooting, and communication skills</li>
<li>Experience testing cloud-integrated systems, preferably on AWS</li>
<li>Bachelor’s degree in Computer Science, Information Systems, or equivalent practical experience</li>
</ul>
<p>Preferred Qualifications</p>
<ul>
<li>Experience testing security-focused AI or threat analytics systems</li>
<li>Familiarity with ML observability tools, vector databases, or data processing frameworks</li>
<li>Experience testing behavior analysis, video, or image-processing pipelines</li>
<li>Knowledge of zero-trust principles and secure data handling</li>
<li>Experience working with LLM providers or AI guardrail frameworks</li>
</ul>
<p><strong>Benefits</strong></p>
<ul>
<li>Medical, Dental & Vision (inclusive of domestic partnerships)</li>
<li>Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life</li>
<li>Voluntary Short/Long Term Disability Insurance</li>
<li>401K (Roth/Traditional)</li>
<li>A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)</li>
<li>Above market annual bonuses</li>
</ul>
<p>Keeper Security, Inc. is an equal opportunity employer and participant in the U.S. Federal E-Verify program. We celebrate diversity and are committed to creating an inclusive environment for all employees.</p>
<p>Classification: <strong>Exempt</strong></p><div class="content-conclusion"><p> </p>
<p class="pf0"><strong><span class="cf0">Keeper Candidate Privacy Notice</span></strong></p>
<p class="pf0"><span class="cf1">This notice explains how Keeper Security processes your personal data during recruitment. Depending on the role and location, the Controller of personal data (the organization responsible for determining why and how personal data is processed) will be Keeper Security Inc. (US), Keeper Security EMEA Ltd. (Ireland), or Keeper Security APAC K.K (Japan).</span></p>
<p class="pf0"><span class="cf1"> 1. </span><strong><span class="cf0">Data We Collect</span></strong></p>
<p class="pf0"><span class="cf1">Information You provide:</span></p>
<ul>
<li class="pf1"><span class="cf1">Contact details, CV/resume, cover letter</span></li>
<li class="pf1"><span class="cf1">Employment history, qualifications, work eligibility</span></li>
<li class="pf1"><span class="cf1">Application responses and uploaded documents</span></li>
</ul>
<p class="pf0"><span class="cf1">Information We generate:</span></p>
<ul>
<li class="pf1"><span class="cf1">Interview notes, assessments, communications</span></li>
<li class="pf1"><span class="cf1">Scheduling information</span></li>
</ul>
<p class="pf0"><span class="cf1">Information From Others:</span></p>
<ul>
<li class="pf1"><span class="cf1">Recruiter/referral information who submit your profile</span></li>
<li class="pf1"><span class="cf1">References (with your consent, before final offer)</span></li>
<li class="pf1"><span class="cf1">Public professional profiles</span></li>
<li class="pf1"><span class="cf1">Background verification (post offer)</span></li>
</ul>
<p class="pf0"><span class="cf1"> Voluntary Diversity and Equal Opportunity Information</span></p>
<ul>
<li class="pf1"><span class="cf1">We may ask you to voluntarily provide diversity information including race/ethnicity, gender, disability status and veteran status (US). Providing this information is optional and Keeper collects this data in order to comply with EEOC and similar requirements</span></li>
</ul>
<p class="pf0"><span class="cf1"> 2. </span><strong><span class="cf0">How We Use Your Data</span></strong></p>
<ul>
<li class="pf1"><span class="cf1">Assess your application and suitability</span></li>
<li class="pf1"><span class="cf1">Manage interviews and recruitment workflow</span></li>
<li class="pf1"><span class="cf1">Consider you for other/future roles (we may seek your consent to keep your information on our systems beyond the retention period specified)</span></li>
<li class="pf1"><span class="cf1">Comply with employment law obligations</span></li>
</ul>
<p class="pf0"><span class="cf1"> 3. </span><strong><span class="cf0">Legal Basis</span></strong></p>
<ul>
<li class="pf1"><span class="cf1">Legitimate Interests (recruitment management, security and integrity of the hiring process)</span></li>
<li class="pf1"><span class="cf1">Contracting steps (for progressed candidates)</span></li>
<li class="pf1"><span class="cf1">Legal and regulatory compliance obligations; explicit consent where required</span></li>
</ul>
<p class="pf0"><span class="cf1"> 4. </span><strong><span class="cf0">Who We Share Information With</span></strong></p>
<p class="pf0"><span class="cf1">Internal: </span></p>
<ul>
<li class="pf1"><span class="cf1">HR, hiring managers, interviewers*, IT support for system administration</span></li>
</ul>
<p class="pf0"><span class="cf1">*Note - diversity and equal opportunity data is not shared with hiring managers.</span></p>
<p class="pf0"><span class="cf1">Third Parties:</span></p>
<p class="pf0"><span class="cf1">Service providers who assist with:</span></p>
<ul>
<li class="pf1"><span class="cf1">Applicant tracking, recruitment systems and assessment providers</span></li>
<li class="pf1"><span class="cf1">Background verification vendors (post offer)</span></li>
<li class="pf1"><span class="cf1">Recruitment agencies (where applicable)</span></li>
<li class="pf1"><span class="cf1">Tools to support communication, collaboration and to securely store your data</span></li>
</ul>
<p class="pf0"><span class="cf1">Keeper ensures that all our third parties are contractually bound to protect your personal data with adequate safeguards in place.</span></p>
<p class="pf0"><span class="cf1"> 5. </span><strong><span class="cf0">International Transfers</span></strong></p>
<p class="pf0"><span class="cf1">Your data may be accessed by Keeper entities globally as needed for the purposes of hiring and decision making. We protect any such data transfer between Keeper entities using appropriate safeguards under applicable data protection laws. </span></p>
<p class="pf0"><span class="cf1">6. </span><strong><span class="cf0">Security</span></strong></p>
<p class="pf0"><span class="cf1">We implement appropriate technical and organizational measures to protect your data, consistent with our industry leading security standards.</span></p>
<p class="pf0"><span class="cf1">7. </span><strong><span class="cf0">Retention</span></strong></p>
<p class="pf0"><span class="cf1">We keep your data for 24 months from your last application activity, then delete or anonymize it. </span></p>
<p class="pf0"><span class="cf1">Exceptions:</span></p>
<ul>
<li class="pf1"><span class="cf1">You opt into our talent database for further retention by providing consent (extended retention)</span></li>
<li class="pf1"><span class="cf1">You're hired (transfers to employee records)</span></li>
</ul>
<p class="pf0"><span class="cf1"> 8. </span><strong><span class="cf0">Your Rights</span></strong></p>
<p class="pf0"><span class="cf1">You have the following rights and can contact us at the email below to exercise them:</span></p>
<ul>
<li class="pf1"><span class="cf1">Access, correct, or delete your data, subject to applicable law and retention requirements</span></li>
<li class="pf1"><span class="cf1">Object to or restrict processing</span></li>
<li class="pf1"><span class="cf1">Withdraw consent (where applicable)</span></li>
<li class="pf1"><span class="cf1">Request data portability</span></li>
<li class="pf1"><span class="cf1">Lodge a complaint with your data protection authority</span></li>
</ul>
<p class="pf0"><span class="cf1">If you become an employee, your rights regarding your employee record are governed by our internal Employee Privacy Notice and certain data will be retained as required under relevant laws such as employment or tax law. </span></p>
<p class="pf0"><span class="cf1">When you request access to your personal data, some information may be redacted if it includes the personal data of other individuals or information that we must protect in order to preserve their privacy rights.</span></p>
<p class="pf0"><span class="cf1"> 9. </span><strong><span class="cf0">Automated Decisions</span></strong></p>
<p class="pf0"><span class="cf1">Keeper does not make hiring decisions using solely automated processing.</span></p>
<p class="pf0"><span class="cf1">10. </span><strong><span class="cf0">Contact</span></strong><span class="cf1"> - Candidates can send privacy questions to: [email protected]</span></p>
<p> </p></div>