SWE-Bench Task Auditor
$70 - $90 per hourTemporary
Mercor
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity, and provide clear, rubric-based written feedback.
Basic Qualifications
- 3+ years professional software engineering
- Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
- Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
- Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
- Familiarity with SWE-Bench (Verified) or similar repository benchmarks
- Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
- Prior code-review or task-grading experience
Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the SWE-Bench Task Auditor in San Francisco, CA vacancy
$70 - $90 per hour
...Evaluate the quality, correctness, and production-readiness of Kubernetes tasks used to train and evaluate a frontier AI lab's models. You'll assess cluster-operations scenarios, manifest correctness, and failure-mode troubleshooting , and provide clear, rubric-based written...SuggestedContract work- ...Evaluate the quality, correctness, and cloud-architecture soundness of AWS serverless and infrastructure-as-code tasks used to train and evaluate a frontier AI lab's models. You'll assess multi-service serverless designs, IaC fidelity, and cross-service integration correctness...SuggestedTemporary workLocal area
Mercor
San Francisco, CA19 days ago - ...Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate a frontier AI lab's models. You'll assess experiment design, model-selection reasoning, and evaluation methodology , and provide clear, rubric-based...SuggestedTemporary work
Mercor
San Francisco, CA19 days ago $70 - $90 per hour
...include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: SWE-Bench Task Auditor Type: Contract Compensation: $70–$90/hour Location: Remote Role Responsibilities Evaluate the...SuggestedContract workSummer workRemote work- Direct message the job poster from Insight Global Job Overview Insight Global is seeking experienced Senior Associates and Managers to support a Big 4 firm during its DAT busy season. This role focuses on IT SOX compliance, control remediation, and project management for...SuggestedContract work3 days per week
$54k - $58k
...agreements. Duties/Responsibilities: Analyze reports for detailed findings; coordinate all analytical, technical and research tasks. Manipulate and organize data via Excel. Gather payroll and other data from employers either at their location and/or remotely...Work at officeRemote workFlexible hoursDay shiftMiller Kaplan
San Francisco, CA1 day ago$61.72k
...competitive service position will be at the discretion of the agency and is neither guaranteed nor implied. In this position as an Auditor, you will work as an advanced trainee performing work assignments and completing training, serving as a support auditor on a team,...Permanent employmentFor contractorsTraineeshipInternshipWork at officeCustoms and Border Protection
San Francisco, CA1 day ago$50.46k
...competitive service position will be at the discretion of the agency and is neither guaranteed nor implied. In this position as an Auditor, you will work as an advanced trainee performing work assignments and completing training, serving as a support auditor on a team,...Permanent employmentFull timePart timeFor contractorsApprenticeshipTraineeshipInternshipWork at officeLocal areaTrial periodFlexible hoursCustoms and Border Protection
San Francisco, CA1 day ago- auditor sustainable seafood chain of custody Emeryville, California Full Time Audit Coordination Schedule and confirm audit date with... ...written communication experience Strong organization, multi-tasking, and prioritization skills Proficient with business software,...Full timeWork experience placement
- ...Employment Type: Full-Time, Mid-Level Department: Financial CGS is seeking a highly skilled Auditor to provide general auditing and accounting services in support of fraud investigations involving potentially complex financial transactions and complex organizations attempting...Full timeWork experience placementInterim roleWork at officeLocal areaFlexible hours
CGS Federal (Contact Government Services)
San Francisco, CA1 day ago - ...The Judicial Council of California is accepting applications for the position of Auditor with the Audit Services office. The Auditor conducts compliance audits of superior courts and other judicial branch entities based on the Judicial Council’s policies and procedures...Work at officeRemote workFlexible hoursNight shift2 days per week
Judicial Branch of California
San Francisco, CA8 hours ago - ...a commitment to innovation to help clear the way for millions of Americans to achieve more. About the Role We are looking for a Sr Auditor who will thrive in a fast-paced environment and adapt easily to change. In this role, you will bring internal audit and financial services...Work experience placementWork at officeLocal areaRemote workRelocationFlexible hours
Tally Inc
San Francisco, CA2 days ago - ...Microsoft Word, Excel, Access and PowerPoint. Knowledge of macros, pivot tables, query design, and other techniques for automating tasks and increasing productivity. Ability to work independently with minimal supervision and also as a strong team member. Ability...Work at office
AHMC Healthcare
Daly City, CA1 day ago - ...audit and risk decisions. We turn it into reviewable results and auditor-ready workpapers in minutes instead of days. The Opportunity... ...product, run office hours during peak testing periods, and build a bench of power users inside each account Advise on the audit work...Work at officeShift work
$166k - $170k
...SecurityPal, Inc. San Francisco, CA Information Security Auditor Duties Lead a team that provides comprehensive vendor assessments to evaluate security risks and compliance with standards and regulations. Serve as the main point of contact for clients...Work experience placementRemote workSecuritypalhq
San Francisco, CA3 days ago- ...a hotel in Wine Country. Where the opportunity to be a partner is a clear path, not just a promise. We are searching for a Senior Auditor. As a Senior Auditor with our firm, you’ll be responsible for completing client engagements while developing strong working relationships...Casual workWork at officeLocal areaRemote workWork from home
$80k - $150k
Senior Auditor Employment Type: Full Time, Mid-level Department: Financial Services CGS is seeking a Senior Auditor who will assist the district legal staff by conducting medical claims data analysis, forensic investigations, financial damages, statistical sampling, and...Full timeFlexible hours- jobr.pro is seeking an IA Lead for Stablecoins and Digital Assets to design and lead a comprehensive internal audit strategy. This critical role requires extensive experience in auditing, risk management, and thorough knowledge of stablecoin mechanics. The candidate will...
Jobr
San Francisco, CA1 day ago $8,265 - $10,785 per month
Job Posting:Lead Utility Auditor-Evaluator Public Utilities Commission JC-533190 - JC-533190 - Lead Utility Auditor-Evaluator SENIOR MANAGEMENT AUDITOR $8,265.00 - $10,785.00 per Month New to State candidates will be hired into the minimum salary of the classification...Permanent employmentFull timeTemporary workWork at officeRemote work2 days per week$74k
...Bench General Manager - Retail Cafe - Fort Mason POSITION: Bench General Manager - Retail Cafe - Fort Mason CLASSIFICATION: Exempt REPORTING RELATIONSHIPS: Reports to the District Manager Direct Reports: Shift Leads, Baristas, and Food Prep Positions...Full timeWork at officeFlexible hoursShift workWeekend workAfternoon shiftEarly shiftFt Mason - Equator Coffees
San Francisco, CA15 hours ago$101.8k - $208.2k
...service delivery and the successful execution of projects. Define performance expectations, evaluate the performance of staff IT auditors, provide feedback and participate in training. Facilitate use of technology-based tools or methodologies to review, design and/...Full timeLocal areaWorldwide$99.5k - $165.5k
...experience managing restoration/remedial action projects including developing, maintaining, and enforcing the QC program for individual task orders. Strong interpersonal skills are a must, including the ability to communicate clearly and confidently, both written and...Contract workTemporary workFor contractorsWork from homeAPTIM
San Francisco, CA2 days ago- Job Description Job Description Company Description Medicine moves forward here. So will you. ADAMAS Consulting helps the world's pharmaceutical and biotechnology companies uphold the highest standards of quality, compliance and patient safety. Since 2007 we...FreelanceRemote work
Adamas Consulting
San Francisco, CA20 days ago $70 - $90 per hour
..., Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: AWS Serverless & Infrastructure-as-Code Task Auditor Type: Contract Compensation: $70–$90/hour Location: Remote Role Responsibilities Evaluate the quality and...Contract workSummer workLocal areaRemote work$50 - $55 per hour
...generated content in wholesale and technical sales to strengthen reasoning and rigor in model outputs . Review complex sales tasks for alignment with domain principles and methodologies. Provide clear, structured feedback to AI research teams to improve...Contract workSummer workRemote work- ...opportunities to complete your registration Merchandising & Audits available. See all information pertaining to rate of pay and tasks to be completed on the CCMI website (link below) Must have merchandising experience This is not a daily job, nor will it lead...Extra incomeFull timePart timeImmediate start
$105k
...to support continuous improvement and risk mitigation. Position Summary Reporting to the Director of Internal Audit, the Senior IT Auditor is responsible for evaluating technology and business related controls across IT focused and integrated audits, performing control...Work at officeRemote work$65 - $85 per hour
..., enhancements, and upgrades, as well as performing related activities for new implementations and functionality. You will also be tasked with creating training materials for system support and change management, and will collaborate on and often lead cross-functional...Hourly payContract workLocal areaFlexible hoursJobot
San Francisco, CA15 hours ago- ...3 years experience preferred Junior Compliance Officer Tasks At a minimum, the Junior Compliance Officer shall perform the... ...queries, prepare spreadsheets, and conduct research. Assist Auditors and Criminal Investigators with evaluation of Employment...Contract workFor contractorsLocal areaRemote workWeekend work
- A state government agency is seeking a Paralegal - Health Quality Enforcement Senior Legal Analyst in San Francisco. This full-time role involves high-level legal work in state and federal court litigation, case management, and document analysis. The ideal candidate is...Full timeFlexible hours
California Department of Justice
San Francisco, CA3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to SWE-Bench Task Auditor. Be the first to apply!
Related searches
- inventory auditor San Francisco, CA
- lease auditor San Francisco, CA
- safety auditor San Francisco, CA
- data auditor San Francisco, CA
- remote auditor San Francisco, CA
- auditor San Francisco, CA
- coding auditor San Francisco, CA
- sales auditor San Francisco, CA
- remote nurse auditor San Francisco, CA
- field auditor San Francisco, CA