SWE Benchmark Auditor for AI Model Evaluation
Dorado
Dorado seeks a software engineering quality assessor to evaluate the quality, correctness, and reproducibility of benchmark tasks used for frontier AI lab models. You will audit repository-level tasks, reference patches, test harnesses, and grading integrity, and provide rubric-based feedback. Ideal candidates will have 3+ years in professional software engineering, real open-source contribution or maintainer experience, and fluency in Python plus at least one of Java, Go, TypeScript, or C++. #J-18808-Ljbffr Dorado
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the SWE Benchmark Auditor for AI Model Evaluation in Brooklyn, NY vacancy
$70 - $90 per hour
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches,... ...Qualifications Familiarity with SWE-Bench (Verified) or similar...SuggestedRemote job$70 - $90 per hour
...for finding and building careers in AI training and data labeling. It... ...contributors review code, assess model outputs, and evaluate benchmark tasks so AI systems can become more... ...The Role OpenTrain is recruiting a SWE-Bench Task Auditor to evaluate repository-level software...SuggestedHourly payContract workPart timeFor contractorsRemote workFlexible hours- ...world's first real-time speech AI platform capable of accent... ...team combines deep expertise in model innovation and systems engineering... ...'s model families, build the evaluation infrastructure to measure it... ...meaningful progress, not just benchmark performance. Develop novel quantitative...Suggested
- Own dynamic model routing: the right model for each task, under... ...cost and policy ceiling. Dipp AI's strategy is decoupled orchestration... ..., capability profiles, evaluation harnesses, and the cost... ...behaviour. Skepticism about benchmarks and the discipline to construct...Suggested
$149 per hour
...team will provide more details.Vice President - Technical AI Foundation Model EngineerRole SummaryThe VP, Technical AI Foundation Model... ...(RAG) pipelines and semantic search capabilities.Evaluate, benchmark, and recommend foundation models based on performance, cost...SuggestedFull timeWork at officeLocal areaRemote work1 day per week- ...control environments for our firm's cloud and AI environments. Your responsibilities will... ...to the audit scope, risks and issues Evaluate, test and report on the adequacy and effectiveness... .... Experience working with either threat modeling, risk assessing or auditing cloud...
Fairygodboss
Jersey City, NJ3 days ago - Progressive Insurance seeks an internal IT auditor to join Corporate Internal Audit. You will evaluate technology and information security risks and controls, support audit engagements, perform testing, analyze results, and help identify opportunities to strengthen controls...Work at office
- ...- Design and optimization of AI-ready cloud environments: Modern AI workloads... ...we need expertise in securely connecting models, APIs, vector databases, and pipelines to... ...organization to pilot new technologies, evaluate vendor solutions, and rapidly implement new...
3B Staffing LLC
Jersey City, NJ2 days ago - Who we are Odyssey is an AI lab pioneering general world models: causal, multimodal systems that learn to predict and interact with the world over long... ...one system, data, architecture, training, inference, evaluation, rather than as six specialties, and go wherever the bottleneck...
- ...LifeSciences Corporation is seeking a Senior Auditor - IT to lead complex risk-based IT audits... ..., and applications. The role evaluates controls, governance, data integrity, and... ...emerging technology risks with emphasis on AI governance and responsible use. The successful...
- ...agentic revolution in travel. Sabre is an AI-native technology leader, backed by one... ...management practices. We are seeking a Staff IT Auditor to join our global Internal Audit team,... .... This team protects enterprise value by evaluating risks, strengthening technology controls...WorldwideFlexible hours
- We’re looking for a Senior IT Internal Auditor. The Senior IT Internal Auditor supports the... ...audit initiatives by identifying risks, evaluating control environments, and providing recommendations... ...emerging technology, cyber, and AI‑related risks. Experience with...Work at officeHome office
- OpenTrain AI is seeking a Multiline P&C Underwriter for a remote, part-time contractor role open to candidates in the United States, Canada, and the United Kingdom. You will design underwriting scenarios, write golden reference responses, and assess AI-generated outputs...Remote jobPart timeFor contractors
$67.3k - $89.7k
...already begun. As an internal IT auditor in our Corporate Internal... ...members and business partners to evaluate technology and information... ...experience with DevOps delivery models and tooling (e.g., GitHub, Azure... ...Experience evaluating AI/LLM-enabled processes for SOX...Temporary workWork experience placementH1bWork at officeFlexible hours- Sabre Corporation in Dallas, TX seeks a Staff IT Auditor to join its global Internal Audit team. You will evaluate IT general controls, system access, and risk across... ...and innovation. You'll leverage data analytics and AI tools to extract data, perform sampling, and...
- Chubb Ltd. seeks a VP, AI Operating Model and Execution to design, deploy, and scale AI infrastructure and agent-based workflows across the enterprise. You will bridge users, technologists, and senior leaders to demonstrate how AI can transform how work gets done. You...
- ...best-in-class.As a Risk Management Quant Modeling Director-Executive Director in Model... ...technical leadership and insight and to evaluate modeling choices using both evidence and... ...model validation workflows (e.g. building benchmark models) JPMorganChase, one of the oldest...
- ...across global operations.As a Principal Technical Program Manager Model & Analytics in Corporate Sector, you will lead complex, multi-... ...documentation, and production showcasesUses enterprise-authorized AI capabilities within the work environment to accelerate program planning...Work at office
$68.91k - $161.54k
...considering: Responsible for developing, verifying, and validating model-based control algorithms for embedded systems using MATLAB/... ...to engineering, all fueled by its market leading capabilities in AI, generative AI, cloud and data, combined with its deep industry expertise...Full timeLocal area- ServiceNow is seeking an experienced Senior Auditor to plan and lead global or regional audits across IT general controls, application controls, and business processes. This remote position reports to the Director, Internal Audit Management and emphasizes proactive, data...Remote work
$70 - $90 per hour
Evaluate the quality, correctness, and methodological rigor of applied machine-learning... ...used to train and evaluate a frontier AI lab's models. You'll assess experiment design, model... ...Preferred Qualifications Competition / benchmark experience (e.g., Kaggle) Graduate research...Remote job- ...within Treasury, you will responsible for building and enhancing models that inform wind-down and business packaging strategies across... ...and improve the framework through sensitivity testing, benchmarking, and structured problem-solvingServe as a primary point of contact...Work at officeVisa sponsorship
- ...seeks certified U.S. medical coders for high-level coding and documentation consulting, acting as subject matter expert to validate AI-generated coding logic and documentation reasoning. This flexible, remote role requires 2+ years of production coding or CDI experience...Remote jobFull timeFlexible hours
- ...Breakout, xStocks, Payward Services and CF Benchmarks - has spent the last 15 years building... ...execute the technology audit program, evaluating the design and operating effectiveness of... ...and privacy, operational resilience, and AI. This is a hands‑on role with real...Local area
Kraken International
Brooklyn, NY1 day ago - ...seeking a highly accomplished Senior Internal Auditor to serve as a strategic advisor in... ...business functions, you will identify risks, evaluate processes, and recommend practical... ...Compiles and visualizes data to support benchmarking, trend analysis, and audit/advisory reporting...Contract workWork experience placementWork at office
WMATA
Brooklyn, NY4 days ago - ...(i.e. risk management, compliance, fraud prevention), external auditors, and regulators, establishing strong working relationships while... ...of internal control concepts, with proven ability to evaluate and determine the adequacy of controls by considering business...Visa sponsorshipFlexible hours
- ...administration platform.As a Senior Auditor on the Private Bank/Wealth... ...activities. You will evaluate risks and controls across wealth... ...Identify opportunities to apply AI-enabled tools, prompt engineering... ...analytics and large language model, as well as champion a culture...Visa sponsorship
- Principal, Model Risk Management (Hybrid - Columbia, SC) The Principal... ...business activities. Evaluate model performance monitoring processes... ...-model risks associated with AI implementations. Advise... ...interacting with regulators, auditors, executive leadership, governance...
- ...judgement to strength JPMorganChase’s internal controls!As a Senior Auditor Associate in our Internal Audit group, you will plan, execute... ...risk assessments, audit planning, audit testing, control evaluation, report drafting and follow-up and verification of issue closure...Visa sponsorship
- As a Technology Senior Auditor (Associate) within the Infrastructure Platforms (IP) Technology Audit team, you will play a key role... ...audit engagements end-to-end (from planning through reporting), evaluating and testing control effectiveness and producing high-quality...Visa sponsorship
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to SWE Benchmark Auditor for AI Model Evaluation. Be the first to apply!
Related searches
- information system auditor Brooklyn, NY
- staff auditor Brooklyn, NY
- coding auditor Brooklyn, NY
- security auditor Brooklyn, NY
- energy auditor Brooklyn, NY
- medical records auditor Brooklyn, NY
- premium auditor Brooklyn, NY
- inventory auditor Brooklyn, NY
- entry level auditor Brooklyn, NY
- remote auditor Brooklyn, NY