Software Engineer, Voice Evaluation [NYC or SF]
$220k - $300kConcurrence
Job Description
Job Description
About Concurrence
Concurrence (formerly Amigo) is building the platform healthcare organizations use to build their own clinical AI agents. Our mission is to make excellent care accessible to every patient, not rationed by the limits of clinician time.
Underneath our platform is the Health World Model, which unifies a health system’s data into a living, event-level record of every patient’s care. On top of it, agents handle work across the patient journey, from intake and scheduling to medication management and clinical decision support.
Our agents have supported more than 10 million patient encounters and are on track to grow tenfold this year. We run natively on Databricks, integrate with every major EHR, and are backed by Madrona, General Catalyst, and Optum Ventures. We’re about 35 people, working in person in New York and San Francisco.
About this roleAs a Software Engineer, Voice Evaluation at Concurrence, you'll build the system that determines whether a voice agent is ready for patients. You'll own realistic audio tests, scoring and the release gate, with authority to block releases that fail it. The tests need to catch failures that a clean transcript can miss, from mishandled interruptions to missed clinical information or a failure to hand off to a person.
What you'll do-
Build simulated callers and call-replay tests that exercise the production voice path with real audio
-
Test phone trees, hold queues, interruptions, conflicting clinical information and appropriate escalation
-
Build scoring systems and calibrate automated judges against human assessments
-
Measure clinical failure rates and uncertainty, and test whether simulation results predict behavior on real calls
-
Define release criteria with model, runtime, product and clinical colleagues, own the gate and block changes that fail those criteria
-
Turn production failures into regression tests, verify fixes and monitor behavior after release
-
Maintain held-out evaluation sets so models are tested on examples they did not train against
-
You've personally built and owned evaluation or simulation for production agents at scale. Voice and audio experience is a strong preference
-
You've built agent simulations or replay systems, implemented scoring and caught a concrete failure before release
-
Your evaluation results determined what shipped, and you led that work
-
You can explain what a score captures, what it misses and how you checked it against real behavior
-
You write reliable software that runs at volume and follow failures through to a verified fix
-
You use coding agents in your work and independently verify their output
-
You're low ego, direct, and can hold the line on release criteria with evidence
-
You can work on site in New York City or San Francisco
-
Experience calibrating automated judges against human labels
-
Load testing and concurrent calls using audio
-
Speech or audio machine learning
-
Experience in healthcare or another regulated environment
Health & Wellness
-
Comprehensive health, dental, and vision insurance
-
Daily catered lunch and dinner
-
Mental health support and wellness coaching
-
Flexible wellness stipend for fitness, therapy, or personal growth
-
Annual learning budget for courses, books, or conferences
-
Conference attendance budget for professional development
-
Annual team offsite
-
Academic collaboration opportunities
-
Unlimited PTO
-
Patients Win, We Win
If patients aren't getting better care, we haven't earned the right to scale. Every internal decision gets pressure-tested: does this make patients' lives better? If we can't draw the line, we question why we're doing it.
-
High Standards, High Care
We hold a high bar for the team because patients are counting on us to get this right. But high standards only work with genuine investment in each other. You can take risks, admit mistakes, and challenge ideas. Not despite our standards, but because of them.
-
Thoughtful Urgency
We move fast by default, but speed without judgment is recklessness. The discipline is knowing which decisions are reversible vs. not. In healthcare AI, the companies that win will be fast everywhere they can be and careful everywhere they must be. We build the muscle to do both.
-
Intensely Measured
We instrument patient outcomes, provider ROI, system performance, and clinical accuracy. But data without action is surveillance. Every metric should have an owner, a threshold, and a response plan. If we're measuring something but never acting on it, we stop measuring it.
-
Low ego: Politics and territory don't interest you. The best ideas win, regardless of who has them.
-
Direct: You say the hard thing, challenge ideas openly, and commit fully once decided.
-
High agency: You thrive on trust rather than instruction. When you see something is broken, you fix it. You don’t file tickets and wait for someone else.
-
Bar of excellence: You hold yourself to a bar most people wouldn't, and you want teammates who do the same.
-
Skeptical: You push back on rules that don’t make sense and question assumptions that haven’t earned their place.
Compensation Range: $220K - $300K
$75 - $105 per hour
...document the evidence behind your conclusion. You’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent’s code, an overly restrictive assertion, an unclear...SuggestedHourly payImmediate startRemote work$184k - $259.44k
...seeking a highly skilled and motivated Software Engineer, ARC (Architecture, Reliability, &... ...support work 3-4 days a week from the DC, SF, NYC, or STL office.Compensation packages... ...allows us to ensure a fair and thorough evaluation of all applicants.About Us:At Scale,...SuggestedFull timeWork at office3 days per week$175.77k - $214.83k
...NY/NJ area and willing to work from our NYC office two days per week.WHAT YOU BRING:5... ...applications.Certification-level fluency in data engineering, big data, and database operations,... ...their rightful place and where every voice matters. Where uniqueness isn’t just accepted...SuggestedWork at officeRemote workFlexible hours2 days per week$174k - $252k
...and maintain reusable eval infra blocks, such as streaming evaluation, autorater pipelines, and metric aggregation.Make large-scale... ...platforms).Experience developing in Python and C++.Google's software engineers develop the next-generation technologies that change how billions...Suggested$157k - $235k
...digital services.Snap Engineering teams build fun and technically... ...product requirements, evaluate trade-offs, and... ...of post-Bachelor’s software development experience... ...diverse backgrounds and voices working together will... ...future.Zone A (CA, WA, NYC):The base salary range...SuggestedFull timeLive inWork at officeLocal area$10k
...for a product-minded backend engineer to help build the... ...iteration.What You Need5+ years of software engineering experience building... ...systems, including approvals, evaluations, tool execution, and... ...moreRelocation expense coverage to NYC or SF (if needed)CanadaGroup...Full timeWork at officeHome officeFlexible hours$118k - $175k
...initiating a search for a Lead Civil Engineer to join our growing Northeast... ...contribute to the design of NYC‑based site civil... ...issues or opportunities. Analyze, evaluate, and interpret data obtained... ..., discipline-specific design software (i.e., Autodesk Civil 3D, Bentley...For subcontractorWork at officeLocal areaRelocationFlexible hours- ...team OpenAI’s Forward Deployed Engineering team partners with customers... ...looking for customer-focused software engineers to build effective... ...feedback. This role is based in NYC. We use a hybrid work model... ...many different perspectives, voices, and experiences that form...Full timeInternshipWork at officeRelocation package
- ...work 100x faster without heavy engineering overhead. About the role... ...~0-2 years of professional software development experience (at... ...Comfortable working on-site in NYC (4–5 days/week) to foster a... ...AI : Familiarity with evaluating AI models or integrating multi...
$10k
...make real decisionsStrong Python and software engineering; your eval code is reproducible and fastStatistical... ...agentic tasksPublished benchmark or evaluation research (NeurIPS Datasets &... ...and moreRelocation expense coverage to NYC or SF (if needed)CanadaGroup medical, dental...Full timeWork at officeHome officeFlexible hours- ...markets, powered by MyPrize's proprietary AI personalization engine. With SOC-2 Type 1 certification, MyPrize maintains... ...entertainment. The Need: We're seeking a Senior Software Engineer to work out of our NYC office. This is an exciting opportunity to join a fast-...Work at officeRemote workFlexible hours
$52 - $58 per hour
...cultivating a work environment where all team members belong and have the opportunity to thrive. Interns work side-by-side with top engineers in the industry while having autonomy from the get-go. They contribute to user-facing products and are able to see their work go...Full timeWork experience placementSummer workInternshipSummer internshipWork at officeLocal areaImmediate startWork from home3 days per week$209k - $313k
...digital services.Snap Engineering teams build fun and technically... ....We’re looking for a Software Engineer, iOS to join... ...new featuresCarefully evaluate the technical... ...diverse backgrounds and voices working together will... ...future.Zone A (CA, WA, NYC):The base salary range...Full timeLive inWork at officeLocal area$10k
...the place to do it.About the RoleAs a software engineer on the AI Solutions team, you will co-... ...GenAI, including problem decomposition, evaluation, and deployment trade-offs.Strong... ...and moreRelocation expense coverage to NYC or SF (if needed)CanadaGroup medical, dental...Full timeWork at officeHome officeFlexible hours- ...Trading Technology Lead | Leading Proprietary Trading Firm | NYC/Philly | $1M+ Comp This role pays $1M+. So let’s be clear about... ...for its front-office trading technology group. You’ll lead 6 engineers sitting with traders, growing to 8+. You own the architecture, the...Full time
- ...Team OpenAI's Applied AI Engineering team helps... ...through prototyping, evaluation, production launch, and... ...role is based in our SF or NYC office. We use a hybrid... ...deployed engineering, software engineering, customer... ...different perspectives, voices, and experiences that...Work at officeRelocation package
- Engineering Manager/Technical Lead We're looking for an Engineering Manager... ...to help build Maven AGI's Voice Agent platform. Maven Voice... ...and co-develop with a team of software engineers building Maven's... ...across customers and use cases Evaluate and adopt technologies that improve...Work at officeHome office
$200k - $260k
...Concurrence meets the real world. Our core engineering team builds the platform. We take it... ...actually needs on top of it, as durable software that runs in production. As a Senior... ...workflows. Experience with Databricks, voice agents, or forward deployed or customer...Full timeFlexible hours- ...W2 Role | Healthcare experience is preferred. Sr. AI Engineer NYC, NY (Full Onsite) AI Engineer with strong hands-on experience... ...grounding, and context orchestration patterns. Define evaluation frameworks and prompt testing methodologies to measure agent...Full time
$148.75k - $192.5k
...Role OverviewCboe Data Vantage is looking for an experienced software engineer who is excited about combining strong engineering skills... ...optimize, and debug real-time financial analytics software Evaluate, tune, and improve model performance, stability, and accuracy...Full timeWork at office$180k - $220k
...Senior Software Engineer (Voice AI) Location: New York City | In-office The Mission “Leading healthcare into the agentic era.” Every... .... Partner with the quality engineering team on shared evaluation tooling and cross-product regression coverage. Own Voice...Full timeWork at office$220k - $260k
...Optum Ventures. We’re about 35 people, working in person in New York and San Francisco. About this role As a Senior Full Stack Engineer at Concurrence, you'll build the applications that clinical teams, researchers, and customers use every day: real-time oversight...Full timeFlexible hours- ...improvements for some of the world's largest retailers.As a Full Stack Engineer, you'll work across every layer of the platform, building... ...of their work quickly.Required Skills & Experience3+ years of software engineering experience with meaningful experience across both...Flexible hours
$50 - $70 per hour
Opportunity Details Full Time NYC SCA Senior Developer/Tech Lead New York, NY (Hybrid... ...developers to foster a culture of engineering best practices. Responsibilities to include... ...abreast of emerging technologies and evaluate their potential benefits for the organization...Hourly payFull timeContract workTemporary workInternshipLocal areaImmediate startFlexible hours$10k
...the place to do it.About the RoleGrowth engineering builds the systems and applications... ...mission, and values!What You’ll DoBuild software that helps grow Ramp to its next millions... ...and moreRelocation expense coverage to NYC or SF (if needed)CanadaGroup medical, dental,...Full timeWork at officeHome officeFlexible hours$276k - $414k
...digital services.Snap Engineering teams build fun and technically... ...for a Principal Software Engineer to join the... ...product requirements, evaluate trade-offs, and... ...diverse backgrounds and voices working together will... ...future.Zone A (CA, WA, NYC):The base salary range...Full timeLive inWork at officeLocal area$208k - $312k
...agents and people to ship and run software, built in the open and trusted... ...We’re looking for a Software Engineer to join our Platform team. You... ...of one of our offices (SF, NY, London, or Berlin), the role... ...record-keeping requirements. Evaluate, integrate, and help operate...Work at officeRemote workWork from homeWorldwideMonday to FridayFlexible hours$197.3k - $225.1k
AI Engineer 4 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for... ...Design, develop, test, deploy, and support AI software components including foundation model training,...Full timePart timeLocal area$229.9k - $262.4k
AI Engineer 5 (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation) At Capital One, we are creating responsible and reliable AI systems, changing banking for... ...Design, develop, test, deploy, and support AI software components including foundation model training,...Full timePart timeLocal area- ...and we’re looking for a Lead Solutions Engineer in our NYC headquarters. As a Lead Solutions... ...Proof of Concepts (POCs) and technical evaluations. You excel at managing multiple high-stakes... ...team members. Having an opinion and voicing your ideas is not only welcome but...Contract workWork at officeWorldwideHome office3 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer, Voice Evaluation [NYC or SF]. Be the first to apply!
- junior software developer New York, NY
- senior software design engineer New York, NY
- software web developer New York, NY
- senior software engineer New York, NY
- software sales engineer New York, NY
- software developer fintech New York, NY
- software support engineer New York, NY
- security software engineer New York, NY
- oracle software engineer New York, NY
- software developer night shift New York, NY








