Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Observability & Evaluation Engineer

Kasmo Inc

Job Description

Job Description




Description:
Charlotte, NC (Onsite)
Observability & Evaluation Engineer will build telemetry, tracing, dashboards, evaluation suites, alerts, service objectives, runbooks, and readiness evidence for Tachyon agent releases. This role ensures production AI systems can be monitored, evaluated, improved, and supported with clear operational visibility.
Key Responsibilities
Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
Create runbooks and support documentation for priority agent releases.
Analyze production behavior and recommend improvements to reliability, performance, and quality.
Required Qualifications
7 years of engineering experience with observability, monitoring, test automation, platform operations, or AI/ML systems.
5 years of strong hands-on Python experience.
5 years of Experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
5 years of Understanding of LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
5 years of Experience working in Agile engineering teams and production support environments.
Required Skills / Knowledge
Python, telemetry, tracing, monitoring, dashboards, alerting, SLOs, evaluation frameworks, test automation, and production operations.
Understanding of LLMs, agents, RAG, prompt performance, retrieval quality, latency, and reliability metrics.
Experience with observability tools and open telemetry concepts.
Preferred Qualifications
Experience with GenAI observability, AI evaluation tools, ML monitoring, or platform reliability engineering.
Experience in regulated environments with evidence and readiness documentation.
Kubernetes, cloud platforms, and CI/CD experience.
Expected Outcomes
Operational dashboards and evaluation suites for priority agent releases.
Clear readiness evidence, alerts, SLOs, and runbooks.
Improved quality, reliability, and trust in production Agentic AI systems.

Vacancy posted 21 days ago
Similar jobs that could be interesting for youBased on the Observability & Evaluation Engineer in Charlotte, NC vacancy
  •  ...Dynatrace Observability Engineer, Contract, W-2 Job Description: We are seeking an experienced Dynatrace Observability Engineer to design...  ...and DevOps teams to implement end-to-end observability. Evaluate and implement Dynatrace Intelligence, Davis AI, automation,... 
    Suggested
    Contract work

    DataEdge Consulting

    Charlotte, NC
    2 days ago
  •  ...Position: Senior Observability AIOps Engineer Location: Charlotte/ Dallas/ New Jersey Duration: 12+ months Contract Primary Objective: Build, integrate, and manage monitoring and observability platforms using AIOps to reduce alert noise and accelerate root-cause... 
    Suggested
    Contract work

    KMM Technologies, Inc.

    Charlotte, NC
    12 days ago
  • $60k - $85k

     ...is seeking an Early Career Structural Engineer to join our team. This position offers...  ...projects. Conduct field investigations and evaluations of existing structures to support...  ...drawing review, responses to RFIs, site observations, and evaluation of proposed changes.... 
    Suggested
    For contractors
    Local area
    Flexible hours

    Lead Dog Recruiting

    Charlotte, NC
    2 days ago
  •  ...Job Description Job Description ENGINEER IV - CIVIL SITE PRACTICE Three Notch Group...  ...with construction-phase management and observation in accordance with contract documents. Define complex problems, collect and evaluate data, establish facts, and draw valid conclusions... 
    Suggested
    Contract work
    Work at office
    Local area
    Night shift
    Afternoon shift

    THREE NOTCH GROUP INC

    Charlotte, NC
    25 days ago
  •  ...At UES, we're a team of more than 4,000 engineers, scientists, geologists, inspectors,...  ...slope stability, and geotechnical risk evaluation. Partner directly with clients, contractors...  ...perception, and focus) and smell for observation and safety purposes. #LI-MD1... 
    Suggested
    Contract work
    For contractors
    Work at office

    Universal Engineering Sciences

    Pineville, NC
    3 days ago
  •  ...Company: We are looking for experienced Engineers (PE) to join our growing team, with an...  ...include the design, analysis and evaluation of both new and existing building structures...  ...including climbing ladders, performing visual observations from afar/rooftops/scaffolding/confined... 
    Full time
    Temporary work
    For contractors
    Local area

    Britt, Peters and Associates

    Charlotte, NC
    4 days ago
  • $103.71k - $138.28k

     ...AI‑ready connectivity, join us today. The Role The Field Engineering Quality Control Engineer is responsible for overseeing and...  ...new position or your current role. Background results will be evaluated on a case-by-case basis. Pursuant to the San Francisco Fair... 
    Full time
    Temporary work
    For contractors

    Lumen

    Charlotte, NC
    1 day ago
  • $101.61k - $152.66k

     ...Energy‑as‑a‑Service (EaaS), we integrate engineering, design, construction, financing, and...  ...applicable. Performs analysis, design, and evaluation of mechanical systems (HVAC, hydronic,...  ..., to document existing conditions and observe construction progress. Light work... 
    Full time
    Work at office
    Remote work

    ENFRA

    Charlotte, NC
    4 days ago
  •  ...Description Job Description At Optima Engineering , an employee-owned MEP engineering...  ...visits and field investigations to evaluate existing mechanical systems. Provide...  ...submittal review, RFI responses, field observations, and closeout documentation. Participate... 
    Flexible hours

    Optima Engineering

    Charlotte, NC
    a month ago
  •  ...Senior Mechanical Engineer Our client is currently seeking a skilled and experienced...  ...execution. Conduct preliminary project evaluations and prepare detailed reports, meeting...  ...systems and facilities, and field observation and reporting. ~ Extensive knowledge... 

    Interfuse Staffing

    Charlotte, NC
    4 days ago
  •  ...A leading comprehensive engineering design and consulting firm, SSR provides innovative solutions...  ...ensuring accuracy, and neatness of work Evaluates the work of design colleagues and...  ...during construction to monitor progress and observe installed construction; makes comments... 
    For contractors
    Local area

    Smith Seckman Reid

    Charlotte, NC
    19 hours ago
  •  ...Timmons Group is seeking a Civil Project Engineer I/II to join our Traffic Analysis and...  ..., corridor studies, access management evaluations, and intersection control evaluations,...  ...and cost estimates;  ~ Conduct field observations, including traffic counts, queue and... 
    Work at office
    Local area
    Flexible hours

    Timmons Group

    Charlotte, NC
    12 days ago
  • $64k - $80k

     ...We are currently hiring a Structural Engineer to join our Building Engineering Division...  ...well as investigation and performance evaluation. This position will work closely under...  ...drawings, respond to RFI’s and perform field observations during CA phase. Salary Range: $64... 
    Temporary work
    Currently hiring
    Work at office
    Local area
    Immediate start
    Flexible hours

    LaBella Associates

    Charlotte, NC
    5 days ago
  • $113.61k - $189.31k

     ...Job Overview: LPL Financial is seeking a leader to oversee Observability Engineering and Operations Monitoring capabilities across the...  ...efforts. Partner with Technology Operations leadership to evaluate and recommend monitoring technologies that support business... 
    Full time
    Work from home

    LPL Financial

    Charlotte, NC
    12 days ago
  • $75k - $80k

     ...world. Protecting its most vital resources. The future needs engineers who are ready to get out in the field, solve real-world...  ...system testing, quality assurance activities, and performance evaluations Provide on-site and remote support during project implementation... 
    Full time
    For contractors
    Internship
    Work at office
    Local area
    Remote work
    Flexible hours

    Emerson

    Charlotte, NC
    2 days ago
  •  ...Certerra, formerly Southern Engineering & Testing, is seeking a Geotechnical Engineer for...  ...and design calculations. Design and evaluate shallow and deep foundations, retaining...  ...provide construction-phase consultation and observation services. Mentor and train junior... 
    Work at office
    Local area

    Certerra

    Charlotte, NC
    20 days ago
  •  ...Site Reliability & Environment Monitoring Engineer (Azure / Dynatrace / ServiceNow) We...  ...to establish and evolve our enterprise observability and monitoring strategy across cloud...  ...integration, and alerting architecture. Evaluate and recommend tooling, integration... 
    Full time
    Shift work
    Day shift

    100 CRC Insurance Group, LLC

    Charlotte, NC
    19 hours ago
  • $85 - $90 per hour

     ...Job Title: Mechanical / Piping Engineer Pipe Stress Specialist Job Description...  ...industry codes and specialized software to evaluate stresses, displacements, and support loads...  ...coordination. Field visits may include observing existing equipment and piping systems, verifying... 
    Contract work
    Temporary work
    Work at office

    Actalent

    Charlotte, NC
    2 days ago
  •  ...opening for a motivated and driven Structural Engineer (PE)in Charleston, SC or Savannah, GA....  ...field work for assessments and construction observation in a variety of environments. Ability to solve problems and evaluate a variety of options. Ability to periodically... 
    For contractors
    Immediate start

    Thomas & Hutton

    Charlotte, NC
    21 days ago
  •  ...Description Job Description 2027 SUMMER INTERN - GILES FLYTHE ENGINEERS WANT AN INTERNSHIP WHERE YOU CAN SEE ENGINEERING IN THE...  ...buildings, or structures. Complete forensic structural engineering evaluations and reports. Work on a variety of HOA and commercial... 
    For contractors
    Summer work
    Internship
    Summer internship
    Work at office
    Local area
    Work visa

    Giles Flythe Engineers

    Charlotte, NC
    11 days ago
  •  ...relate to customers and each other as we Make it Happen! Field Engineer: Primary Responsibility: Estimate quantity take offs for...  ...elevations. Field coordination with blasters for scope of work evaluation of bid versus actual work. Project engineering necessary to... 

    Maine Drilling & Blasting Inc

    Charlotte, NC
    4 days ago
  • $32 - $36 per hour

     ...Job Description Job Description Job Title: HVAC Engineering Staff Lab Technician Job Description We are seeking an experienced...  ...of our hiring process, including sourcing, screening, and evaluating candidates. AI helps assess applications and qualifications,... 
    Contract work
    Temporary work
    Apprenticeship
    Shift work
    Day shift
    Afternoon shift

    Actalent

    Charlotte, NC
    2 days ago
  •  ...Your OpportunityAs a Senior Structural Engineer your role is to work with general supervision...  ...Lead. You will perform and develop evaluations, analyses, calculations, design...  ...field activities including, structural observations, monitoring, and other fieldworkWork in... 
    Full time
    Temporary work
    Part time
    For contractors
    For subcontractor
    Casual work
    Work at office
    Local area
    Remote work
    Flexible hours

    Stantec

    Charlotte, NC
    2 days ago
  •  ...As a Project Engineer at Graycor, you will enroll in our Engineering Mentoring Program designed to expose you with the critical hands-...  ...materials, and track and expedite their delivery. Track and evaluate subcontractor and vendor performance. Interact with subcontractors... 
    Full time
    Contract work
    Temporary work
    For subcontractor
    Internship

    Graycor

    Charlotte, NC
    2 days ago
  • $125k - $165k

     ...Job Description Job Description Position: Professional Civil Engineer, Charlotte Location: Charlotte, NC (fully remote through mid...  ..., coordinate, and monitor assigned engineering projects Evaluate surveys, field notes, maps, aerial photographs, and other materials... 
    Temporary work
    Remote work

    Noor Staffing Group

    Charlotte, NC
    13 days ago
  •  ...Develops design and technical solutions along, with preparation of engineering analysis and calculations for private improvements and public...  ...to determine a project’s economic feasibility Review, evaluate, and verify for accuracy and viability of designs, data, reports... 
    Temporary work
    Work at office
    Relocation package
    Flexible hours

    Becker Morgan

    Charlotte, NC
    25 days ago
  •  ...projects, roadway and utility, and streetscape enhancements Evaluate design/construction -related deficiencies associated with...  ...assigned Position Requirements: Bachelor's Degree in Civil Engineering required Minimum of five (5) years of civil/site development... 
    Night shift

    SEA, Ltd.

    Charlotte, NC
    15 days ago
  •  ...Automation Engineer – Robotics & Industrial Automation Employment Type: Full-Time | Direct Hire Industry: Industrial Automation...  ...depending on experience and the specific opportunity. Develop and evaluate automation solutions involving industrial robotics, CNC... 
    Full time

    Unify Recruit

    Charlotte, NC
    13 days ago
  • $166k - $203k

     ...Fortinet is growing and we are looking to hire Professional Services Engineers to join our team. The dedicated Professional Services...  ...configurations and implementing best practices Performing interoperability evaluation of devices that interact with Fortinet products Training and... 
    Full time
    Remote work
    Worldwide
    Flexible hours
    Shift work

    Fortinet

    Charlotte, NC
    4 days ago
  •  ...developers. You will work closely with the engineers within our team, architects, Security...  ...AI capabilities through automation, observability, and self-service capabilities.Collaborate...  ...management.Practical experience evaluating the quality and reliability of AI-generated... 
    Full time

    Vanguard

    Charlotte, NC
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Observability & Evaluation Engineer. Be the first to apply!