Observability & Evaluation Engineer
Kasmo Inc
Job Description
Job Description
Description:
Charlotte, NC (Onsite)
Observability & Evaluation Engineer will build telemetry, tracing, dashboards, evaluation suites, alerts, service objectives, runbooks, and readiness evidence for Tachyon agent releases. This role ensures production AI systems can be monitored, evaluated, improved, and supported with clear operational visibility.
Key Responsibilities
Implement observability and telemetry for LLM-powered applications, agents, tools, and platform services.
Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety signals.
Develop dashboards, alerts, traces, metrics, service objectives, and reporting for production readiness.
Work with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
Create runbooks and support documentation for priority agent releases.
Analyze production behavior and recommend improvements to reliability, performance, and quality.
Required Qualifications
7 years of engineering experience with observability, monitoring, test automation, platform operations, or AI/ML systems.
5 years of strong hands-on Python experience.
5 years of Experience with dashboards, metrics, alerts, traces, logs, SLOs, and production monitoring.
5 years of Understanding of LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment approaches.
5 years of Experience working in Agile engineering teams and production support environments.
Required Skills / Knowledge
Python, telemetry, tracing, monitoring, dashboards, alerting, SLOs, evaluation frameworks, test automation, and production operations.
Understanding of LLMs, agents, RAG, prompt performance, retrieval quality, latency, and reliability metrics.
Experience with observability tools and open telemetry concepts.
Preferred Qualifications
Experience with GenAI observability, AI evaluation tools, ML monitoring, or platform reliability engineering.
Experience in regulated environments with evidence and readiness documentation.
Kubernetes, cloud platforms, and CI/CD experience.
Expected Outcomes
Operational dashboards and evaluation suites for priority agent releases.
Clear readiness evidence, alerts, SLOs, and runbooks.
Improved quality, reliability, and trust in production Agentic AI systems.
- ...Dynatrace Observability Engineer, Contract, W-2 Job Description: We are seeking an experienced Dynatrace Observability Engineer to design... ...and DevOps teams to implement end-to-end observability. Evaluate and implement Dynatrace Intelligence, Davis AI, automation,...SuggestedContract work
- ...Position: Senior Observability AIOps Engineer Location: Charlotte/ Dallas/ New Jersey Duration: 12+ months Contract Primary Objective: Build, integrate, and manage monitoring and observability platforms using AIOps to reduce alert noise and accelerate root-cause...SuggestedContract work
$60k - $85k
...is seeking an Early Career Structural Engineer to join our team. This position offers... ...projects. Conduct field investigations and evaluations of existing structures to support... ...drawing review, responses to RFIs, site observations, and evaluation of proposed changes....SuggestedFor contractorsLocal areaFlexible hours- ...Job Description Job Description ENGINEER IV - CIVIL SITE PRACTICE Three Notch Group... ...with construction-phase management and observation in accordance with contract documents. Define complex problems, collect and evaluate data, establish facts, and draw valid conclusions...SuggestedContract workWork at officeLocal areaNight shiftAfternoon shift
- ...At UES, we're a team of more than 4,000 engineers, scientists, geologists, inspectors,... ...slope stability, and geotechnical risk evaluation. Partner directly with clients, contractors... ...perception, and focus) and smell for observation and safety purposes. #LI-MD1...SuggestedContract workFor contractorsWork at office
- ...Company: We are looking for experienced Engineers (PE) to join our growing team, with an... ...include the design, analysis and evaluation of both new and existing building structures... ...including climbing ladders, performing visual observations from afar/rooftops/scaffolding/confined...Full timeTemporary workFor contractorsLocal area
$103.71k - $138.28k
...AI‑ready connectivity, join us today. The Role The Field Engineering Quality Control Engineer is responsible for overseeing and... ...new position or your current role. Background results will be evaluated on a case-by-case basis. Pursuant to the San Francisco Fair...Full timeTemporary workFor contractors$101.61k - $152.66k
...Energy‑as‑a‑Service (EaaS), we integrate engineering, design, construction, financing, and... ...applicable. Performs analysis, design, and evaluation of mechanical systems (HVAC, hydronic,... ..., to document existing conditions and observe construction progress. Light work...Full timeWork at officeRemote work- ...Description Job Description At Optima Engineering , an employee-owned MEP engineering... ...visits and field investigations to evaluate existing mechanical systems. Provide... ...submittal review, RFI responses, field observations, and closeout documentation. Participate...Flexible hours
- ...Senior Mechanical Engineer Our client is currently seeking a skilled and experienced... ...execution. Conduct preliminary project evaluations and prepare detailed reports, meeting... ...systems and facilities, and field observation and reporting. ~ Extensive knowledge...
- ...A leading comprehensive engineering design and consulting firm, SSR provides innovative solutions... ...ensuring accuracy, and neatness of work Evaluates the work of design colleagues and... ...during construction to monitor progress and observe installed construction; makes comments...For contractorsLocal area
- ...Timmons Group is seeking a Civil Project Engineer I/II to join our Traffic Analysis and... ..., corridor studies, access management evaluations, and intersection control evaluations,... ...and cost estimates; ~ Conduct field observations, including traffic counts, queue and...Work at officeLocal areaFlexible hours
$64k - $80k
...We are currently hiring a Structural Engineer to join our Building Engineering Division... ...well as investigation and performance evaluation. This position will work closely under... ...drawings, respond to RFI’s and perform field observations during CA phase. Salary Range: $64...Temporary workCurrently hiringWork at officeLocal areaImmediate startFlexible hours$113.61k - $189.31k
...Job Overview: LPL Financial is seeking a leader to oversee Observability Engineering and Operations Monitoring capabilities across the... ...efforts. Partner with Technology Operations leadership to evaluate and recommend monitoring technologies that support business...Full timeWork from home$75k - $80k
...world. Protecting its most vital resources. The future needs engineers who are ready to get out in the field, solve real-world... ...system testing, quality assurance activities, and performance evaluations Provide on-site and remote support during project implementation...Full timeFor contractorsInternshipWork at officeLocal areaRemote workFlexible hours- ...Certerra, formerly Southern Engineering & Testing, is seeking a Geotechnical Engineer for... ...and design calculations. Design and evaluate shallow and deep foundations, retaining... ...provide construction-phase consultation and observation services. Mentor and train junior...Work at officeLocal area
- ...Site Reliability & Environment Monitoring Engineer (Azure / Dynatrace / ServiceNow) We... ...to establish and evolve our enterprise observability and monitoring strategy across cloud... ...integration, and alerting architecture. Evaluate and recommend tooling, integration...Full timeShift workDay shift
$85 - $90 per hour
...Job Title: Mechanical / Piping Engineer Pipe Stress Specialist Job Description... ...industry codes and specialized software to evaluate stresses, displacements, and support loads... ...coordination. Field visits may include observing existing equipment and piping systems, verifying...Contract workTemporary workWork at office- ...opening for a motivated and driven Structural Engineer (PE)in Charleston, SC or Savannah, GA.... ...field work for assessments and construction observation in a variety of environments. Ability to solve problems and evaluate a variety of options. Ability to periodically...For contractorsImmediate start
- ...Description Job Description 2027 SUMMER INTERN - GILES FLYTHE ENGINEERS WANT AN INTERNSHIP WHERE YOU CAN SEE ENGINEERING IN THE... ...buildings, or structures. Complete forensic structural engineering evaluations and reports. Work on a variety of HOA and commercial...For contractorsSummer workInternshipSummer internshipWork at officeLocal areaWork visa
- ...relate to customers and each other as we Make it Happen! Field Engineer: Primary Responsibility: Estimate quantity take offs for... ...elevations. Field coordination with blasters for scope of work evaluation of bid versus actual work. Project engineering necessary to...
$32 - $36 per hour
...Job Description Job Description Job Title: HVAC Engineering Staff Lab Technician Job Description We are seeking an experienced... ...of our hiring process, including sourcing, screening, and evaluating candidates. AI helps assess applications and qualifications,...Contract workTemporary workApprenticeshipShift workDay shiftAfternoon shift- ...Your OpportunityAs a Senior Structural Engineer your role is to work with general supervision... ...Lead. You will perform and develop evaluations, analyses, calculations, design... ...field activities including, structural observations, monitoring, and other fieldworkWork in...Full timeTemporary workPart timeFor contractorsFor subcontractorCasual workWork at officeLocal areaRemote workFlexible hours
- ...As a Project Engineer at Graycor, you will enroll in our Engineering Mentoring Program designed to expose you with the critical hands-... ...materials, and track and expedite their delivery. Track and evaluate subcontractor and vendor performance. Interact with subcontractors...Full timeContract workTemporary workFor subcontractorInternship
$125k - $165k
...Job Description Job Description Position: Professional Civil Engineer, Charlotte Location: Charlotte, NC (fully remote through mid... ..., coordinate, and monitor assigned engineering projects Evaluate surveys, field notes, maps, aerial photographs, and other materials...Temporary workRemote work- ...Develops design and technical solutions along, with preparation of engineering analysis and calculations for private improvements and public... ...to determine a project’s economic feasibility Review, evaluate, and verify for accuracy and viability of designs, data, reports...Temporary workWork at officeRelocation packageFlexible hours
- ...projects, roadway and utility, and streetscape enhancements Evaluate design/construction -related deficiencies associated with... ...assigned Position Requirements: Bachelor's Degree in Civil Engineering required Minimum of five (5) years of civil/site development...Night shift
- ...Automation Engineer – Robotics & Industrial Automation Employment Type: Full-Time | Direct Hire Industry: Industrial Automation... ...depending on experience and the specific opportunity. Develop and evaluate automation solutions involving industrial robotics, CNC...Full time
$166k - $203k
...Fortinet is growing and we are looking to hire Professional Services Engineers to join our team. The dedicated Professional Services... ...configurations and implementing best practices Performing interoperability evaluation of devices that interact with Fortinet products Training and...Full timeRemote workWorldwideFlexible hoursShift work- ...developers. You will work closely with the engineers within our team, architects, Security... ...AI capabilities through automation, observability, and self-service capabilities.Collaborate... ...management.Practical experience evaluating the quality and reliability of AI-generated...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Observability & Evaluation Engineer. Be the first to apply!







