Monitoring And Evaluation Specialist
Full-time
KTek Resourcing
About the Role
We are seeking a skilled GenAI / LLM Observability & Evaluation Engineer to help ensure the reliability, performance, and quality of production AI agents and LLM-powered applications.
The ideal candidate will have hands-on experience implementing LLM evaluation frameworks, distributed tracing, telemetry, monitoring dashboards, automated testing, and production reliability practices .
Required Skills
- Hands-on experience with LLMs, Generative AI, or Agentic AI applications.
- Experience with LLM evaluation, agent testing, and performance analysis.
- Strong Python programming and test automation skills.
- Experience implementing observability, tracing, telemetry, and monitoring solutions.
- Experience building metrics dashboards and analyzing application performance.
- Understanding of SLOs, production operations, incident response, and operational readiness.
Preferred Skills
- LLM evaluation tools such as DeepEval, Ragas, Promptfoo, or equivalent.
- AI observability platforms such as LangSmith, Langfuse, Arize Phoenix, or equivalent.
- OpenTelemetry, OpenInference, Prometheus, Grafana, Datadog, or Splunk.
- Experience with LangChain, LangGraph, Google ADK, or similar agent frameworks.
- Familiarity with CI/CD pipelines, automated regression testing, and release quality gates.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Monitoring And Evaluation Specialist in Charlotte, NC vacancy
- ...Observability & Monitoring Analyst *Contract with potential extensions/conversion* Hybrid: 3x a week onsite in Charlotte, NC Standard... ...-call operations through better observability and alerting Evaluate current monitoring practices to identify noise and propose...SuggestedContract work
Optomi
Charlotte, NC1 day ago - ...Fargo is seeking a Senior Wealth Banking Specialist (SAFE) within Wealth & Investment... ...credit score is not included as part of the evaluation. Successful candidates must also meet... ...decisions. There is emphasis on proactive monitoring, governance, risk identification and...SuggestedFull timeWork experience placementWork at officeImmediate start
Wells Fargo
Charlotte, NC6 days ago - ...DescriptionProSidian Seeks a Learning Management System (LMS) Specialist (Contract Contingent) in Charlotte, NC / Remote to support... ...Candidates shall work to support requirements for PR0001 Monitoring, Evaluation, And Learning (MEL) Support and will manage Learning Management...SuggestedFull timeContract workTemporary workFor contractorsRemote workFlexible hours
Prosidian Consultng
Charlotte, NC2 days ago - ...capable Senior Protective Intelligence Specialist to support the protection of senior Vanguard... ...intelligence, and real-time OSINT monitoring during sensitive movements.Coordinate protective... ...) skills: identifying, collecting, evaluating, and analyzing publicly available...SuggestedFull timeImmediate startWorldwideWeekend workAfternoon shiftEarly shift
Vanguard
Charlotte, NC2 days ago $101.61k - $135.64k
...success.OverviewThe Senior Project Controls Specialist is a senior individual contributor... ...identifies emerging performance issues, evaluates root causes, and recommends corrective... ...assessment, quantification, documentation, and monitoring of project cost, schedule, resource,...SuggestedContract workFor subcontractorWork at officeENFRA
Charlotte, NC6 days ago$22 per hour
...Foreign Material Intrusion Prevention Monitor This position is posted on a continuous basis and does not have a fixed application close date. Applications are reviewed on an ongoing basis. A Day in the Life and the Impact You'll Make: DEKRA FME Services is a specialty...Hourly paySeasonal workRemote workRelocationLong distanceShift workNight shiftAfternoon shiftDEKRA
Belmont, NC4 days ago$175k
...Veterinarian Opportunity Carolina Veterinary Specialists (CVS) – Charlotte, NC Carolina... ...coagulation), SurgiVet multi-parameter monitoring systems, and three high-resolution... ...weekly priorities and expectations Evaluate and monitor protocols for the daily running...Relocation packageNight shiftDay shiftCarolina Veterinary Specialists Charlotte
Charlotte, NC3 days ago$25 - $35 per hour
Ad Ref: FAR-02413 Earning Potential: While pay is per-inspection, efficient contractors typically earn the equivalent of $25-$35 per hour. Do not fill out any form or use any other apply button on this page. Use the blue link below to submit your application. ...Hourly payPrice workExtra incomeFull timeFor contractorsLive inLocal areaFlexible hoursFAR Inspections
Matthews, NC4 days ago- ...Job Description Job Description SUMMARY: The Testing and Monitoring Compliance Analyst II is responsible for evaluating, and executing complex compliance reviews across key business operations. This position collaborates in the development and enhancement...Work at office
First Bank
Charlotte, NC20 days ago $175k
...Veterinarian Opportunity Carolina Veterinary Specialists is looking to add an Emergency... ...coagulation), SurgiVet multi-parameter monitoring systems, and three high-resolution... ...weekly priorities and expectations Evaluate and monitor protocols for the daily running...Local areaRelocation packageNight shiftDay shiftAccreditation Council For Graduate Medical Education
Charlotte, NC3 days ago$20 per hour
...Patient Access and Prior Authorization Specialists! Job Description This role serves as a... ...traditional insurance verification. You will monitor and guide AI-driven calls between... ...process, including sourcing, screening, and evaluating candidates. AI helps assess...Contract workTemporary workInternshipWork at officeRemote work- ...Description Cardiovascular & Electrophysiology Specialist IV Location: Charlotte, NC Shift... ..., including scrubbing, circulating, monitoring, cardiac mapping, device programming,... ...a Pacemaker System Analyzer (PSA) to evaluate electrode lead signals and conductivity...Full timeShift work
Staff Icons
Charlotte, NC2 days ago $77k - $133k
...strategic credit decisions. You'll leverage sound judgment to evaluate business, operational, and risk-related challenges, develop solutions... ...making sound risk decisions. There is emphasis on proactive monitoring, governance, risk identification and escalation, as well as...Full timeWork experience placementWork at officeRelocation package- Description Purpose: The intervention specialist is responsible for all aspects of Special... ...interferes with their or others’ learning and monitor progress; Assist with designing... ...Determination Review Pr - 04 Referral for Evaluation Pr - 05 Parent Consent for Evaluation...
$77k - $133k
...Fargo is seeking a Supervisory Control Specialist to join our Annuities and Insurance Supervision... ...Support branch offices with on-going monitoring to ensure compliance with internal... ...strategies and complex products Evaluate resources to monitor risk and exercise...Ongoing contractFull timeWork experience placementWork at officeImmediate startRemote work- ...Participates in audit and performance improvement activities by monitoring and reporting to appropriate staff and the Quality Improvement... ...needs assessment, meeting quality standards for services, and evaluation of customer satisfaction. Assumes responsibility for related...Full timeWork at officeShift work
- ...of health, safety and security solutions and services. This full-time opportunity functions in our state-of-the-art 24/7 Alarm Monitoring Center as an Alarm Monitoring Representative working a Mid-Shift, 2nd, or 3rd shift schedule which includes holidays and weekends...Full timeTemporary workWork experience placementImmediate startShift workNight shiftAfternoon shift
Securitas Electronic Security Inc
Charlotte, NC3 days ago - ...Job Description Job Description Our Alarm Monitoring Advisors also known as our Central Station Operators are an integral part of what we do (and yes, you'll be the one saying, "CPI Security – IDENTIFY YOURSELF!"). YOU make every day safer for our community...Temporary workLocal areaFlexible hours
CPI Security
Charlotte, NC7 days ago - ...requirements and specifications. This work is accomplished by performing inspections, surveillance / audits, document reviews and ongoing monitoring at our client construction site. Lead/performs auditing services and assessments of equipment suppliers, contractors, vendors,...Hourly payContract workFor contractorsWork experience placementCasual workH1b
Consulting Solutions
Charlotte, NC2 days ago $64.8k - $97.2k
...Arcadis is seeking an Industrial Hygiene Specialist to join our Sustainable Operations team... ...and mold, industrial hygiene exposure monitoring, building characterization assessments,... ..., indoor air quality and mold evaluations, and chemical exposure assessments. Perform...Full timePart timeFor subcontractorWork at officeShift workNight shiftARCADIS
Charlotte, NC1 day ago$100k
...Regional Manager. Carolina Veterinary Specialists (CVS) in Charlotte, North Carolina is... ...coagulation), SurgiVet Multi-Parameter monitoring systems, 3 Toshiba high-resolution ultrasound... ...weekly priorities and expectations Evaluate and monitor protocols for the daily...InternshipLocal areaRelocation package$114.4k
...handle production changes while coordinating with the necessary stakeholders. Conduct research on new products related to fault monitoring. Build, test, and maintain network fault monitoring systems. Develop self-healing automation with engineering teams, write...Hourly payFull timeLive inWork at officeLocal areaFlexible hours3 days per week- ...SpecialistNovant Health is seeking an Cardiovascular Invasive Specialist for a procedural role to collaborate with other healthcare professionals... ...Lab and Interventional Cardiology with advanced skills in monitoring, managing, and caring for patients with severe cardiac...
Novant Health
Charlotte, NC1 day ago $1,000 per month
...veterinary technician! PetVet Care Centers' Carolina Veterinary Specialists - Matthews is currently seeking an ER Veterinary Technician... ...and manage staff schedules to support patient care needs Monitor workflow efficiency and minimize overtime Perform...Local areaRelocation packageMonday to FridayShift workRotating shiftWeekend workAfternoon shiftPetVet Care Centers
Matthews, NCa month ago- ...nature-based products. ~ Role Description The Operational Specialist is a part-time, on-site role based in Charlotte, NC. This... ...fulfillment to ensure efficiency and accuracy. The role includes monitoring workflows, tracking key operational metrics, and identifying...Full timePart timeWork at office
Nature Provides
Mint Hill, NC4 days ago - ...innovative organization.Job SummaryThe Principal Project Controls Specialist supports fixed-bid construction projects related to... ...including cost control and analysis, schedule development and monitoring, impact identification, earned value management, and oversight...Contract workFor contractorsWork experience placementWork at office
MasTec
Charlotte, NC5 days ago $38 - $57 per hour
...Description I'm hiring for a Cardiovascular & Electrophysiology Specialist in North Carolina! \n The Cardiovascular & Electrophysiology... ..., therapies, and procedures (including scrub assistant, monitoring and circulating) under the direction of the physician; administers...Permanent employmentFull timeRelocation packageShift workKA Recruiting Inc.
Charlotte, NC7 days ago- ...program or portfolio level. The Principal Project Controls Cost Specialist is a critical position as the expert/ lead the project... ...monthly status reports as provided by client and its vendors. # Monitoring compliance with department requirements and Enterprise Policy...Full timeContract workFor contractorsWork experience placement
Mindlance
Charlotte, NC4 days ago - ...Senior Medical Director, Clinical Development (Medical Monitoring) About the Company Rapidly growing organization in the oncology & hematology industry Industry Hospital & Health Care Type Privately Held About the Role The Company is seeking a Senior...
$105.4k - $207.8k
...and regulatory adherence across our platforms for our clients. Working closely with cross-functional stakeholders, this role will monitor regulatory changes, manage risks, and enhance our organization’s approach to content safety, user trust, and online integrity.Recruiting...Local areaVisa sponsorshipDeloitte
Charlotte, NC4 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Monitoring And Evaluation Specialist. Be the first to apply!
Related searches
- monitor tech Charlotte, NC
- quality assurance monitor Charlotte, NC
- monitor Charlotte, NC
- security monitor Charlotte, NC
- clinical research monitor Charlotte, NC
- program monitor Charlotte, NC
- pool monitor Charlotte, NC
- mathematics specialist Charlotte, NC
- configuration specialist Charlotte, NC
- enterprise solutions specialist Charlotte, NC