Senior AI Software Engineer

Public Trust: None

Requisition Type: Pipeline

Your Impact

Own your opportunity to work alongside federal civilian agencies. Make an impact by delivering solutions that improve citizen services millions depends on.

Job Description

GDIT is seeking a highly motivated and experienced Senior AI Software Engineer to support the development, modernization, integration, and sustainment of our on-premises, AI-powered enterprise chatbot platform supporting the U.S. Department of State in Sterling, VA.

The selected candidate will design, develop, and integrate software solutions leveraging Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), vector databases, document processing technologies, and modern AI inference frameworks.

This role requires a strong foundation in software engineering, Python development, API integration, containerization, and AI application architecture. The engineer will help transition existing AI services toward a scalable, containerized architecture while introducing new capabilities that enhance system performance, reliability, security, and user experience.

The ideal candidate is a hands-on software engineer who can work across the AI application stack, from backend development and model serving integration to data processing, automated testing, and deployment.

The individual will collaborate with AI/ML engineers, data engineers, infrastructure specialists, cybersecurity teams, and government stakeholders to deliver secure, reliable AI capabilities in highly controlled, classified, and air-gapped environments.

Key Responsibilities

1. AI Application Development & Integration

• Design, develop, test, and maintain Python-based applications, microservices, and APIs supporting an on prem chatbot.
• Extend, configure, and integrate OpenWebUI with enterprise AI services and backend systems.
• Develop custom OpenWebUI Functions, Tools, and integrations to support mission-specific use cases.
• Integrate approved LLMs into chatbot using OpenAI-compatible APIs and inference frameworks such as vLLM and Ollama.
• Develop reusable software components for AI model access, prompt management, response processing, and application workflows.
• Collaborate with AI/ML engineers to improve model inference performance, response quality, and application reliability.
• Support multimodal AI use cases, including document understanding, optical character recognition, and vision-language model integration.
• Develop backend capabilities to support secure enterprise AI workflows.

2. Retrieval-Augmented Generation (RAG) & Document Processing

• Design, implement, and optimize RAG workflows for enterprise document search and question answering.
• Integrate document processing technologies such as Docling, Tesseract, and vision-language models into automated ingestion pipelines.
• Develop document chunking, metadata extraction, embedding generation, and retrieval workflows.
• Integrate embedding models and reranking models to improve retrieval accuracy and relevance.
• Work with PostgreSQL and pgvector to support vector search, semantic retrieval, and document indexing.
• Develop APIs and background processing services for document ingestion and knowledge base management.
• Collaborate with data engineers to ensure accurate storage, indexing, and retrieval of enterprise information.
• Implement testing and evaluation approaches to measure RAG accuracy, retrieval relevance, and answer quality.

3. AI Model Serving & Inference Integration

• Integrate and maintain AI inference services using vLLM, Ollama, and compatible model-serving frameworks.
• Support the deployment and configuration of approved open-weight LLMs, embedding models, rerankers, and vision-language models.
• Develop software integrations using OpenAI-compatible REST APIs.
• Assist in optimizing inference configurations, including context length, GPU memory utilization, concurrency, and model serving performance.
• Support the transition from existing inference frameworks to improved architectures as technologies mature.
• Collaborate with AI/ML engineers to troubleshoot inference errors, model loading issues, and integration failures.
• Develop benchmarking and performance-testing utilities for AI workloads.

4. Containerization & Platform Modernization

• Support the migration of DS Chat components from traditional deployments into containerized services.
• Develop, configure, and maintain container images using Podman or Docker.
• Create and maintain Podman Compose configurations defining application services, networking, volumes, and environment variables.
• Integrate services such as OpenWebUI, vLLM, Docling, PostgreSQL, pgvector, PgBouncer, and Redis.
• Develop scripts and automation for repeatable installation, deployment, upgrades, and recovery.
• Implement health checks, readiness checks, logging, and service dependency management.
• Troubleshoot container networking, authentication, storage, GPU access, and service connectivity.
• Collaborate with infrastructure and cybersecurity teams to improve deployment consistency and security.

5. Agentic AI & Future Capabilities

• Research, prototype, and develop agentic AI capabilities for future DS Chat releases.
• Implement controlled tool-calling and multi-step AI workflows.
• Develop integrations that allow approved AI models to interact with authorized enterprise APIs and services.
• Explore emerging technologies involving AI orchestration, workflow automation, and Model Context Protocol (MCP).
• Design secure execution boundaries, permissions, audit logging, and approval mechanisms for AI-driven actions.
• Collaborate with stakeholders to evaluate new AI capabilities for mission relevance, security, and operational feasibility.
• Implement safeguards to support human oversight and prevent unauthorized actions by AI-enabled systems.

6. Software Engineering, Testing & DevSecOps

• Apply software engineering best practices, including modular architecture, code reviews, documentation, and version control.
• Develop automated unit, integration, regression, and performance tests using frameworks such as pytest.
• Build integration tests validating communication between OpenWebUI, inference services, document processing components, and databases.
• Support CI/CD processes appropriate for disconnected and classified environments.
• Troubleshoot production issues through application logs, metrics, traces, and system diagnostics.
• Integrate operational logging and monitoring capabilities with tools such as Splunk.
• Assist with vulnerability remediation, dependency management, software supply chain reviews, and container security assessments.
• Produce technical documentation, deployment procedures, architecture diagrams, and operational runbooks.
• Participate in Agile development activities, including sprint planning, technical design reviews, and demonstrations.

7. Security & Classified Environment Support

• Design software solutions suitable for secure, on-premises, air-gapped government environments.
• Implement secure authentication integration, role-based access controls, API authorization, and data protection measures.
• Ensure AI services communicate only through authorized internal network paths.
• Support software packaging and offline deployment using preapproved dependencies, container images, and model artifacts.
• Collaborate with cybersecurity personnel to address security findings and support Risk Management Framework (RMF) activities.
• Assist with technical documentation and evidence required for TRB/CCB reviews and Authority to Operate (ATO) processes.
• Follow approved procedures for handling sensitive and classified information.

Required Qualifications

• Minimum Secret Clearance, no sponsorship
• Bachelor's degree in Computer Science, Software Engineering, Artificial Intelligence, Computer Engineering, or a related technical field, or equivalent practical experience.
• 5+ years of professional software engineering experience, including backend application development.
• Strong proficiency in Python and experience developing production-quality software.
• Experience developing RESTful APIs and integrating distributed application services.
• Hands-on experience integrating LLMs or generative AI capabilities into software applications.
• Understanding of transformer-based models, inference APIs, prompt engineering, and LLM application development.
• Experience with RAG concepts, including document ingestion, embeddings, vector search, and retrieval pipelines.
• Experience working with relational databases such as PostgreSQL.
• Familiarity with container technologies such as Podman or Docker.
• Experience with Linux-based development and application deployment.
• Proficiency with Git, automated testing, debugging, and software development best practices.
• Experience troubleshooting applications across multiple interconnected services.
• Strong analytical, problem-solving, communication, and technical documentation skills.
• Ability to work effectively with cross-functional teams and government stakeholders.

Preferred Qualifications

• Experience with OpenWebUI, including configuration, integration, custom Functions, Tools, or application extensions.
• Hands-on experience with vLLM, Ollama, Hugging Face Transformers, or other LLM inference technologies.
• Experience serving or integrating open-weight LLMs in on-premises environments.
• Experience with NVIDIA GPUs, CUDA, GPU memory management, and AI inference optimization.
• Experience with PostgreSQL pgvector, vector similarity search, and semantic retrieval.
• Familiarity with embedding models, rerankers, and cross-encoder architectures.
• Experience with Docling, Tesseract, or document intelligence technologies.
• Experience implementing production RAG pipelines and evaluation frameworks.
• Familiarity with agentic AI frameworks, tool calling, and Model Context Protocol (MCP).
• Experience developing or maintaining containerized microservice architectures.
• Familiarity with Redis, PgBouncer, and asynchronous background processing.
• Experience with Red Hat Enterprise Linux and Windows Server.
• Experience with FastAPI, Pydantic, pytest, and related Python technologies.
• Experience using Splunk for application monitoring and troubleshooting.
• Familiarity with software supply chain security, vulnerability scanning, and container hardening.
• Experience supporting federal government systems, classified networks, or air-gapped deployments.
• Knowledge of federal cybersecurity requirements, RMF, and ATO processes.

Work Requirements

Years of Experience

7 + years of related experience

* may vary based on technical training, certification(s), or degree

Certification

Travel Required

Less than 10%

Citizenship

U.S. Citizenship Required