Data Engineer

Data EngineerSoftware Guidance & Assistance, Inc., (SGA), is searching for a Data Engineer for a contract assignment with one of our premier regulatory clients in Rockville, MD or Tysons, VA. The Data Engineer works with moderate supervision across two equally weighted domains: (1) large-scale data pipeline development processing market events in a cloud environment, and (2) design and development of agentic AI systems including LLM-powered regulatory data assistants, MCP servers, and agent harness architectures. This position contributes to overall product quality throughout the software development lifecycle. Responsibilities include:Build and maintain ETL/ELT pipelines using Apache Spark, Hive, and Trino across S3-based data lake environmentsDevelop and optimize SQL for large-scale surveillance datasets including window functions, multi-table joins, and complex aggregationsBuild and engineer big data systems (EMR-on-EC2, EMR-on-EKS) and develop solutions on analytical platforms (SageMaker, Domino, Dataiku)Participate in data quality monitoring, anomaly detection, and production incident investigationDevelop AI agent systems using AWS Bedrock and agent frameworks (Strands Agents SDK, LangChain/LangGraph, or equivalent)Build agent harness architectures combining LLM reasoning with deterministic execution - skill/RAG-based SQL generation and structured output validationImplement agent memory, context management, and tool integration (MCP servers, API connectors, data catalog lookups) across the data lakeBuild evaluation frameworks for agent accuracy - paraphrase robustness, routing precision, and structural consistencyStay informed of advances in LLM frameworks (LangGraph, Google ADK, AWS Strands) and emerging AI capabilitiesWrite clean, well-tested code; contribute to CI/CD Jenkins pipelines and infrastructure-as-code on AWSEnsure secure handling of RCI and sensitive regulatory data across both data pipelines and agent outputs - auditable execution tracesAdhere to FINRA and team standards for secure development practices and technology policiesPartner across teams, communicate technical information at the appropriate level, and maintain documentation on Confluence/WikiActively learn from senior team members; contribute to process improvement in line with FINRA's values of collaboration, expertise, innovation, and responsibilityRequired Skills:Data Engineering & Big Data TechnologiesExperience building data pipelines using Apache Spark (PySpark preferred) and SQLExperience with SQL query engines (Hive, Trino/Presto, or similar) and cloud data platforms (AWS S3, EMR, Lambda)Real-world experience with debugging and mitigation strategiesGenerative AI & Agentic SystemsPractical experience building LLM-powered agent systems that use tools and produce structured outputs (not just chatbot interfaces)Hands-on experience with at least one agent framework: LangChain, LangGraph, AWS Strands, or equivalentWorking knowledge of prompt engineering, RAG architectures, and context/memory managementExperience with foundation model APIs (Anthropic Claude, Amazon Nova, OpenAI, or similar)Memory Architecture: Understanding of agent memory tiers - working memory, episodic memory, semantic memory - and strategies for context persistence, pruning, and retrieval across sessionsAgent Harness Design: Familiarity with harness patterns that wrap LLM reasoning with deterministic guardrails, tool routing, verification loops, and graceful degradationAI Tool ProficiencyHands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.)Experience with spec-driven development - using structured specifications to guide AI code generation, review, and validationAbility to leverage AI pair programming for code suggestions, debugging, refactoring, and automated test generationCloud TechnologiesExperience with AWS services like S3, EMR, EMR on EKS, Lambda, Bedrock, Step Functions, etc.Hands-on experience using S3 with Spark (e.g., dealing with file formats, consistency issues)Familiarity with AWS Bedrock for foundation model invocation, knowledge bases, guardrails, and agent orchestrationExposure to Google Cloud Vertex AI (model garden, grounding, agent builder) or equivalent managed AI platformsFamiliarity with AWS monitoring and logging tools (CloudWatch, CloudTrail) for production workloadsProgramming - PythonProficiency in Python for data engineering and automationAbility to write clean, modular, and performant codeExperience with functional programming concepts (e.g., immutability, higher-order functions)Strong understanding of collections, concurrency, and memory managementSQL Skills (Window Functions, Joins, Complex Queries)Proficiency with SQL window functions, multi-table joins, and aggregationsAbility to write and optimize complex SQL queriesExperience handling edge cases like NULLs, duplicates, and orderingGood to HaveAWS Bedrock AgentCore (memory, identity, tool gateway)Model Context Protocol (MCP) server development and integrationAgent evaluation harnesses and agentic patterns (draft-verification, compile-style generation)Fine-tuning foundation models for domain-specific tasks (LoRA, PEFT, or managed fine-tuning via Bedrock/Vertex AI)Local model execution with Ollama, vLLM, or similar for development and experimentationVector databases (FAISS, Pinecone, OpenSearch)Docker, Kubernetes, and Amazon EKS for containerized workloadsInfrastructure as Code (Terraform, CloudFormation)Experience with CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, ArgoCD)Experience with monitoring and observability tools (Prometheus, Grafana, ELK stack)AWS certifications (AI Practitioner, Solutions Architect, or Kubernetes certifications like CKA/CKAD)Education / Experience RequirementsBachelor's degree in Computer Science, Data Science, Information Systems, or related discipline with at least two (2) years of related experience; or equivalent training and/or work experience; past Financial Services industry experience preferredDemonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutionsExtensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworksStrong written and verbal technical communication skillsDemonstrated ability to develop effective working relationships that improved the quality of work productsAbility to maintain focus and develop proficiency in new skills rapidlyAbility to work in a fast paced environmentSGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work.