Senior Data Engineer

Key Responsibilities Lead pipeline architecture decisions for assigned workstreams, including batch and real-time ingestion, transformation, and delivery to analytics and AI layers. Design, build, and maintain scalable ETL/ELT workflows with strong data quality, lineage, monitoring, and observability built in from the start. Establish data contracts and pipeline standards for the projects you own, and ensure those standards are followed by other contributors on the workstream. Ensure data reliability, performance, and scalability across platforms; proactively surface bottlenecks and recommend improvements before they affect delivery. Support both batch and streaming ingestion patterns, including real-time data pipeline design and implementation. LLM Enablement & AI Data Foundations Design and implement data pipelines that support LLM and AI use cases, including: Document and unstructured data ingestion Data preprocessing, enrichment, and embedding generation Vector store integration and retrieval-optimized data structures Lead embedding pipeline architecture and vector store configuration in collaboration with developers building LLM features. Ensure data freshness, lineage, and governance for AI-powered systems. Optimize data structures and retrieval patterns to support efficient LLM context usage. Contribute to RAG pipeline design and maintain awareness of foundation model data requirements across active engagements. Cloud Architecture (AWS) Architect and provision AWS services to support data and AI workloads, including S3, Glue, Glue Catalog, Redshift, Athena, EMR, Kinesis, MSK, Lambda, Step Functions, and EventBridge. Lead FedRAMP-compliant architecture design for data environments where required; apply security and access control patterns using Lake Formation and IAM. Contribute to reusable reference architectures for data lakes, warehouses, streaming systems, and AI-ready platforms. Partner with platform and DevOps teams to ensure secure, cost-effective, and scalable cloud deployments. Apply infrastructure-as-code and automation practices to data platform provisioning and maintenance. Client & Stakeholder Engagement Participate in client discovery and requirements-gathering sessions, translating operational needs into concrete data architecture recommendations. Communicate clearly with both technical and non-technical stakeholders, adapting depth and language to the audience without losing precision. Assess and document client data readiness for analytics and AI adoption; identify gaps and propose remediation paths to the lead architect or engagement manager. Support the technical narrative during delivery, contributing to solution design documents, architecture diagrams, and client-facing documentation. Build working trust with client counterparts through consistency, follow-through, and clear expectation-setting. Technical Delivery Leadership Own the data engineering workstream within a project delivery plan, including task decomposition, estimation, and sequencing in Jira. Understand how individual tickets connect to the larger delivery arc, and surface dependencies or risks before they block progress. Facilitate technical planning and review ceremonies for the data workstream; provide clear updates to project managers and technical leads. Partner with project managers and product owners to keep the data engineering track aligned with contractual and delivery constraints. Coordinate across engineering, platform, analytics, and ML teams to ensure data pipelines meet downstream requirements. Product & Data Readiness Support Support the evolving data architecture behind our product capabilities, including predictive and real-time ML systems. Assess and improve internal and client data readiness for analytics and AI adoption. Translate business, product, and client needs into scalable data architectures that can be maintained and extended by the broader team. Document data architectures, pipeline designs, and integration patterns to support transparency and reuse across engagements. Staff Development & Knowledge Sharing Mentor junior and core-level data engineers through code reviews, architecture critiques, and hands-on guidance on active projects. Identify skill gaps in team members and work with technical leadership to address them through structured coaching or pairing. Contribute to internal playbooks, onboarding materials, and engineering standards that reduce tribal knowledge and improve team consistency. Help team members understand the business context behind their work, connecting individual tasks to client outcomes and our technical strategy. Collaboration & Communication Collaborate closely with developers building LLM features to ensure data pipelines meet AI requirements. Work with product, analytics, and technical leadership to align data strategy with organizational and project goals. Communicate findings, risks, and architectural decisions clearly in both written and verbal form, including client-facing documentation. Contribute to our proposal efforts and technical volume sections as a subject matter contributor. #J-18808-Ljbffr