Senior Data Engineer

Senior Data Engineer

Position Overview: The team is building a new enterprise data orchestration layer to centralize and standardize how the organization receives, processes, and moves data between systems. The architecture is new and barely in production, so they want engineers who can help establish modern, standardized patterns rather than simply maintain an established platform.

• Build and maintain data pipelines within the new orchestration/data platform.
• Work heavily with cloud-native data technologies.
• Help modernize existing batch-oriented data processes.
• Participate in architecture/design discussions, troubleshooting, and potentially DevOps.
• Take ownership of areas beyond strictly assigned development work.

Top Skills:

• AWS (Lambda, Glue, EMR, S3, DynamoDB, SNS/SQS, EventBridge)
• Databricks
• Spark / PySpark
• SQL
• Python

Nice to Have Skills:

• Terraform

Job Profile Summary: The Senior Data Engineer is responsible for the design, development, delivery, and support of enterprise data products, data integration solutions, and cloud-native data platforms. The role works across structured and unstructured data, enabling analytics, operational reporting, interoperability, and digital business capabilities through scalable and reliable data engineering solutions. This position focuses on the design and implementation of modern data pipelines, cloud-native integration patterns, event-driven architectures, and data transformation capabilities. The Senior Data Engineer serves as a technical leader within delivery teams and collaborates closely with architects, application engineers, product owners, analysts, and business stakeholders to deliver secure, governed, and scalable data solutions. The ideal candidate brings strong experience in Databricks, cloud data platforms, distributed processing frameworks, and enterprise integration patterns, with the ability to design and implement solutions that support both operational and analytical use cases.

• Design, develop, deploy, and support scalable data pipelines, integration solutions, and data products supporting enterprise business capabilities.
• Design and implement cloud-native data processing solutions leveraging AWS services, Databricks, and distributed data processing frameworks.
• Develop and support batch, API, file-based, and event-driven integration patterns that enable secure and reliable data exchange across enterprise and partner ecosystems.
• Lead technical design activities for data engineering solutions, including data models, transformation frameworks, ingestion patterns, and consumption layers.
• Develop/optimize complex data transformation logic using SQL, Spark, Python, and related technologies.
• Build and support data ingestion, enrichment, quality, lineage, and observability capabilities that improve trust, transparency, and operational supportability.
• Implement and enhance CI/CD pipelines, infrastructure automation, testing strategies, and DevOps practices for data platforms and data products.
• Partner with business stakeholders, application teams, and architects to translate business requirements into scalable technical solutions.
• Ensure data solutions meet enterprise standards for security, governance, reliability, performance, and maintainability.
• Troubleshoot complex data processing and integration issues while driving continuous improvement in platform performance, resiliency, and operational efficiency.
• Mentor less experienced engineers and contribute to engineering standards, reusable patterns, and best practices.

Additional Responsibilities:

• Participate in technology evaluations, proof-of-concept initiatives, and architecture reviews.
• Contribute to the development of reusable integration, transformation, and data platform patterns.
• Share technical expertise through mentoring, documentation, and knowledge-sharing activities.
• Stay current on emerging cloud, data engineering, AI, and interoperability technologies.

Required Skills and Experience:

• 5+ years of related professional experience. All relevant experience including work, education, transferable skills and military experience will be considered.
• Direct experience in data engineering, data integration, or data platform development.
• Essential Candidate Qualities: Creative thinking, strong communication skills, curiosity, self-motivated, time management skills.
• High school diploma (or equivalency) and legal authorization to work in the U.S.

Preferred Technical Experience Data Engineering & Analytics:

• Databricks
• Apache Spark / Spark SQL
• Delta Lake
• Python
• SQL
• Data modeling and schema design
• Data quality, lineage, and observability
• Experience working with operational and analytical data platforms
• Data modeling and performance optimization are a plus

Cloud & DevOps:

• AWS (Lambda, Glue, EMR, S3, DynamoDB, SNS/SQS, EventBridge)
• Infrastructure as Code (Terraform preferred)
• CI/CD pipelines using GitLab, GitHub Actions, Jenkins, or similar tools

AI and Emerging Technologies (Nice to have):

• Exposure to AI-assisted development tools such as Microsoft Copilot, GitHub Copilot, Claude, etc
• Interest in applying AI capabilities to software development, automation, testing, and operational processes.

Ideal Candidate Profile:

• Comfortable working across data engineering, integration engineering, cloud development, and platform enablement.
• Demonstrated experience with Databricks and Cloud-based data platforms.
• Experience designing and implementing enterprise-scale data pipelines, integration solutions, and data transformation processes.
• Experience designing and implementing modern cloud data platforms utilizing Databricks, Apache Spark, Delta Lake, or similar technologies.
• Strong SQL skills with experience developing and optimizing complex transformation and analytical workloads.
• Experience working with cloud-native platforms and distributed data processing technologies.
• Able to lead technical workstreams and influence engineering direction within a team.
• Effective mentor and technical partner across engineering, architecture, and business stakeholders.
• Strong collaboration, problem-solving, and communication skills.
• Comfortable operating in agile product teams and cross-functional environments.