Principal Software Engineer - Insights & Intelligence

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Principal Software Engineer - Insights & Intelligence based in United States. This role sits at the intersection of software engineering, data engineering, DevOps, and enterprise analytics. You will help build and operate secure, reliable data infrastructure that supports critical business decisions. The position focuses on making production data pipelines highly available, observable, maintainable, and resilient. You will develop scalable data integrations and automated deployment workflows across modern cloud and data platforms. Working alongside senior engineers and architects, you will turn technical designs into production-grade data services. You will also help strengthen engineering standards around testing, monitoring, security, and operational excellence. This is an opportunity to make a broad impact on an organization’s data, analytics, and AI transformation. Accountabilities - Design, implement, and maintain monitoring, alerting, and logging capabilities for production data pipelines, with a focus on availability, reliability, and operational visibility. - Build and maintain CI/CD pipelines for data applications and infrastructure using Git, GitHub Actions, GitLab CI/CD, Databricks Asset Bundles, or comparable technologies. - Enable rapid, repeatable, controlled deployments across development, testing, and production environments. - Proactively identify, diagnose, and resolve workflow failures, performance bottlenecks, data-quality issues, infrastructure incidents, and other production challenges. - Apply software engineering and DevOps practices to improve platform resilience, maintainability, scalability, and supportability.

- Design, develop, and optimize scalable data integration solutions that ingest, transform, and deliver information from diverse enterprise source systems. - Establish high standards for code quality, architecture, documentation, automated testing, and production support across data pipelines and integration components. - Partner with senior data engineers, platform engineers, and architects to translate technical designs into stable, secure, and well-engineered data services. - Contribute to observability, incident response, performance optimization, and reliability practices for enterprise-scale data platforms. - Help strengthen automation, infrastructure management, deployment processes, and operational standards across the data engineering environment. Requirements - Bachelor’s degree in Computer Science, Software Engineering, or a related technical discipline with at least 5 years of relevant professional experience, or a Master’s degree with at least 3 years of relevant experience. - Demonstrated professional experience in software development, DevOps, platform engineering, data engineering, or a closely related field. - Advanced proficiency in Python, including the ability to develop robust, maintainable, reusable, and well-tested production code. - Strong experience developing CI/CD workflows using GitHub Actions, GitLab CI/CD, or comparable automation platforms. - Demonstrated understanding of CI/CD practices for software systems, data platforms, or infrastructure. - Strong knowledge of source control, automated testing, deployment automation, production support, and software engineering best practices. - Proven ability to troubleshoot complex production environments, including workflow failures, data-quality problems, performance bottlenecks, and infrastructure incidents. - Experience with infrastructure as code in a major cloud environment using Terraform, REST APIs, or comparable technologies is preferred. - Hands-on experience with Databricks, Apache Spark, and AWS services such as Amazon S3, AWS Batch, Amazon MSK, or Apache Kafka is highly desirable.