Lead Data Engineer

About The Company Schwarz Digits is a leading technology company dedicated to creating the technological foundation for digital decision-making across Europe. As the IT and Digital division of the Schwarz Group, we develop and manage IT infrastructures for retail sectors such as Lidl and Kaufland, as well as Schwarz Produktion and PreZero. Additionally, we operate as an independent market provider, assisting companies throughout Europe in their digital transformation journeys. Our core competencies encompass Cloud Computing, Cyber Security, Data & AI, Communication, and Workspace solutions. At Schwarz Digits, we foster innovation, agility, and security, enabling our team members to contribute to shaping the future of digital commerce and enterprise technology. Join us to be part of a dynamic environment where your expertise drives impactful change and supports the digital independence of businesses across Europe. About The Role We are seeking a highly skilled Data Engineer to join our team at Schwarz Digits. In this role, you will be responsible for designing, developing, and maintaining scalable data pipelines and infrastructure that support analytics and machine learning initiatives. Your expertise will help ensure data integrity, security, and accessibility across various platforms and teams. You will work at the intersection of modern data engineering practices and cloud-native technologies, primarily focusing on Google Cloud Platform (GCP). Your contributions will directly influence how data is harnessed to enable smarter business decisions and innovative solutions. The ideal candidate will have a strong background in Python development, experience with batch and streaming ETL pipelines, and a solid understanding of modern data lakehouse architectures. You will collaborate closely with data scientists, product managers, and IT teams to deliver high-quality data solutions that meet evolving business needs. Qualifications Minimum of four years of professional experience in Data or Software Engineering. Extensive knowledge of Python development, emphasizing clean code principles and software best practices. Practical experience with Java Runtime Environment (JRE) and software testing methodologies. Hands-on experience in developing batch and streaming ETL pipelines using frameworks such as Apache Spark or similar technologies. Deep understanding of schema evolution, data lakehouse storage structures, and modern open-table formats like Apache Iceberg or Delta Lake. Proficiency in orchestrating complex workflows with Apache Airflow. Experience with Apache Kafka for real-time messaging and event processing. Knowledge of containerized application deployment within cloud environments, including Docker and Kubernetes, preferably on Google Cloud Platform. Familiarity with DevOps practices, including version control with Git, automated testing, and CI/CD pipeline implementation. Strong attention to detail and a proactive approach to ensuring data integrity and quality. Ability to work effectively in agile team environments, demonstrating collaboration and communication skills. Responsibilities Design, develop, and optimize scalable ETL and ELT data pipelines to support analytics and machine learning workloads on Google Cloud Platform. Build and maintain data infrastructure utilizing open-table formats such as Iceberg and Delta Lake, ensuring data governance, access control (RLS/CLS), and catalog management (Unity Catalog). Implement and manage streaming data solutions and event-driven architectures using Apache Kafka and Apache Spark. Write high-quality, maintainable, and well-tested code, employing best practices in software design and testing (unit, integration, and end-to-end tests). Maintain and monitor data workflows and pipelines via Apache Airflow, ensuring smooth operation and timely data delivery. Deploy and manage containerized applications using Docker and Kubernetes, with a focus on cloud-native deployment strategies. Develop and maintain CI/CD pipelines to automate deployment processes and ensure continuous integration and delivery. Collaborate closely with cross-functional teams including data scientists, product owners, and business stakeholders to translate requirements into robust data solutions. Monitor system health, troubleshoot issues proactively, and implement improvements to prevent system outages or data inconsistencies. Benefits At Schwarz Digits, we offer a comprehensive benefits package designed to support our employees' well-being and professional growth. You will have access to flexible working hours and remote work options, fostering a healthy work-life balance. We invest in your development through ongoing training, certifications, and opportunities to attend industry conferences. Our competitive salary packages are complemented by performance-based bonuses and comprehensive health insurance coverage. Additionally, you will be part of a vibrant, innovative environment that encourages knowledge sharing and continuous learning. We also provide modern workspaces equipped with the latest technology to facilitate productivity and collaboration. Join us to benefit from a supportive company culture that values diversity, inclusion, and your individual contributions. Equal Opportunity Schwarz Digits is an equal opportunity employer committed to fostering an inclusive environment for all employees. We believe in diversity and are dedicated to providing equal employment opportunities regardless of race, gender, age, religion, sexual orientation, disability, or any other characteristic protected by law. We promote a workplace where everyone can thrive, contribute their unique perspectives, and grow professionally. Our commitment to diversity and inclusion is integral to our success and innovation as a company. #J-18808-Ljbffr AgileGrid Solutions