Data Engineer

Job Description

Insight Global is seeking a Data Engineer to join the Digital Data Platform team at one of our largest agriculture clients. This team is responsible for building and supporting the enterprise data platform that enables more than a dozen data product teams across the organization. The environment consists of a large-scale AWS ecosystem supporting data ingestion, transformation, governance, and analytics.

This individual will be responsible for designing, developing, and optimizing enterprise data pipelines using AWS Glue, PySpark, Apache Iceberg, Redshift, and AWS-native data services. They will own the full lifecycle of data product delivery, from source ingestion through transformation, validation, governance, and analytics consumption. The ideal candidate has strong experience building scalable cloud-based data solutions, implementing data quality frameworks, managing data warehouse integrations, and applying data modeling best practices.

Day-to-day responsibilities include developing ETL/ELT pipelines, managing Apache Iceberg tables, optimizing complex SQL workloads, implementing governance controls through Lake Formation, troubleshooting production issues, onboarding new data sources, and collaborating with business and technical stakeholders to deliver reliable, production-ready data products.

Key Responsibilities

• Design, build, and optimize ETL/ELT data pipelines using AWS Glue (PySpark) and Apache Iceberg

• Develop and maintain enterprise data products from ingestion through consumption-ready analytics datasets

• Build and support Amazon Redshift integrations, including schema management, stored procedures, materialized views, and database migrations

• Create and optimize complex SQL transformations, reporting views, and data quality validations

• Design dimensional data models including star schemas, snowflake schemas, and slowly changing dimensions

• Manage Apache Iceberg table lifecycle activities including schema evolution, partition evolution, compaction, and file cleanup

• Implement and maintain data governance controls through AWS Lake Formation, LF-TBAC permissions, and PII classification standards

• Build automated data quality monitoring and validation frameworks

• Partner with data product teams to onboard new data sources and establish data contracts

• Troubleshoot data pipeline failures and resolve issues related to Glue jobs, DMS replication, and data freshness

• Execute and validate production deployments while ensuring data accuracy and platform reliability

• Participate in CI/CD processes and support continuous improvement of the enterprise data platform

The exact pay rate for this position ranges from $60-$70 an hour based on several factors including location, experience-level, etc. Health, dental, vision and a 401k option is included in this role.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

• 5+ years of hands-on Data Engineering experience building enterprise ETL/ELT solutions

• Expert-level SQL skills including complex joins, window functions, CTEs, query optimization, and performance tuning

• Strong Python development experience

• Experience developing and optimizing PySpark workloads

• Hands-on experience with AWS Glue, including job authoring, crawlers, bookmarks, and large-scale data processing

• Experience with Amazon Redshift including schema design, stored procedures, external schemas, materialized views, and performance optimization

• Experience with Apache Iceberg or similar open table formats

• Strong understanding of dimensional data modeling and analytics-focused database design

• Experience with AWS data services including:

o Athena

o Lake Formation

o S3

o DMS

o Lambda

o EventBridge

• Experience implementing data governance and access controls

• Experience using GitHub and CI/CD deployment processes

• Strong data quality mindset with a focus on accuracy, completeness, and reliability

• Ability to independently own projects and deliver production-ready solutions • Experience with Data Mesh architecture principles

• Experience with Lakehouse architecture

• Experience with AWS DMS and CDC ingestion patterns

• Terraform or Infrastructure-as-Code experience

• Experience with AWS DataZone, ER/Studio, or similar data catalog/lineage tools

• Knowledge of AI-assisted development tools such as GitHub Copilot or Amazon Kiro

• Experience supporting SageMaker or machine learning platforms

• Docker and containerization experience

• Agriculture, retail, or agribusiness industry experience