Senior Data Engineer

Salary: $182,000 - 182,000 per year Requirements:

• We accept a masters degree in Computer Science, Information Technology, Engineering, or a related field, plus two years of experience in the position offered or in another occupation where the required experience was gained.
• Two years of experience building PySpark ETL workflows on AWS EMR and Glue, reading from and writing to S3, and using partitioning, predicate pushdown, and broadcast joins to deliver scheduled analytics-ready fact and dimension tables.
• Two years of experience authoring and tuning complex SQL on Greenplum or Postgres, including window functions, ROW_NUMBER-based deduplication, distribution keys, AO/AOCO compression, and VACUUM/ANALYZE.
• Two years of experience managing the AWS Glue Data Catalog, querying curated data layers with Athena, and designing S3 prefix, partition, and lifecycle strategies to reduce query time and storage costs.
• Two years of experience implementing CI/CD with CodeCommit, CodeBuild, and CodePipeline, parameterizing environments, and using CloudFormation for infrastructure as code and repeatable, auditable deployments with fast rollbacks.
• Two years of experience building observability with CloudWatch logs, metrics, alarms, and job health checks, and creating SLA dashboards and alerts to detect failures and reduce mean time to recovery.
• Two years of experience enforcing data quality with SQL and Python unit tests, reconciliation checks for row counts, sums, and duplicates, referential-integrity checks, and documented user acceptance testing evidence for change control.
• Two years of experience applying data security and governance controls, including normalizing and encrypting sensitive identifiers, using KMS-managed keys, enforcing least-privilege IAM, and maintaining audit logging.
• Two years of experience integrating LLMs, including AWS Bedrock, with PySpark and SQL to suggest transformations, generate test cases, and summarize validation results.
• Two years of experience developing internal data applications with AWS Amplify and DynamoDB or Athena to manage lookup keys or brand lists and provide stakeholders with controlled self-service access.

• We develop and execute software tests to identify problems and determine their causes.
• We write and optimize complex SQL scripts and build Python scripts for large-scale transformations on Greenplum.
• We create test plans, perform thorough unit and integration testing, and provide reproducible, well-documented results.
• We design and maintain AI-enabled PySpark data pipelines on AWS, primarily using EMR, Glue, Athena, DynamoDB, and S3, to process large datasets at scale.
• We improve and tune jobs for performance, cost, and compliance, and manage data partitioning, cataloging, and lineage.
• We automate deployments and infrastructure with CloudFormation and parameterized configurations, and schedule batch workloads to keep pipelines timely and reliable.
• We build internal dashboards and data products, such as Amplify applications, and set up monitoring and alerts to identify issues early.
• We handle client tickets and translate ambiguous requests into clear technical tasks.

• AI
• AWS
• AWS Glue
• CI/CD
• CloudWatch
• ETL
• IAM
• Python
• PySpark
• REST
• SQL
• Security
• Cloud

More:

We offer a role based in our Chicago, Illinois office, with attendance required on certain days each week and remote work permitted for the rest of the week. The position focuses on software testing, large-scale data engineering, and the development of reliable, governed data products.

last updated 42 week of 2026