Senior Data Engineer

Job Description

Insight Global is looking to add a 2 Data Engineers to our client's team who are headquartered in the St. Louis area but can sit remotely. They will be responsible for moving applications from one environment to another in addition to ensuring every component has been upgraded and will also be responsible for modernizing the framework as well. They will need someone to come in and support development efforts across the enterprise and will interact with multiple different application owners & software vendors so communication is very important. They will be using Apache Spark, Apache Iceberg, and Apache Airflow for ETL pipelines. This person will    transform and integrate EBCDIC Mainframe data into Hive and Impala tables using Precisely Connect for Big Data, optimize data transformation processes for performance, scalability, and reliability, and ensure data consistency, accuracy, and quality across the ETL pipelines. This person will function as the point of escalation for ETL loads, data quality issues, and production support issues, monitor ETL workflows and troubleshoot issues to ensure smooth production operations and implement proactive monitoring, alerting, and reporting solutions to improve pipeline reliability. This individual will be responsible for supporting and rebuilding legacy ETL jobs (currently not using ACID transactions) with modern solutions using Apache Spark and Apache Iceberg to support ACID transactions. They will be overseeing design, and implement workflows for automating data pipelines using Apache Airflow as well as establishing and enforcing best practices for ETL code development, version control, and deployment using Azure DevOps.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

5+ years of Data Engineering experience

Experience with Apache Spark, Airflow or Iceberg

Python programming experience

Production Support experience

Linux experience

Strong communication skills Experience with Precisely Connect

Experience with Hadoop

Experience with Cloudera