Data Engineer II

Job Description

About the Role:

This role is centered on modernizing customer journey and platform reliability analytics by converting legacy Redshift and sandbox work into governed, production-grade Databricks data products. The strongest candidate will be highly autonomous, technically deep in PySpark and SQL optimization, and capable of leaving behind maintainable assets for broader engineering teams.

Day to day:

The Senior Analytics Engineer will work closely with the core analytics team to assess existing Redshift queries, sandbox datasets, clickstream events, and transactional log sources tied to customer digital journeys and platform reliability. They will reverse-engineer complex legacy logic, identify performance and structural issues, and rebuild that logic as scalable PySpark pipelines and governed analytics data products in Databricks. The engineer will also standardize schemas, documentation, naming, and output structures so that the resulting assets are reliable, efficient, and ready for long-term ownership by engineering and offshore teams. The role is designed for someone who can become productive quickly, operate independently, and materially improve data quality, compute efficiency, maintainability, and downstream analytics usability.

Responsibilities:

• Pull and analyze raw data from Amazon Redshift and leverage Databricks/PySpark to transform and optimize datasets for downstream analytics consumption.

• Refactor and modernize existing Redshift and sandbox-based data logic into scalable, maintainable PySpark pipelines.

• Evaluate existing SQL queries and transformation logic, identify inefficiencies, and implement performance improvements to increase analytics usability and processing efficiency.

• Work with large-scale datasets (hundreds of millions of rows) using both snapshot and incremental processing patterns.

• Build, maintain, and optimize production-grade analytics data products that support customer experience, operational reporting, and platform reliability initiatives.

• Partner with the analytics team to streamline analysis of checkout experience and operational datasets, helping improve visibility into customer journey issues and system performance.

• Standardize data structures, schemas, metadata, and governance practices to support long-term maintainability and engineering ownership.

• Audit unfamiliar datasets and existing codebases, reverse-engineer business logic, and recommend scalable solutions with minimal onboarding.

• Create clean, well-documented data assets and technical solutions that can be successfully handed off to engineering and offshore support teams.

Communicate technical findings, optimization opportunities, and architectural recommendations to business and technical stakeholders.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

Must Haves:

• 5+ years of hands-on experience designing, optimizing, and maintaining large-scale data processing workloads using PySpark and Databricks

• Expert-level PySpark, including DataFrames, Structured Streaming, and Delta Lake

• Advanced SQL expertise with complex, high-volume queries in AWS Redshift or PostgreSQL environments

• Experience reverse-engineering and refactoring legacy SQL/PySpark logic into optimized, modular, scalable pipelines

• Ability to architect reliable, production-grade analytics data products with schema enforcement, metadata documentation, governance, and standardized naming conventions

• Experience parsing and modeling complex semi-structured data, including transactional logs, clickstream events, API errors, and performance logs

• Strong understanding of analytics data modeling for attribution, conversion funnels, behavioral analytics, and platform performance metrics

Ability to audit unfamiliar schemas and deliver production-ready code with minimal onboarding or business context Plusses:

• Experience in travel, e-commerce, logistics, or another fast-paced, enterprise-scale environment

• Experience transitioning experimental sandbox data work into production-ready assets for long-term maintenance

• Familiarity with standardizing data product outputs using lowercase/underscore naming conventions, schema controls, and detailed metadata

• Experience creating technical assets and documentation that can be handed off to broader engineering or offshore support teams

Strong communication, collaboration, and partnership skills