Data Engineer

Job Description Data Pipeline Development Develop and maintain ETL/ELT pipelines in Databricks using Python (PySpark) and SQL Build data pipelines aligned to medallion architecture (Bronze, Silver, Gold layers) Implement transformations, aggregations, and data enrichment logic to support downstream analytics Ensure all pipelines are scalable, efficient, and production-ready Workflow Migration (Alteryx → Databricks) Convert Alteryx workflows into Databricks-native pipelines Translate legacy business logic into optimized Spark-based transformations Support migration of both: Simple workflows (standard transformations) Complex workflows (multi-step logic, flat-file dependencies) Validate output to ensure high parity with legacy systems Data Ingestion & Processing Build and maintain ingestion pipelines for: Structured data sources Flat files and semi-structured datasets Implement robust ingestion frameworks to handle schema changes and automate data loads Optimize ingestion processes for performance, cost, and reliability Testing, Quality & Optimization Execute unit testing, SIT, and regression testing for pipelines Support data validation by comparing legacy vs. new system outputs Identify and resolve defects, data discrepancies, and performance issues Optimize pipeline performance (runtime, cost, scalability) Documentation & Best Practices Write clear technical documentation (code comments, runbooks, README files) Follow established coding standards, version control, and CI/CD processes Contribute to continuous improvement of development standards and reusable frameworks Collaboration & Agile Delivery Work within an Agile Scrum environment (2-week sprints) Partner with Data Architects, Product teams, and SMEs to understand requirements Participate in sprint ceremonies (standups, planning, retrospectives) Support backlog refinement and provide technical input on implementation approaches We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: Skills and Requirements 4–8+ years of experience in data engineering / ETL development Strong hands-on experience with: Databricks (Spark / PySpark) Azure data ecosystem (ADLS, compute, storage) Proficiency in: Python and SQL for data engineering Building and optimizing data pipelines Experience working with: Large datasets and distributed processing frameworks Data ingestion from multiple source types (including flat files) Strong understanding of: Data transformation logic, ETL patterns, and data quality validation Experience working in Agile environments Experience converting workflows from Alteryx or similar ETL tools Familiarity with medallion/layered data architectures Exposure to Tableau or BI tools Experience with CI/CD pipelines (GitHub, DevOps tools) Prior experience in supply chain or analytics platforms #J-18808-Ljbffr