Data Engineer

Data Engineer This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Data Engineer based in United States. This role offers an opportunity to help build and evolve a scalable, self-service data and AI ecosystem within a modern enterprise environment. You will contribute to data integration, ETL, cloud-based pipelines, and analytics solutions that support critical business needs. Working primarily with AWS and Databricks, you will develop reliable batch and near-real-time data pipelines. The position combines hands-on engineering with production support, troubleshooting, automation, and continuous improvement. You will collaborate closely with senior engineers, architects, analytics teams, infrastructure specialists, business stakeholders, and external partners. This is a strong opportunity to deepen your expertise in modern data engineering technologies while contributing to high-quality, dependable data solutions. The role is remote, with candidates in the Chicago metropolitan area preferred for occasional office collaboration. Accountabilities: Participate throughout the Software Development Life Cycle for data integration, ETL, and cloud-based data pipeline initiatives, including requirements analysis, design, development, testing, deployment, and production support. Design, develop, test, deploy, and support batch and near-real-time data pipelines using AWS and Databricks. Build data transformations and integration solutions using SQL, Python, Spark, and Delta Lake. Develop and maintain Databricks notebooks, workflows, jobs, Delta tables, and related data components. Contribute to requirements analysis, estimation, solution design, code reviews, testing, production deployment, and technical documentation. Create unit tests and support integration, reconciliation, performance, regression, and data quality testing. Monitor data pipelines, ETL processes, and cloud platform components, investigating production issues and supporting timely resolution. Diagnose data integration and pipeline failures, participate in root cause analysis, and contribute to corrective and preventive actions. Maintain source-to-target mappings, transformation rules, data flow diagrams, troubleshooting guides, and operational documentation. Collaborate with senior engineers, architects, analytics teams, infrastructure teams, business stakeholders, and external partners to deliver scalable data solutions. Support platform upgrades, deployment automation, monitoring improvements, and operational process enhancements. Participate in a scheduled on-call rotation and provide occasional evening or weekend support for production environments and project requirements. Stay current with AWS, Databricks, Spark, Python, SQL, Delta Lake, Unity Catalog, and emerging data engineering practices. Identify automation, process improvement, and operational efficiency opportunities that strengthen platform reliability and delivery quality. Requirements: Bachelor's degree in Information Technology, Computer Science, Statistics, Economics, Mathematics, Engineering, or another quantitative or technical discipline. 1–3 years of experience in software development, data engineering, ETL development, data integration, analytics, or a related technical field within a complex enterprise environment. Experience or exposure to data integration, ETL processes, data warehousing, or cloud-based data platforms. Working knowledge of SQL and relational databases for data extraction, transformation, and analysis; Oracle and PL/SQL experience is a plus. Basic programming experience with Python, SQL, or similar languages used in data engineering and analytics. Working knowledge of Apache Spark, preferably PySpark, and distributed data processing concepts. Familiarity with Databricks, Delta Lake, Databricks Workflows and Jobs, and cloud-based data engineering technologies. Basic understanding of AWS, including services such as Amazon S3, IAM, and cloud-native data storage concepts. Understanding of data warehousing, data modeling, data mapping, data transformation, data quality, governance, and integration principles. Familiarity with modern data platform concepts, including data lakes, data warehouses, lakehouse architecture, Medallion Architecture, and batch and streaming processing. Exposure to incremental data loading, Change Data Capture (CDC), streaming data fundamentals, and scalable data integration patterns. Familiarity with Git-based version control, CI/CD concepts, automated deployments, environment management, code reviews, and software development best practices. Basic understanding of data security, access controls, governance, data lineage, compliance, and enterprise data lifecycle management. Familiarity with Databricks Unity Catalog or similar governance tools is preferred. Understanding of cloud cost management, performance monitoring, and optimization concepts is a plus. Familiarity with Agile methodologies and tools such as Jira is beneficial. Strong analytical, problem-solving, troubleshooting, communication, and collaboration skills, with the ability to learn new technologies quickly. Demonstrated attention to detail and commitment to delivering reliable, maintainable, and high-quality solutions. Familiarity with data visualization concepts, Microsoft Office Suite, enterprise data platforms, and transactional system integrations is preferred. Asset leasing industry experience, particularly in the rail sector, is a plus. Relevant certifications such as Databricks Certified Data Engineer, AWS Certified Data Analytics, or Azure Data Engineer are preferred. Working knowledge of Unix shell scripting is a plus. Ability to accommodate occasional travel and scheduled production support outside standard working hours when required. Benefits: Annual salary range of $80,800–$96,000 USD. Eligibility for a short-term incentive plan, subject to applicable plan terms. Remote work arrangement, with periodic access to an office environment for candidates in the Chicago metropolitan area. Opportunities to work with modern cloud and data technologies, including AWS, Databricks, Spark, Python, SQL, and Delta Lake. Career development opportunities through hands-on experience, training, self-development, and exposure to enterprise data and AI initiatives. Benefits and employee programs designed to support professional and personal well-being. Equal opportunity employment and a commitment to an inclusive workplace. Jobgether