Data Engineer

Data Engineer
Develops cloud-based data pipelines and solutions using Azure Databricks, PySpark, SQL, and data lake technologies.

Job #136614

Manufacturing

$50 - $52 an hour

Chicago, Illinois

Apply Now

Talascend is currently seeking a Data Engineer for a contract opportunity with our client in Chicago, Illinois .


Overview


This role involves designing and developing data solutions that are strategic for the business, utilizing the latest cloud data technologies such as Azure Databricks, Python, PySpark, SparkSQL, Azure functions, Delta Lake, and Azure DevOps CI/CD. The Data Engineer will work in a fast-paced environment supporting the enterprise data needs.


Responsibilities

• Design, architect, and develop solutions leveraging cloud big data technology to ingest, process, and analyze large, disparate data sets to meet business requirements.
• Create data management and persistence solutions for application use cases via relational and non-relational databases, enhancing data processing capabilities.
• Develop proofs of concept to influence platform architecture, product management, and software engineering teams, and facilitate migration to modern technology platforms.
• Design data lake solutions for structured and unstructured data, providing technical guidance for migration and platform modernization.
• Contribute to CI/CD processes, adhere to development best practices, and foster discipline within the data engineering team.
• Build data pipelines, perform data cleansing, normalization, and transfer, and optimize ETL workflows using Azure Databricks and PySpark.
• Conduct testing phases, including unit, system integration, regression, and user acceptance testing.
• Translate business requirements into detailed technical solutions, and document and communicate data processes clearly.
• Implement data security measures, including encryption, access controls, and compliance with data regulations, ensuring data integrity and confidentiality.
• Troubleshoot data-related issues and assist in resolution processes.
• Build agent-driven workflows for data loading and transformation, leveraging MCP framework and agent orchestration.

Qualifications

• Minimum of 5 years of experience as a data developer using Python, PySpark, Spark SQL, and SQL Server, with a strong foundation in ETL concepts.
• Bachelor's degree in Information Science, Computer Science, Mathematics, Statistics, or a related quantitative discipline.
• Experience with Azure Cloud Platform, Databricks, and Azure storage solutions.
• Effective written and verbal communication skills, including technical documentation.
• Strong analytical and organizational skills.

Preferred Qualifications

• Experience working with unstructured, semi-structured data, and data lake environments; familiarity with data streaming and event-driven data pipelines.
• Hands-on experience with real-time or near real-time data processing.
• Knowledge of JSON, Parquet, and other file formats.
• Experience with NoSQL databases such as Hbase, MongoDB, Cosmos DB.
• Cloud experience on Azure or AWS platforms.
• Knowledge of Python with Spark, Spark Streaming, Azure SQL Server, Cosmos DB/MongoDB, Azure Event Hubs, Azure Data Lake Storage, and Azure Search.
• Reliable, self-motivated team player capable of managing multiple projects in a fast-paced environment.

We thank all applicants for their interest. However, only those qualified individuals who closely meet the qualifications of the position will be contacted. The details of the position are only a summary, other duties may be assigned as necessary.


Background Check and Drug Screen may be required.