Data Engineer

Data Engineer

Our client is seeking a highly skilled and experienced Data Engineer to join their Product Development team. This is a great opportunity for someone who specializes in building large-scale, global software, cloud-based technologies to access timely, accurate and relevant data. An ideal candidate will have built scalable analytics platform that can easily accommodate and integrate different data sources and provide comprehensive data capabilities to all client products and delivery channels globally.

What you’ll be doing

You will play a crucial role in designing, developing, and maintaining our data analytics systems. Your primary focus will be on leveraging your expertise in Python, PySpark, Scala programming languages, and other data engineering tools to ensure the efficient processing, fast performance, and analysis of large-scale datasets.

Responsibilities:

• Designing, implementing, and maintaining scalable data pipelines using airflow and build Kafka based data platform and Microservice based restful API’s.
• Develop and build data models in relational and graph databases, and proficiency on using of Elastic Search.
• Developing efficient data models and schemas for optimal data storage and retrieval.
• Building and optimizing data processing systems for both batch and real-time data ingestion, transformation, and analysis.
• Monitoring and troubleshooting data pipeline performance issues, identifying bottlenecks, and implementing necessary optimizations.
• Ensuring data security and privacy by implementing appropriate data governance practices and adhering to data protection regulations.
• Proficiency on multi-cloud native technologies, AWS, Azure, GCP.
• Collaborating with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements and ensure data accuracy, completeness, and integrity.
• Collaborating with infrastructure and platform teams to deploy and manage data infrastructure in both cloud and on-premises environments.
• Staying up to date with the latest advancements and trends in data engineering and evaluating new technologies, machine learning and AI tools for potential adoption.
• Mentoring and providing technical guidance to junior / intern team members, promoting a culture of knowledge sharing and continuous learning.

What your background looks like

1. BS/MS Degree in Computer Science, Engineering, or related technical discipline, or equivalent practical experience.

2. 2-3+ years of proven experience as a Data Engineer or similar role, working with large scale data and analytics system.

3. Strong proficiency in PySpark, Scala, and Python, with solid understanding of their respective ecosystems and libraries.

4. Experience with other data engineering tools and frameworks such as Kafka, Hadoop, Airflow, Spark, SQL, etc.

5. Proficiency in working with distributed computing systems, cloud platforms (AWS, Azure, GCP) for data storage, processing, and deployment.

6. Solid understanding of data modeling concepts, relational, and NoSQL databases, data lakes, and data warehousing principles.

7. Knowledge of data governance practices, data security, and privacy regulations (e.g., GDPR, CCPA, etc.) is desirable.

8. Strong analytical and problem-solving skills, with the ability to troubleshoot and optimize complex pipelines.

9. Experience with agile development framework and tools (e.g., Jira).

10. Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.

11. Self-motivated, detail-oriented, and able to prioritize and manage multiple tasks in a dynamic environment.