Data Engineer

Data Engineer

Cape Town, WC, South Africa

About the Job

Remote, Contract position

Minimum Requirements:

• BSc Engineering/ Computer Science/ relevant IT qualification
• 4+ years experience in a Data domain role (Data engineering) / Data modelling experience in relevant environment
• Data warehouse technical experience definition/implementation/integration.
• Strong programming skills in Python and DBA skills (SQL/PSQL/DynamoDB or other).
• Experience with data pipeline and ETL tools and reporting/analytics tools including, but not limited to, any of the following combinations (1) SSIS and SSRS, (2) ETL Frameworks, (3) Data conformance, (4) Caching, (5) Spark (6) AWS data builds.
• Experience with data modelling, data governance, and data quality.
• Strong problem-solving skills and ability to work in a fast-paced environment.
• Strong communication skills and ability to work in a team.
• Expertise in Machine Learning (ML) and deep learning frameworks.
• Explaining the thinking behind simple ML algorithms.
• Proficiency in all aspects of model architecture, data pipeline interaction, and metrics interpretation.

Ideal:

• Experience with Big Data technologies such as Hadoop and Spark.
• Experience with containerization technologies such as Docker and Kubernetes.

Responsibilities:

• Develop and implement portfolio Data modelling, assurance and utilisation strategies and frameworks that align with enterprise approved governance, data and technology strategy and the Data COE.
• Lead the implementation of these strategies within the portfolio.
• Responsible for reporting activities at a portfolio level that provides insights into the portfolio data assurance landscape, strategy and roadmap and key metrics and indicators.
• Serve as though leader and guide in the data domain by sharing knowledge identifying problems, patterns, trends, and support the development of relevant BI and MI solutions.
• Design and implement scalable and robust processes for ingesting and transforming complex datasets.
• Contribute to the development of architectural frameworks, apply architecture principles, and drive the development of data architecture models within the organisation.
• Design and develop data models using dimensional modelling and data vault techniques and ensure stated business requirements are met by these models.
• Focused on data stewardship and curation, the data engineer enables the data scientist to run their models and analyses to achieve the desired business outcomes.
• Architect, train, validate and test advanced analytics / machine learning models, using enterprise-grade software engineering practices.
• Collaborate with data scientists and analysts to understand data requirements and ensure that data models and prompt engines function as expected and data is accessible and usable.
• Design, develops and maintain automated scalable data pipelines that improve estate performance, stability and auditability.
• These include data pipelines for ETL processing.
• Monitor and troubleshoot data pipeline issues.
• Define, implement and integrate with enterprise data lake and data warehouse solutions (cloud and on-premises).
• Ingest large, complex data sets that meet functional and non-functional requirements.
• Enable the business to solve the problem of working with large volumes of data in diverse formats, and in doing so, enable innovative solutions.
• Engineer data in the appropriate formats for downstream customers, risk and product analytics or enterprise applications.
• Proficiency in managing test data, ensuring data integrity, and maintaining data privacy and security standards.
• Provide technical leadership and mentorship to junior, intermediate, and senior data specialists.
• Leadership and Team Management
• Strong leadership skills to guide and mentor squads, setting clear goals and expectations for team members.
• Competence in managing and coordinating efforts, including resource allocation, workload distribution, and task prioritisation.
• Ability to foster a collaborative and innovative team culture that promotes excellence in data engineering practices.
• Stakeholder Communication
• Manage stakeholder communication, providing regular updates on data activities, milestones, and risks.
• Excellent communication and presentation skills for effectively conveying data status, data-driven insights, and recommendations to stakeholders at all levels.
• Ability to collaborate with cross-functional teams and provide visibility into data/modelling-related matters.
• Ethical and Compliance Awareness
• Understanding of ethical considerations in data engineering, including data privacy, security, and confidentiality.
• Awareness of industry-specific compliance standards, regulations, and best practices, and the ability to ensure adherence in data engineering processes.
• Skill in conducting ethical and compliant testing and data assurance activities.
• Continuous Learning and Adaptability
• Commitment to staying updated with emerging data engineering trends, technologies, and industry developments.
• Willingness to pursue certifications, training, and continuous learning opportunities to enhance and adapt data engineering skills.
• Ability to quickly adapt to evolving project requirements and data paradigms.

Or refer someone