Data Engineer

About the job Data Engineer iVoyant is looking for multiple Data Engineers to join our client Digital Team. The Data Engineers will work with the client functional SMEs within the Digital and the Business team to understand the requirements and prepare technical design. They will ensure the availability and validity of the data to ensure that it fits with the requested business rules. The Data Engineers will create and maintain the Source-to-Target data mappings, Logical and Physical Data Model and, business and technical test cases to ensure data quality. In addition, they will also assist a team of developers in developing the data pipeline (using Informatica BDM or Python) to ensure ongoing data collection, consolidation, and management. The Data Engineers will aid in the optimization of operations by manipulating and aggregating the disparate data sources into a format that is easily digestible by both data scientists and statistically adept colleagues. Their core responsibility will be to combine large volumes of disparate complex data, conduct quality checks on the data, manipulate the data and ensure continuous access to a clean format of the operational data for data scientists and other stakeholders. Responsibilities Take ownership of the Functional Data Domain assigned to the Data Engineer. Gather knowledge of the business, functions, and their data analysis needs along with business's relationship with the Data. Work with Functional team to understand requirements, data sources and business rules. Identify, Analyze and Profile the data across various source systems to understand and ensure that data fits the business requirements. Raise any data anomalies and work with the functional users to close any anomalies or data gaps. Identify relationships across multiple source systems; and create reference data with the help of data stewards. Design the Dimensional Data Model, Data Engineering Pipelines, Data Quality and Validation rules based on the business requirements and rules that would include any error handing, exception management and data quality routines to expose the anomalies in the data. Finalize the technical Source-to-Target mapping to develop the data pipelines from source systems to the dimensional model. Work with the team of Developer(s) for the technical implementation of the Data Pipelines. Ensure that the Unit and Integration tests complete successfully. Drive the QA/UAT with the Functional users and ensure a deployment to higher environments. Designs and jointly develops the data architecture with data architect and ensures security and maintenance Identifies gaps and implements solutions for data security, quality and automation of processes Supports maintenance, bug fixing and performance analysis along data pipeline Diagnoses existing architecture and data maturity and identifies gaps Gather requirements, assess gaps and build roadmaps and architectures to help the analytics driven organization achieve its goals Qualifications 7+ years of experience in data engineering. 7+ years of working on data analysis using SQL, PowerBI and other equivalent tools 4+ years of experience in data modeling, data analysis and working with business. 3+ years of experience in leading a team of developers and drive technical solutions. Required Skills Knowledge of SQL, databases and tables structure to query the data from various places, integrate and present the results to the business Knowledge of ETL, data pipelines using Informatica (DEI/BDM) or equivalent technologies. Strong critical thinking and problem solving skills Success at working on cross-functional teams to meet a common goal Self-starter with a high sense of urgency Other Skills Knowledge of Big Data platforms (e.g. Hadoop, Map/Reduce, Spark, HBase, HDInsight, Data Bricks, Hive) and with programming languages like UNIX shell scripting, Python etc. Knowledge of reporting and BI packages e.g., PowerBI, Tableau, SAP BO etc. Education Bachelor's Degree in Computer Science, Engineering, and/or background in Mathematic and Statistics; Master's or other advanced degree a plus iVoyant