Data Engineer II

Data Solutions EngineerAs a part of our Big Data Product team, the Data Solutions Engineer will be responsible for developing and validating Big data products and applications which runs on the large Hadoop cluster. The qualified engineer will be developing and testing ETL process, Migrating different applications to cloud, developing Data validation tools used for performing quality assessments and measurements on different data sets that feed BDAI products.The candidate will be involved in:Perform data migration and conversion activities on different applications and platforms.Design, development and testing of data ingestion pipelines, perform end to end automation of ETL process for various datasets that are being ingested into the big data platform.Perform data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.Develop transformation logic, interfaces and reports as needed to meet project requirements. Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)Tuning performance optimization of data pipelines.Develop unit and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.Integrate automated processes into continuous integration workflows.Contribute to data quality assurance standards and procedures.Required Skills:Bachelors degree in Computer Science or equivalent education/training 4- 5 years of Software development and testing experience. 3+ years of Working experience on tools like Spark, HBase, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc. 3+ years of programming experience in Scala, Java or Python Experience with some cloud environment - AWS or GCP is mustExperience with development and automated testing in a CI/CD environment.Knowledge of GIT/Jenkins and pipeline automation is a must.Experience with developing and testing real-time data-processing and Analytics Application Systems. Strong knowledge in SQL development on Database and/or BI/DWStrong knowledge in shell scripting Experience in Web Services - API development and testing.A solid understanding of common software development practices and tools. Strong analytical skills with a methodical approach to problem solving applied to the Big Data domainGood organizational skills and strong written and verbal communication skills.Desired Skills:Working experience on large migration Projects is a big plus. Working experience on Google Cloud platform is a big plusDevelopment expereicne for tools and utilities for monitoring and alert set etc.Experience in load and performance testing, familiarity with testing tools such as JMeter. Familiarity with project Management and bug tracking tools, i.e., JIRA or a similar tool.