Data Engineer

Here's What You'll Do

• Collaborate with business stakeholders and other technical team members to acquire and migrate data sources that are most relevant to business needs and goals.
• Demonstrate deep technical and domain knowledge of relational and non-relation databases, Data Warehouses, Data lakes among other structured and unstructured storage options.
• Determine solutions that are best suited to develop a pipeline for a particular data source.
• Develop data flow pipelines to extract, transform, and load data from various data sources in various forms, including custom ETL pipelines that enable model and product development.
• Write custom scripts to extract data from unstructured/semi-structured sources.
• Provide clear documentation for delivered solutions and processes, integrating documentation with the appropriate corporate stakeholders.
• Identify and implement internal process improvements for data management (automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability).
• Stay current with and adopt new tools and applications to ensure high quality and efficient solutions.
• Build cross-platform data strategy to aggregate multiple sources and process development datasets.

Essential Qualifications / Requirements:

• Bachelor's degree in Computer Engineering, Computer Science or related discipline, Master's Degree preferred.
• 7+ years of ETL design, development, and performance tuning using ETL tools such as SSIS/ADF in a multi-dimensional Data Warehousing environment.
• 3+ years of experience with setting up and operating data pipelines using Python or SQL
• 7+ years of advanced SQL Programming: PL/SQL, T-SQL
• 5+ years of Enterprise Data & Analytics solution architecture
• 3+ years of strong and extensive hands-on experience in Azure, preferably data heavy / analytics applications leveraging relational and NoSQL databases, Data Warehouse and Big Data
• 3+ years of experience with Azure Data Factory, Azure Synapse Analytics, Azure Analysis Services, Azure Databricks, Blob Storage, Databricks/Spark, Azure SQL DW/Synapse, and Azure functions.
• 2+ years of experience in defining and enabling data quality standards for auditing, and monitoring.
• Strong analytical abilities and a strong intellectual curiosity
• In-depth knowledge of relational database design, data warehousing and dimensional data modeling concepts
• Deep understanding of REST and good API design.
• Strong collaboration and teamwork skills & excellent written and verbal communications skills.
• Self-starter and motivated with ability to work in a fast-paced development environment.
• Agile experience highly desirable.
• Proficiency in the development environment, including IDE, database server, GIT, Continuous Integration, unit-testing tool, and defect management tools.
• Working experience with Python, Spark, Pyspark.
• Strong Leadership capabilities.

Preferred Skills:

• 2+ years of experience with Big Data Management (BDM) for relational and non-relational data (formats like json, xml, Avro, parquet, copybook, etc.)
• Knowledge of Dev-Ops processes (CI/CD) and infrastructure as code.
• Knowledge of Master Data Management (MDM) and Data Quality tools.
• Experience developing REST APIs.
• Kafka
• Knowledge of key machine learning concepts & MLOPS

Technologies we use: Databricks, Azure SQL DW/Synapse, Azure Tabular, Azure Data Factory, Azure Functions, Azure Containers, Docker, DevOps, Python, PySpark, Scripting (Powershell, Bash), Git, Terraform, Power BI, Snowflake

Required Skills : • Strong Azure data engineering skills • Need person who can do all traditional ETL pipeline builds with knowledge of Azure Databricks, Data Factory, Snowflake, PySpark and SQL. • Ability to provide direction to offshore team members

Basic Qualification :

Additional Skills :


Background Check : Yes


Drug Screen : Yes


Notes :
Selling points for candidate :
Project Verification Info :
Candidate must be your W2 Employee :Yes
Exclusive to Apex :No
Face to face interview required :No
Candidate must be local :No
Candidate must be authorized to work without sponsorship ::No
Interview times set : :No
Type of project :
Master Job Title :
Branch Code :