Big Data Engineer

Big Data Engineer

The Big Data Engineer is a vital member of a collaborative team, responsible for designing, engineering, maintaining, testing, evaluating, and implementing big data infrastructure, tools, projects, and solutions for the client. This role involves working closely with the team to leverage cutting-edge database technologies for the swift retrieval of results from vast datasets. The engineer will select and integrate big data frameworks and tools to meet specific needs and manage the entire lifecycle of large datasets to extract valuable insights.

Key Responsibilities:

• Design and implement scalable big data solutions tailored to client's needs.
• Maintain and enhance existing big data infrastructures to meet client's unique requirements.
• Test and evaluate new big data technologies and frameworks for compatibility with client systems and goals.
• Collect, store, process, manage, analyze, and visualize large datasets to derive actionable insights.
• Collaborate with team members to integrate big data solutions with existing client systems.
• Ensure data integrity and security across all platforms used within Client.
• Develop and optimize data pipelines for ETL/ELT processes specific to Client's data needs.
• Document technical solutions and maintain comprehensive records in line with Client standards and protocols.
• Stay updated with the latest trends and advancements in big data technology relevant to Client's strategic initiatives.

Required Qualifications:

• Thorough understanding of cloud computing technologies, including IaaS, PaaS, and SaaS implementations.
• Skilled in exploratory data analysis (EDA) to support ETL/ELT processes.
• Proficiency with Microsoft cloud products, including Azure and Fabric.
• Experience with tools such as Data Factory and Databricks.
• Ability to script in multiple languages, with a strong emphasis on Python and SQL.

Preferred Qualifications:

• Experience with data visualization tools.
• Proficiency with Excel and Power BI.
• Familiarity with Delta Lake.
• Knowledge of Lakehouse Medallion Architecture.

Skill Matrix:

Skill - Required No. of Years - Actual Years of Experience

IaaS, PaaS, or SaaS data or AI implementations within Microsoft Azure - 3

Exploratory Data Analysis (EDA): Proficiency in EDA techniques to support ETL/ELT processes - 3

Implementation of Development and Production Workflows in Azure Data Factory and Databricks - 2

Strong scripting skills in Python and SQL - 3

Expert proficiency of data engineering creation in Microsoft Fabric - 1

Expert proficiency with Excel and Power BI - Desired -2

Expert proficiency of Delta Lake format and protocol - Desired - 1

Expert understanding of the Data Lakehouse Medallion Architecture - Desired - 1

About Us:

InterSources Inc. is a Small, Woman, and Minority-Owned Business Enterprise, ISO/IEC 27001, SOC 2 Type 2 certified company with massive 18+ years of diversified experience in providing IT Consulting Services, Artificial Intelligence, Data Analysis, Application Development, Cloud Services, Cybersecurity, Digital Marketing, ERP Management, Custom Software Development, Web Development, UI/ UX Design, System Integration, QA Support etc. We make reasonable accommodations for clients and employees, and we do not discriminate based on any protected attribute including race, religion, color, national origin, gender sexual orientation, gender identity, age, or marital status. We also are a Google Cloud and Oracle partner company