Azure Data Engineer

Azure Data Engineer

Location: Remote (until further notice), but will be working onsite about 50-60% of the time in Houston, TX

Type: Contract to Hire

We are looking for an Azure Security Data Engineer with strong technical expertise in Big Data, who is interested in joining our team to create and manage our data infrastructure and tools, including collecting, storing, processing and analyzing a range of security data and data systems.

As a key member of this team, you will be working as part of the small empowered, customer-centric agile squad to deliver a consolidated and easy to use data platform for reporting and analytics

Responsibilities:

• Collect and process raw data at scale for a variety of projects and initiatives.
• Read, extract, transform, stage and load data to selected tools and frameworks as required and requested.
• Build ingestion pipelines and ETL jobs to consume data feeds from security source systems (e.g. Azure feeds, O365, AD, Syslog, APIs).
• Designing and implementing Continuous Integration / Continuous Delivery (CI/CD) solution using company standards for the squad.
• Creation and managing of data pipelines, transformation, and efficient storage.
• Build monitoring and alerting mechanisms for data pipelines and transformations.
• Managing infrastructure for Big Data and data pipelines.
• Build flexible data solutions that support the ingestion, or the consumption needs of our data lake and its customers.
• You will be a pivotal member of the squad and can shape and support the success of this service.

Requirements:

• Working knowledge of developing using the Azure analytics components including Data Lake, Power BI, Data Factory, Azure Data Explorer, Azure Synapse, Data Warehouse, and Data Bricks.
• General understanding of Agile and DevOps development methodology and concepts as applied to data driven analytics projects. Including CI/CD Coding, security testing best practice and standards.
• Experience in DevOps in Azure cloud environments.
• Experience with designing, building, and operating analytics solutions using Azure cloud and serverless technologies.
• Data Management experience e.g. data profiling, large volume data handling.
• Experience in automated data driven testing.
• General knowledge of scripting Languages such as Python, Scala, or PowerShell (at least one of them).
• General knowledge of database programming including relational (SQL, Maria) and nonrelational (NoSQL, MongoDB, Cassandra) databases.

Nice to Have:

• Experience with security related data sets.
• Object Oriented Programming knowledge is a plus.
• General understanding of Machine Learning Models.