Data Engineer

Data Engineer

JOB RESPONSIBILITIES


The primary focus of a Data Engineer will be expanding and optimizing our data and data pipeline architecture and optimizing data flow and collection for cross-functional teams. The Data Engineer will support our software developers, database architects, and data analysts on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products.


Essential Responsibilities include the following. Other duties and special projects may be assigned.

• Create and maintain optimal data pipeline architecture
• Assemble complex, large data sets that meet business requirements
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Keep our data separated and secure across national boundaries through multiple AWS regions.
• Build the infrastructure required for optimal ETL (extraction, transformation, and loading) of data from various data sources using SQL and AWS 'big data' technologies.
• Work with stakeholders teams to assist with data-related technical issues and support their data infrastructure needs.
• Create data tools for analytic team members to build and optimize our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer operational efficiency and other critical business performance metrics
• Work with development team and other departments to create, improve, and maintain agents, procedures, and data reports

Supervisory Responsibilities: None

QUALIFICATIONS


Required

• 5+ years of experience in a Data Engineer role
• Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases and dynamic procedures
• Experience building and optimizing 'big data' data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
• A successful history of manipulating, processing, and extracting value from large disconnected datasets
• Strong project management and organizational skills
• Working knowledge of message queuing, stream processing, and highly scalable 'big data' data stores
• Experience supporting and working with cross-functional teams in a dynamic environment.

Desired

• Master's in Computer Science, Information Technology, or Information Systems
• Experience with data pipeline and workflow management tools: Azkaban, Airflow, Luigi, etc.
• Experience with stream-processing: Storm, Spark-Streaming, etc.
• Experience with AWS services: EC2, RDS, Redshift
• Experience with data tools: Hadoop, Kafka, Spark, etc.
• Experience with languages: Python, Java, C++, Scala, etc.
• Exposure to MySql and AdvantageDB
• Experience with SSIS and SSRS
• Experience with technology and platform migrations like MySql to SQL or SQL to Postgres