Lead Data Engineer

Job Description: As part of the Network Analytics team, the Sr Data Engineer role is crucial in executing advanced analytic techniques to enhance security and operational efficiency of client enterprise networks. This position focuses on constructing robust ETL pipelines using Spark to ensure data availability and quality for analytics tasks. The role requires expertise in data engineering, leveraging AWS for scalable solutions and enhancing operational efficiency. The role also requires expertise in statistical analysis and machine learning, utilizing generative AI to build data-driven, automated systems that enhance operational efficiency. Roles & Responsibilities: Design, develop, and maintain scalable ETL pipelines to ensure data quality and availability Implement monitoring and alerting solutions to ensure data pipeline reliability and performance Develop and manage deployment pipelines to facilitate continuous integration and delivery of data engineering solutions Implement data integration solutions to support analytics and reporting needs Execute the complete analytics lifecycle for problem solving, including: Algorithm traditionalization Model validation Model prototyping Data exploration Data grooming Survey varied data sources for analytic relevance, including: External sources accessed via API Flat files Relational databases Distributed file systems Interpret, synthesize and communicate results of analyses to effect action and changes within the organization Collaborate across teams to integrate analytic products with existing production architecture, develop, execute, and evaluate courses of action, and socialize results Help teach and explain techniques and tools used to a broad set of business Required Qualifications: Skills/Abilities and Knowledge Ability to read, write, speak and understand English Strong communication and presentation skills Expertise in data engineering languages such as Scala (preferred) or Java, with proficiency in Python Experience with BigData tools, particularly Spark Proficiency in building and managing ETL pipelines Expert-level quantitative analysis skills including interpretation of model results, consideration of causality, treatment of multicollinearity The ability to work in compiled, high-performance languages (e.g., Scala, Java, C++) Experience with relational databases Strong understanding of relational databases and SQL, and familiarity with NoSQL databases Broad experience and solid theoretical foundation on the modeling process using a variety of algorithmic techniques, including Machine Learning, and Graph/Network Analytics Data pre-processing, exploratory data analysis using a variety of techniques Basic understanding of data architecture, data warehouse, and data marts Demonstrated ability and desire to continually expand skill set, and learn from and teach others Preferred Qualifications: Experience with Linux-based operating systems Knowledge of other relevant techniques such as text analysis and text mining Bachelors degree (or relevant experience) in computer science, mathematics, statistics, physics, operations research, or other quantitatively-focused fields Masters: 6+ Years Work Experience Bachelors: 8+ Years Work Experience #J-18808-Ljbffr