Data Modeler

Data Modeler

The ideal candidate will have a proven track record as a senior/self-starting data engineer in implementing data ingestion and transformation pipelines for large scale organizations. We are seeking someone with technical skills in Databricks development, performance tuning, and optimization. The candidate will assist in the design and development of high performant data ingestion pipelines from multiple sources using Databricks. The candidate will be involved in all stages of integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is always maintained. Candidate will have extensive experience with commercial and open source relational and non-relational data repositories.

•Provide assignments and direction to team members

•Develop scalable and re-usable frameworks for ingestion and transformation of large data sets.

•Design and implement data ingestion pipelines from multiple sources ensuring the quality and consistency of data is always maintained.

•Work with event based / streaming technologies to ingest and process data.

•Work with other members of the project team to support delivery of additional project components (API interfaces, Search).

•Stream and Batch processes in Databricks.

•Work within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.

•Convert existing Informatica ETL code to Databricks code whenever it is feasible/required.

•Create and maintain Databricks queries to support dashboarding and/or reporting activities.

•SQL query development and optimization as required to support various reporting needs.

•Performance monitoring and diagnosis of the ingest pipeline and suggest continual improvement.

Position is remote within US.

Project Specific Qualifications:

•Bachelor's Degree from an accredited college or university required; an additional four (4) years of related work experience can substitute for a degree

•At least seven (7) years of relevant experience required

•At least two (2) years of experience with Databricks.

•Experience leading small teams

•Expertise in designing and deploying data applications on cloud solutions, such as Azure or AWS

•Comprehensive understanding of data management best practices including demonstrated experience with data profiling, sourcing, and cleansing routines utilizing typical data quality functions involving standardization, transformation, rationalization, linking and matching.

•Experience in building ETL / data warehouse transformation processes

•Hands on experience in performance tuning and optimizing code running in programming languages such as PySpark and Python

•Good understanding of SQL, T-SQL and/or PL/SQL

•Demonstrated analytical and problem-solving skills particularly those that apply to a big data environment

•Experience with Apache Kafka for use with streaming data / event-based data

•Experience with other Open-Source big data products Hadoop (incl. Hive, Pig, Impala)

•Experience working with structured and unstructured data

•Experience working in an Agile DevSecOps environment

•Experience of working with relational databases: (SQL Server, PostgreSQL)

•Experience with non-relational / NoSQL data repositories (incl. MongoDB, Cassandra, Neo4J)

Preferred experience:

•Knowledge of IRS business systems and data

•Experience working in a command line environment and a general understanding of Red Hat Linux OS, or other Unix-like OS.

•Databricks certification

•Expertise with ELK stack and/or Splunk.

•Knowledge of FedRAMP, FISMA and other Federal IT security guidelines.

Additional requirements as per Contract/Client:

•Candidates must meet requirements to obtain and maintain an IRS Minimum Background Investigation (MBI) clearance (active IRS Moderate Risk MBI clearance is a plus).

•Candidates must be a US Citizen or a Legal Permanent Resident (Green Card status) for at least three (3) years, and Federal Tax compliant