Data Engineer

Data Engineering Pipeline Onboarding Specialist

Onboard clients via the components of our data engineering pipeline, which consists of UIs, Azure Databricks, Azure ServiceBus, Apache Airflow, and various container-based services configured through IUs, SQL, PL/SQL, Python, YAML, Node, and Shell, with code managed in GitHub, deployed through Jenkins, and monitored through Prometheus, Grafana.

Work as a part of our client implementation team to ensure the highest standards of product configuration that meet client requirements. Test and troubleshoot data pipeline using sample and live client data. Utilize Jenkins, Python, Groovy Scripts and Java to automate these tests. Must be able to parse logs to determine next actions. Work with product teams to ensure the product is configured appropriately. Utilize dashboards for Kubernetes/OpenShift to diagnose high level issues and ensure services are healthy. Support Implementation immediately after go live, work with O&M team to transition support to that team. Participate in daily AGILE meetings. Estimate project deliverables. Configure and test REST APIs and utilize manual tools to interact with APIs. Work with data providers to clarify requirements and remove roadblocks. Drive automation into everyday activities.

Qualifications:

• 3+ years of experience working with SQL (preferably Oracle Pl/SQL and SparkSQL) and data at scale.
• 3+ years of ETL experience ensuring source to target data integrity.
• Familiar with various file types (Delimited Text, Fixed Width, XML, JSON, Parque).
• 1+ years of coding experience with one or more of the following languages; Java, C#, Python, NodeJS using Git, with practical experience with working collaboratively through Git branching strategies.
• 1+ years of experience with Microsoft Azure cloud infrastructure, DataBricks, DataFactory, DataLake, Airflow and Cosmos Database.
• 1+ years of experience in reading and configuring YAML.
• 1+ years of experience with ServiceBus, setting up ingress and egress within a subscription, or relevant Azure Cloud services administrative experience.
• 1+ years of experience with Unit Testing, Code Quality tools, CI/CD Technologies, Security and Container Technologies.
• 1+ years of Agile development experience and knowledge of Agile ceremonies and practices.

Required Skills: Strong SQL Experience Experience with PySpark and SparkSQL Experience building Data Pipelines Azure Environment Experience including: Azure DataFactory Azure Data Lakes Azure Databricks

Basic Qualification: SQL Spark SQL Pyspark building data pipelines Azure Datafactory Databricks Data Lakes

Additional Skills: SQL Spark SQL Pyspark building data pipelines Azure Datafactory Databricks Data Lakes

Long Term a lot of growth and promotion opportunities. Background Check: Yes Drug Screen: Yes Notes: 100% Remote 6 month contract to hire- candidates should be willing to convert to FTE no subvend candidates should be willing to work CST schedule