Data Engineer

Role: Data Engineer
Location: Atlanta, GA
Term: Full Time







Job Description

Must Have Technical/Functional Skills

Must Have Skills:

• Databricks (Notebooks, Delta Lake, Delta Live Tables, Unity Catalog)
• Python and PySpark
• Spark SQL and Apache Spark
• ETL/Data Pipeline Development
• Big Data technologies: Hadoop, Spark, Kafka, Hive, HDFS, Sqoop
• Strong SQL querying and data modeling
• Data ingestion, transformation, cleansing, and validation
• Cloud platforms (AWS/GCP)
• Data warehouse and large-scale data processing experience
• Experience with streaming data using Kafka and Spark Streaming
• Agile methodologies (Scrum, Kanban, SAFe)

Preferred Skills:

• Scala / Scala-Spark
• IBM DataStage, Tibco, or similar ETL tools
• AWS services: S3, Glue, Lambda, Kinesis/MSK
• BI Tools: Tableau, Power BI, Spotfire, OAC
• Databases: Teradata, DB2, BigQuery, Mainframe
• Git version control
• Data governance and retention modeling

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL solutions.
• Build and optimize data ingestion frameworks for structured and unstructured data.
• Implement data quality, validation, standardization, and governance processes.
• Develop data infrastructure integrating multiple heterogeneous data sources.
• Support analytics, reporting, machine learning, and business intelligence initiatives.
• Work with Data Scientists and BI teams to prepare and curate data for analytics.
• Develop scalable solutions for Big Data processing and streaming workloads.
• Analyze data and provide actionable insights and recommendations to business stakeholders.
• Gather requirements from business teams and translate them into technical solutions.
• Lead and mentor junior data engineers.
• Collaborate across multiple projects as the Data Engineering SME.
• Drive adoption of Data Engineering best practices and standards.