Big/ Data Engineer - GA

Position: Big Data Scientist

Length: 24+ months (Possible extensions based of performance)

Location: Atlanta, GA

Shift: Onsite or Remote for the right person

Interview Mode: Video 1 st and Onsite face to face 2 nd

Notes:

• Local candidates.
• Remote for the right candidate that is a great fit to the role.
• Linkedin must match Resume, or resume will not be reviewed. No altered resumes or profiles!

Job Description:

Data Engineers own the architecture, design, and delivery of scalable analytics and streaming solutions that turn enterprise data into trustworthy, governed datasets for BI and advanced analytics. They participate across the full data engineering lifecycle-gathering and clarifying requirements with business partners and modelers, shaping lakehouse and pipeline designs, and building and operating production ingestion and transformation on Databricks using Delta Lake, Delta Live Tables, Unity Catalog, Apache Spark (Spark SQL / PySpark), Kafka (including managed patterns such as Confluent Cloud), AWS services (for example S3 and supporting integration patterns), and SQL - while applying strong engineering practices for quality, observability, and maintainability. Success depends on advanced hands-on proficiency with these technologies and the ability to communicate clearly in writing and conversation, partnering across the organization to translate business questions and data needs into durable requirements and measurable outcomes.

Education:

• Bachelor's Degree required. Preferably in Information Systems, Computer Science, Electrical Engineering, Computer Information Systems or related field

Client is seeking a Senior Data Engineer who enjoys collaborating across the organization to deliver reliable, scalable data solutions. Join a skilled team

focused on modern data platforms, where you will help shape and operate our Databricks-based lakehouse and streaming analytics capabilities.

This role blends strong business and analytical judgment with hands-on engineering: you will partner with stakeholders to understand problems, translate them into clear requirements, design pragmatic architectures, and deliver production-grade ingestion and transformation pipelines - with ownership through deployment and steady-state operations. You will work with teams across the company to clarify data and BI needs, perform structured requirements discovery alongside Data Modelers, align with BI development on scope and estimates, and help guide delivery through completion while promoting engineering quality and consistency.

Qualifications Required:

• 5+ years of professional experience in data engineering (or equivalent depth), including manipulating, processing, and extracting value from large datasets in production settings.
• Databricks (must have): hands-on delivery on the Databricks platform, including developing and operating Databricks Jobs/Workflows, notebooks, and production pipelines using Apache Spark (Spark SQL and/or PySpark) on Databricks.
• Delta Lake on Databricks (must have): building and maintaining Delta Lake tables (for example: incremental processing, merges/upserts, schema evolution, and performance patterns aligned with Delta best practices).
• Delta Live Tables / DLT (must have): designing, implementing, and operating DLT pipelines (Python or SQL as used in your standards), including expectations around pipeline dependencies, incremental processing, and operational monitoring as supported by DLT.
• Databricks governance (must have): practical experience with Unity Catalog (or equivalent Databricks governance patterns strongly aligned to UC), including secured sharing concepts and catalog/table governance as implemented in your environment.
• 4+ years hands-on Apache Spark engineering for analytics workloads (batch and/or streaming), with ability to implement reliable transformations and troubleshoot performance issues.
• 3+ years working with Apache Kafka (or managed equivalents such as Confluent Kafka) for high-volume event/stream processing, including operational considerations (throughput, lag, scaling, replay scenarios).
• 3+ years building and operating AWS capabilities supporting analytics platforms (for example: S3, IAM-aligned access patterns, integration with compute/storage patterns common to lakehouse architectures; familiarity with services such as Glue/Lambda/MSK as applicable).
• Strong SQL: ability to write and optimize intermediate-to-advanced queries and translate business logic into reliable datasets.
• Demonstrated experience delivering ETL/ELT pipelines in a Databricks lakehouse environment; comfort with incremental loads, schema evolution, and data quality checks.
• Experience working in Agile delivery (Scrum/Kanban/SAFe or similar): backlog refinement, iterative delivery, dependency coordination.

Preferred:

• Experience with Snowflake , or other large-scale analytical databases (helpful for hybrid/platform comparisons).
• NoSQL exposure (Cassandra or similar) where operational systems feed analytics platforms.
• Experience with enterprise messaging/integration ecosystems (TIBCO EMS, IBM MQ, etc.) in addition to Kafka-helpful when bridging operational feeds into analytics landing zones.