Senior Data Engineer - onsite

Position: Big Data Engineer

Length: 24+ months (Possible extensions based of performance)

Location: Atlanta, GA

Shift: Hyrbrid

Interview Mode: Video 1 st and Onsite face to face 2 nd

Job Description:

Data Engineers own the architecture, design, and delivery of scalable analytics and streaming solutions that turn enterprise data into trustworthy, governed datasets for BI and advanced analytics. They participate across the full data engineering lifecycle-gathering and clarifying requirements with business partners and modelers, shaping lakehouse and pipeline designs, and building and operating production ingestion and transformation on Databricks using Delta Lake, Delta Live Tables, Unity Catalog, Apache Spark (Spark SQL / PySpark), Kafka (including managed patterns such as Confluent Cloud), AWS services (for example S3 and supporting integration patterns), and SQL - while applying strong engineering practices for quality, observability, and maintainability. Success depends on advanced hands-on proficiency with these technologies and the ability to communicate clearly in writing and conversation, partnering across the organization to translate business questions and data needs into durable requirements and measurable outcomes.

Education:

• Bachelor's Degree required. Preferably in Information Systems, Computer Science, Electrical Engineering, Computer Information Systems or related field

Client Corporation is seeking a Senior Data Engineer who enjoys collaborating across the organization to deliver reliable, scalable data solutions. Join a skilled team

focused on modern data platforms, where you will help shape and operate our Databricks-based lakehouse and streaming analytics capabilities.

This role blends strong business and analytical judgment with hands-on engineering: you will partner with stakeholders to understand problems, translate them into clear requirements, design pragmatic architectures, and deliver production-grade ingestion and transformation pipelines - with ownership through deployment and steady-state operations. You will work with teams across the company to clarify data and BI needs, perform structured requirements discovery alongside Data Modelers, align with BI development on scope and estimates, and help guide delivery through completion while promoting engineering quality and consistency.

Responsibilities

• Define data requirements; discover, integrate, and wrangle large volumes of structured and semi/unstructured data; validate outcomes using appropriate tooling in our data environment.

• Support standardized datasets and ad hoc analysis needs; build mechanisms to ingest, validate, normalize, cleanse, and enrich data for downstream consumption.

• Contribute to data policies and technical controls where needed (access patterns, retention considerations, and synthesizing/anonymizing sensitive attributes in line with enterprise standards).

• Implement rigorous data quality approaches for new and evolving sources; iterate with analytics partners to improve sourcing, preparation, and trust in datasets used for insights and modeling.

• Champion data engineering best practices (testing, observability, operational readiness) and contribute practical guidance on analytics preparation and visualization-friendly dataset design where applicable.

• Partner closely with data science and business intelligence teams to deliver data models and pipelines that support reporting, research, and machine learning workflows.

• Build pipelines that clean, transform, aggregate, and publish data from disparate systems into curated layers suitable for BI and advanced analytics.

• Use Databricks, SQL, Spark, scripting, and AWS services to integrate systems reliably and efficiently.

• Apply solid knowledge of data architecture principles and help lead initiatives from requirements through implementation-balancing correctness, performance, cost, and maintainability.

Qualifications Required

• 5+ years of professional experience in data engineering (or equivalent depth), including manipulating, processing, and extracting value from large datasets in production settings.

• Databricks (must have): hands-on delivery on the Databricks platform, including developing and operating Databricks Jobs/Workflows, notebooks, and production pipelines using Apache Spark (Spark SQL and/or PySpark) on Databricks.

• Delta Lake on Databricks (must have): building and maintaining Delta Lake tables (for example: incremental processing, merges/upserts, schema evolution, and performance patterns aligned with Delta best practices).

• Delta Live Tables / DLT (must have): designing, implementing, and operating DLT pipelines (Python or SQL as used in your standards), including expectations around pipeline dependencies, incremental processing, and operational monitoring as supported by DLT.

• Databricks governance (must have): practical experience with Unity Catalog (or equivalent Databricks governance patterns strongly aligned to UC), including secured sharing concepts and catalog/table governance as implemented in your environment.

• 4+ years hands-on Apache Spark engineering for analytics workloads (batch and/or streaming), with ability to implement reliable transformations and troubleshoot performance issues.

• 3+ years working with Apache Kafka (or managed equivalents such as Confluent Kafka) for high-volume event/stream processing, including operational considerations (throughput, lag, scaling, replay scenarios).

• 3+ years building and operating AWS capabilities supporting analytics platforms (for example: S3, IAM-aligned access patterns, integration with compute/storage patterns common to lakehouse architectures; familiarity with services such as Glue/Lambda/MSK as applicable).

• Strong SQL: ability to write and optimize intermediate-to-advanced queries and translate business logic into reliable datasets.

• Demonstrated experience delivering ETL/ELT pipelines in a Databricks lakehouse environment; comfort with incremental loads, schema evolution, and data quality checks.

• Experience working in Agile delivery (Scrum/Kanban/SAFe or similar): backlog refinement, iterative delivery, dependency coordination.

Preferred

• Experience with Snowflake , or other large-scale analytical databases (helpful for hybrid/platform comparisons).

• NoSQL exposure (Cassandra or similar) where operational systems feed analytics platforms.

• Experience with enterprise messaging/integration ecosystems (TIBCO EMS, IBM MQ, etc.) in addition to Kafka-helpful when bridging operational feeds into analytics landing zones.