Senior Data Engineer

Senior Data Engineer

Location - Dallas, TX and San Ramon, CA and New York city, NY (Remote option is available)

Our rate: $55 on C2C / $50 on W2

Customer – Centene

We are seeking an experienced Senior Developer to support the project initiative. The ideal candidate will have strong hands-on experience developing cloud-native, configuration-driven, event-based, and data-processing solutions using AWS, Kafka, Python, and Java.

This role requires the ability to design scalable data pipelines, configurable business-rule frameworks, APIs, microservices, and containerized applications while ensuring performance, reliability, observability, and maintainability.

Key Responsibilities

• Design, develop, and maintain scalable components for the PISCES HUB platform.
• Build AWS-based data pipelines, event-driven workflows, APIs, microservices, and integration services.
• Develop Kafka producers and consumers, including appropriate topic, partitioning, schema, offset-management, retry, and error-handling strategies.
• Design and implement configuration-driven systems, including:
• Metadata-driven pipelines and workflows
• Dynamic business-rule engines
• Configurable APIs and data-integration frameworks
• Template-based code-generation solutions
• Parameterized services and reusable components
• Design and optimize configuration interpreters using JSON, YAML, and XML.
• Implement configuration validation, versioning, auditing, rollback, backward compatibility, and error-handling capabilities.
• Develop high-quality, reusable, and maintainable software using Python and/or Java.
• Design SQL and NoSQL data models for storing, retrieving, auditing, and versioning business rules and system configurations.
• Build and integrate solutions using Amazon EKS, AWS Lambda, Amazon ECS, Amazon S3, Amazon RDS, and Amazon DynamoDB.
• Develop REST APIs, asynchronous services, batch-processing components, and event-based integrations.
• Implement automated unit, integration, component, and contract testing.
• Integrate application components with CI/CD pipelines and GitOps deployment workflows.
• Build, package, and deploy applications using Docker and Kubernetes.
• Implement application logging, monitoring, distributed tracing, dashboards, and alerting using Dynatrace.
• Troubleshoot production issues and perform root-cause analysis across application, integration, data, and infrastructure layers.
• Conduct code reviews and enforce engineering, security, performance, and maintainability standards.
• Collaborate with architects, product owners, quality analysts, data engineers, DevOps engineers, and other developers.
• Use Windsurf and other AI-assisted engineering tools to accelerate development, refactoring, automated testing, troubleshooting, and technical documentation.

Required Technical Skills

• Strong proficiency in Python or Java; hands-on experience with both is preferred.
• Strong experience developing AWS-based pipelines and cloud-native applications.
• Hands-on experience with Apache Kafka and event-driven architectures.
• Proven experience building:
• Metadata-driven pipelines and workflows
• Dynamic business-rule engines
• Configuration-driven applications
• Configurable API and data-integration frameworks
• Template-based code-generation frameworks
• Parameterized and reusable services
• Strong experience designing and optimizing JSON-, YAML-, and XML-based configuration interpreters.
• Strong understanding of configuration schemas, validation, versioning, auditability, backward compatibility, and rollback.
• Strong knowledge of SQL and NoSQL database modeling.
• Experience designing data models for business rules, metadata, configuration versioning, and audit history.
• Hands-on experience with:
• Amazon EKS
• AWS Lambda
• Amazon ECS
• Amazon S3
• Amazon RDS
• Amazon DynamoDB
• Strong understanding of REST APIs, microservices, asynchronous processing, and distributed systems.
• Experience with CI/CD pipelines, Git, and GitOps practices.
• Hands-on experience with Docker, Kubernetes, and container orchestration.
• Strong working knowledge of Dynatrace for observability, monitoring, performance analysis, and troubleshooting.
• Working knowledge of Windsurf or similar AI-assisted development tools.

Preferred Qualifications

• Experience developing high-volume, fault-tolerant, distributed platforms.
• Experience with Kafka schema management and serialization formats such as Avro, JSON, or Protobuf.
• Knowledge of data lineage, schema evolution, data contracts, and data-quality controls.
• Experience with infrastructure-as-code and deployment tools such as Terraform, Helm, Argo CD, or equivalent technologies.
• Familiarity with design patterns for retry management, dead-letter queues, idempotency, caching, and fault tolerance.
• Experience implementing secure APIs and cloud services using authentication, authorization, encryption, and secrets-management practices.
• AWS Developer, AWS Solutions Architect, Kubernetes, Java, or Python certifications are an advantage.

Key Competencies

• Strong software-design and problem-solving abilities.
• Ability to translate business requirements into configurable and reusable technical solutions.
• Strong understanding of scalable and resilient distributed-system design.
• Effective debugging and root-cause-analysis skills.
• Commitment to clean code, automated testing, documentation, and engineering standards.
• Ability to work independently and mentor junior developers.
• Strong collaboration and communication skills.
• Ability to deliver effectively within an Agile product-engineering environment.