Data Engineer

Data Engineer

Location – Malvern, PA – Hybrid Duration – 6+ months Interview – Video

Required Technical Skills: Python, SQL, AWS Web services (Glu, S3, Lambda)

Core Programming Skills:

• Expert proficiency in Python, with experience in building data pipelines and back-end systems.
• Advanced knowledge of SQL for querying and optimizing large datasets.
• AWS Cloud Services Expertise: DynamoDB, S3, Athena, GlueETL, Lambda, ECS, Glue Data Quality, EventBridge, Redshift Machine Learning, OpenSearch, and RDS.

API and Resilience Engineering:

• Proven expertise in designing fault-tolerant APIs using Swagger/OpenAPI, GraphQL, and RESTful standards.
• Robust understanding of distributed systems, load balancing, and failover strategies.
• Monitoring and Orchestration: Hands-on experience with Prometheus and Grafana for observability and monitoring.

Job Duties:

• Senior Data Engineer – 7+ Years of Experience
• We are seeking a highly experienced Senior Data Engineer with 7+ years of expertise in designing, building, and optimizing robust data solutions. The ideal candidate must possess top-tier skills in Python, AWS services, API development, and TypeScript, and have significant hands-on experience with anomaly detection systems.
• The candidate should have a proven ability to work at both strategic and tactical levels, from designing data architectures to implementing them in the weeds.

Key Responsibilities:

• Data Pipeline Development
• Independently design, build, and maintain complex ETL pipelines, ensuring scalability and efficiency for large-scale data processing needs.
• Manage pipeline complexity and orchestration, delivering high-performance data products accessible via APIs for business-critical applications.
• Archive processed data products into data lakes (e.g., AWS S3) for analytics and machine learning use cases.
• Anomaly Detection and Data Quality
• Implement advanced anomaly detection systems and data validation techniques, ensuring data integrity and quality.
• Leverage AI/ML methodologies, including Large Language Models (LLMs), to detect and address data inconsistencies.
• Develop and automate robust data quality and validation frameworks.
• Cloud and API Engineering
• Architect and manage resilient APIs using modern patterns, including microservices, RESTful design, and GraphQL.
• Configure API gateways, circuit breakers, and fault-tolerant mechanisms for distributed systems.