Big Data Engineer

Job Description

\n

Title : Big Data Engineer
Location : DC Metro Area
Target Start Date : ASAP
Type :contract
Pay Rate: DOE

\n

We are seeking an experienced Data Engineer to design, develop, and optimize large-scale data processing pipelines in a cloud-based environment. The ideal candidate will have strong hands-on experience with Big Data technologies, AWS cloud services, and software engineering best practices.

\n

This role requires a highly analytical problem-solver who can build scalable, reliable data solutions, troubleshoot complex processing challenges, and collaborate with cross-functional teams to deliver high-quality enterprise applications.

Key Responsibilities \n

\n
• \n

  Design, develop, and maintain scalable data pipelines using Big Data technologies such as Apache Spark, Hadoop, Hive, and Trino.

\n
• \n

  Collaborate with engineering teams and business stakeholders to translate requirements into efficient technical solutions.

\n
• \n

  Optimize data processing workflows for performance, scalability, reliability, and cost efficiency.

\n
• \n

  Troubleshoot complex data processing issues, including data skew, resource constraints, job failures, and data quality discrepancies.

\n
• \n

  Develop and maintain automated unit, integration, and end-to-end tests to ensure data accuracy and system reliability.

\n
• \n

  Monitor, support, and troubleshoot production data pipelines, proactively identifying opportunities for improvement.

\n
• \n

  Leverage AWS cloud services to build and maintain large-scale data processing solutions.

\n
• \n

  Apply software engineering best practices, including CI/CD, test automation, code reviews, and configuration management.

\n

Required Qualifications \n

\n
• \n

  Bachelor's degree in Computer Science, Engineering, or a related technical field with 5 years of relevant experience, or an advanced degree with 2 years of experience.

\n
• \n

  Strong hands-on experience with Big Data technologies, particularly Apache Spark , Hadoop, Hive, or Trino.

\n
• \n

  Experience designing, developing, and optimizing large-scale data processing pipelines.

\n
• \n

  Hands-on experience with AWS services such as S3, EMR, Glue, Lambda, and Athena .

\n
• \n

  Experience troubleshooting distributed data processing challenges, including data skew, memory limitations, resource allocation, and job failures.

\n
• \n

  Understanding of distributed computing concepts and experience working with high-volume datasets, ideally at petabyte scale.

\n
• \n

  Strong software development fundamentals, including object-oriented programming, database technologies, and automated testing.

\n
• \n

  Familiarity with CI/CD pipelines, build automation, and software configuration management.

\n
• \n

  Strong analytical and problem-solving skills, including the ability to independently investigate data discrepancies and validate processing results.

\n
• \n

  Excellent communication skills and the ability to collaborate effectively with technical and nontechnical stakeholders.

\n

Preferred Qualifications \n

\n
• \n

  Advanced degree in Computer Science, Engineering, or a related discipline.

\n
• \n

  Experience processing and optimizing petabyte-scale datasets in distributed computing environments.

\n
• \n

  Familiarity with financial services, capital markets, regulatory reporting, or other data-intensive industries.

\n
• \n

  AWS certifications or additional cloud computing credentials.

\n
• \n

  Demonstrated interest in emerging technologies through technical projects, hackathons, open-source contributions, or technical publications.

\n

Ideal Candidate \n

The ideal candidate combines strong software engineering fundamentals with deep expertise in Big Data processing and AWS cloud technologies. They are comfortable troubleshooting complex distributed systems, optimizing Spark workloads, and independently solving challenging data engineering problems. They bring a proactive mindset, strong communication skills, and a commitment to building scalable, reliable, enterprise-grade data solutions.

\n

Welcome to ConsultNet, a premier national provider of technology talent and solutions. Our expertise spans across project services, contract-to-hire, direct search, and managed services onshore, nearshore, and hybrid. For over 25 years, we have connected thousands of consultants with meaningful roles through a personal, communication-driven approach, partnering with a diverse client base to build high-performing teams and create lasting impact. Our comprehensive service offerings cover a wide range of technology and engineering positions across key markets nationwide. Learn more at .

\n

We champion equality and inclusivity, proudly supporting an Equal Opportunity Employer policy. We welcome applicants regardless of Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other status protected by law.