Big Data Engineer

Big Data Engineer Software Guidance & Assistance, Inc., (SGA), is searching for a Big Data Engineer for a contract assignment with one of our premier regulatory clients in the DMV or NY/NJ area. Responsibilities: As a Data Engineer on our team, you'll design, develop, and maintain large-scale data processing pipelines using Big Data technologies. In this role, you will: Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. Optimize and enhance existing data pipelines for performance, scalability, and reliability. Write and maintain automated unit, integration, and end-to-end tests. Monitor and troubleshoot production data pipelines to identify and resolve issues Required Skills: Bachelor's degree with 5+ years of experience, or an advanced degree with 2+ years of experience. Hands-on experience with Big Data technologies such as Hadoop, Spark, Hive, and Trino. Solid understanding of common data challenges, including data skew and mitigation strategies, working with petabyte-scale data volumes, and troubleshooting job failures caused by resource limitations, bad data, or scalability constraints. Industry experience with AWS services such as S3, EMR, Glue, Lambda, and Athena. Demonstrated expertise in object-oriented and database technologies, applied to deliver enterprise-quality solutions. Familiarity with industry-leading software engineering practices, including test automation, build automation, and configuration management frameworks. Strong software development background, with the ability to contribute to technical discussions. Strong analytical and problem-solving skills, with the ability to independently investigate data discrepancies and validate system outputs. Strong communication skills and the ability to quickly learn new technical problems and articulate them clearly. Preferred Skills: Major in Computer Science or a related field; advanced degree preferred. Knowledge of SEC/FINRA regulatory rules and reporting requirements, or a strong interest in learning about US financial market structure and the evolving regulatory landscape. Participation in hackathons, technical blog posts, or AWS certifications. By applying for a job with SGA, you agree to allow SGA to process your application for this and future opportunities in accordance with our Privacy Policy. Also, to ensure timely processing, you agree to be contacted by our AI recruiter via email, text, or phone. Message frequency varies and data rates may apply, but you can reply STOP to any SMS message to opt-out of texts and may contact SGA at View email address on click.appcast.io to opt-out of AI communications. The choice not to engage with AI will not adversely impact your consideration for placement. AI is not used to make any hiring determinations. SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at . SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. SGA