Big Data Engineer

Software Guidance & Assistance, Inc. (SGA, Inc.)

Software Guidance & Assistance, Inc., (SGA), is searching for a Big Data Engineer for a CONTRACT assignment with one of our premier Regulatory clients in Rockville, MD or Tysons, VA.
We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. In this role, you will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have deep expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop, Spark etc
Responsibilities :
Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).
Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.
Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
Optimize and enhance existing data pipelines for performance, scalability, and reliability.
Develop automated testing frameworks and implement continuous testing for data quality assurance.
Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
Work with data scientists and analysts to support data-driven decision-making across the organization.
Ability to write and maintain automated unit, integration, and end-to-end tests
Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.
Required Skills :
Bachelor's degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience; Master's degree and past Financial Services industry experience preferred.
Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.
Past experience with developing enterprise quality solutions in an iterative or Agile environment.
Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks.
Strong written and verbal technical communication skills.
Demonstrated ability to develop effective working relationships that improved the quality of work products.
Should be well organized, thorough, and able to handle competing priorities.
Ability to maintain focus and develop proficiency in new skills rapidly.
Ability to work in a fast paced environment.
Experience with object oriented programming languages such as Java, Scala or Python.
Essential Technical Skills:
AI Tool Proficiency: Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.)
Technical Background: Strong software development background with ability to contribute to technical discussions
Agile Methodology: Extensive experience with Scrum, Kanban, and continuous improvement practices
Big Data technologies
Experience with Big data technologies such as Hadoop, Spark, Hive & Trino
Evaluate understanding of common issues like:
Data skew and strategies to mitigate it.
Working with massive data volumes in PetaBytes.
Troubleshootingjob failures due to resource limitations, bad data, scalability challenged.
Look for real-world debugging and mitigation stories.
AI Skills
Prompt Engineering: Proficiency in crafting effective prompts for AI coding assistants and analysis tools
AI Workflow Design: Experience redesigning development processes to leverage AI capabilities
Data Analysis: Ability to interpret AI-generated insights and translate them into actionable team improvements
Change Management: Experience leading teams through AI adoption and workflow transformation
SQL Skills (Window Functions, Joins, Complex Queries)
Assess comfort with SQL window functions, multi-table joins, aggregations.
Provide examples or ask them to write/optimize SQL queries on the spot.
Probe how they handle edge cases like NULLs, duplicates, ordering, etc.
Apache Spark (Development, Internals & Tuning)
Test their understanding of Spark's core architecture — executors, tasks, stages, DAG.
Focus on Spark performance tuning techniques: partitioning, caching, broadcast joins, etc.
Ask scenario-based questions on troubleshooting slow running/stuck jobs or resource issues in Spark.
Explore their experience optimizing Spark jobs for large-scale datasets.
Cloud Technologies
Check exposure to AWS services like S3, EMR, Glue, Lambda, Athena, etc.
Ask how they've used S3 with Spark (e.g., dealing with file formats, consistency issues).
EKS, Serverless knowledge, etc.
Programming - Python or Scala
Assess ability to write clean, modular, and performant code.
Look for experience in functional programming concepts (e.g., immutability, higher-order functions).
Ask about real-world use cases where they wrote scalable data processing code.
Evaluate understanding of collections, concurrency, and memory management.
Preferred Skills :
Experience with managing production data pipelines/ETL systems
Experience with CI/CD
Experience writing test cases
AWS certifications
By applying for a job with SGA, you agree to allow SGA to process your application for this and future opportunities in accordance with our Privacy Policy. Also, to ensure timely processing, you agree to be contacted by our AI recruiter via email, text, or phone. Message frequency varies and data rates may apply, but you can reply STOP to any SMS message to opt-out of texts and may contact SGA at View email address on click.appcast.io to opt-out of AI communications. The choice not to engage with AI will not adversely impact your consideration for placement. AI is not used to make any hiring determinations.
SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at .
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.

Apply

Vacancy posted 2 days ago

Similar jobs that could be interesting for youBased on the Big Data Engineer in Rockville, MD vacancy

Big Data Engineer
...Role: Big Data Engineer Location: Rockville, MD or Tysons, VA or Jericho, NY or New York, NY or Jersey City, NJ and Woodbridge, NJ Duration: Long Term We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large...
Big data
Work experience placement
Unisys
Rockville, MD
1 day ago
Big Data Engineer
Job-ID29514256Reference26-02235Remote50% RemoteTitle: Senior Data Engineer Location: Rockville, MD Target Start Date: ASAP Type: contract Pay... ...software engineering fundamentals, hands-on experience with Big Data technologies, and the ability to solve complex technical challenges...
Big data
Contract work
Immediate start
ConsultNet
Rockville, MD
6 hours ago
Big Data Engineer
...Job Description Job Description Title : Big Data Engineer Location : DC Metro Area Target Start Date : ASAP Type :contract Pay Rate: DOE We are seeking an experienced Data Engineer to design, develop, and optimize large-scale data processing...
Big data
Contract work
Immediate start
ConsultNet
Rockville, MD
14 days ago
Data Engineer
...opportunity with strong work-life balance. As a Senior Backend Engineer, you'll lead through excellent code, solve complex problems, set... ...environment, utilizing distributed microservices architecture, big-data technologies (HIVE, Presto, Spark, Hadoop), and full-stack...
Big data
Unisys
Rockville, MD
22 days ago
Senior Data Engineer
...Guidance & Assistance, Inc., (SGA), is searching for a Senior Data Engineer for a CONTRACT assignment with one of our premier Regulatory... ..., and maintain large-scale data processing pipelines using Big Data technologies. In this role, you will: Collaborate with...
Big data
Contract work
Software Guidance & Assistance
Rockville, MD
15 hours ago
Senior Engineer/Senior Azure Data Engineer Azure Synapse, Databricks & Power BI
...Description Job Summary: We are looking for a highly skilled Azure Data Engineer to design and implement modern data solutions using the Azure... ...: Experience in building end-to-end data pipelines for big data and analytics . Knowledge of data governance and...
Big data
Macpower Digital Assets Edge Private Limited
Rockville, MD
16 days ago
BIg Data Engineer
...Description: Hybrid 3 days onsite in McLean, VA Our client seeks a Data Engineer to lead data management and big data or machine learning engineering initiatives. The role will focus on building and optimizing data pipelines and platforms using Python, PySpark,...
Big data
Hourly pay
Local area
Eliassen Group
McLean, VA
2 days ago
Senior/Lead Big Data Engineer
...We’re hiring: Senior/Lead Big Data Engineer We’re looking for a senior- or lead-level Big Data Engineer who can take on complex technical challenges and communicate solutions clearly. This is a hybrid role based in Rockville, MD, or Tysons Corner, VA . Must...
Big data
Full time
Unisys
Tysons Corner, VA
2 days ago
Data Scientist / Data Engineer
...Data Scientist / Data EngineerJTEC Consulting LLC focuses on successfully delivering solutions... ...is actively hiring Data Scientists, Data Engineers, and Data Analysts to support critical... ...: Support data-centric solutions, big data analytics, and data modeling in secure...
Big data
Local area
JTEC Consulting LLC
McLean, VA
5 days ago
Senior Data Engineer
...We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. In this role, you will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and...
Big data
Full time
Unisys
McLean, VA
2 days ago
AWS Data Engineer
$70 - $75 per hour
...Job Title: Senior Data Engineer Location: Mclean, VA/ Richmond, VA - Hybrid Duration: 07+ Months Contract with Extension Job id... ...Ecosystem: Deep hands-on experience with AWS Glue and Amazon EMR. Big Data Architecture: Proven track record of building scalable...
Big data
Full time
Contract work
PTR Global
McLean, VA
4 days ago
Data Science and Engineering Intern
...and citizen services. We're a nonprofit engineering, applied research, and advanced technology... ...- and help shape the future. MITRE's Data Science and Engineering department is looking... ...Distributed Data Processing and Big Data Architectures (e.g., Spark, Hadoop,...
Big data
Full time
Internship
Local area
The MITRE Corporation
McLean, VA
4 days ago
BigID Data Engineer
$62k - $141k
...Job Number: R0249404 BigID Data Engineer The Opportunity: Ever-expanding technology like IoT, machine learning, and artifi cia l intelligence... ...You Have: Experience with Databricks Experience with big data ecosystems Experience with Infrastructure-as-Code tools...
Big data
Full time
Contract work
Part time
Work at office
Local area
Remote work
Booz Allen Hamilton
McLean, VA
15 hours ago
Data Engineer
$62k - $141k
...Job Number: R0249294 Data Engineer The Opportunity: Do you want to work at the forefront of advanced technology and solve complex data... ...experience with Python and SQL ~3+ years of experience with big data technologies, including Databricks and distributed...
Big data
Full time
Contract work
Part time
Local area
Remote work
Booz Allen Hamilton
McLean, VA
15 hours ago
Data Engineer
$110k - $160k
OverviewWe are looking for seasoned Data Engineer to work with our team and our clients to develop enterprise grade data platforms, services... ...experience in Data Engineering with experience in tools such as:Big data tools: Hadoop, Spark, Kafka, etc.Relational SQL and NoSQL...
Big data
Steampunk
McLean, VA
2 days ago
Senior Data Engineer
...and citizen services. We're a nonprofit engineering, applied research, and advanced technology... ...future. Department Summary: MITRE's Data Science and Engineering department is... ...Engineering Distributed Data Processing and Big Data Architectures, such as Spark, Hadoop...
Big data
Internship
Local area
3 days per week
The MITRE Corporation
McLean, VA
2 days ago
Data Engineer - Databricks
$125k - $160k
OverviewWe are looking for seasoned Data Engineer to work with our team and our clients to develop enterprise grade data platforms, services... ...experience in Data Engineering with experience in tools such as: Big data tools: Databricks, Apache Spark, Delta Lake, etc....
Big data
Steampunk
McLean, VA
15 hours ago
Data Engineer, Mid
$62k - $141k
Data Engineer, MidThe Opportunity:Ever-expanding technology like IoT, machine learning, and artificial intelligence means that there’s more... ...than ever before. As a data engineer, you know that organizing big data can yield pivotal insights when it’s gathered from disparate...
Big data
Full time
Contract work
Part time
Local area
Remote work
Booz Allen Hamilton
McLean, VA
15 hours ago
Lead Specialist, Federal Data Engineer
...and design, develop, and optimize architectures to tackle the Big Data and Data Science needs for federal and local government agenciesUtilize... ...college/universityAbility to rapidly ingest, transform, engineer, and visualize data for both ad hoc and product-level (e.g., automated...
Big data
Local area
KPMG
McLean, VA
4 days ago
Distinguished Data Engineer- Card
Join to apply for the Distinguished Data Engineer- Card role at Capital One Distinguished Data Engineers are individual contributors who strive... ...machine learning models 3+ years of experience implementing big data processing solutions on AWS 3+ years of experience in building...
Big data
Full time
Part time
Capital One Group
Mc Lean, VA
4 hours ago
Data Engineering Intern - Summer 2027
$20 - $24 per hour
...government agencies with cutting-edge AI/ML and data solutions. We are a leader in autonomous... ...Americans. About the Role: As a Data Engineering Intern at Raft, you’ll play a hands-on... ...in cloud-based data platforms and big data technologies. Eagerness to learn about...
Big data
Hourly pay
Part time
For contractors
Work experience placement
Summer work
Internship
Live in
Raft Company Website
Mc Lean, VA
4 days ago
Data Engineer
...Job Description Job Description Data Engineer Employment Type: Full-Time, Mid-level Department: Business Intelligence CGS is... ...manipulate data (Python, R, SQL, SAS). -Strong knowledge of big data analysis and storage tools and technologies. -Strong understanding...
Big data
Full time
Flexible hours
Contact Government Services, LLC
Rockville, MD
5 days ago
Cloud Big Data Software Engineer
...Cloud Big Data Software EngineerWe are seeking a skilled Cloud Big Data Software Engineer to join our team and play a pivotal role in designing and developing cloud-based big data solutions. In this role, you will work with cutting-edge technologies to build scalable systems...
Big data
Temporary work
For contractors
Immediate start
Cymertek
McLean, VA
1 day ago
Mid Data Engineer
$62k - $141k
...Job Number: R0236780 Data Engineer, Mid The Opportunity: Ever-expanding te chn ology like IoT, machine learning, and artifi cia l intelligence... ...than ever before. As a data engineer, you know that organizing big data can yield pivotal insights when it's gathered from...
Big data
Full time
Contract work
Part time
Local area
Remote work
Booz Allen Hamilton
McLean, VA
2 days ago
Data Engineering Intern - Summer 2027
$20 - $24 per hour
...government agencies with cutting-edge AI/ML and data solutions. We are a leader in autonomous... .... About the Role: As a Data Engineering Intern at Raft, you'll play a hands-on role... ...in cloud-based data platforms and big data technologies. Eagerness to learn...
Big data
Hourly pay
Part time
Work experience placement
Summer work
Internship
Live in
Raft Company Website
McLean, VA
7 days ago
AI Big Data Engineer AWS Data Engineer
...This is Kumar from Tek Leaders Inc hope you are doing great. Please find the below Job Description. Role: AI Big Data Engineer Location: Tysons VA or Rockville MD ( Hybrid Position & In-person interview ) Duration: Contract to hire Job...
Big data
Full time
Contract work
Internship
Tek Leaders Inc
Rockville, MD
a month ago
Government and Infrastructure - AI and Data - Data Engineer - Top Secret Clearance
$91.4k - $136.9k
...Government and Infrastructure - Technology Consulting - AI & Data - Data Engineer - Top Secret ClearanceFrom strategy to execution, the Government... ...better working world.EY delivers unparalleled service in big data, business intelligence, and digital analytics built on a...
Big data
For contractors
Summer holiday
Work at office
Local area
Immediate start
Flexible hours
EY (Ernst & Young)
McLean, VA
15 hours ago
Government and Infrastructure - AI & Data - Data Engineer
$128.4k - $192.5k
...Government and Infrastructure - Technology Consulting - AI & Data - Data Engineer - Senior ConsultantFrom strategy to execution, the Government... ...build a better working world.EY delivers unparalleled service in big data, business intelligence, and digital analytics built on a...
Big data
For contractors
Summer holiday
Work at office
Local area
Flexible hours
EY (Ernst & Young)
McLean, VA
1 day ago
Data Engineering Team Lead
$130k - $160k
OverviewWe are looking for a seasoned Data Engineering Team Lead to work with our team and our clients to develop enterprise grade data platforms... ...in Data Engineering, including experience with tools such as:Big data tools: Hadoop, Spark, Kafka, etc.Relational SQL and NoSQL...
Big data
Steampunk
McLean, VA
1 day ago
Cloud Data Engineer
$116.2k - $229.1k
Position Summary Cloud Data Engineer Join our AI & Engineering team in transforming technology platforms, driving innovation, and helping... ..., including architecting and governing enterprise-scale Big Data, cloud, and AI/ML platforms across Azure, AWS, GCP, and on...
Big data
Local area
Deloitte
McLean, VA
6 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!