Big Data Engineer

Software Guidance & Assistance, Inc. (SGA, Inc.)

Software Guidance & Assistance, Inc., (SGA), is searching for a Big Data Engineer for a CONTRACT assignment with one of our premier Regulatory clients in Rockville, MD or Tysons, VA.
We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale data processing systems. In this role, you will work closely with cross-functional teams to architect data pipelines, implement data integration solutions, and ensure the performance, scalability, and reliability of big data platforms. The ideal candidate will have deep expertise in distributed systems, cloud platforms, and modern big data technologies such as Hadoop, Spark etc
Responsibilities :
Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).
Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.
Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
Optimize and enhance existing data pipelines for performance, scalability, and reliability.
Develop automated testing frameworks and implement continuous testing for data quality assurance.
Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
Work with data scientists and analysts to support data-driven decision-making across the organization.
Ability to write and maintain automated unit, integration, and end-to-end tests
Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.
Required Skills :
Bachelor's degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience; Master's degree and past Financial Services industry experience preferred.
Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.
Past experience with developing enterprise quality solutions in an iterative or Agile environment.
Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks.
Strong written and verbal technical communication skills.
Demonstrated ability to develop effective working relationships that improved the quality of work products.
Should be well organized, thorough, and able to handle competing priorities.
Ability to maintain focus and develop proficiency in new skills rapidly.
Ability to work in a fast paced environment.
Experience with object oriented programming languages such as Java, Scala or Python.
Essential Technical Skills:
AI Tool Proficiency: Hands-on experience with AI development tools (GitHub Copilot, Q Developer, ChatGPT, Claude, etc.)
Technical Background: Strong software development background with ability to contribute to technical discussions
Agile Methodology: Extensive experience with Scrum, Kanban, and continuous improvement practices
Big Data technologies
Experience with Big data technologies such as Hadoop, Spark, Hive & Trino
Evaluate understanding of common issues like:
Data skew and strategies to mitigate it.
Working with massive data volumes in PetaBytes.
Troubleshootingjob failures due to resource limitations, bad data, scalability challenged.
Look for real-world debugging and mitigation stories.
AI Skills
Prompt Engineering: Proficiency in crafting effective prompts for AI coding assistants and analysis tools
AI Workflow Design: Experience redesigning development processes to leverage AI capabilities
Data Analysis: Ability to interpret AI-generated insights and translate them into actionable team improvements
Change Management: Experience leading teams through AI adoption and workflow transformation
SQL Skills (Window Functions, Joins, Complex Queries)
Assess comfort with SQL window functions, multi-table joins, aggregations.
Provide examples or ask them to write/optimize SQL queries on the spot.
Probe how they handle edge cases like NULLs, duplicates, ordering, etc.
Apache Spark (Development, Internals & Tuning)
Test their understanding of Spark's core architecture — executors, tasks, stages, DAG.
Focus on Spark performance tuning techniques: partitioning, caching, broadcast joins, etc.
Ask scenario-based questions on troubleshooting slow running/stuck jobs or resource issues in Spark.
Explore their experience optimizing Spark jobs for large-scale datasets.
Cloud Technologies
Check exposure to AWS services like S3, EMR, Glue, Lambda, Athena, etc.
Ask how they've used S3 with Spark (e.g., dealing with file formats, consistency issues).
EKS, Serverless knowledge, etc.
Programming - Python or Scala
Assess ability to write clean, modular, and performant code.
Look for experience in functional programming concepts (e.g., immutability, higher-order functions).
Ask about real-world use cases where they wrote scalable data processing code.
Evaluate understanding of collections, concurrency, and memory management.
Preferred Skills :
Experience with managing production data pipelines/ETL systems
Experience with CI/CD
Experience writing test cases
AWS certifications
By applying for a job with SGA, you agree to allow SGA to process your application for this and future opportunities in accordance with our Privacy Policy. Also, to ensure timely processing, you agree to be contacted by our AI recruiter via email, text, or phone. Message frequency varies and data rates may apply, but you can reply STOP to any SMS message to opt-out of texts and may contact SGA at View email address on click.appcast.io to opt-out of AI communications. The choice not to engage with AI will not adversely impact your consideration for placement. AI is not used to make any hiring determinations.
SGA is a technology and resource solutions provider driven to stand out. We are a women-owned business. Our mission: to solve big IT problems with a more personal, boutique approach. Each year, we match consultants like you to more than 1,000 engagements. When we say let's work better together, we mean it. You'll join a diverse team built on these core values: customer service, employee development, and quality and integrity in everything we do. Be yourself, love what you do and find your passion at work. Please find us at .
SGA is an Equal Opportunity Employer and does not discriminate on the basis of Race, Color, Sex, Sexual Orientation, Gender Identity, Religion, National Origin, Disability, Veteran Status, Age, Marital Status, Pregnancy, Genetic Information, or Other Legally Protected Status. We are committed to providing access, equal opportunity, and reasonable accommodation for individuals with disabilities in employment, and our services, programs, and activities. Please visit our company EEO page to request an accommodation or assistance regarding our policy.

Apply

Vacancy posted 8 hours ago

Similar jobs that could be interesting for youBased on the Big Data Engineer in Rockville, MD vacancy

Big Data Engineer
...Role: Big Data Engineer Location: Rockville, MD or Tysons, VA or Jericho, NY or New York, NY or Jersey City, NJ and Woodbridge, NJ Duration: Long Term We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale...
Big data
Work experience placement
Unisys
Rockville, MD
1 day ago
Big Data Engineer (Rockville)
...Role: Big Data Engineer Location: Rockville, MD or Tysons, VA or Jericho, NY or New York, NY or Jersey City, NJ and Woodbridge, NJ Duration: Long Term We are seeking a highly skilled and experienced Big Data Engineer to design, develop, and optimize large-scale...
Big data
Part time
Work experience placement
Unisys
Rockville, MD
2 days ago
Backend Big Data Engineer
...Backend Big Data Engineer Location: Rockville, MD or McLean, VA (Hybrid & locals only) F2F Interview W2 role (Open for sponsorship) Is this the role you are looking for If so read on for more details, and make sure to apply today. Backed Tools: Java, Python...
Big data
Local area
Orbien LLC
Rockville, MD
6 days ago
Big Data Developer
...Hiring: Big Data Engineer | Rockville, MD / Tysons, VA Seeking 5+ years of hands-on Big Data experience with Hadoop, Spark, Hive, Trino, SQL, Python/Scala , and petabyte-scale data processing . Strong AWS experience with S3, EC2, EMR/SMR , plus Spark performance...
Big data
KMM Technologies, Inc.
Rockville, MD
3 days ago
Data Engineer
$125k - $160k
OverviewWe are looking for seasoned Data Engineer to work with our team and our clients to develop enterprise grade data platforms, services... ...experience in Data Engineering with experience in tools such as:Big data tools: Hadoop, Spark, Kafka, etc.Relational SQL and NoSQL...
Big data
Steampunk
McLean, VA
7 days ago
Senior Data Engineer
$125k - $165k
OverviewWe are looking for seasoned Senior Data Engineer to work with our team and our clients to develop enterprise grade data platforms, services... ...in Data Engineering with experience in tools such as:Big data tools: Hadoop, Spark, Kafka, etc.Relational SQL and NoSQL...
Big data
Steampunk
McLean, VA
3 days ago
Junior Big Data Engineer (AI-Forward)
Software Guidance & Assistance, Inc., (SGA), is searching for a Junior Big Data Engineer (AI-Forward) for a CONTRACT assignment with one of our premier Regulatory clients in the DMV area. We are seeking an early-career Big Data Engineer who works AI-first. This...
Big data
Contract work
Internship
Software Guidance & Assistance, Inc. (SGA, Inc.)
Tysons Corner, VA
4 days ago
Data Engineer - Databricks
$125k - $160k
OverviewWe are looking for seasoned Data Engineer to work with our team and our clients to develop enterprise grade data platforms, services... ...experience in Data Engineering with experience in tools such as: Big data tools: Databricks, Apache Spark, Delta Lake, etc....
Big data
Steampunk
McLean, VA
3 days ago
Data Engineer, Mid
$62k - $141k
Data Engineer, MidThe Opportunity:Ever-expanding technology like IoT, machine learning, and artificial intelligence means that there’s more... ...than ever before. As a data engineer, you know that organizing big data can yield pivotal insights when it’s gathered from disparate...
Big data
Full time
Contract work
Part time
Work at office
Local area
Remote work
Booz Allen Hamilton
McLean, VA
6 days ago
BigID Data Engineer
$62k - $141k
BigID Data EngineerThe Opportunity:Ever-expanding technology like IoT, machine learning,... ...available today than ever before. As a data engineer, you know that organizing data can yield... ...Experience with DatabricksExperience with big data ecosystemsExperience with Infrastructure...
Big data
Full time
Contract work
Part time
Work at office
Local area
Remote work
Booz Allen Hamilton
McLean, VA
6 days ago
Cleared - Data Engineer (Multiple Levels)
$90.7k
ResponsibilitiesNoblis is seeking a cleared Data Engineer to support our customer in Bethesda, MD. SETA.The Data Engineer will serve as a subject... ...emerging technologies such as data lakes, cloud storage, and big data analyticsAnalyze datasets to identify data quality issues...
Big data
Full time
Contract work
Part time
Local area
Remote work
Noblis
Bethesda, MD
3 days ago
Senior Data Engineer
...and citizen services. We’re a nonprofit engineering, applied research, and advanced technology... ...shape the future.Department Summary:MITRE’s Data Science and Engineering department is... ...EngineeringDistributed Data Processing and Big Data Architectures, such as Spark, Hadoop...
Big data
Internship
Local area
3 days per week
Mitre
McLean, VA
6 days ago
Sr Databricks Data Engineer
$116.2k - $229.1k
Position Summary Join Deloitte’s AI & Engineering practice and help organizations transform... ...technology platforms, modernize data environments, and unlock value through innovation... ...Lead: Evaluate, pilot, and integrate new big data and analytics technologies, ensuring...
Big data
Local area
Visa sponsorship
Deloitte
McLean, VA
5 days ago
Data Engineer
$62k - $141k
Data EngineerThe Opportunity: Ever-expanding technology like IoT, machine learning, and artificial... ...today than ever before. As a data engineer, you know that organizing data can yield... ...maintaining scalable data stores that supply big data in forms needed for business...
Big data
Full time
Contract work
Part time
Local area
Remote work
Booz Allen Hamilton
McLean, VA
6 days ago
Lead Specialist, Federal Data Engineer
...and design, develop, and optimize architectures to tackle the Big Data and Data Science needs for federal and local government agenciesUtilize... ...college/universityAbility to rapidly ingest, transform, engineer, and visualize data for both ad hoc and product-level (e.g., automated...
Big data
Local area
KPMG
McLean, VA
5 days ago
TS/SCI NiFi Data Architect & Engineer
B4Corp is seeking a Software Engineer in Bethesda, MD, to design and optimize NiFi data pipelines for enterprise ETL, including integration with cloud services and big data ecosystems. The role requires strong NiFi expertise, Java and Python coding, and collaboration with...
Big data
B4Corp
Bethesda, MD
2 days ago
Government and Infrastructure - AI & Data - Data Engineer
$128.4k - $192.5k
...Government and Infrastructure - Technology Consulting - AI & Data - Data Engineer - Senior ConsultantFrom strategy to execution, the Government... ...build a better working world.EY delivers unparalleled service in big data, business intelligence, and digital analytics built on a...
Big data
For contractors
Summer holiday
Work at office
Local area
Flexible hours
EY (Ernst & Young)
McLean, VA
4 days ago
Government and Infrastructure - AI and Data - Data Engineer - Top Secret Clearance
$91.4k - $136.9k
...Government and Infrastructure - Technology Consulting - AI & Data - Data Engineer - Top Secret ClearanceFrom strategy to execution, the Government... ...better working world.EY delivers unparalleled service in big data, business intelligence, and digital analytics built on a...
Big data
For contractors
Summer holiday
Work at office
Local area
Immediate start
Flexible hours
EY (Ernst & Young)
McLean, VA
7 days ago
Data Engineering Team Lead
$130k - $160k
OverviewWe are looking for a seasoned Data Engineering Team Lead to work with our team and our clients to develop enterprise grade data platforms... ...in Data Engineering, including experience with tools such as:Big data tools: Hadoop, Spark, Kafka, etc.Relational SQL and NoSQL...
Big data
Steampunk
McLean, VA
3 days ago
Data Engineer
...Hybrid in McLean, VA Tuesday-Thursday Our client seeks two Data Engineers to build and support batch-oriented data pipelines for AML and... ...or data engineering with hands-on coding. ~4+ years with big data technologies such as Spark, Hadoop ecosystem, or EMR. ~...
Big data
Hourly pay
Temporary work
Local area
Eliassen Group
McLean, VA
8 days ago
Cloud Analytics Engineer - Flexible Schedule
...Corporation is seeking a Cloud Analytic Software Engineer to design, develop, and optimize cloud-... ...environment. You will build scalable data pipelines, process large datasets, and collaborate... ..., cloud platforms (AWS/Azure/GCP), and big data tools like Hadoop/Spark, with a focus...
Big data
Flexible hours
Cymertek
Mc Lean, VA
1 day ago
Government and Infrastructure - AI and Data - AI Automation Engineer
$91.4k - $136.9k
...Government and Infrastructure - Technology Consulting - AI & Data - AI Automation Engineer - ConsultantFrom strategy to execution, Government and... ...better working world.EY delivers unparalleled service in big data, business intelligence, and digital analytics built on...
Big data
For contractors
Summer holiday
Work at office
Local area
Immediate start
Flexible hours
EY (Ernst & Young)
McLean, VA
6 days ago
Government and Infrastructure - AI and Data - AI Automation Engineer - Manager
$150.7k - $251.2k
...Government and Infrastructure - Technology Consulting - AI & Data - AI Automation Engineer - ManagerFrom strategy to execution, Government and... ...better working world.EY delivers unparalleled service in big data, business intelligence, and digital analytics built on...
Big data
For contractors
Summer holiday
Work at office
Local area
Immediate start
Flexible hours
EY (Ernst & Young)
McLean, VA
6 days ago
Data Engineer
...Job Description Job Description Data Engineer Employment Type: Full-Time, Mid-level Department: Business Intelligence CGS is... ...manipulate data (Python, R, SQL, SAS). -Strong knowledge of big data analysis and storage tools and technologies. -Strong understanding...
Big data
Full time
Flexible hours
Contact Government Services, LLC
Rockville, MD
8 days ago
Senior Data Engineer
$131.3k - $237.35k
...intelligence! We are seeking an experienced and innovative Senior Systems Engineer to support the Maru Program with our Intelligence Community... ...a direct impact on national security.What You'll DoThe Senior Data Engineer will lead the design and implementation of a scalable,...
Full time
Contract work
Leidos
Gaithersburg, MD
3 days ago
Senior Data Engineer (Teamcenter)
$101.4k - $154.65k
...responsible for managing and organizing data to support all business processes to achieve... ...with the applications and integration engineers to meet all necessary compliance and governance... ...ETL/ELT processes. Understand the big picture, set the right scope, and...
Temporary work
Work at office
Flexible hours
MESO SCALE DIAGNOSTICS, LLC.
Rockville, MD
7 days ago
AI Big Data Engineer AWS Data Engineer
...This is Kumar from Tek Leaders Inc hope you are doing great. Please find the below Job Description. Role: AI Big Data Engineer Location: Tysons VA or Rockville MD ( Hybrid Position & In-person interview ) Duration: Contract to hire Job...
Big data
Full time
Contract work
Internship
Tek Leaders Inc
Rockville, MD
7 days ago
Government and Public Sector - AI & Data - Data Engineer - Senior Consultant
$106.9k - $176.5k
...Government and Public Sector - Technology Consulting - AI & Data - Data Engineer - Senior Consultant From strategy to execution, the Government... ...wide variety of clients delivering the latest data science and big data technologies and practices to design, build and maintain...
Big data
For contractors
Summer holiday
Work at office
Local area
Flexible hours
EY
Mc Lean, VA
3 days ago
Senior Software Engineer - Data
$140k - $220k
...About the Role:We are seeking an experienced Sr Software Engineer to join our fast-paced, collaborative data team. In this role, you will have broad authority to... ...opportunity to solve challenging problems and make a big impact as we invest in making technographics a first-...
Big data
Worldwide
ZoomInfo
Bethesda, MD
3 days ago
Data Scientist SME
$131.3k - $237.35k
...Turn Mission Data into Mission Advantage. Are you ready to turn complex data into insights... ...intersection of data science, software engineering, artificial intelligence, machine... ...tecchnologies). ~ Strong experience with big data processing tools (e.g., Spark, PySpark...
Big data
Full time
Contract work
Remote work
Leidos
Gaithersburg, MD
1 day ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!