Big Data Engineer

Purple Drive

Overview:

Job Title: Big Data Engineer

Location: Atlanta, GA / Tampa, FL/ Dallas, TX

Job Summary

We are seeking an experienced Big Data Engineer to design, build, and optimize large-scale data pipelines and distributed systems across cloud and on-prem platforms. The ideal candidate will have strong expertise in Spark, Hadoop ecosystems, cloud data services, ETL/ELT design, streaming platforms , and best practices for scalable data processing.

Responsibilities

Design, develop, and maintain big data pipelines using Spark (PySpark/Scala), Hadoop, Kafka, and distributed computing frameworks.
Build and optimize ETL/ELT pipelines for structured and unstructured data across cloud and on-prem data platforms.
Work with cloud technologies (AWS, Azure, or GCP) including Data Lake, Databricks, EMR, Glue, Dataflow, Synapse, or Snowflake.
Develop Delta Lake / Lakehouse architecture for high-performance ingestion and processing.
Build real-time streaming solutions using Kafka, Spark Streaming, Kinesis, or Event Hub.
Collaborate with data architects, analysts, and application teams to gather data requirements and deliver scalable solutions.
Implement and maintain CI/CD pipelines , automated jobs, and orchestration using Airflow/Azure Data Factory/Glue Workflows.
Optimize data pipelines for performance, cost efficiency, and reliability.
Ensure data quality, validation, governance, and lineage best practices across the ecosystem.
Troubleshoot and resolve production issues in a high-availability environment.

Required Skills & Qualifications

5-8+ years of experience as a Big Data Engineer or Data Engineer.
Strong hands-on skills with Spark (PySpark or Scala), Hadoop, Hive, HDFS, MapReduce .
Experience working with Databricks, EMR, Glue, Synapse, DataProc , or equivalent big-data compute engines.
Proficiency in Python or Scala for data engineering.
Experience with Kafka or other event-streaming technologies.
Strong understanding of cloud data architectures (AWS S3/Glue/EMR | Azure ADLS/ADF/Databricks | GCP BigQuery/DataProc).
Solid SQL skills and experience with relational + NoSQL databases.
Experience with version control (Git) and CI/CD tools (Jenkins, Azure DevOps, GitHub Actions).
Hands-on experience with Airflow , ADF, or other orchestration and scheduling tools.
Familiarity with data modeling, data governance, and best practices for data quality.

Apply

Vacancy posted 4 days ago

Similar jobs that could be interesting for youBased on the Big Data Engineer in Tampa, FL vacancy

Big Data Systems Engineer
$142.4k - $180k
...Title: Big Data Systems Engineer Belong, Connect, Grow, with KBR! KBR's National Security Solutions (NSS) team provides high-end engineering and advanced technology solutions to our customers in the intelligence and national security communities. In this position...
Big data
Contract work
Temporary work
Local area
Remote work
Work from home
Relocation package
KBR
Tampa, FL
1 day ago
Big Data Developer
...Spark, Scala,SQL ,Hive, Impala, Hadoop, Splunk, Autosys . 3. Data Analysis and Data Wrangling skills when dealing with Huge Volume.... ...Performance analysis, troubleshooting and resolution including Cloudera Big Data eco system. 5. Experience with Linux-based...
Big data
Full time
Flexible hours
Northern Base
Tampa, FL
2 days ago
Global Trade and Working Capital Solutions Technology - Data & AI Engineering Director
$170k - $300k
...are on a mission to harness the power of data to drive innovation, create exceptional customer... ....The Opportunity We are seeking an Engineering Director to join our development team.... ...and CI/CD pipelinesExperience with modern big data technologies for data processing and...
Big data
Full time
Citigroup
Tampa, FL
1 day ago
Senior Data Platform Engineer
$107.12k - $160.68k
...We are seeking a highly skilled and motivated Senior Data Platform Engineer, AVP (Python/PySpark, Microsoft SSIS, SQL Server, ETL & Database... ...using scheduling/orchestration platforms such as AutoSys. Big Data Platforms: Work with Hadoop ecosystem technologies including...
Big data
Full time
Citi
Tampa, FL
1 day ago
Big Data/PySpark Engineering Lead - Vice President
$113.84k - $170.76k
...scalable, fault-tolerant batch and real-time data processing pipelines. Develop robust... ...environment. ETL/ELT Transformation: Re-engineer existing stored procedures and complex... ...consistency between legacy outputs and new big data results. Schema Evolution: Map and...
Big data
Full time
Flexible hours
Citi
Tampa, FL
3 days ago
Data Engineer
...Job Description Job Description Data Engineer Employment Type: Full-Time, Mid-level Department: Business Intelligence CGS is... ...manipulate data (Python, R, SQL, SAS). -Strong knowledge of big data analysis and storage tools and technologies. -Strong understanding...
Big data
Full time
Flexible hours
Contact Government Services, LLC
Tampa, FL
24 days ago
Data Engineer
...free counseling, legal, and financial services.\n \n\nNow Hiring: Data Engineer\nAll Climate First Bank employees must be willing to embrace... ...shared-data approaches are appropriate.\n\nData Consolidation and Big Data\n\n Be capable of creating and working with Big Data...
Big data
Full time
Temporary work
Part time
Immediate start
Remote work
Climate First Bank
Tampa, FL
1 day ago
Databricks Data Engineer
$86.7k - $170.9k
Position Summary Join Deloitte’s Core AI & Data practice and help organizations modernize data platforms, strengthen enterprise... ...and artificial intelligence capabilities. As a Databricks Data Engineer, you will support the design, build, and optimization of cloud-based...
Local area
Visa sponsorship
Deloitte
Tampa, FL
2 hours ago
Data Engineer- Greater Tampa
...Job Description Job Description MaxHealth is seeking a Data Engineer for developing, maintaining, and supporting enterprise data pipelines and data integration solutions within MaxHealth's cloud-based analytics ecosystem. This role works closely with Data Platform...
Work at office
Weekday work
MAXhealth
Tampa, FL
16 days ago
Data Engineer (BI)
...Job Description Job Description Bridgeway is seeking a Data Engineer (BI) to design, develop, and maintain our data warehouse infrastructure and the semantic models, dashboards, and reporting that sit on top of it. This role involves working closely with analysts,...
Remote work
Bridgeway Benefit Technologies
Tampa, FL
28 days ago
Data Engineer
...Location: Tampa, FL Security Clearance: TS/SCI clearance required Responsibilities: Design and implement a high-throughput data pipeline architecture integrating structured and unstructured logistics data into a centralized data lake Leverage Apache Kafka...
TalentBridgeX
Tampa, FL
6 days ago
Data Engineer - MDM
$88k - $110k
...position is at Philip Morris International The selection process will be fully managed by Philip Morris International. -- MDM Data Engineer- Tampa, Florida Be a part of a revolutionary change! At Philip Morris International (PMI), we’ve chosen to do something...
Full time
Philip Morris International
Tampa, FL
more than 2 months ago
Data Engineer
...Job Description Job Description Graham Technologies is seeking a highly skilled Data Engineer to design, build, and maintain enterprise data solutions that support advanced analytics and modernize large-scale data environments. In this role, the Data Engineer will...
Flexible hours
Graham Technologies
Tampa, FL
2 days ago
Data Center Engineer
...Datacenter Engineer The engineer in this position should have a wealth of experience engineering/installing datacenter equipment. The... ...cohesive package. This would include the complete engineering of data centers including, but not limited to, infrastructure, overhead...
For contractors
Work at office
Local area
ADEX
Tampa, FL
3 days ago
Junior Data Engineer
...POSITION Junior Data Engineer LOCATION Remote but must live in EST or CST time zone DURATION Contract -5 months through Sept 2025 then could convert full time INTERVIEW TYPE Video VISA RESTRICTIONS...
Full time
Contract work
Live in
Remote work
Relocation
3B Staffing LLC
Tampa, FL
3 days ago
Data Engineer
...POSITION Data Engineer LOCATION Hybrid in either Dallas TX or Richmond VA DURATION 6+Months PAY RATE EW TYPE VISA RESTRICTIONS N/A REQUIRED SKILLS Minimum Qualifications: ~7+ years with Microsoft SQL Server (2016-2022) in an enterprise...
Free visa
RIT Solutions, Inc.
Tampa, FL
13 hours ago
Junior/Entry Level Data Engineer
$100k
...Cloud World/Oracle Java one (Las Vegas) -2023/2022 and at Gartner Data Analytics Summit (Florida)-2023 All Positions are open for... ...developers, Data analysts/ Data Scientists, and Machine Learning engineers for full-time positions with clients. Who Should Apply...
Full time
H1b
SynergisticIT
Tampa, FL
12 hours ago
Data & Analytics Engineer
...and high-frequency M&A. This roll-up strategy necessitates an A+ data & analytics team. Apex Service Partners was founded in 2019... ...$4.5 billion in capital commitments. Job Description – Data Engineer (Data & Analytics) Mission of the Role: Apex is looking...
APEX Service Partners
Tampa, FL
2 days ago
EDAT- Senior Data Security Engineer - Tampa, FL
...established and growing capabilities across Intelligence, Analytics, Engineering, Mission Support, and Communications disciplines. Founded in 20... .... Join our team. Barbaricum is currently seeking a Senior Data Security Engineer to serve as a technical leader supporting the...
For contractors
Work at office
Barbaricum
Tampa, FL
7 days ago
Data Scientist (TS/SCI req'd)
...operating firm of Command Holdings, is seeking Data Scientists to support USSOCOM's mission... ...scale data processing, and performs data engineering, data processing, and modeling techniques... ..., or Snowflake. Experience with big data processing frameworks such as Apache...
Big data
Full time
Contract work
Work at office
Local area
Visa sponsorship
Work visa
Flexible hours
WWC Global LLC
Tampa, FL
a month ago
Data Security Engineer
...Data Security Engineer Minimum 7 years Knowledge in process design and implementation Strong security tool experience (M365 security tools, Pureview, Imperva,) Excellent understanding of cyber security and risk Very good knowledge about encryption and database...
RIT Solutions
Tampa, FL
4 days ago
Distinguished AI and Data Security Engineer
...Want in? Join the #VTeamLife.What you'll be doing: Verizon’s Exposure and Vulnerability Management (EVM) Engineering and Reporting team is seeking a Sr. AI and Data Engineer to lead the development of our security data architecture, web infrastructure, and artificial intelligence...
Full time
Temporary work
Part time
Work experience placement
Work at office
Work from home
Shift work
3 days per week
Verizon
Tampa, FL
3 days ago
SQL Data Engineer (W2 ONLY)
$40 - $50 per hour
Our client is searching for a Data Engineer for a remote temp-to-perm position.***All candidates must work directly for TEKsystems and no sponsorship will be available now or in the future***Duties:Pipeline Design: Construct resilient data processing pipelines in Python...
Permanent employment
Contract work
Temporary work
Remote work
TEKsystems
Tampa, FL
2 hours ago
Data Engineer
$114.3k - $135.1k
...more. Join us to drive positive, lasting change that moves missions and the government forward! AFS is searching for a Data Engineer to support the design, development, and maintenance of data pipelines, data models, and analytics platforms. This role is ideal...
Full time
Live in
Work at office
Local area
Accenture Federal Services
Tampa, FL
4 days ago
Sr Python Developer
...tools. Build, test, and optimize prompt engineering strategies for LLM use cases. Integrate with external data sources (databases, APIs, vector databases like... ...). Experience in data pipelines, ETL, or big data frameworks . Education: ~ Bachelor'...
Big data
Purple Drive
Tampa, FL
12 hours ago
Advisory Seasonal Intern, Federal Technology Enablement - PhD Data Science/Data Engineer | Summer 2027
$86k - $100k
...full power of AI and digital innovation, we deliver intelligent, data-driven solutions to help our clients navigate change and... ...Seasonal Intern, Federal Technology Enablement - PhD Data Science/Data Engineer for our Advisory practice. Responsibilities : Conduct research...
Summer work
Internship
Seasonal work
H1b
Work at office
Local area
Remote work
Night shift
KPMG LLP
Tampa, FL
4 days ago
Advisory Seasonal Intern, Federal Technology Enablement - PhD Data Science/Data Engineer | Winter 2027
$86k - $100k
...full power of AI and digital innovation, we deliver intelligent, data-driven solutions to help our clients navigate change and... ...Seasonal Intern, Federal Technology Enablement - PhD Data Science/Data Engineer for our Advisory practice. Responsibilities : Conduct research...
Internship
Seasonal work
H1b
Work at office
Local area
Remote work
Night shift
KPMG LLP
Tampa, FL
4 days ago
Data Layer Engineer
...Job Description Job Description Data Layer Engineer Full TimeProfessional Doral, FL Position Summary The Data Layer Engineer supports mission-critical objectives under the referenced work order by designing, implementing, and optimizing data transformation...
For contractors
Endurion
Tampa, FL
a month ago
QA Automation
...PySparkGood to have exposure to Java with SeleniumExperience in ETL / Data Warehouse TestingBasic knowledge of tools such as Control Center and Ab InitioProficiency in SQL / OracleStrong working knowledge of Unix and Shell ScriptingExposure to Hive and Big Data ecosystem...
Big data
Futran Tech Solutions Pvt. Ltd.
Tampa, FL
5 days ago
QA Engineer
....Good to have exposure to Java with Selenium.Experience in ETL / Data Warehouse Testing.Basic knowledge of tools such as Control Center... ...Ab Initio.Proficiency in SQL / Oracle.Strong working knowledge of Unix and Shell Scripting.Exposure to Hive and Big Data ecosystem....
Big data
Futran Tech Solutions Pvt. Ltd.
Tampa, FL
5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!