Big Data Engineer

Guru Schools

Irving, TX

Overview:

Job Title : Big Data Engineer

Location : Irving TX (hybrid role) (Only local candidates)

Duration : 12 Months

Roles and responsibilities:

Lead in design, development and testing of data ingestion pipelines, perform end to end validation of ETL process for various datasets that are being ingested into the big data platform.
Perform data migration and conversion validation activities on different applications and platforms.
Provide the technical leadership on data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
Contribute to development of transformation logic, interfaces and reports as needed to meet project requirements.
Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)
Lead in design and develop validation framework and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
Assist Manager in project planning, validation strategy development
Provide support in User acceptance testing and production validation activities.
Provide technical recommendations for identifying data validation tools, recommend new technologies to improve the validation process.
Evaluate existing methodologies and processes and recommend improvements.
Work with the stakeholders, Product Management, Data and Design, Architecture teams and executives to call out issues, guide and contribute to the resolutions discussions.

Must have :

8+ years of Software development and testing experience.
4+ years of Working experience on tools like Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
4+ years of programming experience in Scala, Java or Python
Experience in technical leading and mentoring the teams
Experience with developing and testing ETL, real-time data-processing and Analytics Application Systems.
Strong knowledge in Spark SQL, Scala code development in big data Hadoop environment and/or BI/DW development experiences.
Strong knowledge in shell scripting Experience in Web Services - API development and testing.
Experience with development and automated framework in a CI/CD environment.
Experience with cloud environments - AWS or GCP is a plus.
Knowledge of GIT/Jenkins and pipeline automation is a must.
A solid understanding of common software development practices and tools.
Strong analytical skills with a methodical approach to problem solving applied to the Big Data domain
Good organizational skills and strong written and verbal communication skills.

Nice to have :

Working experience on large migration Projects is a big plus.
Working experience on Google Cloud platform is a big plus
Development experience for tools and utilities for monitoring and alert set etc.
Familiarity with project Management and bug tracking tools, i.e., JIRA or a similar tool

Skills:

ETL,,Spark SQL,

Apply

Vacancy posted 3 days ago

Similar jobs that could be interesting for youBased on the Big Data Engineer in Irving, TX vacancy

Principal Data Engineer
...Principal Data Engineer We are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer... ...to enhance query performance and scalability. Leverage big data technologies and frameworks (e.g., Hadoop, Spark, Hive) to...
Big data
Local area
TXU Energy
Irving, TX
1 day ago
Data Engineer
...and risks on the project or closure of escalations. Required Skills and Experience 9+ years of experience in Data Engineering and Data Analytics. Expert in Big Data Technologies including Spark, Kafka, Kudu, Impala, HBase, and Spark Scala programming. Experience with GCP...
Big data
For contractors
TechDigital Group
Irving, TX
5 days ago
Sr. DATA Engineer
...Job Title: Sr. DATA Engineer Work Location Irving TX Job Description Seeking a Senior professional with 3 to 5 years of experience in Python Scala and Spark SQL to design and optimize big scalable data processing solutions within the Databricks Spark SQL Scala Python...
Big data
Temporary work
Local area
LTM
Irving, TX
4 days ago
Senior Data Engineer
...d love to meet you! ABOUT THE ROLE We are looking for a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse... ..., Polars , PySpark or DuckDB ; 2+ years of experience with Big Data technologies ( Spark , Snowflake ); Expert‑level knowledge...
Big data
Work at office
Remote work
Visa sponsorship
Work visa
Flexible hours
AgileEngine
Irving, TX
1 day ago
Data Engineer
Data Engineer - MEM SQL Location: New Jersey / Irving, TX / Tampa, FL We are looking for an experienced Data Engineer with strong expertise... ...Experience with cloud platforms (AWS / Azure / GCP) Knowledge of Big Data technologies (Spark, Kafka, etc.) Telecom domain...
Big data
Qode
Irving, TX
5 days ago
Sr Data Engineer
...service while fostering opportunity and growth for associates. Sr Data Engineer A Sr. Data Engineer is an experienced professional responsible... ...AWS Redshift, Azure Synapse, or Google BigQuery. Knowledge of big data technologies, including Hadoop, Spark, Kafka, and HBase,...
Big data
Local area
Lennar
Irving, TX
3 days ago
Databricks Data Engineer
Job Title Databricks Data Engineer Location Irving, TX (Onsite) Type Full Time with TCS Experience Required 8+ years Job Description As... ...solutions for data generation, collection, and processing in a Big Data environment using predominantly PySpark/Python. Your typical...
Big data
Full time
VOLTO Consulting
Irving, TX
2 days ago
AWS Data Engineer
$125k - $140k
...seeking a highly skilled and motivated AWS Certified Engineer to design, build, and optimize scalable data solutions within the Amazon Web Services (AWS) ecosystem... .... The ideal candidate will have strong expertise in big data processing using PySpark and a deep...
Big data
Tata Consultancy Services
Irving, TX
2 days ago
Databricks Data Engineer
$125k - $140k
Job Title: Databricks Data Engineer Overview We are seeking a highly skilled and motivated Databricks Certified Engineer to design, build, and... ...(AWS), Microsoft Azure, or Google Cloud Platform (GCP). Big Data Concepts: Deep understanding of distributed computing, data...
Big data
Tata Consultancy Services
Irving, TX
2 days ago
Senior Data Engineer
Job Title Big data Developer Work Location: Irving, Texas Job Summary Seeking a Senior Python Data Engineer with 3 to 5 years of experience specializing in Apache Spark and Python to design and implement scalable data solutions Job Description Develop and maintain...
Big data
Temporary work
Local area
LTM
Irving, TX
1 day ago
Lead Data Engineer
...Lead Data Engineer We are Lennar Lennar is one of the nation's leading homebuilders, dedicated to making an impact and creating an extraordinary... ..., Glue, Lambda, AppFlow, Cloudwatch etc. AWS certification is big plus Snowflake Data Cloud Account Administration Virtual...
Big data
Live in
Local area
Aimwel
Irving, TX
3 days ago
Senior Data Engineer
$157.41k - $207.8k
...the design, construction, and upkeep of AI-enabled data pipelines. Collaborate closely with project managers, engineers, and analysts to elevate the data ecosystem.... ...solutions. Enhance real‑time data insights for big data processing and analytics. Support BI initiatives...
Big data
For contractors
Work at office
Local area
Remote work
Avaya Corporation
Irving, TX
3 days ago
Data Engineer
...Strong Experience in SQL Experience with data pipeline and workflow management tool: GCP... ...(may not be expert) To work on data engineering solutions to leverage best practices and... ...variety of data sources using SQL, and GCP ‘big data’ technologies. Work with data and analytics...
Big data
Programmers.io
Dallas, TX
3 days ago
Data Engineer
...understand their challenges Ask questions and present ideas to solve data problems with code Begin with a set of technical specifications... ...Experience with data integration from different sources into Big Data systems is preferable Hands‑on experience managing AWS services...
Big data
Jobtailor
Coppell, TX
4 days ago
Data Engineer with DevOps
...Job Title: Senior Data Engineer with DevOps Duration : Full Time Location : Pittsburgh, PA, Cleveland, OH, or Dallas, TX Work... ...Qualifications ~5+ years of experience in data engineering and big data processing ~ Strong expertise in Apache Spark (Spark...
Big data
Hourly pay
Full time
System One Holdings, LLC
Farmers Branch, TX
12 hours ago
Palantir Data Engineer
9+year of experience in data engineering, machine learning, or AI deployment. Proficiency with Palantir Foundry (pipelines, code workbooks, ontology... .... Hands-on experience with Unix/Linux environments and big data tools. Familiarity with ML Ops, model retraining, and monitoring...
Big data
InfoVision Inc.
Dallas, TX
2 days ago
Principal Data Engineer
...About the Role We are seeking a Microsoft Fabric Data Engineer with 7+ years of experience for a lead role. The ideal candidate will be... ...storage and analytics, integrating with Synapse Data Warehousing for big data processing and reporting. Cross-functional...
Big data
Brillio
Dallas, TX
4 days ago
Senior Data Engineer
...environment and you take the time to celebrate both the small & big wins. INCLUSION is a core tenant of your personal beliefs. A... ...and creating change. Position Lantern is looking for a Senior Data Engineer to join our Data Engineering team. The ideal candidate will have...
Big data
Temporary work
Work at office
Lantern
Dallas, TX
5 days ago
Data Engineer
...: Business Intelligence CGS is seeking a passionate and driven Data Engineer to support a rapidly growing Data Analytics and Business Intelligence... ...to manipulate data (Python, R, SQL, SAS) Strong knowledge of big data analysis and storage tools and technologies Strong...
Big data
Full time
Flexible hours
CGS Federal (Contact Government Services)
Dallas, TX
4 days ago
Data & ML Engineer
...Data & ML Engineer Brand: Luxottica Wholesale North America Location: Dallas, TX, US, 75204 Store #: A00152 Tech Srvcs Wholesale DAL-T Position... ...and ML pipeline issues Mentoring engineers on cloud-native, big data, and MLOps practices Basic Qualification Bachelor'...
Big data
Minimum wage
Full time
Local area
EssilorLuxottica SA
Dallas, TX
11 hours ago
Lead Data Engineer
...Lead Data Engineer page is loaded## Lead Data Engineerlocations: Dallas, TXtime type: Full timeposted on: Posted Todayjob requisition id: R... ...Lead Data Engineer to play a critical role in advancing NMC[2]’s big data engineering and analytics capabilities. This role will...
Big data
Temporary work
Flexible hours
NorthMark Strategies LLC
Dallas, TX
4 days ago
Sr. Data Engineer
...other Tue and Thu in the office. We are looking for a savvy Data platform Engineer to join our growing team of Data platform management. The candidate... ...technologies Microsoft SQL server, Teradata. 2+ years big data experience, Hadoop, Hive, Spark. Expert knowledge of...
Big data
Long term contract
Work at office
Rapinno Tech
Dallas, TX
2 days ago
Lead Data Engineer
...breakthroughs that shape the industries of tomorrow. Its engineers build critical infrastructure to eliminate... ...innovation. The Position We are seeking a Lead Data Engineer to play a critical role in advancing NMC²’s big data engineering and analytics capabilities. This...
Big data
NorthMark Compute & Cloud
Dallas, TX
4 days ago
Senior Data Engineer
We are seeking a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data platforms that support business intelligence... ...concepts and dimensional modeling. Experience with big data technologies such as Apache Spark, Apache Kafka, and Apache...
Big data
Fixity Technologies
Dallas, TX
3 days ago
Data Engineer
Data Engineer Who you are You appreciate the client relationship and understand that delivering an excellent experience to the client is critial... ...a wide variety of data sources using SQL, Fabric, and Azure big data technologies. Keep our data separated and secure. Create...
Big data
Full time
Work at office
Hirebridge
Dallas, TX
1 day ago
Senior Data Engineer
$79.6k - $139.3k
Senior Data Engineer Position Description: We are seeking a highly motivated and experienced Senior Data Engineer to design, develop, and optimize large scale data transformation solutions on modern Big Data platforms. The ideal candidate is passionate about technology...
Big data
Local area
CGI Technologies and Solutions, Inc.
Dallas, TX
1 day ago
Senior Big Data Platform Engineer (DevOps)
A technology solutions firm in Dallas is seeking a Big Data Infrastructure DevOps/Support Administrator. This role involves supporting and developing the Enterprise Infrastructure Big Data Service using Java, Python, and Hadoop. Ideal candidates will have a strong background...
Big data
Right Talent Right Now
Dallas, TX
4 days ago
Google Cloud Data Engineer
...Google Cloud Data Engineer Are you ready to step up to the New and take your technology expertise to the next level? Join and help transform... ...data pipelines from ingestion to consumption within a hybrid big data architecture, using Java, Python, Scala etc. Minimum 1...
Big data
Work experience placement
ClifyX
Dallas, TX
1 day ago
Service Delivery Center, AI & Data, Machine Learning Engineer (MLE) - Manager
$76.2k - $174.1k
...Financial Services Organization - AI and Data - Service Delivery Center EY is the only... ...and developing solutions leveraging big data technology (Hadoop, Spark, etc.) to... ...years of related work experience in AI/ML engineering or MLE/ML Ops Experience working with...
Big data
Work experience placement
Summer holiday
Flexible hours
EY
Dallas, TX
3 days ago
Lead Consultant Data Engineer (DHS)
$138k - $185k
Lead Consultant Data Engineer (DHS) Arlington, Virginia, United States Excella is a transformative technology firm that helps organizations... ...optimization in a cloud‑based environment. Hands‑on experience with big data technologies such as Spark, Kafka, and file formats like...
Big data
Work from home
Home office
Flexible hours
慨正橡扯
Arlington, TX
3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!