Big Data Engineer
Guru Schools
Overview: Job Title : Big Data Engineer Location : Irving TX (hybrid role) (Only local candidates) Duration : 12 Months Roles and responsibilities:
- Lead in design, development and testing of data ingestion pipelines, perform end to end validation of ETL process for various datasets that are being ingested into the big data platform.
- Perform data migration and conversion validation activities on different applications and platforms.
- Provide the technical leadership on data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
- Contribute to development of transformation logic, interfaces and reports as needed to meet project requirements.
- Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)
- Lead in design and develop validation framework and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
- Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
- Assist Manager in project planning, validation strategy development
- Provide support in User acceptance testing and production validation activities.
- Provide technical recommendations for identifying data validation tools, recommend new technologies to improve the validation process.
- Evaluate existing methodologies and processes and recommend improvements.
- Work with the stakeholders, Product Management, Data and Design, Architecture teams and executives to call out issues, guide and contribute to the resolutions discussions.
- 8+ years of Software development and testing experience.
- 4+ years of Working experience on tools like Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
- 4+ years of programming experience in Scala, Java or Python
- Experience in technical leading and mentoring the teams
- Experience with developing and testing ETL, real-time data-processing and Analytics Application Systems.
- Strong knowledge in Spark SQL, Scala code development in big data Hadoop environment and/or BI/DW development experiences.
- Strong knowledge in shell scripting Experience in Web Services - API development and testing.
- Experience with development and automated framework in a CI/CD environment.
- Experience with cloud environments - AWS or GCP is a plus.
- Knowledge of GIT/Jenkins and pipeline automation is a must.
- A solid understanding of common software development practices and tools.
- Strong analytical skills with a methodical approach to problem solving applied to the Big Data domain
- Good organizational skills and strong written and verbal communication skills.
- Working experience on large migration Projects is a big plus.
- Working experience on Google Cloud platform is a big plus
- Development experience for tools and utilities for monitoring and alert set etc.
- Familiarity with project Management and bug tracking tools, i.e., JIRA or a similar tool
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Big Data Engineer in Irving, TX vacancy
- ...Position: Big Data Engineer Location: Remote Role Summary Excellent Scala, PySpark, Java coding experience for REST API and SOAP webservices Excellent AWS experience (Batch and API microservices) using EMR/EC2/ECS/S3/Airflow/Step Functions, AWS...Big dataRemote work
United IT Solutions
Irving, TX3 days ago - • Analyze and understand data sources & APIs• Design and Develop methods to connect & collect... ..., Payments Required Technical Skills • Big Data-Hadoop, NoSQL, Hive, Apache Spark•... ...web-APIs• Exposure to machine learning engineering• Exposure to NLP and text processing• Experience...Big data
- Job Title: Sr. DATA Engineer Work LocationTampa FLJob Description:Mandatory Karat InterviewRole Overview We are seeking a highly skilled and... ...AWS ecosystem The ideal candidate will have strong expertise in big data processing using PySpark and a deep understanding of data...Big data
- ...strong experience in building distributed data query and analytics solutions across... ...movement.Collaborate with business and data engineering teams to translate requirements into scalable... ...Have:Experience with Databricks Spark or big data ecosystems.Exposure to Python for...Big data
$125k - $140k
...Roles & Responsibilities Job Title: Data Engineer -ML Job Description: We are looking for a highly skilled and passionate... ...Ops/LLM Ops and distributed systems. Experience with Big Data platform like Cloudera Hadoop and Could platforms like AWS...Big dataTata Consultancy Services
Irving, TX2 days ago- ...specialize in leveraging advanced technologies such as AI, cloud, and data-led innovation to help our clients accelerate growth and unlock... ...veteran; or disability.Technical/Domain Skill 1Technology|Big Data - Data Processing|PySpark Technical/Domain Skill 2Technology...Big dataFull timeTemporary workRelocation
- ...SummaryWe are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer, you will be... ...techniques to enhance query performance and scalability. Leverage big data technologies and frameworks (e.g., Hadoop, Spark, Hive) to...Big dataFull timeLocal area
- ..., Impala, HBase spark scala batch pipelines GCP, AWS - data pipelines Oozie, Airflow, Composer - scheduling tools... ...preferred. Job Description ~ Experience in data engineer, Data Analytics ~ Big Data Technologies Expert in Python ~(Developer level Expertise...Big data
Omni Inclusive
Irving, TX2 days ago - ...are currently available and apply online.Job SummaryAs a Lead Data Engineer, you will be responsible for leading the design and implementation... ...tools such as Tableau or Power BI.•Experience with big data technologies such as Hadoop, Spark, or Kafka.•Experience gained...Big dataFull timeLocal area
$125.76k - $188.64k
...ProfessionalCompany: CitiCiti is seeking a highly skilled and experienced Senior Data Engineer to join our dynamic and innovative technology team. The ideal... ...-native services for data storage, processing, and analytics.Big Data Technologies: Work extensively with big data frameworks...Big dataFull time$125.76k - $188.64k
...autonomous individuals. We are seeking a Senior Data Developer /Lead to join one of our small,... ...will possess deep technical expertise in big data technologies, including real-time... ...diverse data requirements from engineers and scientists into detailed technical specifications...Big dataFull time$125.76k - $188.64k
...skilled Starburst Expert to join our dynamic data team. The ideal candidate will have over... ...leadership and mentorship to data engineers and analysts, fostering best practices in... ...in data engineering, data warehousing, or big data analytics projects.Minimum of 3 years...Big dataFull time$125k - $140k
...Roles & Responsibilities Job Title: Data Engineer Job Description: We are seeking a highly skilled and motivated Data Engineer... ...and efficient data solutions. Qualifications: • Big Data Frameworks Expertise: Demonstrated high proficiency in Apache...Big dataTata Consultancy Services
Irving, TX4 days ago$120.5k - $231k
...always. Want in? Join the #VTeamLife.What you’ll be doing...Data science is the engine that drives informed decision-making and unlocks the hidden... ...by developing and performing complex analyses using big data technologies. Develop high-scale systems, applications...Big dataFull timeTemporary workPart timeWork experience placementWork at officeWork from homeShift work3 days per week- ...meet you! ABOUT THE ROLE We are looking for a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse... ..., PySpark or DuckDB ; - 2+ years of experience with Big Data technologies ( Spark , Snowflake ); - Expert-level knowledge...Big dataWork at officeRemote workVisa sponsorshipWork visaFlexible hours
AgileEngine
Irving, TX2 days ago - ...Job Title: GCP Data Engineer work Location: Irving, TX Duration: 8+ Months Job Description: Big data expert with 6+ years experience in Hadoop Big data ecosystem Spark - Batch & Streaming (Python,Scala ) Apache Kafka hands on experience...Big data
Ruri Software Technologies LLC
Irving, TX4 days ago $92.82k - $109.2k
...career. Try new things, learn new skills and discover what you excel at—all from Day One.Job DescriptionResponsible for working on big data/analytics projects that gather and integrate large volumes of data, performs analysis, interprets results and develops actionable...Big dataFull timeWork experience placementLocal area$157.41k - $207.8k
...Family: Research and Development Job Function: Software Engineering About the Job Senior Data Engineer Location: Irving, Texas Job Duties:... ...actionable solutions. Enhance real-time data insights for big data processing and analytics. Support BI initiatives...Big dataFor contractorsWork at officeLocal areaRemote workAvaya
Irving, TXa month ago- ...Responsibilities: Responsible for developing strategies for effective data analysis and reporting. Selects, configures, and implements... ...: ~ Senior Developer for development of IT project using Big Data/Hadoop technologies (Kafka, Storm, Hive, Elastic search)....Big dataContract workImmediate start
Pyramid Consulting
Irving, TX5 days ago - ...Sr. Data Engineer At Orion180, we shine because of how we do it. Our mission is to deliver an exceptional insurance experience through innovative... ...flows, and parameterizing pipelines in Azure Data Factory. ~ Big Data Compute: Proven experience processing large-scale data...Big dataWork at office
Orion180
Irving, TX5 days ago - ...and risks on the project or closure of escalations. Required Skills and Experience 9+ years of experience in Data Engineering and Data Analytics. Expert in Big Data Technologies including Spark, Kafka, Kudu, Impala, HBase, and Spark Scala programming. Experience with GCP...Big dataFor contractors
- ...Implement and operationalize modern AI-enabled data capabilities on Google Cloud to ingest,... ..., and distribute data for a variety of big data apps Leverage AI/Agentic frameworks... ...5 plus years of experience in data engineering including hands‑on experience working with...Big dataH1b
$157k - $176.1k
...DescriptionU.S. Bank is seeking the position of Senior Software Engineer - Data Engineering in Irving, TX.Essential Responsibilities:The... ...reliability and proactive issue resolution; implementing scalable big data solutions, utilizing Spark for in-memory computation and...Big dataBank staffLocal areaWork from home- Job Title: Sr. DATA Engineer Work Location Irving TX Job Description Seeking a Senior professional with 3 to 5 years of experience in Python Scala and Spark SQL to design and optimize big scalable data processing solutions within the Databricks Spark SQL Scala Python...Big dataTemporary workLocal area
$125.76k - $188.64k
...Texas, United StatesSalary: $125 760,00 - $188 640,00Category: Technology, Applications Support, ProfessionalCompany: CitiThe Big Data Support Engineer Lead is a strategic professional who stays abreast of developments within own field and contributes to directional...Big dataFull timeShift work- ...Analytics solutionSQL experience: querying data and sharing what insights can be... ...verbal, and writing skills.Understanding of big data platforms and architectures, data stream... ...4 years of hands-on experience with data engineering requiredAdditional Knowledge and SkillsWorking...Big dataWork at officeFlexible hours2 days per week
- ...Title: Big Data Engineer Location: Dallas, TX (Hybrid) JD: Summary We are looking for an experienced Big Data Engineer with 7-10 years of hands-on experience to design, build, and maintain scalable data pipelines and processing systems in an on-premises...Big data
Wise Skulls
Dallas, TX4 days ago - ...GCP Senior Big Data Engineer/Architect Looking for a Senior Big Data Engineer/Architect on Google Cloud Platform to help strategize, architect and implement various solutions to migrate data hosted on our on-prem platform to Google cloud Platform (GCP). The architect...Big data
Samprasoft
Irving, TX10 hours ago - ...Hadoop - HDFS, Hive, MapReduce, HBase2+ years of experience with NOSQL database - HBase1+ years of working knowledge in Kafka/Storm data pipelines6+ years of experience in working in a Data warehouse environment4+ years of Linux skills2+ years in code documentation and...Big data
- ...environment and you take the time to celebrate both the small & big wins. INCLUSION is a core tenant of your personal beliefs. A... ...help you navigate our interview process.Lantern is looking for a Data Engineer II to join our Data Engineering team. The ideal candidate will...Big data
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!
Related searches
- aws data engineer Irving, TX
- big data devops engineer Irving, TX
- big data developer Irving, TX
- software data engineer Irving, TX
- data center engineer Irving, TX
- big data cloud engineer Irving, TX
- hadoop big data developer Irving, TX
- big data engineer Irving, TX
- senior data integration developer Irving, TX
- data developer Irving, TX