Big Data Engineer
Guru Schools
Overview: Job Title : Big Data Engineer Location : Irving TX (hybrid role) (Only local candidates) Duration : 12 Months Roles and responsibilities:
- Lead in design, development and testing of data ingestion pipelines, perform end to end validation of ETL process for various datasets that are being ingested into the big data platform.
- Perform data migration and conversion validation activities on different applications and platforms.
- Provide the technical leadership on data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
- Contribute to development of transformation logic, interfaces and reports as needed to meet project requirements.
- Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)
- Lead in design and develop validation framework and integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
- Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
- Assist Manager in project planning, validation strategy development
- Provide support in User acceptance testing and production validation activities.
- Provide technical recommendations for identifying data validation tools, recommend new technologies to improve the validation process.
- Evaluate existing methodologies and processes and recommend improvements.
- Work with the stakeholders, Product Management, Data and Design, Architecture teams and executives to call out issues, guide and contribute to the resolutions discussions.
- 8+ years of Software development and testing experience.
- 4+ years of Working experience on tools like Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce, etc.
- 4+ years of programming experience in Scala, Java or Python
- Experience in technical leading and mentoring the teams
- Experience with developing and testing ETL, real-time data-processing and Analytics Application Systems.
- Strong knowledge in Spark SQL, Scala code development in big data Hadoop environment and/or BI/DW development experiences.
- Strong knowledge in shell scripting Experience in Web Services - API development and testing.
- Experience with development and automated framework in a CI/CD environment.
- Experience with cloud environments - AWS or GCP is a plus.
- Knowledge of GIT/Jenkins and pipeline automation is a must.
- A solid understanding of common software development practices and tools.
- Strong analytical skills with a methodical approach to problem solving applied to the Big Data domain
- Good organizational skills and strong written and verbal communication skills.
- Working experience on large migration Projects is a big plus.
- Working experience on Google Cloud platform is a big plus
- Development experience for tools and utilities for monitoring and alert set etc.
- Familiarity with project Management and bug tracking tools, i.e., JIRA or a similar tool
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Big Data Engineer in Irving, TX vacancy
- ...Big Data EngineerLocation: RemoteRole SummaryExcellent Scala, PySpark, Java coding experience for REST API and SOAP webservicesExcellent AWS experience (Batch and API microservices) using EMR/EC2/ECS/S3/Airflow/Step Functions, AWS API GatewayExcellent Batch Application...Big data
United IT
Irving, TX4 days ago - Job Title: Sr. DATA Engineer Work LocationIrving TXJob Description:Seeking a Senior professional with 3 to 5 years of experience in Python Scala and Spark SQL to design and optimize scalable big data processing solutions within the Databricks Spark SQL Scala Python Java...Big data
- • Analyze and understand data sources & APIs• Design and Develop methods to connect & collect... ..., Payments Required Technical Skills • Big Data-Hadoop, NoSQL, Hive, Apache Spark•... ...web-APIs• Exposure to machine learning engineering• Exposure to NLP and text processing• Experience...Big data
- Job Title: Sr. DATA Engineer Work LocationTampa FLJob Description:Mandatory Karat InterviewRole Overview We are seeking a highly skilled and... ...AWS ecosystem The ideal candidate will have strong expertise in big data processing using PySpark and a deep understanding of data...Big data
- Job Title: Senior Specialist - Data EngineeringWork Location:Irving, TXJob Summary:Seeking... ...SQL to design and optimize scalable big data processing solutions within the Databricks... ...and data pipelines.Collaborate with data engineers data scientists and stakeholders to...Big data
$107.12k - $160.68k
...responsible for building, implementing, and supporting enterprise data and ETL systems in close coordination with the Technology... ...shell scripting.Banking domain experience is required.Exposure to Big Data technologies, AI/ML trends, and compliance or regulatory knowledge...Big dataFull time- ...SummaryWe are seeking a highly skilled and experienced Principal Data Engineer to join our team. As a Principal Data Engineer, you will be... ...techniques to enhance query performance and scalability. Leverage big data technologies and frameworks (e.g., Hadoop, Spark, Hive) to...Big dataFull timeLocal area
- ...Required Skills Data Science, Machine Learning, AI, Big Data/ Hadoop, Python, Cloud- AWS/GCP. What were looking for... You are an accomplished Data... ...Degree in Computer Science, Statistics, Math, Economics, Engineering or Data Science related fields with four or more years...Big dataWork experience placement
Rapinno Tech
Irving, TX4 days ago - OverviewThe Infosys Engineering unit is dedicated to amplifying product-centric value delivery... ...engineering solutions. We focus on adopting a data-first approach to engineering and value-... .../SQL Technical/Domain Skill 5Technology|Big Data|Oracle BigData Appliance Work...Big dataFull timeTemporary workRelocation
- ...are currently available and apply online.Job SummaryAs a Lead Data Engineer, you will be responsible for leading the design and implementation... ...tools such as Tableau or Power BI.•Experience with big data technologies such as Hadoop, Spark, or Kafka.•Experience gained...Big dataFull timeLocal area
- ...Data EngineerWe are seeking a highly skilled and motivated Data Engineer to play a pivotal role in designing, building, and optimizing our next-generation scalable data... ...sound and efficient data solutions.Qualifications:Big Data Frameworks Expertise: Demonstrated high...Big data
United IT
Irving, TX4 days ago - ...Job Title: Lead Data Engineer Location: Irving, TX - Hybrid Must have: ~ Expert in Databricks, PySpark and Azure data services... ...technologies such as Terraform or Pulumi. ~ Exposure to big data tools (Spark, Hadoop, Kafka) preferred. ~ Experience with...Big data
Jobs via Dice
Irving, TX1 day ago $157.41k - $207.8k
Senior Data EngineerAvaya LLC is seeking a Senior Data Engineer (Irving, TX) to spearhead the design, construction, and upkeep of our AI-enabled data pipeline. Collaborate... ...solutions. Enhance real-time data insights for big data processing and analytics. Support BI...Big dataLocal areaRemote workAvaya
Irving, TX3 days ago$125.76k - $188.64k
...ProfessionalCompany: CitiCiti is seeking a highly skilled and experienced Senior Data Engineer to join our dynamic and innovative technology team. The ideal... ...-native services for data storage, processing, and analytics.Big Data Technologies: Work extensively with big data frameworks...Big dataFull time- ...Job Title: GCP Data Engineer work Location: Irving, TX Duration: 8+ Months Job Description: Big data expert with 6+ years experience in Hadoop Big data ecosystem Spark - Batch & Streaming (Python,Scala ) Apache Kafka hands on experience...Big data
Ruri Software Technologies LLC
Irving, TX1 day ago - Data EngineerMandatory skills: Scala, Python, PySpark, Kudo, Kafka, and data engineering background. Experience in data engineer, data analytics; big data technologies expert in Python. Developer level expertise expected. Programming experience in Python, R programming,...Big data
Omni Inclusive
Irving, TX2 hours ago - Big Data EngineerLocation: Irving, TX (Day 1 Onsite-Hybrid)HexawarePrimary Skills: Hadoop, Hive & Spark FrameworkResponsibilities:Experience with big data processing and distributed computing systems like Spark.Implement ETL pipelines and data transformation processes....Big data
Intersources
Irving, TX2 hours ago - ...Sr. Data EngineerAt Orion180, we shine because of how we do it. Our mission is to deliver... ...production-ready assets for real-time analytics engines, portfolio risk platforms, and client-... ...pipelines in Azure Data Factory.Big Data Compute: Proven experience processing...Big dataWork at office
Orion180
Irving, TX2 days ago - ...Role : Data Engineer 3: Location: Chandler, AZ / Charlotte, NC / Minneapolis, MN / Irving, TX (Hybrid) Job Type: W2 Contact... ...Cloud to ingest, transform, and distribute data for a variety of big data apps. Leverage AI/Agentic frameworks to automate data...Big data
Stellent IT LLC
Irving, TX2 days ago - ...Job Description Job Description Job Title: Data Engineer – MEM SQL Location: New Jersey / Irving, TX / Tampa, FL Job Description:... ...Experience with cloud platforms (AWS / Azure / GCP) Knowledge of Big Data technologies (Spark, Kafka, etc.) Telecom domain...Big data
Qode
Irving, TX14 days ago $125k - $140k
...Description We are looking for a highly skilled and passionate Data Engineer with strong Python/PySpark experience and understanding of AL/ML... ...with ML Ops/LLM Ops and distributed systems. Experience with Big Data platform like Cloudera Hadoop and Could platforms like AWS...Big data- ...Lead Data EngineerLennar is one of the nation's leading homebuilders, dedicated to making... ...FutureThe primary mission of the Lead Data Engineer role is to help our business evolve into a... ..., Cloudwatch etc.AWS certification is big plusSnowflake Data CloudAccount AdministrationVirtual...Big dataLive in
Lennar
Irving, TX2 days ago $55 - $60 per hour
...AI-Enabled Data Engineer (GCP)Duration: 12+ Months ContractLocation: Chandler, AZ | Charlotte, NC | Minneapolis, MN | Las Colinas, TX (Hybrid... ...to ingest, transform, and distribute data for a variety of big data appsLeverage AI/Agentic frameworks to automate data management...Big dataHourly paySaxon Global
Irving, TX2 hours ago- ...Position: Big Data Developer Location: Irving, TX Duration: Long Term Responsibilities: 1). Develop software using Hadoop technologies like Spark, Scala, Python, Hbase, Hive, Cloudera. 2). Fine tune application performance...Big data
3B Staffing LLC
Irving, TX12 hours ago - ...expanding our efforts into complementary data technologies for decision support in areas... ...execution. Other data sets support search engine-based analytics. Exploration and deployment... ...Journey Analytics solutions Understanding of big data platforms and architectures, data...Big dataWork experience placement
$125.76k - $188.64k
...autonomous individuals. We are seeking a Senior Data Developer /Lead to join one of our small,... ...will possess deep technical expertise in big data technologies, including real-time... ...diverse data requirements from engineers and scientists into detailed technical specifications...Big dataFull time$141.34k - $144.2k
...Health Company, is hiring for the following role in Irving, TX: Data Engineer to Analyze data engineering problems and develop, build and... ...requires two (2) years of experience with each of the following: “Big data” platforms including: Azure, Amazon Web Services (AWS), or...Big dataHourly payFull timeTemporary workWork at officeLocal areaRemote work- NICE CXone Cognigy Aentic AI DeveloperLocation: RemoteJob Description:Minimum 3 years of experience in Conversational AI, Virtual Agent, Chatbot, Bot Development, or Contact Center AI platforms.Hands-on experience with Cognigy.AI and NICE CXone / NICE Mpower Agentic AI...Big data
United IT
Irving, TX4 days ago $125.76k - $188.64k
...Texas, United StatesSalary: $125 760,00 - $188 640,00Category: Technology, Applications Support, ProfessionalCompany: CitiThe Big Data Support Engineer Lead is a strategic professional who stays abreast of developments within own field and contributes to directional...Big dataFull timeShift work- ...Analytics solutionSQL experience: querying data and sharing what insights can be... ...verbal, and writing skills.Understanding of big data platforms and architectures, data stream... ...4 years of hands-on experience with data engineering requiredAdditional Knowledge and SkillsWorking...Big dataWork at officeFlexible hours2 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Big Data Engineer. Be the first to apply!
Related searches
- aws data engineer Irving, TX
- data science developer Irving, TX
- sr information security engineer Irving, TX
- senior data engineer Irving, TX
- data engineer machine learning Irving, TX
- junior data engineer remote Irving, TX
- data center engineer Irving, TX
- data developer Irving, TX
- senior data center engineer Irving, TX
- big data developer Irving, TX