Lead Data Engineer-PySpark, Redshift, Airflow, AWS
Zortech Solutions
Lead Data Engineer
We are looking for senior candidates with at least 12+ years experience. We are looking for tech experts who can lead the project as well with good communication and stakeholder management skills. Must have onshore-offshore coordination experience.
Here are the key expectations from the tech perspective: Primary skills are PySpark, RedShift, Airflow, AWS
Location: 777 Old Saw Mill River Rd, Tarrytown, NY 10591 (100% onsite role) – Please share profile local to NY/NJ only.
Duration: 6+ Months
Job Description:
- Candidate should have 12+ years of experience in Data Engineering. Must have strong work experience with onshore-offshore model
- Designing, creating, testing and maintaining the complete data management & processing systems.
- Candidate need to have in depth understanding of how data pipelines are built
- Typical challenges with fetching data from various sources. How incremental/CDC data flows are handled.
- How do you ensure data quality
- How do you do Data profiling
- Hands-on experience with PySpark, Redshift (SQL) and Airflow at minimum
- Strong hands-on with required tech skills, flexible, right attitude to play the lead role
- Should be able to design and document data model at various levels
- Working closely with the stakeholders.
- Building highly scalable, robust & fault-tolerant systems.
- Knowledge of Hadoop ecosystem and different frameworks inside it – HDFS, YARN, MapReduce, Apache Pig, Hive, Flume, Sqoop, ZooKeeper, Oozie, Impala and Kafka
- Must have experience on SQL-based technologies (e.g. MySQL/ Oracle DB) and NoSQL technologies (e.g. Cassandra and MongoDB)
- Should have Python/Scala/Java Programming skills
- Discovering data acquisitions opportunities
- Finding ways & methods to find value out of existing data.
- Improving data quality, reliability & efficiency of the individual components & the complete system.
Vacancy posted 5 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer-PySpark, Redshift, Airflow, AWS in Tarrytown, NY vacancy
- ...Client: Persistent Role: Lead Data Engineer-PySpark, RedShift, Airflow, AWS Location: 777 Old Saw Mill River Rd, Tarrytown, NY 10591 (100% onsite role) - Ideal to look for local profiles. Duration: 6+ Months Job Description:...Amazon Web ServiceWork experience placementLocal areaFlexible hours
Zortech Solutions
Valhalla, NY3 days ago - ...Onsite Lead Data Engineer Location: Tarrytown, NY (100% onsite) Duration: 6+ Months Job Description: Location... ...in Real-time processing Framework (Apache Spark), PySpark and in AWS Redshift, Apache Airflow and EMR Must have experience on SQL-based technologies...Amazon Web ServiceWork experience placementLocal area
Zortech Solutions
Tarrytown, NY5 days ago - ...Lead Data Engineer Location: Tarrytown NY 10591 (Hybrid role – 3 days onsite 2 days WFH) Duration: 6-12+ Months Job Description: Must have AWS, Apache Airflow, Pyspark, Redshift Candidate should have 12+ years of experience in Data Engineering Designing...Amazon Web ServiceWork experience placementWork from home
Zortech Solutions
Tarrytown, NY5 days ago - ...Overview Role: Data Architect Location: Tarrytown NY 10591 (100% Onsite) Skills: Spark, PySpark, AWS Redshift, Airflow, EMR, Python Job Description: Candidate should... ...12+ Years of experience in data engineering architecture for large-scale platforms...Amazon Web ServiceContract workWork experience placement
E-Solutions
Tarrytown, NY2 days ago - ...requirement for a A WS Data Architect contract... ...of our client. AWS Data Architect... ...NY Skills: PySpark, AWS Redshift, EMR, Databricks, Python... ...of experience in data engineering architecture for large... ...AWS Redshift, Apache Airflow and EMR etc Strong...Amazon Web ServiceHourly payContract workWork experience placementLocal areaImmediate start
E-Solutions
Tarrytown, NY4 days ago - ...Data Engineer Global Development is embarking on a Digital Transformation... ...' experience developing and leading the implementation of data... ...platforms (e.g., Snowflake, Redshift, BigQuery, Databricks) and programming... ...in cloud architecture (e.g. AWS, Azure, GCP) and DevOps...Amazon Web ServiceRemote workWorldwide
Regeneron
Tarrytown, NY3 days ago - Kforce Inc is looking for a skilled data engineer to build, enhance, and optimize Palantir Foundry pipelines. The ideal candidate will have... ...strong background in developing scalable data pipelines using PySpark and Databricks, with experience integrating large-scale...
- ...talented Team. Job Title: Senior Data Engineer Location(s): White Plains, NY... ...Design and implement data pipelines across AWS, Azure, and Google Cloud. - Develop SAP... ...implementations. - Familiarity with Python, PySpark for data processing and automation. -...Amazon Web Service
Ampcus
White Plains, NY5 days ago - ...looking for a skilled SQL Senior / Data Engineer + Power BI (L2) . Join our... ...on a project with a leading transportation and logistics... ...data pipelines across Azure and AWS environments Write and execute... ...basics Pipeline tools: Apache Airflow or Azure Data Factory — DAG/...Amazon Web Service
- ...A leading biotechnology company in New York is seeking a Process Development Engineer III to innovate data platforms and optimize bioprocess workflows. The ideal candidate will have strong programming skills and experience in bioprocess development. Responsibilities include...
BioSpace, Inc.
Tarrytown, NY2 days ago - ...have to offer our associates, please click here. Overview The Data Engineer, Research Informatics plays a critical role in enabling data-driven... ...data transformation and pipeline development. Experience with AWS (S3, compute services), Git, and DevOps workflows....Amazon Web Service
ViziRecruiter
Yonkers, NY5 days ago - ...Kforce has a client in Armonk, NY that is seeking an EMS Data Engineer to support the design, integration, and optimization of data pipelines... ...-on experience with cloud platforms such as Microsoft Azure, AWS, or GCP Familiarity with time-series data and industrial protocols...Amazon Web ServiceHourly payContract workPart time
Kforce
Armonk, NY5 days ago $128.6k - $210k
...Principal Software Engineer The Regeneron Genetics Center is one of the largest... ...and work with large-scale genetic data. Cloud-based data pipeline work on AWS, keeping high-volume genomic data... ...& Databases: PostgreSQL or MySQL, Redshift or Snowflake, ETL orchestration...Amazon Web ServiceLocal areaRegeneron
Tarrytown, NY5 days ago- Overview The Senior Data Engineer role within the data engineering vertical will play a pivotal role at the forefront of our data and analytics... ...techniques, and performance analysis & tuning. Experience with AWS Cloud, S3, DevOps and Git management. Experience with EPIC...Amazon Web Service
- 2000 Montefiore Health System, Inc. is seeking a Data Engineer in Yonkers, NY to enable data-driven clinical research. The role primarily involves... ...in data engineering, strong SQL skills, and familiarity with AWS and cloud platforms. This position offers an opportunity to...Amazon Web Service
- ..., we are looking to the future of being a data driven organization and tech-enabled care... ...impact with the healthcare system. The Data Engineer role within the data engineering vertical... ...analysis & tuning. Experience with AWS Cloud, S3, DevOps and Git management. Experience...Amazon Web Service
- ...available cloud-based solutions and estimation on AWS and Azure Cloud. Experience in Azure Databricks and ADF, Azure Synapse and PySpark. Participate in pre-sales activities,... ...Experience with integration of different data sources with Data Warehouse and Data Lake...Amazon Web Service
The Dignify Solutions, LLC
Purchase, NY5 days ago - ...should have at least 7 years of experience in AI/ML, overseeing projects while collaborating with various stakeholders. A Bachelor's degree in a related field is required along with expertise in big data platforms such as AWS, Redshift, and Snowflake. #J-18808-LjbffrAmazon Web Service
New York Life
White Plains, NY2 days ago $90 - $100 per hour
...document processing (e.g., OCR, NLP, data extraction), and workflow... ...Pipelines: Tools like Apache Airflow, Kubeflow, or MLFlow... ...platforms like Apigee, Kong, or AWS API Gateway for secure and scalable... ...solution architecture, software engineering, or a related role. ~ Proven...Amazon Web ServicePermanent employmentFull timeTemporary workWork at officeRemote workHelix Tech IT Services
Tarrytown, NY2 days ago- ...Tibco Spotfire Bi Engineer Location: 777 Old Saw Mill River Rd, Tarrytown, NY 10591 (3 days onsite/week) -... ...Spotfire, Big Query, Iron Python, ELK, JavaScript, AWS Strong experience in BI Reporting tools and data integration, manipulation and presentation Data...Amazon Web ServiceLocal areaRelocation3 days per week1 day per week
Zortech Solutions
Tarrytown, NY5 days ago $127k
...Senior Data Engineer Job Locations US-NY-Rye Job ID 2026-8436 Category... ...capabilities. Reporting to the Lead Data Engineer, you will drive technical... ...strategy, execution plans) and Python (PySpark, pipeline frameworks, testing). ~ Deep...Work at officeLocal areaNew York Blood Center
Rye, NY7 days ago- ...ETL/Data Engineer Montvale, New Jersey, United States Orion Innovation is a premier, award... ...: We are seeking a Technical ETL Lead to design, build, and manage scalable data... ...experience with: Azure Databricks (Spark, PySpark) Data Lakes (Azure Data Lake Storage)...
Orion Innovation
Montvale, NJ10 days ago $60 - $80 per hour
...modern cloud architecture with complex biological data systems. Core Responsibilities Systems Implementation: Lead the integration and configuration of Commercial-... ...of Continuous Integration (CI) workflows. AWS Ecosystem: Practical experience with Lambda, Step...Amazon Web ServiceLinks Technology Solutions Inc
Tarrytown, NY2 days ago$160k - $200k
...Yonkers, New York Department: D&A - Data & Analytics Work Shift: Day (... ...- $200,000.00 Montefiore is a leading academic medical center... .... The Senior Manager for Data Engineering will play a pivotal role in building... ...products. Experience with AWS Cloud, S3, DevOps, Git, Oracle...Amazon Web ServiceShift work- ...looking to the future of being a data driven organization and tech-... ...The Senior Manager for Data Engineering is a hands‑on role that will play... ...external sources. You will lead our suite of enterprise data products... ...& tuning. Experience with AWS Cloud, S3, DevOps, Git...Amazon Web Service
- ...· Design, architect, and implement AWS IAM solutions , including policies, roles, permissions, and federation. · Provide strategic guidance on identity lifecycle management, authentication, authorization, and access governance. · Collaborate with security, cloud...Amazon Web ServicePermanent employmentContract workLocal area
ETHEREUM TECHNOLOGIES LLC
Sleepy Hollow, NY2 days ago - ...models tailored for biomedical data, optimizing training pipelines... .... MLOps & Deployment Lead the development of CI/CD... ...Collaborate with data scientists and engineers to optimize model performance... ...FastAPI, and cloud platforms (AWS, Azure, GCP). Experience...Amazon Web Service
United IT Solutions
Tarrytown, NY4 days ago - ...strategy, experience design, and engineering, with a unique combination of agility... ...We are seeking a Technical ETL Lead to design, build, and manage scalable data pipelines and modern data... ...with: Azure Databricks (Spark, PySpark) Data Lakes (Azure Data Lake Storage...
$150.5k - $245.5k
...The Data Enablement and Analytics (DEA) team, within the PAPD (Product... ...data marts. Partner with data engineering teams to design streaming and... ...product manager and program Lead – define our roadmap and... ...data platforms: Expertise in AWS, Snowflake, Databricks or comparable...Amazon Web ServiceLocal area- ...Workbooks and Transforms Develop and maintain Foundry Ontology objects, properties, and links Design and implement scalable data pipelines using PySpark and Databricks Develop workflow or platform logic using TypeScript Integrate large-scale datasets across Snowflake, ODS,...Hourly payContract work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer-PySpark, Redshift, Airflow, AWS. Be the first to apply!
Related searches