PySpark Data Engineer — ETL & Data Lake Architect
VSG Business Solutions LLC
VSG Business Solutions LLC in New York seeks a Data Engineer to build and manage ETL pipelines for a Data Lake and Snowflake environments. You will design scalable data processing solutions and optimize Spark/PySpark applications in a distributed data platform. Strong Python and live-coding skills are essential, along with hands-on experience in ETL, Control-M, Airflow, and AWS Big Data services. Java is not required, scripting knowledge is welcome. #J-18808-Ljbffr VSG Business Solutions LLC
Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the PySpark Data Engineer — ETL & Data Lake Architect in New York, NY vacancy
- ...and be available on EST hours. Senior Data Architect/Engineer Summary of Role We are seeking an... ...scalable data warehouse, data lake, and lakehouse architectures. Establish... ...Design, develop, and maintain scalable ETL/ELT pipelines that ingest, transform...SuggestedFull timeRemote work
TeamBuilder
New York, NYmore than 2 months ago - BSL Consulting seeks a senior data engineer to design and manage large-scale data pipelines on AWS. You will build data lakes, data warehouses, and ingestion architectures, while guiding a team of engineers to deliver scalable analytics solutions. Ideal candidates have...SuggestedFull time
- Capgemini is looking for an experienced AWS Data Engineer to join their team in New York, NY. The role involves designing, developing, and maintaining... ...should have a robust background in data engineering, including ETL development and cloud technologies. Additionally, they should...Suggested
- ...Technology Enabled Services.Job DescriptionThe Data Lake engineer should come from the background of a Big Data Architect. The position will be in support of data management... ...is required.Knowledge of other industry ETL tools (including No SQL) such Cassandra/Drill/Impala...SuggestedPermanent employmentFull timeH1bFlexible hours
- AgileEngine is seeking a Senior Data Engineer to design and build scalable data lakes, warehouses, and lakehouse architectures supporting a thematic research platform. You will implement Python-based ETL/ELT pipelines, orchestrate with Airflow, and work with Snowflake,...SuggestedRemote jobFlexible hours
- ...across a federated network of data, and interact across the network... ...is looking for talented Data Engineers who are passionate about data... ...including dataset evaluation, ETL pipelines, and continuous data... ...Experience with different data lake technologies (AWS Data Lake,...
- What success looks like in this role: The Sr. Data Lake Support Engineer will support database operations, ETL processes, cloud services, and enterprise applications within a secure environment. This role combines SQL development, database administration, Azure support...
$97.17k - $126.32k
...that mindset.About the Role:The Data Engineer designs, develops, and... ...Develop and maintain robust ETL (Extract, Transform, Load) and... ...that leverage SQL, Python, and PySpark to support enterprise data processing... ...Data Factory, Azure Data Lake Storage, Azure SQL, SQL...Full timeLocal areaVisa sponsorship$141.9k - $190.3k
Job Posting Title:Sr Data EngineerReq ID:10147075Job... ...global organization of engineers, product developers,... ...and maintain scalable ETL/ELT pipelines for acquisition... ...using Databricks, PySpark, SQL, and Unity Catalog... ...Databricks, including Delta Lake, Unity Catalog, data...Full time- ...full-time position for a Principal Data Engineer / Solution Architect specializing in Microsoft Fabric and... ...ADLS Gen2, Databricks Unity Catalog, PySpark in Fabric, and related Azure ecosystem... ...~ Deep familiarity with Azure Data Lake Storage Gen2 (ADLS Gen2) , including...Full timeTemporary workWorldwideShift work
Fusemachines
New York, NY10 days ago - ...career in Advisory.KPMG is currently seeking a Sr. Associate, Data Science Engineer to join our Consulting practice.Responsibilities:Translate... ..., deployment, and MLOps, with hands-on expertise in Python, PySpark, SQL, Databricks, and Snowflake, while partnering with clients...H1bLocal area
- ...Senior AWS Data EngineerSoftware Guidance... ...a Senior AWS Data Engineer for a contract assignment... ...and AWS Data Lake solutions supporting... ...software engineers, architects, and business stakeholders... ...optimize scalable ETL/ELT pipelines using... ...Functions, EMR, PySpark, and related AWS...Contract workRemote work
SGA
New York, NY5 days ago $168k - $252k
...in large-scale IT and engineering, we combine market-leading... ...spanning technology, data sciences, consulting and... ..., with a focus on ETL, ELT, ALT, lambda, kappa... ...SQL, DBT, Python, Spark, PySpark, Java, JavaScript, Scala... ...notebooks, jobs, Delta Lake, orchestration, optimization...Flexible hours$112.29k - $161.28k
...meets media scale, where data becomes direction,... ...of data warehouse/data lake solutions utilized by internal... ...solutions include ETL, data modeling, data architecture... ...and mentorship to data engineers and analystsAct as a... ...: SQL, Python, Spark (PySpark), JavaETL / Integration...Temporary workFreelanceFlexible hours- A leading mental health service in New York is seeking a Senior Data Engineer. This role includes designing scalable data architectures, managing large datasets, and leading ETL processes. The ideal candidate will have 4-6 years of experience in data engineering, with proficiency...Remote job
- Kinect, a leading private markets investment firm, seeks a Senior Data Engineer to design and scale the data infrastructure supporting its... ...jurisdictions, enabling trusted insights. The role emphasizes scalable ETL pipelines, data quality, governance, and the use of AI to...
- A mental health technology service in New York is seeking a Senior Data Engineer to lead the design and development of scalable data architectures. This role involves managing ETL processes, collaborating with teams to support data-driven decision-making, and enhancing...Work at officeRemote workFlexible hours2 days per week1 day per week
$175k - $250k
Millennium is seeking a Commodities Content Engineer based in New York to develop robust ETL workflows on their commodities data platform. The role involves using Python and SQL for data processing and analytics collaboration with various teams. Candidates should have...- A leading technology firm is seeking a Data Engineer to design and develop scalable data architectures for advanced intelligence analysis. The role involves implementing ETL workflows, ensuring data quality, and collaborating with analysts to meet mission needs. Candidates...
$135k
...together the rigor of data-driven analysis with the... ...seeking a Senior Data Engineer with deep expertise in... ...grade data pipelines and ETL/ELT processes that feed... ...e.g., warehouses, data lakes, and data models) that... ...engineering libraries (e.g., PySpark, Apache Airflow) and...Permanent employmentH1bLocal areaWorldwideHome office- Mandatory Skills Azure Data Bricks Azure Data Lake Python Pyspark Desired Skill: Bloomberg data acquisition and transformation Design, develop, and maintain... ...Databricks platform. Implement data integration and ETL processes using Azure Data Factory. Develop and optimize...
- Greetings ! Role:Senior Data Engineer Experience:10+ Years... ...and optimize scalable ETL/ELT pipelines using Python, SQL, and PySpark. Build batch and real-... ...solutions. Collaborate with architects, analysts, and data... ...DevOps) Git Docker Data Lake / Lakehouse...Remote work
$180k - $225k
A Career with Point72’s Data Engineering Technology Team As Point72 reimagines the future... ...maintain robust data pipelines and ETL workflows in Databricks to support quantitative... ...in Databricks, including Spark (PySpark or Scala), Delta Lake, and notebook-based development...Work experience placement- Data Engineer (601/602/603) Must-Have Skills: Strong Python programming (core Python, not just PySpark) Expert-level Apache Spark / PySpark Hands-on ETL development with large-scale data pipelines Strong Data... ...(EMR, Glue) Snowflake Data Lake Control-M Airflow Strong software...
- An established industry player is seeking a skilled data engineer to design and maintain scalable data pipelines using Azure technologies. This... ...sources into the Azure Databricks platform, implementing ETL processes, and ensuring data quality and integrity. The successful...
- We are seeking a Data Engineer with 5+ years of experience in designing,... ...Databricks, Apache Spark, and ETL/ELT development, along with... ...Spark. Develop and maintain data lakes and cloud-based data... ...experience with Apache Spark (PySpark preferred). Experience building...
- Capgemini in New York, NY seeks an experienced AWS Data Engineer to design, build, and optimize scalable data pipelines on AWS. You will work... ..., SQL and Python/Scala/Java, and knowledge of data modeling, ETL/ELT, and Spark/Hadoop. Capgemini offers a comprehensive benefits...
- ...an experienced Senior Data Engineer with 8+ years of hands... ...expertise in Python/SQL, ETL/ELT, cloud data... ...data warehouses, data lakes, lakehouses, and dimensional... ...such as Apache Spark/PySpark . Develop data... ...Collaborate with data architects to translate business...
- Create/manage ETL pipelines for the data lake and Snowflake; automate data analysis/aggregation work. Design... ...data movement. Must-Have Skills: PySpark (Strong hands-on experience) AWS... ...required. Ideal Candidate: Senior Big Data Engineer with strong PySpark, AWS Glue/EMR,...Live in
- Motion Recruitment Partners is seeking a Data Engineer for an onsite role in Arlington, VA. The ideal candidate will have 3-5 years of experience... ...initiatives. Experience with PostgreSQL, SQL, Python, ETL processes, data replication, and reporting tools is preferred....
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to PySpark Data Engineer — ETL & Data Lake Architect. Be the first to apply!
Related searches
- junior data engineer remote New York, NY
- director data engineering New York, NY
- hadoop big data developer New York, NY
- data engineer intern New York, NY
- senior data quality engineer New York, NY
- senior cloud data engineer New York, NY
- data engineer contract New York, NY
- data infrastructure engineer New York, NY
- junior data developer New York, NY
- associate data engineer New York, NY