Senior AWS Data Engineer

Meet The TeamHitachi Vantara is looking for an experienced Senior AWS Data Engineer to be a part of, the Enterprise Data Platform team. The Enterprise Data Platform team is responsible for the data lake that powers a lot of the analytics at Hitachi Vantara. The Senior Data Engineer will be building and testing data pipeline architectures within the AWS cloud environments, as well as identifying, designing, and implementing internal process improvements for automating manual processes, optimizing data delivery and re-designing infrastructure for greater scalability.For this role, the Senior Data Engineer will be highly focused on working on data ingestion, extracting data from different sources, and making it available for analytics and reporting, using Python and other tools. This role will require working with a combination of new initiatives and development, as well as supporting current production systems built by the team. An important part of this role will be to provide the team guidance on best practices while developing code in Python for AWS cloud services. In addition, the Senior AWS Data Engineer will look at constantly improving the code and infrastructure in line with industry best practices. If you enjoy working within data and cloud environments, apply today! What You Will Be DoingBuild, test, maintain data pipeline architecture in the AWS cloud environment.Assemble large, complex data sets that meet functional / non-functional business requirements.Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and cloud-based big data technologies.Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.Working knowledge of REST and implementation patterns pertaining to Data and Analytics.Build processes supporting data transformation, data structures, metadata, dependency, and workload management.Working knowledge of message queuing, stream processing, and highly scalable big data stores (Kafka, Kinesis, Storm) and ETL orchestration tools such as Airflow.Working on cloud-based big data automation and orchestration solution AWS/Azure DevOps - CI/CD.What You Bring To The Team Mandatory Requirements: 6+ years of Python/Spark/SQL programming experience5+ years of Data Ingestion experience: Ingestion from various sources (REST, data systems, incremental data, etc.) both on-prem and in the cloud.Ingestion- use of AWS services (Lambda, Athena, S3, Glue or Redshift) required.Must have experience loading data from REST API's and database systems.Must have experience with ETL's using Glue.Knowledge of AWS DevOps - CI/CD.Nice to Have Requirements: Production experience within Salesforce.DevOps pipeline in AWS using CloudFormation.Creating conceptual, logical and physical data modelsEnsure data is transferred from its source of origin provided by the data steward (when available)Understand the business use of the information being transferred to the data warehouse to support adaptable logical and physical data modelsDevelop and implement a metadata management strategy (first priority is to harness metadata for lineage and transformation information -including standards for data asset naming conventions)Ability to encrypt and mask dataAbility to write requirements and translate business needs into technical requirementsDocument repository for requirements, technical architecture, and metric derivationIdentify the most used data entities in the environment, what teams are using them and how they are being usedIdentifying access approval requirements and management based on data classificationStandardize naming conventions in reports aligned to business terminology & definitionsPowerBI standards so they are not used as data marts As required by the equal pay and transparency acts, the expected base salary for this position is: Tier One Location (including New York City and California Bay Area): $120k -$160k Tier Two Location (including Colorado, Seattle, and the rest of California): $110k - $150k Tier Three Location (including the rest of Washington State): $100k - $130k The expected pay is determined based on a variety of factors including, but not limited to, depth of experience in the practice area. Employees are eligible to participate in Hitachi Vantara's bonus/variable/commission pay programs, where applicable, and are subject to the program's conditions and restrictions. Our CompanyHitachi Vantara is part of the Global Hitachi family. We balance innovation with an open, friendly culture and the backing of a long-established parent company, known for its ethical reputation. We guide customers from what's now to what's next by unlocking the value of their data and applications to solve their digital challenges, achieving outcomes that benefit both business and society.Our people are our biggest asset, they drive our innovation advantage as we strive to offer a flexible and collaborative workplace where they can thrive. Diversity of thought is welcomed, and our employee base is represented by several active Employee Resource Group communities. We offer industry leading benefits packages (flexible working, generous pension, and private healthcare) and promote a creative and inclusive culture. If driving real change gives you a sense of pride and you are passionate about powering social good, we'd love to hear from you.Our ValuesWith Japanese Roots Going Back Over 100 Years, Our Culture Is Founded On The Values Of Our Parent Company Expressed As The Hitachi SpiritWe are proud to say we are an equal opportunity employer and welcome all applicants for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran, or disability status.Wa - Harmony, Trust, RespectMakoto - Sincerity, Fairness, Honesty, IntegrityKaitakusha-Seishin - Pioneering Spirit