Senior Data Engineer

Job Description We are seeking a Senior Data Engineer who is experienced with data pipelines and data wrangling to join a nationally recognized renewable energy provider. You will expand and optimize existing data infrastructure, working closely with IT and Data Science team members. This person ideally has previous experience as a Software Engineer and can work effectively and communicate between the two disciplines. Note: The company is unable to offer visa sponsorship. The Senior Data Engineer will: Create, maintain, and enhance the infrastructure for data extraction, transformation, and loading (ETL), and API integrations. Develop robust data pipeline operations, ensuring scalability and reliability. Automate manual tasks, fine-tune data delivery mechanisms, and optimize data storage and queries to minimize costs within GCP. Design and implement error logging and process monitoring solutions, leveraging Google Cloud Platform technologies. Innovate internal processes by adopting top-tier DevOps practices. Refine version control infrastructure, implement GitHub Actions for automated processes, and establish CI/CD pipelines. Transition our bash processes to a containerized framework using Google Cloud Platform's CloudRun service. Proactively address challenges, supporting the team with process testing, bug fixes, and code reviews. Enhance our API using best practices in data architecture, engineering efficiency, and data security. Collaborate with the Analytics team to implement machine learning infrastructure. Maintain open-source software supporting clean and renewable energy initiatives. Strengthen IAM (Identity and Access Management) policies and collaborate on network security measures. Conduct regular security audits and risk assessments. Monitor security logs and alerts, responding to suspicious activities. Ad hoc projects and duties, as needed. Here’s what you’ll need to succeed A bachelor's degree in computer science or another engineering / data-intensive discipline (MS preferred). At least 5 years of relevant professional experience, including building, maintaining, and optimizing ‘big data' pipelines. Strong experience with Bash Scripting, Python, Linux operating systems, and relational databases, ideally GCP BigQuery. Demonstrated success maintaining and/or developing data or software engineering infrastructure for data analysis and sharing, and machine learning. Front end (TypeScript, Vue3, Vuetify3, Google Maps / DeckGL) and backend (Golang) software development. Google Cloud Services, such as GCP, or other cloud platform experience. Solid understanding and experience with CRMs and the backend infrastructure, preferably Salesforce, Salesforce Bulk Python API. Extensive experience with batch-processing systems, data pipeline and workflow management tools (e.g. Luigi, Airflow), containerized applications and container based orchestration environments (e.g., Docker, Kubernetes), and infrastructure as code, such as Terraform. Experienced with version control tools, such as Github. #J-18808-Ljbffr