Lead Data Engineer

Data Integration And Ingestion Specialist

Responsible for technical design, writing, support and operations of efficient, scalable, and extensible data integrations and ingestions. Implement effective solutions from the ground up and investigate anomalies in a mature running data platform and environment. Work with business, developers, analysts, architecture and product owners to develop data integration and ingestion solutions that are critical to the enterprise analytics, reporting and application environments leveraged by all areas of the business. Perform all data platform related configuration, software and server installations, upgrades and patching partnering with the necessary Infrastructure, IT Security and other teams as needed.

Major Responsibilities:

• Implement enhancements for prioritized data integration and ingestions work efforts partnering with the business to ensure they meet the specified requirements.
• Implement critical and non-critical system data integration and ingestion fixes for the data platform and environment. Ensuring root cause resolution to identified problems. Monitor and support the campaign management environment and processes in order to meet the daily SLA.
• Recommend and code efficient and effective data integration and ingestion solutions for challenging problems for medium to large work efforts of medium to high complexity. Translate business requirements, data mappings into data integrations and ingestions code solutions.
• Support Release and Change Management processes along with other UPT Compliance initiatives. Work effectively with Technology Operations/Infrastructure teams to ensure continued operations and maintenance of the data integrations and ingestions as well as the data platform and environment.
• Understands and actively participates in Environmental, Health & Safety responsibilities by following established UO policy, procedures, training and team member involvement activities.
• Performs other duties as assigned.

Education:

• Technical degree, Bachelor's degree; or highly relevant equivalent experience is required.

Experience:

• 7+ years of demonstrated experience working as part of large Information Technology teams and/or consulting organizations partnering with clients/business groups in a role developing and supporting a complex data environment; or equivalent combination of education and experience. Strong understanding of software development lifecycle
• Experience in a transaction-based environment is a MUST
• Proficient with scripting languages, data ingestion tools and corresponding tools, preferably Python, Pip, Pipenv, Bash, Scala, Spark, PySpark ScalaSpark, NIFI, and Java Script, and one object-oriented language like Java/C#
• Able to work in and with Cloud and non-cloud technologies like AWS, Azure, Hadoop etc.
• Experience with one or more relational databases/data technologies like DB2, SQL Server, Snowflake. Able to write and ready complex SQL, stored procedures and leverage backend databases. Should be able to triage database performance issues with SQL being sent to the database. Able to read and explain plans, and understand the complexity.
• Experience working with scheduling, automation, file transfer, monitoring, alerting, notification and logging tools like Airflow, RunDeck, SFTP etc.
• Experience working in Linux environment
• Experience with source control management such as Git and/or SVN
• Strong problem solving skills, and the ability to communicate complex concepts clearly and accurately
• Familiarity with no-SQL databases like Couch, MongoDB, Hbase or Cassandra
• Knowledge of Data Science, Analytics and Business Intelligence/Reporting practices and tools as it pertains to data platforms and environments (R-Studio, Jupyter, Zeppelin, Tableau or DsX)
• Experience interacting with Kafka and API/Service programming and transactional databases
• Experience with principles of data governance and quality as well as tools used (i.e. Ranger, Atlas)
• Understanding of containers, resource management or YARN is desirable
• Able to perform software and server installations, patching and upgrades.
• Participate in an on-call rotation and available to work off-hours, holidays and weekends.
• Must be able to partner with other teams and developers effectively.
• Strong knowledge of systems development and project management Agile methodologies/processes.
• Demonstrated ability to produce high quality technical documentation.
• Able to quickly meet changing business requirements.
• Able to complete work items, validate the results and present deliverable to the business for review with minimal to no defects.
• Able to understand and interpret Entity-Relationship, logical and physical data models.
• Work with change management processes to ensure smooth migration between dev, QA and production environments.
• Monitor and maintain all the system processes.
• Ensure system issues are resolved in a timely manner.
• Strong interpersonal skills, with a demonstrated ability to make effective decisions while working through complex system issues.
• Must be able to utilize and effectively communicate functional and technical components of an initiative to applicable parties both verbally and through documentation.
• Coordinate the necessary operations documentation and standards required for compliance and identified best practices.
• Problem and Change Management ITIL ServiceNow toolset in tracking and providing status on support work activities.
• Attention to detail, good analytical and problem solving skills and critical thinking.
• Self-starter/motivator with a proactive, agile and strategic mindset. Able to work independently.
• Provide work estimates on break-fixes and enhancements.
• Facilitate and lead meetings with end users to discuss issues and open work items.
• Keeps management, team members and business stakeholders informed of critical issues, status, changes, etc.
• Willingness to learn new technologies, tools, and approaches to problem-solving

Your talent, skills and experience will be rewarded with a competitive compensation package.