Big Data DevOps Engineer (Hadoop & AWS EMR) Client: Lending Club

Senior Big Data DevOps Engineer

Must have experience designing, building, scaling and maintaining the infrastructure in datacenter and AWS Big Data analysis tools and machine learning applications.

• Accountability for the end-to-end availability, efficiency, reliability and security of our Big Data infrastructure in AWS.
• Ensure that our developers & analysts are central to the experience of our platform.
• Build custom tools that automate day-to-day operational tasks on the new AWS platform.
• Provide input and accountability for setting the standards of our production environment.
• Agile and flexible when managing multiple and changing priorities.
• Participate in 24X7 on-call rotation investigating and responding to issues.

Requirements:

• Must have deep understanding of Hadoop ecosystem like HDFS, Yarn, Hive, Spark, Oozie, Presto and Ranger
• Excellent knowledge in AWS including EMR, RDS and good understanding on IaaS, PaaS
• Experience in infrastructure automation using Terraform
• Demonstrated proficiency in Bash & Python, Java
• Good understanding of all aspects of JRE/JVM and GC tuning
• Experience providing and implementing observability solutions based on "Logs, Metrics & Graphs AWS Cloud watch, Splunk, Elastic stack, Prometheus, Grafana etc.
• Hands-on experience with RDBMS (Oracle, Mysql) and NoSql (Cassandra, MongoDB)
• Good grasp of the basics of SQL (joins, etc.)
• Conduct post-incident reviews and follow-up RCA resulting from platform issues.
• Experience with configuration management and remote execution tools such as Chef, Puppet, SaltStack, etc.
• Experience with Airflow, Qubole and Hortonworks is a plus
• Experience with cloud native Bigdata platforms like Databricks, Snowflake etc., is a plus.