Senior Big Data Engineer

Role Description

The Senior Big Data Engineer is responsible for designing, building, and delivering highly scalable big data and ETL solutions across distributed environments. This role supports projects at various and unanticipated client worksites throughout the United States, with headquarters based in Chicago, IL. The engineer works closely with cross-functional teams to develop, test, and deploy data solutions that meet business and operational needs, leveraging modern big data frameworks, cloud platforms, and agile development practices. Telecommuting is permitted.

Job Responsibilities

• Design and implement highly scalable ETL applications on Hadoop and Big Data ecosystems.
• Develop new scripts, tools, and methodologies for streamlining and automating ETL workflows.
• Deliver big data projects using Spark, Python, Scala, SQL, and Hive.
• Design and create use-cases and scenarios for functional testing, integration testing, and system testing.
• Work closely with Data Science, QA, Operations, and other teams to deliver on tight deadlines.
• Participate in daily agile and scrum meetings and code reviews.
• Coordinate with cross-functional operational teams for managing data delivery.
• Write efficient, reusable, and well-documented code.
• Prepare technical design documents for solutions.
• Identify and address issues encountered from the data factory and provide timely solutions to incorrect or undesired results or behavior with ILD solutions.

Technical Environment

• Writing ETL Spark applications in PySpark and Scala, Flume.
• Spark architecture, data frames, tuning Spark.
• Relational databases (Oracle, PostgreSQL).
• Python, SQL HQL, Hive.
• Data Bricks.
• Managing software systems using Hadoop, MapReduce, HDFS and all included services.
• Distributed computing principles.
• Big Data querying tools (Pig, Hive, Impala).
• Data-warehousing and data-modeling techniques.
• Core Java, Linux, SQL, scripting languages.
• Cloud platforms (Azure).
• Integration of data from multiple data sources.
• Lambda Architecture.

Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Computer Engineering, or related field plus 5 years of progressive experience as a Software Engineer/Developer or in Software Development required.
• Experience writing ETL Spark applications in PySpark or Scala.
• Experience with spark architecture, data frames, tuning spark.
• Experience with relational databases (Oracle, PostgreSQL).
• Experience with Python, SQL HQL, Hive.
• Experience with Data Bricks.
• Experience managing software systems using Hadoop, MapReduce, HDFS and all included services.
• Experience with distributed computing principles.
• Experience with Big Data querying tools (Hive).
• Experience with data-warehousing & data-modeling techniques.
• Experience with Core Java, Linux, SQL, scripting languages.
• Experience with Cloud platforms (Azure).
• Experience with integration of data from multiple data sources.
• Experience with Lambda Architecture.
• Telecommuting permitted.

Circana Behaviors

• Stay Curious: Being hungry to learn and grow, always asking the big questions.
• Seek Clarity: Embracing complexity to create clarity and inspire action.
• Own the Outcome: Being accountable for decisions and taking ownership of our choices.
• Center on the Client: Relentlessly adding value for our customers.
• Be a Challenger: Never complacent, always striving for continuous improvement.
• Champion Inclusivity: Fostering trust in relationships engaging with empathy, respect, and integrity.
• Commit to each other: Contributing to making Circana a great place to work for everyone.

Location

This position can be located in the following area(s): US Remote.

Compensation

The salary range for this role is 160,500 USD to 200,000 USD. This job is also eligible for bonus pay.

Benefits

• Comprehensive package of benefits including paid time off.
• Medical/dental/vision insurance.
• 401(k) to eligible employees.

Application Process

You can apply for this role through the Circana careers website or Intranet site for internal candidates.