Databricks Data Engineer

Databricks Data Engineer

New York

About Indicium AI

Indicium AI is trusted by the world's leading enterprises to deliver AI into production at scale. We are a global AI-native consultancy with proven experience across Financial Services, Energy & Utilities, Healthcare & Life Sciences, Retail & CPG, and Manufacturing. From strategy, to build, to business outcomes, we unlock value from AI with unmatched clarity, speed, and capability.

Powered by 600+ AI experts serving 50+ enterprise clients from 5 global locations, we work side-by-side with top partners - including Anthropic, Databricks, AWS, OpenAI, and Microsoft - to deliver modern AI with speed and measurable impact.

The Opportunity

Responsible for the development, implementation, and maintenance of scalable, reliable, and high-performance data solutions. Will design and build data pipelines, ensuring the integration, security, and reliability of data from diverse sources to meet the needs of internal and external applications, as well as to support advanced analytics and strategic decision-making for the company.

Responsibilities

• Perform data ingestion/integration from various sources (relational and NoSQL databases, internal and external service APIs, files, and others) and ensure data quality and consistency.
• Implement data storage solutions (Data Warehouses, Data Lakes) and optimize the performance of data queries and processing.
• Managing data loading in distributed storage (whether relational or non-relational)
• Aggregate data using distributed tools that can handle large volumes of data.
• Design, develop, and maintain robust and efficient Electronic Logistics (ELT) pipelines using data engineering best practices.
• Ensure the entire ELT process is functioning correctly through monitoring and metrics such as SLAs.
• Monitor the execution of ELT applications hosted in the cloud and on-premises.
• To ensure data security and governance by implementing data access and quality policies.
• To provision and/or maintain the data infrastructure, ensuring scalability, availability, and security.
• Automate the provisioning and management of infrastructure using Infrastructure as Code (IaC) tools.
• Disseminating DevOps and DataOps best practices within the team.
• Collaborate with teams to deliver data solutions that meet business needs.
• Research and implement new technologies and tools to improve the efficiency and scalability of data solutions.
• Having the freedom and critical thinking skills to propose and question solutions related to data engineering.
• Documenting processes, architectures, and data solutions.

Requirements

• At least 4 years of experience in at least one programming language (Python, Java, Ruby, JavaScript, Scala, etc.);
• Experience with version control systems such as GitHub, GitLab, Bitbucket, etc;
• Advanced knowledge of SQL;
• Advanced knowledge in DBT;
• Experience or knowledge working with Databricks;
• Knowledge of algorithms and data structures;
• Have experience with technologies such as Spark, Kafka, Presto, and/or Airflow and feel confident creating aggregated datasets.
• Experience with data warehouses such as Google BigQuery, Redshift, and/or Snowflake is required.
• Have experience with Infrastructure as Code (IaC) tools, such as Terraform.
• Have experience with cloud infrastructure (AWS, GCP, etc.)
• Have experience with cloud data processing (AWS, GCP, Azure, Snowflake, Databricks)
• Certification in public cloud platforms (AWS, Azure, Snowflake, Databricks, GCP) at the Associate level or equivalent.

Applicants must be U.S. citizens to be eligible for this role.

Why Indicium AI

• Fast-growing start-up organisation with huge opportunity for career growth
• Highly competitive salary package along with company bonus
• A hugely collaborative working environment where every person's viewpoint is considered - a chance to make your mark on the business from day one!
• Financially backed business meaning security and support for new initiatives and global market expansion
• Pick your own Gear! Macbooks, PCs, Accessories!
• Drive your development with a personal learning budget.

The anticipated base salary range for this role is $120,000 - $160,000. In addition to base pay, this position may be eligible for an annual discretionary bonus. An individual's final salary offer will be determined based on a variety of factors, including geographic location, experience, specialized skills, and qualifications. This compensation range is subject to updates or modifications at the company's discretion