Big Data Practice Lead - Spark / Databricks / AWS

Job Overview A global consulting and technology services organization is seeking a Big Data / Spark Principal Architect to lead the design and implementation of large-scale data systems. This senior-level role combines deep technical expertise in Spark, Databricks, and cloud-based data platforms with a strategic understanding of solutioning and consulting across enterprise environments. The successful candidate will architect high-performance data pipelines, guide technical teams, and act as a subject matter expert (SME) for big data and streaming solutions within the Media & Entertainment domain. Key Responsibilities Architecture & Design: Define and implement scalable, high-performance big data architectures using Spark (DataFrame, Spark SQL, PySpark/Scala Spark) and Databricks. Solutioning & Consulting: Lead solution design engagements, act as SME for big data technologies, and provide consulting expertise for enterprise data strategies. Pipeline Development: Build and optimize batch and streaming data pipelines leveraging Spark, AWS Glue, and Kafka (or equivalent streaming tools). Data Governance & Quality: Establish frameworks for data quality, lineage, and compliance across multi-source data ecosystems. Cloud Enablement: Architect and deploy solutions on AWS or Azure, ensuring performance, scalability, and cost efficiency. Practice Development: Contribute to the growth of the Big Data practice through mentoring, pre-sales support, and best practice documentation. Collaboration: Work closely with data engineers, analysts, and business stakeholders to translate requirements into robust technical solutions. Continuous Improvement: Stay current with emerging technologies and promote adoption of modern data engineering practices. Must-Have Requirements 10–18 years of total experience in Big Data Architecture, Engineering, and Consulting. Strong hands-on expertise in Apache Spark (DataFrame, SQL, PySpark/Scala Spark) and Databricks. Proven track record in solutioning and consulting as a Subject Matter Expert (SME) in Big Data. Experience with AWS Glue or Azure Data Services for data ingestion and transformation. Familiarity with streaming technologies such as Kafka, Kinesis, or Flink. Deep understanding of ETL design, data governance, and data quality frameworks. Strong coding skills and willingness to be hands-on with development. Prior experience in Media & Entertainment or related domains. Must be a U.S. Citizen or Green Card Holder (mandatory). Demonstrated job stability and long-term project ownership. Nice-to-Have Skills Exposure to AI/ML integration within big data ecosystems. Experience leading practice development or solution selling for enterprise clients. Knowledge of modern data mesh or lakehouse architectures. Location: USA – Remote (Full-Time) Work Type: Permanent Remote Experience Required: 10 – 18 Years Compensation: USD 150,000 – 200,000 (plus bonus, based on experience and performance) #J-18808-Ljbffr Mogi I/O : OTT/Podcast/Short Video Apps for you