Lead Software Engineer - SparkFlow Framework

Lead Software Engineer – SparkFlow Framework

This is a Contract Opportunity with our company that MUST be worked on a W2 Only. No C2C eligibility for this position. Visa Sponsorship is Available! The details are below.

Job Title: Lead Software Engineer – SparkFlow Framework Contract Length: 12+ Month contract Hybrid work (3 days on site/ 2 days remote Location: CHARLOTTE, NC 28202

We are seeking a Lead Software Engineer to drive the evolution of SparkFlow, an enterprise data processing framework built on Apache Spark. This role focuses on designing composable abstractions, delivering new functional capabilities within the framework, and improving developer experience.

The engineer will also design and deploy AI-enabled capabilities and lead the integration of the SparkFlow framework into the Unity control plane.

Key Responsibilities

• Lead the architectural evolution of the SparkFlow framework, emphasizing modularity, composability, and clear abstraction boundaries.
• Design and implement new functional features within the framework to expand supported use cases and capabilities.
• Balance feature delivery with framework quality, ensuring performance, backward compatibility, and governance standards.
• Improve developer experience by simplifying APIs, configuration models, extension points, and onboarding patterns.
• Design and implement AI-driven solutions that assist developers and enhance framework usability, productivity, and operability.
• Integrate SparkFlow as a first-class framework within the Unity control plane.
• Collaborate with data, platform, and infrastructure teams to ensure consistent adoption across environments.
• Provide technical leadership through design reviews, code reviews, and architectural guidance.

Required Experience

• Deep experience with Apache Spark using Scala and/or Java (Python is a plus).
• Proven success building reusable frameworks or libraries, including ownership of functional feature development.
• Strong skills in API design, abstraction design, and extensibility patterns.
• Experience applying AI/ML or LLM-based techniques to developer tooling or data frameworks.
• Experience working in large-scale enterprise data environments.

Nice to Have

• Experience integrating frameworks into control planes or internal developer platforms.
• Familiarity with Spark SQL–centric architectures and data governance patterns.
• Experience modernizing or replacing legacy ETL frameworks.