Lead Software Engineer

Lead Software Engineer

Will not sponsor. Must be legally authorized to work in the United States without current or future employer sponsorship.

As a Lead Software Engineer, you will be responsible for leading software development initiatives. You will independently design, develop, and test complex software programs and systems. You will also collaborate with team members, mentor junior engineers, and provide technical guidance to ensure the delivery of high-quality software solutions. You will also collaborate with product managers, designers, and other engineers to define, refine, and implement features and enhancements.

Responsibilities:

• Own and evolve large-scale data flows end-to-end-batch processing, transformation, clustering, and publication of datasets and indexes-ensuring they are reliable, documented, and ready for modernization.

• Drive modernization of data-flow architecture: evaluate and introduce new techniques, patterns, and tooling; make build-vs-buy and technology choices; set direction others can implement against.

• Lead solution design for batch and distributed processing: data quality checks, indexing strategies, replay/idempotency patterns, and integration with existing Java/Spring services and APIs.

• Develop and tune Apache Spark workloads: implement batch jobs, diagnose failures, and improve performance through profiling, troubleshooting, and optimization.

• Build and maintain scripting and orchestration at scale using bash (required) and related automation; coordinate long-running jobs through enterprise scheduling and operational runbooks.

• Establish technical standards for batch and data-flow engineering-scripting patterns, monitoring, failure handling, and operational handoff-and influence practices beyond your immediate team.

• Collaborate with product managers, leadership, and engineering teams to align roadmaps with product needs and translate organizational goals into executable technical strategy.

• Troubleshoot and resolve complex production issues in data flows and batch systems; implement preventive, systemic improvements-not one-off fixes.

• Leverage and explore AI-assisted development tools (e.g., GitHub Copilot, Cursor, code generation, smart testing) where appropriate; help assess effectiveness and support adoption.

• Champion agile methodologies, lead technical and design reviews, and foster cross-team collaboration.

• Maintain awareness of security, data governance, and quality standards in an enterprise context.

Qualifications:

• Bachelor's degree in computer science or related discipline, or equivalent work experience.
• 7+ years of software development experience .
• Demonstrated professional strength in Java: shipping, maintaining, and evolving production services-not occasional or peripheral use.
• Strong bash scripting at scale in production data-flow and batch environments; comfort maintaining and extending large script-based systems.
• Hands-on experience with Apache Spark (or Hadoop or Cassanda): developing batch workloads and troubleshooting/tuning performance and reliability (depth in patterns matters; specific cluster or vendor context can be learned on the job).
• Proven experience owning large-scale batch processing and data flows: design, operation, troubleshooting, and evolution-including publishing datasets or indexes at scale.
• Experience with batch orchestration and job scheduling in production (enterprise schedulers, dependency chains, failure recovery, and operational runbooks).
• Ability to work with messy, evolving data: inconsistent schemas, multiple sources, and changing requirements; design for robustness and incremental improvement.