Senior Principal Software Engineer (New York)

We are looking for seasoned engineers who have operated real production systems at scale—not just built prototypes or proof‑of‑concepts.

You have been on‑call. You have debugged live production incidents under pressure. You deeply understand the difference between code that looks clean in a pull request and code that remains stable during peak production traffic, downstream failures, and operational stress.

This is not a research‑only role or a greenfield experimentation position. This is a senior‑to‑principal‑level engineering role focused on building, operating, and evolving critical backend and AI‑powered systems in production environments. You will be expected to take ownership, drive engineering quality, and make sound technical decisions when systems fail—because eventually, they do.

What You’ll Do

Production Ownership: Own and maintain production‑grade Python services end‑to‑end, ensuring reliability, scalability, observability, and robust operational support.

Cloud Architecture: Design, build, and operate Azure‑native distributed systems using Azure Functions, Durable Functions, Event Grid, Service Bus, and supporting cloud infrastructure.

AI & Orchestration: Build and maintain production‑ready AI systems, including RAG pipelines, agentic workflows, and LLM orchestration frameworks (e.g., LangChain, AutoGen, Semantic Kernel).

Observability & Triage: Diagnose and resolve live production issues using distributed tracing, log analytics, profilers, Azure Monitor, Application Insights, and related observability tooling.

Engineering Standards: Establish and uphold strong engineering standards around code reviews, CI/CD, testing strategy, deployment safety, and incident post‑mortem practices.

Polyglot Flexibility: Contribute to secondary technology stacks, including C#, when needed. Deep expertise is not required, but practical working proficiency and transparency around strengths and limitations are expected.

What We’re Looking For

Non‑Negotiable Qualifications

Experience: 8–10+ years of professional backend or distributed systems engineering experience, with a proven track record of hands‑on ownership of systems running in production.

Python Expertise: Strong Python engineering depth, including performance optimization, concurrency, debugging, and production reliability considerations.

Azure Production Experience: Real‑world Azure production experience—beyond certifications or lab environments. You have worked through issues such as cold‑start latency, distributed failure modes, event ordering, scaling bottlenecks, and operational recovery.

Incident Mastery: Ability to clearly explain a past production incident you owned, including the root cause, investigation approach, remediation steps, validation strategy, and long‑term prevention measures.

Strong Additional Signals

Production AI/RAG: Experience shipping and maintaining production LLM or RAG systems beyond initial launch, including handling prompt drift, latency optimization, evaluation degradation, and cost management.

High‑Concurrency Systems: Experience with distributed data systems, including queue‑backed architectures, streaming systems, or large‑scale SQL/NoSQL workloads.

DevOps Culture: Ownership of CI/CD systems and deployment pipelines end‑to‑end, including rollback strategies, deployment safety, and operational automation.

Interview & Work Logistics

Interview Process: Candidates must be open to participating in initial in‑person interviews at our designated office location in the nearby city area.

Work Location: Day‑to‑day employment is strictly hybrid (2‑3 days a week in nearby office).

Integrity & Validation

Experience Verification: All candidates will undergo comprehensive background and experience verification.

Authenticity Over Buzzwords: We strongly value candidates who are honest about both their strengths and limitations. We are looking for genuine engineering depth, operational judgment, and production experience—not keyword‑heavy résumés.

Requirements added by the job poster

• 8+ years of experience in Engineering
• 5+ years of work experience with Microsoft Azure
• 6+ years of work experience with Python (Programming Language)

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next‑generation technology consulting and workforce solutions, specializing in delivering high‑quality talent across digital, engineering, healthcare, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions.

BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high‑impact global initiatives while advancing their careers.