Senior Machine Learning Engineer — Rust, Models & Agent Systems (New York City Metropolitan Area)

This is a Founding Engineer role at Loci for a scrappy, versatile ML generalist who thrives on owning core systems.

We’re looking for a machine learning engineer to help build an AI application that gives users control over their models, tools, and data. The work spans model training and fine-tuning, local inference, agent workflows, tool use, persistent memory, and retrieval. You’ll help decide what belongs in the model and what belongs in the surrounding software, then build and evaluate both.

You’ll own core systems end to end, from investigation and experimentation through architecture, implementation, delivery, and ongoing reliability. You’ll move between model work and product engineering, identify the highest-impact bottleneck, and build practical solutions with limited time and resources. That means writing excellent Rust code, understanding model behavior, making sound architectural decisions, and helping the team turn AI capabilities into dependable product features.

Compensation

$200,000 annual base salary plus equity. We believe in paying a generous base salary and prefer candidates who value equity and long-term ownership in Loci more than maximizing base pay.

What you’ll do

- Train, fine-tune, and evaluate models against product requirements, with clear objectives, reproducible experiments, and responsible use of training data.

- Select models that can support the application’s features within practical limits on memory, latency, compute, and hardware. Establish when a smaller model is sufficient and when a capability requires a different approach.

- Design the boundaries between models and the agent harness: the software that manages context, tools, state, and execution. Decide which responsibilities require model judgment and which should be enforced in code.

- Build reliable, maintainable Rust software for inference, agent workflows, tool execution, memory, and retrieval, using the type system to make invalid states difficult to express.

- Work with llama.cpp and related inference engines to integrate models, diagnose compatibility issues, and improve performance through profiling, quantization, caching, batching, and engine optimizations.

- Write and maintain system prompts, chat templates, tool definitions, and tool-call handling. Design how the harness responds to malformed calls, unsupported capabilities, failures, and uncertain model output.

- Build reproducible evaluation workflows for non-deterministic behavior that run separately from normal development gates. Measure task success, tool-use reliability, regressions, latency, and resource use, and turn findings into engineering decisions.

- Recommend how the harness should manage context, retrieve information, preserve memory, request clarification, recover from errors, and determine when a task is complete.

- Teach the team how models and agent systems behave. Explain limitations, share useful research and practices, and help engineers reason about model and harness interactions.

- Structure code, experiments, documentation, and tickets so that assumptions, responsibilities, results, and next steps are easy to understand.

- Discuss approaches before implementation, explain tradeoffs, communicate progress and blockers clearly, and validate your changes before merge.

What you’ll bring

- Broad practical ML judgment: you can frame an ambiguous product problem, build a useful baseline, prepare data, design an evaluation, diagnose failures, and ship and improve the resulting system. You learn unfamiliar parts of the stack quickly and work across model and software boundaries.

- Hands-on experience training and fine-tuning language models, including dataset preparation, training objectives, parameter-efficient methods, and evaluation. You can recognize overfitting, data leakage, and misleading benchmark results.

- Strong Rust engineering skills, including ownership, traits, async, concurrency, error handling, FFI integration, and performance profiling.

- Practical experience with llama.cpp, local inference, quantized models, and the constraints of running models on consumer hardware.

- Experience profiling and optimizing inference engines, with an understanding of memory use, prefill, decoding, KV caches, and hardware acceleration.

- Experience building agent systems beyond basic API calls: designing tool interfaces, implementing tool calling, managing context and state, and debugging failures across model and application boundaries.

- Strong judgment about model selection and system design. You can determine whether a problem is best addressed through prompting, fine-tuning, retrieval, tool design, or changes to the harness.

- Experience writing system prompts and evaluating their behavior across representative tasks, edge cases, and adversarial inputs.

- An understanding of statistical evaluation and experimental design. You can distinguish a meaningful improvement from sampling noise and explain what an evaluation does and does not establish.

- Excellent written and verbal communication, including clear technical explanations, useful code reviews, actionable tickets, and practical teaching.

- Independent engineering ability. You must be able to design, write, debug, and test software when cloud AI providers or coding assistants are unavailable. AI tools can accelerate your work; your ability to deliver must remain dependable during an outage.

Nice to have

- Contributions to llama.cpp or other inference engines, model architectures, quantization methods, or hardware-specific performance improvements.

- Experience with cloud model services and routing platforms such as OpenRouter, including provider differences, structured outputs, tool calling, fallbacks, latency, and cost.

- Experience with GPU infrastructure for training and inference, reproducible deployments, Kubernetes, Configuration as Code, and CI/CD.

- Experience with multimodal models, on-device speech, or adapting small models for reliable tool use.

Who you are

- You’re scrappy and versatile. You make progress with incomplete information, choose simple approaches that work, and balance fast experimentation with dependable engineering.

- You enjoy owning core systems. You take responsibility for their design, interfaces, performance, reliability, and evolution, and help teammates work effectively with them.

- You value privacy as a design requirement. You make deliberate choices about what training pipelines, models, and applications store, share, and expose.

- You’re self-driven. You investigate unfamiliar problems and carry work through to completion without waiting for detailed instructions.

- You exercise independent judgment, explain your reasoning, and ask for input when it matters. You keep others informed without needing someone to manage every step.

- You approach AI empirically. You test assumptions, examine failures, and change your position when the evidence warrants it.

- You bring order to complexity. You leave code, experiments, documentation, and architecture easier for others to understand and extend.

- You’re passionate about self-sovereign, on-device AI and making it useful on hardware people already own.

We value evidence of how you think and build. Be ready to discuss a model you trained or fine-tuned, a Rust or inference system you’ve owned, and an agent workflow you took beyond a demo. We’ll want to understand how you evaluated success, chose the boundary between model behavior and application logic, and helped others make better engineering decisions.