Principal Engineering Lead — Content & Data Pipeline

We believe in bold ideas, diverse perspectives, and the drive to transform knowledge into impact. Here, your curiosity fuels progress, your voice shapes innovation, and your ambition helps redefine what’s possible within science and learning. We are a culture that obsesses over impact, challenges, and drives what’s next to power infinite possibilities for our customers, colleagues and society at large.

About the Role:

About the role

This role leads Wiley's Content & Data Pipeline function — the platform that takes scholarly content (journals, articles, books) from a wide mix of sources through ingestion, processing, and enrichment, and delivers it to internal AI systems, RAG pipelines, and external customers under the right entitlements. It is a hybrid role: you set technical direction and stay hands-on with architecture and code where it matters, and you also manage and grow a team of engineers who own the day-to-day pipeline work.

The role sits alongside an existing dev lead who runs the current content ingestion and delivery pipeline; you take end-to-end ownership above that, expanding the platform to cover new source types, new delivery surfaces (especially internal AI / RAG and licensed external delivery), and higher standards of quality, freshness, and observability.

The mandate spans the full content lifecycle — ingestion, transformation, enrichment, metadata management, rights-aware delivery, and vendor / supplier integration — across scholarly publications and books. Stakeholders include product, business, licensing partners, and vendors, and part of the job is turning their needs into a coherent, prioritized platform roadmap.

We are looking for someone who leads by doing and by coaching — who codes, reviews, unblocks, and mentors, while also owning delivery, and stakeholder alignment for the team.

What you'll do

Technical direction and architecture

• Own end-to-end architecture for the content and data pipeline: ingestion from vendors and public sources, transformation and enrichment, metadata management, and delivery to internal AI / RAG systems and external customers.

• Run and operate the pipeline in production and manage the content lifecycle — corrections, retractions, version updates, metadata deposits and registry syndication, and rights and entitlements enforced at every delivery channels.

• Set standards for pipeline reliability, data quality, freshness monitoring, testing, observability, and CI/CD across the team's services.

• Codify architectural decisions in ADRs; drive convergence on shared patterns across ingestion, processing, and delivery.

• Stay hands-on — write code, run architecture reviews, prototype, and unblock the team where needed.

Team leadership and delivery

• Manage and coach a team of engineers spanning content ingestion, processing, and delivery.

• Partner closely with the existing dev lead on the current pipeline; align on ownership boundaries and coordinate roadmap execution.

• Own delivery commitments — scope, sequencing, dependencies, and honest tradeoff conversations with stakeholders when reality does not match the plan.

• Run engineering practices that scale: code review discipline, on-call rotation, incident response and postmortems, quality gates in CI.

Stakeholder partnership

• Partner with product and business teams on capability priorities — what content the platform ingests next, what delivery surfaces get built, what quality standards matter.

• Own vendor and licensing-partner relationships for content sources and delivery pipes — technical integration, data contracts, escalation, renewal input.

• Translate stakeholder needs (editorial ops, product, licensing) into a coherent platform roadmap; push back honestly when requests do not fit the platform's direction.

• Communicate technical strategy and tradeoffs upward and outward — not just to peer engineering leaders but to product, business, and executive audiences.

What you'll own

• The Content & Data Pipeline platform end-to-end — ingestion through delivery — including architecture, standards, delivery SLAs, and data quality guarantees to internal and external consumers.

• The Content & Data Pipeline team — build and delivery

• Vendor and licensing-partner technical relationships for content sources and delivery pipes in scope.

• The technical roadmap for expanding the platform to new sources, new delivery surfaces, and higher engineering bars, codified in ADRs and communicated to stakeholders.

Must-have skills

Leadership

• Player-coach engineering leadership at scale — you have led an engineering team while remaining genuinely hands-on with architecture and code.

• Cross-team influence with product, business, vendor, and executive stakeholders. Able to communicate technical strategy in language each audience actually uses.

• Vendor and partner relationship management — technical integration, contract execution, renewal input, escalation.

• Honest tradeoff communication — you make scope, quality, and timeline conversations concrete rather than optimistic.

Technical depth

• Deep production experience designing and operating content or data pipelines end-to-end — ingestion, processing, delivery — at scale.

• AWS at production depth — S3, Lambda, ECS or EKS, SQS / SNS, EventBridge, Step Functions, RDS, DynamoDB. Real IAM and cost fluency.

• Python and SQL at production depth.

• Workflow orchestration in production — Airflow, Step Functions, Dagster, or equivalent — with real experience in idempotency, backfills, retries, dependency management.

• Content transformation experience — XML processing, schema-driven transformation, structured content formats.

• Data quality practice: schema validation, defensive parsing, incremental correctness checks, freshness monitoring, alerting on drift.

• Distributed systems literacy: idempotency, retries, duplicate handling, partial failure, replay.

• Real production experience with API-based and bulk ingestion from external sources — auth, pagination, rate limiting, silent schema changes.

• Testing and CI/CD discipline for pipelines — unit, integration, and data-quality tests in CI; safe deployment patterns.

• Observability practice — logs, metrics, lineage, and the discipline of knowing whether the pipeline is actually healthy vs. just running.

• Active use of AI coding agents in your day-to-day workflow — always with review.

Hugely preferred: content pipeline experience

Scholarly publishing — journals, articles, books — is our domain and our first preference. But the core skill is building pipelines for content, not just data, and that experience carries over from any content-heavy environment.

• You have built or run pipelines that ingest, process, and deliver content in structured formats — XML, HTML, EPUB, PDF — where structure and metadata matter.

• You have dealt with content that gets corrected, updated, or pulled after it ships, and built pipelines that handle that cleanly.

• You have delivered the same content to different consumers in different shapes — feeds, APIs, search, AI systems.

• You have worked with content that has usage rights or access restrictions attached.

• You have partnered with editorial, production, or content-operations teams and earned their trust.

Scholarly-specific knowledge — JATS, DOIs, Crossref, retraction handling — is a strong plus. We can teach the acronyms to someone who already has the pipeline instincts.

Nice to have

• Scholarly publishing specifics — JATS / BITS XML, DOI infrastructure, Crossref deposits, ORCID / ROR identity, MeSH and similar controlled vocabularies.

• BigQuery / GCP experience — useful for the secondary consumer surface, especially as our AI teams standardise there.

• Specialised publishing tooling — XSLT at scale, existing content-management systems, or custom XML processing frameworks.

• Streaming ingestion — Kafka, Kinesis, Pub/Sub — for near-real-time content updates.

• Vector search and embedding pipelines — useful for the internal AI / RAG delivery surface.

• Prior partnership with ML, AI, or agent teams as a downstream consumer of your platform.

• Data catalog and lineage tooling: DataHub, Amundsen, or equivalent.

• Existing relationships in the scholarly publishing ecosystem — vendors, standards bodies, or peer engineering leaders at similar organisations.

How we work

• Player-coach leadership. We lead by doing and by coaching. Purely delegating leaders and purely heads-down principal engineers are both mismatches for this role.

• Content as a first-class engineering concern. Scholarly content has production standards, versioning semantics, and rights implications that generic data pipelines do not — and we engineer to them.

• End-to-end ownership. We own ingestion through delivery. We do not throw work over the wall between ingestion, processing, and delivery teams; we design across them.

• Stakeholder partnership. Product, business, vendors, and executives are partners, not audiences. We share evidence and tradeoffs openly, and we push back honestly when needed.

• Learning agility. The scholarly content and AI landscapes both move quickly. We hire leaders who show up curious about new territory rather than defensive of what they already know.

• Quality is a first-class engineering concern. Data-quality bugs are engineering bugs, not “the upstream source's problem.”

• Agentic engineering is a strong expectation, including at leadership levels. We use AI coding agents actively — always with review — and we set the standards our teams follow.

• Production ownership. Engineers, including engineering leaders, own the platforms they build — including on-call, incident response, and using observability tools to diagnose their own systems.

We power infinite possibilities.

For more than 200 years, we've transformed knowledge into discoveries that shape the world. Today, our global team of innovators, creators, and experts is driving what's next in science, education, and publishing—creating impact that reaches everywhere.

We're not just observers of progress. We're the ones accelerating scientific breakthroughs, advancing learning, and sparking innovation that redefines entire fields and improves lives.

Here, your talent matters. Your ideas have room to grow. And your work creates breakthroughs that can change everything.

Wiley is an equal opportunity/affirmative action employer. We evaluate all qualified applicants and treat all qualified applicants and employees without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, disability, protected veteran status, genetic information, or based on any individual's status in any group or class protected by applicable federal, state or local laws. Wiley is also committed to providing reasonable accommodation to applicants and employees with disabilities. Applicants who require accommodation to participate in the job application process may contact View email address on aiapply.co for assistance.

We are proud that our workplace promotes continual learning and internal mobility. We offer meeting-free Friday afternoons allowing more time for heads down work and professional development, and through a robust body of employee programing we facilitate a wide range of opportunities to foster community, learn, and grow.

We are committed to fair, transparent pay, and we strive to provide competitive compensation in addition to a comprehensive benefits package. The range below represents Wiley's good faith and reasonable estimate of the base pay for this role at the time of posting roles in the United Kingdom, Canada, USA, Austria, Czechia, Denmark, France, Greece, Italy, Netherlands, Romania, or Spain. It is anticipated that most qualified candidates will fall within the range, however the ultimate salary offered for this role may be higher or lower and will be set based on a variety of non-discriminatory factors, including but not limited to, geographic location, skills, and competencies.

When applying, please attach your resume/CV to be considered.

Salary Range:

140,000.00 USD to 200,733.33 USD #LI-CW1