AI Data Engineer

Partnering with an industry-leading global technology pioneer through Aquent, you will have the unique opportunity to shape the future of artificial intelligence. In this dynamic role, you will directly impact the development of next-generation AI models by building and optimizing the critical data pipelines that power them. If you are passionate about transforming massive, unstructured datasets into high-quality training data and want to see your work influence cutting-edge technology used by millions worldwide, this is the perfect opportunity to elevate your career.

The Opportunity

As a key contributor to an advanced applied science team, you will design and operate robust Python-based workflows that turn complex document and text datasets into high-quality inputs for AI model training. This hands-on position bridges the gap between raw data and sophisticated AI, combining OCR processing, advanced data cleaning, and scalable pipeline development. You will collaborate closely with applied scientists to translate complex data requirements into reproducible, high-performance processing workflows, making a tangible impact on the capabilities of modern AI models.

Key Responsibilities

• Develop, maintain, and optimize Python pipelines for data ingestion, OCR processing, text extraction, cleaning, normalization, and deduplication.
• Run and monitor large-scale batch processing jobs, proactively investigating failures and implementing reliable retry and recovery mechanisms.
• Build automated quality checks to identify missing or corrupted files, incomplete OCR outputs, duplicate content, formatting issues, and metadata inconsistencies.
• Prepare training-ready datasets through filtering, text segmentation, tokenization, and conversion into structured formats.
• Manage bulk uploads, downloads, and synchronization between local environments and Azure storage, ensuring data integrity and completeness.
• Organize raw data, intermediate outputs, and released datasets with consistent naming, metadata, versioning, and source traceability.
• Optimize processing throughput, memory usage, storage consumption, and cloud costs using parallel or distributed processing approaches.
• Track dataset statistics, processing coverage, quality metrics, and exceptions to produce clear operational reports.

Required Qualifications

• At least five years of dedicated, hands-on data engineering, data pipeline development, or data science experience with substantial production programming responsibilities (a software development background alone without direct data management experience is not a fit).
• Strong proficiency in Python, including writing reusable modules, debugging, testing, logging, and handling processing errors.
• Demonstrated experience processing large structured and unstructured datasets using memory-efficient, parallel, or distributed techniques.
• Practical experience with Azure Blob Storage or Azure Data Lake Storage, including bulk data transfers, storage organization, and access troubleshooting.
• Familiarity with Python data-processing libraries and formats such as pandas, NumPy, JSON/JSONL, CSV, and Parquet.
• Working knowledge of SQL, Git, Linux environments, and command-line tools.
• Strong troubleshooting skills, meticulous attention to data quality, and the ability to communicate technical issues clearly.
• Educational background in computer science, data science, mathematics, or a related technical field.

Preferred Qualifications

• Prior experience working within large-scale enterprise environments.
• Experience with OCR pipelines, scanned documents, multilingual text, or document-layout processing.
• Experience with Azure Databricks, Azure Machine Learning, Azure Data Factory, or workflow orchestration tools.
• Familiarity with Hugging Face Datasets, tokenization, near-duplicate detection, and training/evaluation data separation.
• Experience with dataset lineage, reproducibility, containerized environments, and CI/CD.

Client Description:

Join a global technology leader driving innovation and empowering people and organizations worldwide to achieve more. Here, you'll collaborate with passionate, diverse teams to create impactful solutions that shape the future. The culture prioritizes growth, inclusion, and purpose—where your ideas are valued, and your work drives real change. Wherever your passion lies and wherever your career is headed—you’ll play a role in projects that touch billions of lives. With flexible work options and a supportive environment, you'll have the tools to thrive both personally and professionally. Be part of something that truly matters.

Why work with AQUENT? Check out our awesome benefits:

Aquent is an equal opportunity employer. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, and other legally protected characteristics.