AWS and Gen AI Engineer

AWS Generative AI Engineer

Location: 100% Remote

Role: AWS GenAI Engineer - Intelligent Document Processing and Data Pipelines

Job Summary

We are seeking an experienced AWS Generative AI Engineer to design and develop intelligent document-processing solutions for large and complex files, including Excel workbooks, PDFs, scanned documents, forms, and semi-structured business content.

The ideal candidate will have strong experience with AWS services, Generative AI, Intelligent Document Processing (IDP), OCR, Python, and scalable data pipelines. The candidate will develop automated processes to identify and handle feature variations, extract meaningful data from complex documents, and organize processed data using a Medallion Architecture.

Modern AWS document-intelligence solutions commonly combine services such as Amazon Textract, Amazon Bedrock, scalable ingestion pipelines, and vector or search-based retrieval components.

Responsibilities

• Design, develop, and deploy AWS-based Generative AI and Intelligent Document Processing solutions.

• Build scalable ingestion pipelines for large volumes of PDFs, Excel files, scanned documents, images, and other unstructured or semi-structured data.

• Develop automated processes to extract tables, fields, entities, formulas, metadata, and business rules from complex Excel workbooks and PDF documents.

• Create reusable frameworks to identify document layouts, schema variations, feature variations, and changing business rules.

• Design processes that automatically route, classify, validate, and transform documents based on content and structure.

• Use Generative AI and Large Language Models to interpret complex document content and generate structured outputs.

• Develop prompt templates, extraction strategies, validation rules, and fallback mechanisms for reliable document processing.

• Build Retrieval-Augmented Generation solutions using embeddings, vector search, document chunking, and metadata enrichment where required.

• Integrate AWS AI/ML services such as Amazon Bedrock, Amazon Textract, Amazon Comprehend, and Amazon SageMaker.

• Develop Python-based services, APIs, Lambda functions, and reusable processing components.

• Implement asynchronous and event-driven processing for large files and long-running document workloads.

• Design retry, checkpointing, error-handling, dead-letter, and reprocessing mechanisms.

• Create data pipelines using AWS Glue, Apache Spark, PySpark, or equivalent distributed processing technologies.

• Implement Medallion Architecture using Bronze, Silver, and Gold layers.

• Store raw, intermediate, and curated data in appropriate AWS storage and analytics platforms.

• Develop data quality checks, reconciliation processes, schema validation, and exception reporting.

• Process large files efficiently by applying partitioning, batching, pagination, compression, and incremental processing techniques.

• Integrate document-processing outputs with databases, data warehouses, APIs, reporting platforms, and downstream applications.

• Monitor pipeline performance, processing accuracy, latency, failures, and AWS resource utilization.

• Implement CI/CD, infrastructure-as-code, automated testing, logging, security, and operational support.

• Collaborate with product owners, data engineers, architects, business users, and compliance teams.

• Create technical documentation, architecture diagrams, data mappings, runbooks, and support procedures.

Required Skills

• Strong experience designing and developing solutions on  AWS .

• Hands-on experience with  Generative AI , Large Language Models, prompt engineering, and document understanding.

• Experience with  Intelligent Document Processing , OCR, document classification, information extraction, and data normalization.

• Strong experience processing complex  PDFs, Excel workbooks, scanned documents, tables, forms, and images .

• Advanced programming experience with  Python .

• Experience with AWS services such as:

  • Amazon S3.

  • AWS Lambda.

  • AWS Step Functions.

  • AWS Glue.

  • Amazon Textract.

  • Amazon Bedrock.

  • Amazon Comprehend.

  • Amazon CloudWatch.

  • IAM and AWS Secrets Manager.

• Experience building scalable ETL/ELT and data ingestion pipelines.

• Strong knowledge of JSON, XML, CSV, Parquet, Excel, PDF, and other structured or semi-structured formats.

• Experience with large-file processing, asynchronous workflows, and distributed processing.

• Experience implementing  Medallion Architecture , including Bronze, Silver, and Gold data layers.

• Strong understanding of data quality, schema evolution, metadata management, and data governance.

• Experience with REST APIs, event-driven architecture, queues, and microservices.

• Familiarity with Git, CI/CD, unit testing, and deployment automation.

• Strong troubleshooting, analytical, communication, and documentation skills.

Preferred Skills

• Experience with Amazon Bedrock foundation models, including Anthropic Claude or Amazon Nova.

• Experience building RAG applications using embeddings, vector databases, and semantic search.

• Experience with Amazon OpenSearch Serverless, Aurora PostgreSQL with pgvector, or other vector stores.

• Experience with Bedrock Knowledge Bases, Agents, Guardrails, and model evaluation.

• Experience with Amazon Textract AnalyzeDocument, AnalyzeExpense, Queries, tables, forms, and asynchronous APIs.

• Experience with PyMuPDF, pdfplumber, Camelot, Tabula, OpenPyXL, Pandas, or similar document-processing libraries.

• Experience with Apache Spark, PySpark, Databricks, or EMR.

• Knowledge of OCR post-processing, confidence scoring, human-in-the-loop validation, and exception management.

• Experience designing metadata-driven and configuration-driven frameworks.

• Knowledge of AWS EventBridge, SQS, SNS, Kinesis, DynamoDB, RDS, Redshift, and Athena.

• Experience with Terraform, AWS CDK, or CloudFormation.

• Familiarity with LLMOps/MLOps, model monitoring, prompt versioning, and AI governance.

• Experience handling PII, confidential data, security controls, and regulatory requirements.