Big Data Engineer

Responsibilities Data Architecture: Design and implement scalable, robust, and efficient data architectures, including data lakes, data warehouses, and data pipelines, to support the organization's data requirements. Data Pipeline Development: Develop and maintain ETL (Extract, Transform, Load) processes and data pipelines to ingest, process, and transform large volumes of data from various sources, ensuring data quality and reliability. Data Modeling: Design and implement data models and schemas to organize and structure data for efficient storage, retrieval, and analysis, considering business requirements and data governance principles. Data Integration: Integrate data from diverse sources, such as databases, APIs, files, and streaming platforms, ensuring seamless data flow and interoperability across systems. Data Transformation and Enrichment: Implement data transformation and enrichment processes to derive insights, perform aggregations, and create derived datasets for analytics, reporting, and machine learning. Data Governance and Compliance: Establish and enforce data governance policies, standards, and best practices to ensure data integrity, security, privacy, and compliance with regulatory requirements. Performance Optimization: Optimize data pipelines and queries for performance, scalability, and efficiency, leveraging techniques such as partitioning, indexing, and query optimization. Collaboration: Collaborate with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver data solutions that meet business needs. Required Experience 3+ years of Data Management experience in an enterprise environment. 2+ years of experience with SQL. 1+ years of experience with AWS S3, Spark and Glue. Knowledge of metadata management, data governance, and data lineage. Preferred Experience Good understanding of Agile Software Development practices and how to leverage these to streamline project execution. The ability to decompose business problems into smaller parts and solve them correctly, considering the impact on other subsystems. AWS certification. Team player with excellent problem-solving and troubleshooting skills. Familiar with software engineering practices, including CI/CD pipelines, Automated Testing, and Performance Engineering. Working knowledge of SDLC, especially regarding improving agile delivery metrics. Strong communication skills. Bachelor’s degree in Computer Science, MIS, or a related field. Financial services experience. Consulting experience. #J-18808-Ljbffr