Senior Data Engineer

Senior Data Engineer Bright Vision Technologies builds innovative software solutions that automate and optimize business operations. Job Summary We are seeking an experienced Senior Data Engineer to architect, design, develop, and maintain enterprise‑grade data platforms, scalable pipelines, and distributed processing systems that support analytics, business intelligence, and machine learning initiatives across multiple domains. The role covers the end‑to‑end data engineering lifecycle, from translating business requirements into robust data architectures to deploying cloud‑native solutions and ensuring their operational longevity. Key Responsibilities Design, build, and continuously improve scalable batch and real‑time data pipelines using Python, SQL, Spark, Scala, or equivalent technologies. Author secure, reusable, production‑quality ETL/ELT workflows that adhere to enterprise coding standards, governance policies, data quality principles, and security best practices. Develop scalable data integration solutions on modern cloud platforms such as AWS, Azure, or Google Cloud using services like Databricks, Snowflake, BigQuery, Redshift, Synapse, Data Factory, Glue, or equivalents. Design and implement robust data architectures, dimensional models, data lakes, data warehouses, and streaming solutions that integrate diverse data sources with consistency, scalability, and high availability. Participate in enterprise data architecture discussions, cloud migration initiatives, technical design reviews, and solution planning, evaluating trade‑offs involving scalability, performance, governance, and cost. Continuously monitor, profile, and optimize ETL processes, Spark jobs, SQL queries, database performance, storage utilization, and pipeline throughput. Maintain metadata management, data cataloging, lineage tracking, schema evolution, and governance frameworks to ensure trusted, discoverable, and compliant data assets. Develop automated testing frameworks for pipelines, ETL workflows, data validation, reconciliation, integration, and performance, ensuring reliable production deployments. Contribute to CI/CD pipeline design, infrastructure automation, and deployment processes using Jenkins, GitHub Actions, Azure DevOps, Terraform, Docker, Kubernetes, or equivalents. Identify and resolve data pipeline bottlenecks, operational risks, technical debt, and architectural weaknesses, driving continuous improvement. Collaborate within Agile/Scrum teams, participating in sprint planning, backlog refinement, daily stand‑ups, architecture discussions, reviews, and retrospectives. Maintain comprehensive technical documentation, including diagrams, pipeline specs, metadata, deployment guides, operational runbooks, and disaster recovery procedures. Required Qualifications Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Software Engineering, Mathematics, or a closely related discipline. Five or more years of professional experience designing, developing, and supporting production‑grade enterprise data engineering solutions. Strong understanding of data structures, database design, distributed computing, data modeling, ETL/ELT methodologies, and data warehousing concepts. Advanced working knowledge of Python, SQL, Spark, Scala, Java, and related frameworks for scalable data processing. Hands‑on experience designing and operating batch and streaming pipelines, data lakes, and cloud‑native platforms using solutions such as Databricks, Snowflake, Apache Spark, Kafka, Airflow, or equivalents. Proven experience with relational and NoSQL databases, including PostgreSQL, SQL Server, Oracle, MySQL, MongoDB, Cassandra, or equivalents, covering schema design, query optimization, and performance tuning. Strong SQL skills and experience designing dimensional models, star and snowflake schemas, data marts, and enterprise‑scale data warehouse solutions. Solid experience with Git‑based version control, CI/CD pipelines, DevOps practices, release management, and Agile software development methodologies. Hands‑on experience deploying data platforms on AWS, Azure, or Google Cloud, covering storage, compute, networking, security, identity management, and integration services. Excellent troubleshooting, analytical thinking, debugging, root‑cause analysis, communication, and documentation skills. Preferred Qualifications Experience designing and implementing event‑driven architectures, real‑time streaming platforms, Kafka, Flink, NiFi, RabbitMQ, or equivalent. Familiarity with containerization, orchestration, Infrastructure as Code, and cloud‑native deployment using Docker, Kubernetes, Terraform, Helm, or similar. Exposure to distributed systems concepts such as eventual consistency, fault tolerance, distributed transactions, replication, and CAP theory. Experience implementing data governance, master data management, lineage, metadata management, data quality automation, security compliance, and DataOps best practices within Agile environments. Equal Employment Opportunity (EEO) Statement Bright Vision Technologies is an equal opportunity employer and values diversity. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, gender identity, gender expression, age, marital or veteran status, pregnancy, disability, or any other protected characteristic. Climate, harassment, or discrimination of any form is prohibited. The company provides reasonable accommodations for religious practices and disabilities. Bright Vision Technologies is an Equal Opportunity Employer, including Disability/Veterans. #J-18808-Ljbffr