Observability Engineer

Job Title Observability Engineer Location Remote Role Overview We are seeking a talented and motivated Observability Engineer to join our POS (Point of Sale) Platform team. In this role, you will be instrumental in enhancing the observability and alerting capabilities of our critical POS and SCO (Self-Checkout) applications. You will be responsible for modifying our Java-based applications to emit necessary telemetry, working with a modern observability stack, including Grafana and BigQuery, to ensure the reliability and performance of our systems. A key aspect of this role will be leveraging AI and machine learning to analyze observability data, identify trends, and proactively address potential issues. Key Responsibilities Enhance Application Metrics: Modify and instrument our Java-based POS and SCO applications to emit the necessary metrics, logs, and traces for robust monitoring and alerting. Develop & Maintain Dashboards: Design, build, and maintain comprehensive Grafana dashboards to visualize key performance indicators (KPIs), metrics, and application health. Query & Analyze Data: Write and optimize complex BigQuery queries to extract and analyze observability data from our applications. Leverage AI for Data Analysis: Utilize AI and machine learning techniques, including prompt engineering, to analyze telemetry data, identify anomalies, and predict potential system failures. Improve Alerting: Develop and refine alerting rules and procedures to ensure timely and actionable notifications for production issues. Troubleshoot & Resolve Issues: Investigate and troubleshoot production incidents, using observability data to identify root causes and implement solutions. Documentation: Create and maintain clear and concise documentation for dashboards, queries, and observability-related processes. Skills and Qualifications Required Skills & Experience Application Development: Proven experience in Java application development and code modification. SQL & BigQuery (BQ): Strong proficiency in writing and optimizing complex SQL queries in Google BigQuery for large datasets. Grafana: Experience in creating and managing complex Grafana dashboards for application monitoring. Data Analysis: Demonstrated ability to analyze complex datasets, identify trends, and draw meaningful conclusions. Observability Concepts: Solid understanding of observability principles, including metrics, logging, and tracing. Communication: Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams. Preferred Qualifications Prompt Engineering: A strong understanding of prompt engineering for interacting with and leveraging large language models (LLMs) for data analysis. Statistical Analysis: Experience with statistical analysis techniques and tools to interpret data and provide insights. Terraform: Experience with Infrastructure as Code (IaC) using Terraform. POS Systems: Familiarity with Point of Sale (POS) or retail systems. CI/CD: Knowledge of CI/CD pipelines and DevOps practices. OpenTelemetry: Familiarity with the OpenTelemetry standard. #J-18808-Ljbffr TalentBridge