Client Overview

Global Provider of HR & Financial, Health & Wealth Solutions with over 25 years of experience

Business Need

Client is redesigning their solutions platform using the next generation architecture. They wanted to develop a Log Aggregation Framework that could monitor & detect failures ahead of time. The framework should

  • Capture distributed microservices logs into a centralized location
  • Acquire data from diverse set of data sources (UI, database, microservices)
  • Provide capability of visualize and search desired logs via GUI
  • Analyze log patterns to determine different workflows of application
  • Identify failure pattern, exports and sends report to respective teams

Key Features

  • ELK (Elasticsearch-Logstash-Kibana) based Log Processing framework with key components including

    • Logstash – a server-side data processing pipeline that ingests data from a multiple sources simultaneously, transforms it, and sends it to Elasticsearch
    • Elasticsearch - a distributed analytics engine to manage data acquired from Logstash
    • Kibana

      • Provides visualization on the data
      • Monitors ELK nodes – Node health, request/response time etc.
      • Email alerts on predefined conditions
    • High Availability – Maximizing uptime of ELK nodes via internal and external load balancers
    • Security - SSL/OpenSSL encryption across each communication between ELK nodes
    • HDFS – Compressed raw data backup for future use

Benefits

Secure centralized Log Processing Framework based on ELK stack with powerful visualization and search functionalities enabling them to deliver high availability through

  • Real time monitoring of core micro services
  • Real time access to failures leading to faster turnaround time
  • Integrated framework to analyze failures/errors from diverse applications through a single interface