With increasing automation across the entire spectrum of business operations, and with the continued deployment and expansion of IT infrastructure, the amount of logs that these systems generate has become humongous. It has become almost impossible to manually process these logs in the event of a failure to take corrective measures. Besides, with increasing scale, the log collection mechanism itself comes under stress, resulting in loss of important information that could have helped in better analysis.
Lumina Open Log Stack is an open source based application that facilitates log collection at scale, while aiding the in-memory translation from unstructured log to structured data. The structured data is then stored in analytics db like Elastic which aides further automated analysis and detection of anomalies in operations of systems. It renders itself for horizontal scaling, thereby bringing the ability to parse/analyze millions of logs per second in a clustered deployment
While the OLS app can be deployed for any domain, it finds its key applications in security domain, aiding in analysis of millions of firewall logs, thereby assisting in threat detection and enabling effective counter measures
- Enables IT departments to collect granular log messages, without worry about loss of logs in translation
- In-memory translation at scale permits seamless conversion from unstructured to structured data, facilitating better offline analysis
- Identification of patterns from large data sets facilitates automating corrective actions and closed-feedback loops
Comments from the Community
An Open Source solutions to the industry's call: "Many currently available network planning tools are inadequate, lacking end-to-end visibility into the network and big data-assisted capabilities, making it harder for Communications Service Providers to maintain Service Level Agreements (SLAs) and Quality of Service for their customers."
- Hardened, open source based log ingestion
- Translators for security log analysis
- Log push support for Elastic DB
- Policy query language and feedback triggers (future)
- Tuned to operate at scale