SUNRISE CRITICAL INFRASTRUCTURE SERIES
Enhancing Digital Transformation with Advanced Anomaly Detection: Introducing LOg MOnitoring System (LOMOS)
In the era of rapid digital transformation, industries worldwide are encountering significant challenges in maintaining robust security postures. The increasing complexity and interconnectivity of modern software systems necessitate sophisticated monitoring tools capable of swiftly detecting anomalies and security-related events. Traditional log monitoring methodologies, reliant on manual rule-based analyses, often struggle to keep pace with dynamic system behaviours and evolving threat landscapes. Addressing this critical need, we present the LOg MOnitoring System (LOMOS), an advanced AI-driven solution for anomaly detection in system logs.
As industries navigate the global digital transformation process, the detection of anomalies within system logs has emerged as a crucial aspect of enhancing security and visualizing vulnerabilities. An effective anomaly detection system significantly improves the security posture of an organization by providing real-time insights into potential threats, vulnerabilities, and system behaviours. LOMOS is designed to fulfil this imperative, offering a robust methodology for monitoring stack conditions and fuelling self-healing mechanisms.
LOMOS is an innovative AI technology and methodology tailored for anomaly detection in logs. This system is engineered to adapt to new data sensitivity concerns, facilitating the creation of informative metrics and variables with significant screening capabilities. By enabling real-time monitoring and automatic deployment, LOMOS ensures timely notifications about security episodes, enhancing the ability to detect and respond to security-related events within deployed application environments.
This innovative technology is fit to the challenges in industry today and tomorrow, including key features that come as a big benefit to SUNRISE in enhancing the security and resilience of critical infrastructure:
- Real-Time Monitoring: LOMOS provides continuous detection of vulnerabilities, threats, and malware in production infrastructures and during software development phases. This capability is crucial for identifying new vulnerabilities that arise when adding new services or features or when existing services become outdated.
- AI-Driven Anomaly Detection: Unlike traditional log monitoring solutions that rely on manual analysis of time series data, LOMOS employs a behavioural model to autonomously identify anomalies indicative of abnormal conditions, including potential security threats. This model leverages state-of-the-art Natural Language Processing (NLP) architectures to characterize log streams and their operational states.
- Transformer-Based Methodology: LOMOS integrates a novel transformer-based anomaly detection methodology, facilitating a smooth and secure digital transformation process. This approach ensures comprehensive adherence to evolving security requirements while supporting the dynamic nature of modern infrastructures.
LOMOS has been successfully implemented and demonstrated in three different industrial contexts, showcasing its efficacy in continuous detection of vulnerabilities and threats. For research purposes within the SUNRISE project, LOMOS has contributed to enhancing critical infrastructure security by integrating machine learning tools designed for this purpose. The system utilizes data from Insiel’s data center, including comprehensive records of user activities, to train its models and improve anomaly detection capabilities.
The initial log parser uses the Drain method to extract log templates from raw data, which are crucial for the anomaly detection process. The anomaly detector trainer employs the LogBERT model, a self-supervised method that excels in learning from normal log data without requiring labelled anomalies. This approach is further enhanced by the SemLogBERT model, which balances high recall with better precision, effectively identifying deviations from normal patterns.
The LOMOS dashboard offers interactive visualizations of extracted log templates and logs with anomaly scores. It provides control panels for model learning and inference, guiding administrators through training and deployment processes. Administrators can monitor logs and anomaly scores, set rules for alerts, and receive notifications via email, Slack, or Microsoft Teams.
LOMOS represents a significant advancement in log monitoring and anomaly detection, leveraging AI and machine learning to provide real-time insights into system vulnerabilities and security threats. By integrating this innovative solution, organizations can enhance their digital transformation efforts, ensuring robust security and compliance with evolving requirements. Join us at the workshop “Next steps in IoT-Edge-Cloud Continuum Evolution: Research and Practice”, co-hosted with Euro-PAR 2024, the 30th international European conference on parallel and distributed computing, happening on August 26-30 in Madrid, Spain. We will be presenting the research paper ” LOMOS: an AI-based runtime security monitoring system fit for the cloud continuum” discussing the power of AI in transforming how industries approach security and system monitoring in an increasingly digital world.
Written by: Joao Pita Costa, Hrvoje Ratkajec, Daniel Vladusic, Justin Činkelj
