What is Log Analysis?

• Compliance. Many governmental or regulatory bodies require organizations to demonstrate their compliance with the myriad of regulations that impact nearly every entity. Log file analysis can demonstrate that HIPAA, PCI, GDPR or other regulation’s mandates are in fact being met by the organization.

• Security enhancements. As cybercrime becomes increasingly organized the need for stronger countermeasures also grows. Event log analysis provides powerful tools for taking proactive measures and enables forensic examinations after the fact if a breach or data loss does occur. Log analysis can utilize network monitoring data to uncover unauthorized access attempts and ensure security operations and firewalls are optimally configured.

• Efficiency. A log analysis framework helps improve efficiency across the organization. IT resources in every department can share a single log repository, and analysis of an organization’s log data can help spot errors or trends in every business unit and department, enabling rapid remediation.

• High availability. Timely action that occurs based on information uncovered by log analysis can prevent an issue from causing downtime. This in turn can help ensure that the organization meets its business goals, and that the IT organization meets its commitments to provide services with a given uptime guarantee.

• Avoiding over- or under-provisioning. While organizations must plan to meet peak demands, log analysis can help project whether there is sufficient CPU, memory, disk, and network bandwidth to meet current demands – and projected trends. Overprovisioning wastes precious IT dollars, and under-provisioning can lead to service outages as organizations scramble to either purchase additional resources or utilize cloud resources to meet flexes in demand.

• Sales and Marketing Effectiveness. By tracking metrics such as traffic volume and the pages that customers visit, log analysis can help sales and marketing professionals understand what programs are effective, and what should be changed. Traffic patterns can also help with retooling an organization’s website to make it easier for users to navigate to the most frequently accessed information.
  

Since logs offer visibility into application performance and health, log analysis lets operations and development teams understand and remedy any performance issues that arise during the course of business operations.

Log analysis serves many important functions including:

• Compliance with governance and regulatory mandates as well as internal policies
• Tracking security breaches and data exfiltration to determine responsible parties and act to remediate breaches.
• Aid in diagnosing and troubleshooting the full stack
• Tracking user behavior anomalies to detect malicious intent or compromised systems
• Assistance in forensic investigation of malware attacks, exfiltration, or employee theft
  

Some regulatory bodies insist that organizations perform log file analysis to be certified as compliant with their regulations, and every organization that wants to improve its cybersecurity posture will need expertise in log analysis to help uncover and remediate cyber threats of all kinds. Some of the regulatory compliance requirements that log analysis helps meet are ISO/IEC 27002:2013, regarding the code of practice for IT security, PCI DSS V3.1 which covers privacy for credit card and other financial information, and NIST 800-137 regarding continuous monitoring for federal IT organizations.

Logs are time-series records of actions and activities generated by applications, networks, devices (including programmable and IoT devices), and operating systems. They are typically stored in a file or database or in a dedicated application called a log collector for real-time log analysis.

A log analysts task is to help interpret the full range of log data and messages in context, which requires normalization of the log data to ensure use of a common set of terminology. This prevents confusion that might arise if one function signals ‘normal’ and other function signals ‘green’ when they both mean that there is no action required.

Generally, log data is collected for the log analysis program, cleansed, structured or normalized and then offered for analysis for the experts to detect patterns or uncover anomalies such as a cyber-attack or data exfiltration. Performing log file analysis generally follows these steps:

1. Data collection: Data from hardware and software probes is collected to a central database
2. Data indexing: Data from all sources is centralize and indexed to speed searchability, enhancing IT professionals ability to rapidly uncover problems or patterns
3. Analysis: Log analysis tools including normalization, pattern recognition, correlation, and tagging can be done either automatically using machine learning tools or manually where needed.
4. Monitoring: A Real-time, autonomous log analysis platform can generate alerts when anomalies are detected. This type of automated log analysis is the underpinning for most continuous monitoring of the full IT stack
5. Reports: Both traditional reports and dashboards are part of a log analysis platform, providing either at-a-glance or historical views of metrics for operations, development, and management stakeholders