Log Management Considerations: What You Need to Know

There are a myriad of interrelated variables and features to take into consideration when evaluating log management solutions. Log management solutions range from inexpensive do-it-yourself open-source to multi-million dollar enterprise solutions. When evaluating log management solutions, it is important to determine the features, flexibility, resources, capacity, and retention required to achieve the desired results.

Open-source vs. Proprietary
There are primarily two main categories of log management platforms- open-source and proprietary. Open source refers to software whose source code is freely available on the internet to download, distributed under various licenses such as BSD, Apache, LGPL, GNU, etc. Open-source software can in almost all cases be used without incurring a fee.
Popular open-source log management vendors include: Elastic, Graylog, LOGalyze

The upside to open-source software is that it’s free and fully customizable. The downside is that even through the software is free, there’s a significant cost associated with training, installing, modifying, operating, and managing open-source systems. Proprietary commercial software is a closely guarded secret that’s typically owned by the vendor. Pricing model around proprietary application vary from vendor-to-vendor, usually requiring both a license and recurring annual maintenance fee.
Popular proprietary commercial software vendors include: Splunk, SumoLogic, Loggly

The upside to proprietary software is that it’s typically known for being feature rich, comprehensive, and supported. The downside is that these solutions are generally very expensive and lack flexibility and customizability.

Hosting Models
SaaS
Software-as-a-Service (SaaS) is centrally hosted software that is licensed on a flat yearly or monthly subscription basis. SaaS applications allow limited modifications but typically, the underlying code can’t be customized to meet the specific needs of the organization. Proprietary solutions typically use a SaaS model. The upside of using the SaaS model is that SaaS vendors provide the hosting, maintenance and upgrades to the log management system that is accessed over the Internet. The downside is that control, upgrades, customizability and ownership are typically limited by the proprietary software vendor. In addition, SaaS platforms are notorious for becoming extremely expensive over time as log data collection and storage requirements expand.
Examples of SaaS log management platforms include: Elastic Cloud, Splunk Cloud, Sumo Logic, Loggly

On-Premise
In the case of on-premise,, the company either buys a proprietary packaged software license or downloads an open-source software package, installing and managing the entire platform either in-house or in a cloud. The on-premise model requires hardware and personnel with the necessary expertise and knowledge of the specific platform to install, configure, and manage on a 24/7 basis. The upside of on-premise software is that the customer has more control and ownership than they would have with a SaaS model. The downside is that if the on-premise logging platform is proprietary, the customer still must bear the cost of hardware, installation and ongoing management and maintenance of the platform.
Examples of on-premise log management platforms include: Elastic Stack, Graylog, Splunk

Managed Services
With the managed services model, the company acquires its own log management software licenses and then hires the managed service provider (MSP) to install, manage, and maintain the software. MSPs will many times deploy using open-source software because of the absence of license fees as well as their high configurability and customizability. The MSP approach gives the company far more flexibility, configurability, and accessibility to both the application and data.

The upside of using a managed service provider is that it’s a cost effective solution in which the customer has far more control and ownership without having to purchase and maintain hardware or dedicate a resource for the installation and ongoing management/maintenance of the platform. Flexibility and customizability are no longer an issues with the MSP. The downside of using a managed service provider is that some MSP’s don’t have the necessary qualifications or experience to adequately install, manage and maintain the log management platform.
Examples of managed services log management platforms include: Mission Managed ELK Stack

When making the decision between SaaS, on-premise, and managed service provider, the primary questions are:

• How much control over the log management system and your data do you require?
• How many resources do you have available that you can dedicate to log management?
• How much budget do you have available to dedicate to log management?

If you need maximum flexibility and you have adequate staff and budget to dedicate to log management, then an on-premise non-proprietary solution is probably the way to go. If you have adequate budget but don’t need the flexibility and control or don’t want to use additional resources to manage your own on-premise solution, then a proprietary SaaS solution would be the choice. If, however you need maximum flexibility, control and customizability, and you don’t have the staff, budget or expertise required to manage an on-premise log management solution then a fully-managed solution from a managed services provider might be a better way to go.

Scale
Scale is one of the more important considerations when selecting a log management solution. Log management costs typically scale with each incremental increase in average daily data ingest. SaaS platforms are known to be the most expensive solutions as average daily data ingest scales upward. Depending on daily data ingest levels, larger companies with high volumes of log data have been known to scale up to millions of dollars per year for their log management solution.

On-premise solutions are also expensive to scale due to the combined costs of infrastructure and personnel. Scaling an on-premise, open-source log management platform requires in-depth knowledge of the platform as well as the time, energy and money to manage non-trivial clusters. The challenges associated with this multiply as the system scales out into a multitude of servers with dozens of time-based indices, and thousands of shards.

Access and Control
Access and control of log data is an extremely important consideration when selecting a log management solution. Access and control requirements are determined by use cases. If for instance one of your applications for log management is regulatory compliance, it’s likely that you’ll want and need full access and control over your logo data.

When considering a SaaS model log management solution some of the key factors the customer may or may not have control over include:

• Timeliness and flexibility of log event availability, alerts, and security updates
• SaaS provider’s compliance with changing regulations
• Security of log data, common key storage and encryption in multi-tenant environments as well as data transport and encryption of that data
• Lack of control over the underlying log management architecture and your data
• Where your backups are stored and how long data is retained
• Frequency of data transmission and the level of compression and bandwidth utilization
• How much of the log data is stored (and discarded) and how much of the log data is searchable and whether archived data is searchable
• What types of log data is supported by the vendor as well as support for devices and the data they generate
• How system upgrades are handled and how much control the customer has over upgrades
• Proprietary vendor data storage formats that require a vendor tool to access and read the data

On-premise non-proprietary solutions give the customer full access and control over the log management solution but require adequate resource to monitor logs regularly and maintain updates.

The Managed Service Provider scenario provides skilled resources for 24/7 monitoring and management while still providing the client full access to an control over the log management system. A big plus to both non-proprietary log management solutions is that log data is typically stored in standard, non-proprietary formats that prevent vendor lock-in and provide more ownership over the customer’s data.

Security
Security is an important consideration when dealing with sensitive data, especially when regulatory requirements are in play. Log management platforms should provide encryption of log data in transit to is destination in addition to data at rest after arriving and placed into storage. It’s important to ask if the log management platform is multi- or single- tenant. Multi-tenant solutions share resources, which result in impacting regulatory requirements.

Customizability
If your log management plans call for a high level of customization you should consider building your own proprietary log management solution or use an open-source platform. Proprietary log management platforms are closed systems with limited configurability and customizability. When bugs are discovered on proprietary platforms, the customer must rely entirely on the vendor to resolve any anomalies and apply any necessary patches or make any necessary upgrades. Since proprietary SaaS log management platform vendors control the versions of their platforms, updates to SaaS platforms typically affect all customers at the same time.

The Managed Services Provider can configure the Centralized Log Management System to meet their specific requirements of the client. Customizability may be in the form of:

• VPN access to infrastructure
• AWS Direct connect to infrastructure
• Direct access to ES for querying data
• Customized front-ends
• Customized code (forked versions of ELK stack)

After taking a close look at the various features of log management solutions such as, flexibility, resources, capacity, and retention, it’s time to start evaluating different log management solution providers. Our Essential Guide to Log Management takes an in-depth look at the top log management vendors. We’ve done the research and put in the time and have created the ultimate comparison so you can see exactly how different vendors stack up against each other.