Inside Auto-Triage

Many enterprises are expecting to continue operating remotely through the end of the year. This new operating environment has significantly increased workloads on security infrastructure operations teams. Luckily, Indeni’s automation capabilities can help take a few things off their plates by troubleshooting and capturing pertinent information required to report an incident.

Explore how Indeni’s Auto-Triage capabilities play a key role in keeping your remote operations secure and fully functional 24/7.

How does Auto-Triage help?

Once Indeni identifies an issue, it can run its own investigative steps, the same ones that are normally run manually. The steps can be as simple as gathering additional contextual diagnostics information, or as in-depth as analyzing and performing common troubleshooting tasks. Applying best practice procedures reduces time to resolution, whether that’s providing an engineer all of the information or by automatically narrowing down the issue with even more prescriptive remediation recommendations.

An example: Log management for maximum visibility

The sudden remote workforce model has changed the operating conditions. Logs are the primary data source for forensics and security incident responses. Not only does log analysis increase security awareness, it rapidly detects failed processes, network outages, or protocol failures. It also helps in the effective management of applications and infrastructure. If log collection doesn’t happen, it is considered a P1 (a high priority) event. For businesses to prevent service disruptions and detect threats, they must rely on logs and they must continuously monitor log collection, at scale.

What can possibly go wrong with log collection?

1. Logging rate is higher than what the device can handle.
2. Devices are stressed due to a high number of connections.
3. Unable to reach the log management server(s).
4. Network connection issues.
5. Limited local storage on the device to temporarily store log data.

Auto-Triage ensures continuous log collection

Indeni continuously monitors log collection by tracking the log-forward discards of a Palo Alto Network device. When logs are discarded, Indeni will immediately notify users and automatically initiate an investigation. 

The first investigative step is to retrieve the logging rate from the device.

Indeni will determine if the logging rate is within the device limit. If the logging rate is exceeding the device rate limit, logs are discarded. This could be a hardware resource limitation. 

Indeni will suggest workarounds such as disabling logging for some types of traffic (DNS and PING). An alternate suggestion is to only log the container page and not subsequent pages. Because URL filtering can potentially generate a lot of log entries, this could be a viable workaround.

If logging rate is within the device limit, Indeni will check session utilization.

Discarding logs can be caused by an increase in traffic rate that exceeds the device limit. Some types of traffic create more sessions (e.g. DNS, PING) and use more resources for session lookups, log at session start, log at session end, just to name a few.  If session utilization is above 70% of the device session table, it is considered high. If this issue persists, this could be an indication of hardware limitations. 

If session utilization is below 70%, Indeni will determine if the log collectors are reachable and connection can be established. 

In conclusion, if logging rate is within the device limit, session utilization is below 70% of the connection table, and the collector is reachable, it would be time to open a trouble ticket with Palo Alto Networks. With the automated investigative steps, users have just gathered the relevant information required to open a case. 

Next Steps

Indeni is here to help you maintain business as usual. If you are a customer, we encourage you to enable Auto-Triage, geared to help you run your security infrastructures smoothly during this challenging time. If you can’t find the Auto-Triage Element (ATE) you’re looking for, you can always submit a request to our community. If you are new to Indeni, we would love to give you the chance to try our automation capabilities in your own environment.