Palo Alto Networks firewalls: Job(s) stuck in pending

This is a real life sample alert from the indeni alert guide for Palo Alto Networks Firewalls.


One or more jobs running on this device have been stuck in “pending” state for more than 30 minutes.

Affected Jobs:

  • EBLRefresh

  • Install

Manual Remediation Steps:
Review the jobs listed above for possible issues. You may want to stop and re-issue the job if possible. For more information read DOC-2259.

How does this alert work?
indeni reviews the current list of jobs on a regular basis by running “show jobs all”. For this alert, indeni looks for jobs that have been stuck in PEND for more than 30 minutes.

Pan(w)achrome for Palo Alto Networks firewalls from the indeni perspective

Pan(w)achrome is a chrome extension written by Luigi Mori, a solutions architect at Palo Alto Networks. The extension lets you connect to your Palo Alto firewalls and keep track of certain vital stats – mostly CPU, memory, traffic and a variety of counters.

This is a good step in the right direction – every product we support today has some sort of tool for visualizing some basic stats. Check Point has SmartView Monitor. Juniper has Junos Space. Fortinet has the capabilities in the Fortigate’s web UI as well as within FortiManager. The nice touch with Pan(w)achrome is that it’s built into the browser (through the extension) so it’s a bit easier to access.

As PAN-OS progresses (see the recent release – 7.0), we, at indeni, hope that an investment into Pan(w)achrome will be made. Firstly by taking ownership of the extension at the R&D level (and not a side project), as well as providing much deeper visibility into important elements of the Palo Alto Networks firewalls.

At indeni, we see tools like this as a great means of providing customers with some visibility. The challenge, though, is that these tools are not capable of analyzing configurations and logs (as these are too complicated and are not graphable) and cannot be used as alerting systems. It’s simply far outside their scope.

So, if you’re looking for in-depth configuration and log analysis, as well as comprehensive operational data collection, give indeni for Palo Alto Networks firewalls a spin. Takes just 45 minutes to set up.

Where does indeni fit in your environment?

Prospective users often try to figure out how indeni will fit in their environment. This document will hopefully help you visualize this better.

Click on the image to enlarge

The world before indeni

Since you’re reading this, we’ll assume you are responsible for some kind of Network Devices – switches, routers, firewalls, load balancers, etc. If you have a sizable deployment, you probably also use a Manufacturer-Provided Management Application (such as Palo Alto Networks Panorama, F5 Enterprise Manager, Check Point Provider-1, Cisco Prime, etc.). These management applications are great for configuring your devices, but generally are somewhat challenged in providing you with insight into the operational health of those devices.

We’d also bet you have a Network Monitoring System to see how your environment is doing – maybe one of the more expensive products (HP NNMi), or the cheaper ones (SolarWinds Orion NPM) or even the free ones (Nagios). These systems are great at telling you what’s up, what’s down and show green or red lights. However, making sense of the data is difficult, and they generally “dumb-it-down” to the lowest common denominator. So the level of visibility you get is low.

You may have your Network Devices send their logs to a Log Server – such as Splunk (if you’ve got more budget) or Kiwi (if less). These are extremely useful at storing logs and helping you sift through them and run complex queries – to troubleshoot an issue once it happens. They are not good at telling you before a problem happens, because they don’t truly understand what the logs mean.

In larger enterprises we also see IT Operations Management systems, which centralize the monitoring information coming out of various systems as well as help with tracking service tickets, assets, etc.

If you are on the security side of the house in larger organizations you may also be using a Firewall Policy Manager, such as Tufin or Algosec, which is great for tracking how well the firewall policy is configured.

Click on the image to enlarge

Where does indeni fit?

indeni was created to deal with specific shortcomings of the setup described above. Namely – that there are many systems out there that are great at collecting data (logs, configs, stats, etc.) but not very good at making sense of it. So, naturally, we’ve invested most of our R&D effort into the mechanisms required to figure out what to do with the data and how to make it actionable. Specifically, providing pin-point insights and suggesting remedial action to ensure problems are averted.

However, we also had another requirement – indeni must be easy to set up and take no more than an hour to get up and running. To achieve this, we needed to have indeni capable of collecting the data it needs on its own (hence the direct arrows from the Network Devices to indeni in the diagram). When indeni finds an issue, it will send alerting information over to the operations’ management systems as well as emails directly to the operations and engineering teams.

To summarize– indeni fits side by side with your current systems to ensure you get actionable alerts while keeping your current operations’ processes the same. It automatically makes sense of the data and runs the troubleshooting steps required to get to the bottom of many issues.

Interested in digging deeper?

Are you ready for 99.9999%? Try indeni for free and see what is lurking in your network.


RX Traffic Drastically Reduced Post Fail Over. Palo Alto Networks Alert Guide

This is a real life sample alert from indeni alert guide for Palo Alto Networks Firewall.


This device is receiving far less traffic than expected. It is receiving 142 packets/sec at the moment, compared to 15921 packets/sec it received a few minutes ago. This can be a result of a fail over of this firewall cluster.

Manual Remediation Steps:

Consider clearing the ARP cache, as detailed in DOC-4575. Review the comments of that DOC.

How does this alert work?

indeni tracks the traffic flow on firewalls to identify situations where there is a sharp decrease in RX traffic (as opposed to TX traffic). Such a drop in RX traffic means the surrounding network equipment isn’t forwarding traffic to the firewall, usually due to ARP issues.

How To Find Out When Your SSL Certificate Expires on F5 BIG-IP DNS

indeni, cisco

Do you know when the SSL certificate expires on your F5 Load balancers?

Every single deployment of LTM ® we’ve encountered has SSL termination included in it. Think about it – it makes sense, it’s one of the strongest advantages of the F5 hardware.

However, every single deployment we’ve encountered also had SSL certificates configured that have expired or were expiring in the next three months. Apparently, staying on top of your SSL certs isn’t as straightforward as you’d want it to be.

So, we thought we’d put in the effort to summarize in a short post how does one get notified, ahead of time, when SSL certificates expire on their F5 BIG-IP DNS LTM:

  • Buy Enterprise Manager – it has a built-in feature for doing this.
  • Get BIG-IQ, can be done there, too.
  • Write a script – read DevCentral and SOL15288.
  • Run indeni – you can get a limited license free and easy by going here. Within 45 minutes you can easily know which SSL certs need refresh, as well as hundreds of other possible issues lurking in your F5 configuration. You can even run it every 6 months or so, to make sure you’re in top shape.

For your information, this is how the alert would look like in indeni:


Some SSL certificates are about to expire or have expired.

Certificates expired or about to expire: expires on November 30, 2016

Manual Remediation Steps:

Replace the SSL certificates with new ones.

For more information on how to manage certificates, refer to Managing SSL Certificates for Local Traffic in the F5 user guide.

How does this alert work?

indeni retrieves the SSL certificates configured on an F5 BIG-IP DNS device and analyzes them: checking their expiration date, their validity (are they self-signed or signed by an internal CA?), etc.

F5 Too many RST packets sent

indeni, cisco

This is a real life sample alert from indeni from our F5 Load Balancing Methods Library


This device is being hit with too many connections that appear to have already been closed or never opened. It is possible the device is under DDoS attack. indeni has found this log message:
May 18 12:49:43 JCNC-ADC1 warning tmm1[11241]: 011e0001:4: Limiting open port RST response from 251 to 250 packets/sec 

Manual Remediation Steps:

Review SOL13151 and review the cause of this sudden increase in unexpected connections.

How does this alert work?

indeni crosses information from the log files with SOL’s listed on to identify when certain logs should receive attention.

Comparing indeni and Check Point’s SmartWorkflow and Compliance blades

The summary:

SmartWorkflow helps you track your rulebase, the Compliance blade helps you identify specific configurations that are not in compliance with known security regulations. Both exist in order to ensure your firewall configuration is secure and so, your network is secure. indeni’s role is to make sure your firewall works – performance, log flow, routing, kernel parameters, SIC connectivity, licensing, contracts, etc.

Therefore, indeni is an amazing fit with your SmartWorkflow and Compliance blades.

The longer version:

This image was copied from


From “The Check Point SmartWorkflow Software Blade provides a seamless and automated process for policy change management that helps administrators reduce errors and enhance compliance. Enforce a formal process for editing, reviewing, approving and auditing policy changes from a single console, for one-stop, total policy lifecycle management.”

What that means is that you have a way of tracking changes made to the firewall rulebase (policy) to ensure that these changes are being done in the correct way. The reason for doing this is simple: the firewall is the gatekeeper to your network and you want to make sure that the traffic it lets in is the one you want it to.

indeni has no ability to track the firewall policy or identify changes to the rules. We have specifically refrained from doing that as we know the market has very capable solutions for dealing with this challenge – some provided by the firewall manufacturer (like SmartWorkflow) and some by third parties.

Compliance Blade:
From “The Check Point Compliance Software Blade monitors your management, Software Blades and security gateways to constantly validate that your Check Point environment is configured in the best way possible. The Check Point Compliance Software Blade provides 24/7 security monitoring, security alerts on policy violations, and out-of-the-box audit reports.”

So the compliance blade is there to make sure you are in compliance with the variety of regulations that you must adhere to – DSD, HIPAA, PCI DSS, etc. Each of those regulations lists a set of requirements that must be followed on your firewalls – from ensuring stateful inspection is used to how connections are timed out. Each regulation has a different set and there’s some overlap between them.

indeni focuses on the operational health of the firewall, as well as OS-level configurations that need to be done for compliance purposes (like what users are defined). Therefore, it augments the compliance blade by ensuring that the firewall isn’t only secure and following security best practices, but also alive and kicking.

The bottom line:

indeni is a great fit with the SmartWorkflow and Compliance blades. It ensures that your security meets standards and regulations and your network is up and running without issues. With indeni, you can find all of the issues listed below, which you cannot find with the SmartWorkflow and Compliance blades:

  1. Gateway cannot access certificate authority
  2. Policy installation resulted in high CPU load cluster may failover
  3. Firewall log file increase rate critical – possible connectivity loss to log server
  4. Firewall kernel table limit approaching or reached
  5. ClusterXL member is in a critical state
  6. Cluster member down due to NIC error
  7. Some received packets have been dropped by NIC (SA#24915)
  8. High memory usage
  9. DNS servers configured but responding too slowly
  10. Use of NTP servers configured but not operational
  11. Firewall Connection Table Limit Approaching or Reached
  12. A NIC has failed recently (SA#24915)
  13. RX traffic drastically reduced post fail over possible ARP issue
  14. Two cluster members differ in their routing tables (SA#66322)
  15. DNS server resolution test failed
  16. NAT connections (fwx_alloc) table limit approaching or reached
  17. Errors have been found in packets transmitted by NIC (SA#24915)
  18. ARP table is approaching its limits (SA#25890)
  19. VPN gateway is dropping unexpected packets (SA#22255)
  20. NIC duplex set to half with speed of 10mbps or 100mbps (SA#24967)

Announcing indeni 5.2: Palo Alto Networks beta, improvements and bugfixes

Welcome 5.2!

In this release we’ve included many improvements to the underlying infrastructure and bugfixes, as well as kicked off the beta for our support of Palo Alto Networks firewalls. Please reach out to our support team to get the updated release. Note that between minor releases (such as 5.1 and 5.2) we make interim releases with new content and bugfixes on a weekly basis. You may have received a previous release of 5.2, which we recommend you upgrade to the newest one announced today.

New products and versions supported:

  • BETA of Palo Alto Networks firewalls running PAN-OS 6.x.x. If you are interested in joining the beta, fill out the form.
  • IK-1675: Support CP R77.30
  • IK-1840: Fortigate: Added support for FortIOS V5.2.1

NOTE: Customers who require support of a given product version prior to the main release can contact and a running build will be provided.

Select new signatures:

  • IK-1677: Firewall is running with a trial license (Check Point)
  • IK-1836: Enhanced “BIG-IP node availability issue detected” (F5)
  • IK-1825: ConfigSync operational status issues (F5)
  • IK-2020: The BIG-IP system is near or out of disk space or inodes (SOL12263, SOL14403) (F5)
  • IK-2021: “Possible multicast or broadcast loop on SFP NICs detected” (F5)
  • IK-1834: “Load balancer node connection limit nearing (or reached)” (F5)
  • IK-1831: “Number of active members in pool is lower than required” (F5)
  • IK-1835: “Pool member connection limit nearing (or reached)” (F5)
  • IK-1827: “SSL transactions per second (TPS) limit nearing or reached” (F5)

Bugs fixed and minor improvements:

  • IK-1674: “A NIC has failed recently (SA#24915)”: reduced the number of log lines shown
  • IK-1518: “Cluster Members Identical Kernel Parameter Values Verification (SA#66322)”: additional dynamic parameters ignored
  • IK-1859: “DNS server resolution test failed” – eliminate false positive in Cisco devices
  • IK-1672: “Errors have been found in packets received by NIC (SA#24915)” triggered for very low packet count
  • IK-1704: “Communication with device suspended due to 2 reboots” false positive
  • IK-1712: “Hardware has reached end of support” is auto-resolving
  • IK-1683: “Hardware temperature sensor reading too high” false positive
  • IK-1391: “High storage usage has been measured” doesn’t show list of large files in Cisco devices
  • IK-1871: “HSRP cluster members differ in VLAN configuration” false positive
  • IK-1858: “License(s) have expired” false positive for CP licenses with expiration “never”
  • IS-1349: “Max SSH Session Count” remains at default
  • IK-1976: “Monitoring Suspended” creating too many alerts
  • IK-1958: “NAT cache (fwx_cache) table limit approaching or reached” false positive
  • IK-1879: “NAT connections (fwx_alloc) table limit approaching or reached” false positive
  • IK-1901: “RX traffic drastically reduced post fail over, possible ARP issue” add specific interface details
  • IK-1919: “SecureXL templates are partially disabled” false positive
  • IK-1914: “Some members of the same cluster are not being monitored” false positive
  • IK-1731: “Some proxy ARPs required by NAT are missing” – signature removed
  • IS-1000: “Some received packets have been dropped by NIC (SA#24915)” – duplicate text in e-mail alert details
  • IK-1628: “Two cluster members differ in their routing tables” failing to create alert
  • IK-1870: “Use of NTP is configured but no servers are defined” false positive
  • IK-1684: “Voltage too high or too low” false positive
  • IK-1702: “Voltage too high or too low” – don’t alert if hi/low limits are unknown
  • IS-1454: Backup: sometimes old backups are not deleted
  • IK-1501: “Proxy ARP is enabled” flapping in Cisco
  • IK-1696: GAiA R77.10: Replace use of ckp_regedit with cpinfo
  • IK-1846: ClusterXL member differences alerts are referring to the wrong cluster members
  • IK-2133: Configuration Check – “Hotfix(es) Installed” does not handle comma delimited string of HFs correctly
  • IS-1077: Connection to SecurePlatform with SSH private key fails
  • IK-1741: Correctly identify device model for CP 21700
  • IK-1742: Correctly identify device model for CP 4400
  • IK-1670: Live Configuration – all NICS are showing as Down
  • IK-1856: Hardware alert false positives from Check Point open server
  • IS-1346: Prevent “service indeni4it start” from starting the application more than one time
  • IK-1690: “Route overlap identified” – don’t alert when next-hop is the same
  • IK-1688: NIC stats alerts (e.g. packet errors) should contain the total number of packets that we compare against
  • IK-2066: SmartCenter degradation due to hanging “fw log” processes
  • IK-1993: SmartCenter backup: use “migrate export” for R75.40 and above
  • IK-2067: Reduce “sshd[xxx]: Did not receive identification string from <indeni server>” in device messages log
  • IS-1037: Update by UPD fails to restart the service
  • IK-1966: Crossbeam discovery failure
  • IS-1453: Backup Report – empty “Failed Backups” section header
  • IS-1348: Scheduled Reports delivery does not follow DST changes
  • IS-1441: F5 – wc should not show the groups common/device_trust_group and common/gtm
  • IS-1036: E-mail Alerts: remove PDFs from e-mail alerts
  • S-1019: Tools-Troubleshooting – add “cpstat os -f sensors” for Check Point firewalls
  • IS-1765: Alert Report – add alert timestamps&nbsp;
  • IS-1060: Alerts e-mails – add alert timestamp