DNS servers configured but responding too slowly: Check Point Firewall

download (1)

 

This is a real life sample alert from the indeni alert guide for Check Point Firewalls.

NOTE: While the alert described here is for Check Point firewalls, the same logic applies with other devices that are sensitive to DNS response times. The difference would be in the information provided.

Description:

DNS is configured on this device, but it is responding to queries more slowly than required. The measured response time for a query for www.indeni.com is 7456 milliseconds while the threshold for alerting is 250 milliseconds.

DNS response time is important for certain functions, such as Domain Objects. For more information, read SK41632.

Possibly Problematic DNS Servers:

8.8.8.8
17.15.201.22

Manual Remediation Steps:

Review the DNS configuration, firewall rules, routing tables and other elements of the network to determine the cause.

How does this alert work?

indeni runs the command “nslookup www.indeni.com” (or the respective command for the given device being analyzed) every hour and times how long it takes to complete it. To avoid false positives, indeni includes a number of mechanisms that ensure the accuracy of the results. The address “www.indeni.com” is configurable.

Get this and thousands of other checks performed algorithmically, 24/7/365. 

Want to see what indeni can find lurking in your environment? Click the pic below to try indeni for free:

Check Point Firewalls
indeni is like insurance for your Check Point Firewalls

 

VPN Gateway Configuration Issues: Clear Packets Drop : Check Point Firewall Alert Guide

This is a real life sample alert from  the World leader in Proactive Network Management for your Check Point Firewalls.

Description:

Certain packets are being dropped. This is happening because the local VPN gateway is receiving packets in the clear while the current configuration states they should be encrypted.

Note that indeni automatically identifies this issue as resolved when a log appears in the SmartView Tracker with the same destination IP address and a Decrypt action (instead of Drop) OR if sufficient time has passed since the error has appeared.

Affected Source IPs:

1.2.244.11
1.2.244.12

Manual Remediation Steps:

Review the VPN configuration on both sides of the VPN tunnel. If the packet should not be encrypted, modify the VPN configuration of the local VPN peer to reflect this (either update the encryption domain or update the list of services for which traffic should not be encrypted). If the packet should indeed be encrypted, update the VPN configuration of the remote peer to ensure it sends the packets through the VPN tunnel.

General tips for VPN debugging can be found on CPShared and SK33327.

How does this alert work?

indeni loads the VPN configuration from the management servers (CMAs, for example) as well the current VPN status from the gateways (‘vpn tu’ command, for example). indeni also listens to the logs coming out of the firewalls that indicate possible VPN issues (such as “encryption failure: cleartext packet should be encrypted”). In the case above indeni tracks the logs for dropped/rejected connections under the IPSec VPN blade to identify when there are drops due to a misconfiguration on either side of the VPN tunnel.

F5 and Palo Alto Networksx: Preemption enabled on cluster member

NOTE: This sample alert is provided for F5 devices. indeni supports a similar alert for Palo Alto Networks firewalls, see DOC-7857.

This is a real life sample alert from indeni

Description:

Preemption in clustering means that if this device’s health is degraded and there’s a fail over in the cluster to another member, and then later this device’s health is restored, it will attempt to assume priority in the cluster. This can have some negative impacts, such as constant fail overs in a cluster (if the health issue resumes) as well as issues with surrounding equipment (that are no longer “sure” which member is the active one).

Manual Remediation Steps:

Disable Auto Failback under Traffic Groups.

How does this alert work?

indeni reviews the configuration for any features that are enabled or disabled that are known to cause issues. This is essentially a “best practice” type alert. In the example above, a best practice for configuring F5 LTMs is being verified.

cphaprob syncstat: What is it? What Can you do with it for Check Point Firewalls?

This is a real life example of interesting Check Point Firewalls commands we found:

 

There is a command in the Check Point  world that isn’t widely known:
cphaprob syncstat

It’s for clusters. It’s there to share some stats on how the clustering mechanism is behaving and help identify certain issues. While there’s a Secure Knowledge article about this, we’ve discovered most users either don’t know the command exists or don’t know what to do with the output. Here’s a sample output:

Sync Statistics (IDs of F&A Peers - 1 2 3 4 5 6 7 ):  Other Member Updates: Sent retransmission requests...................165 Avg missing updates per request................1 Old or too-new arriving updates................5661 Unsynced missing updates.......................0 Lost sync connection (num of events)...........4354 Timed out sync connection .....................1  Local Updates: Total generated updates .......................9180670 Recv Retransmission requests...................1073 Recv Duplicate Retrans request.................2564  Blocking Events................................0 Blocked packets................................0 Max length of sending queue....................4598 Avg length of sending queue....................0 Hold Pkts events...............................1 Unhold Pkt events..............................1 Not held due to no members.....................16 Max held duration (sync ticks).................0 Avg held duration (sync ticks).................11  Timers: Sync tick (ms).................................100 CPHA tick (ms).................................100  Queues: Sending queue size.............................512 Receiving queue size...........................256

So – we’ve done quite a bit of work trying to understand what is possible with the stats. We’ve used some documentation made available by Check Point and others, as well as our own analysis. Interestingly, our indeni Insight service was extremely useful as some things we thought were true turned out to be wrong.

For example – we thought that every sync loss (see “Lost sync connection (num of events)” above) was a problem. But no, in almost every policy installation you’ll have up to 6 of these.

At the end of the day, our experience shows that an in-depth analysis of the syncstat can prove very useful for finding possible clustering issues. So, if you’re running into weird clustering issues with Check Point firewalls, take us for a spin, it’s free. If you got to this page because you saw an alert in indeni and are looking for clarification (this syncstat thing can get complicated) – you can use the comments section at the bottom or contact our support (best to simply copy and paste the alert text in the support ticket).

 

Get this and thousands of other checks performed algorithmically, 24/7/365. 

Want to see what indeni can find lurking in your environment? Click the pic below to reserve your spot at our High Availability Webinar:

Check Point Firewalls
indeni is like insurance for your Check Point Firewalls

F5 SSL transactions per second (TPS) limit nearing or reached

This is a real life sample alert from indeni

Description:

This F5 device is licensed to handle up to 200 SSL transactions per second. It is currently handling 190 SSL transactions per second.

Manual Remediation Steps:

Review your settings and devices to determine if more licenses need to be purchased or you should change the configurations of the nodes or pools. Review SOL6475.

How does this alert work?

indeni monitors the SSL TPS counters and compares them to the licenses installed on the device. There is no need to configure within indeni what the licenses allow, that is fetched automatically.

NIC Duplex Set to Half Instead of Full: Router and Switches Configuration Guide

This is a real life sample alert from the leader in Proactive Network Management software: indeni.

Description:

Some NIC’s duplex is set to half instead of full. This may be an auto-negotiation error.

indeni will re-check this alert every 1 minute. If indeni determines the issue has been resolved, it will automatically be flagged as such.

Affected NICs:

eth5 (Bandwidth: 100M/half, MAC Address: 00:1C:7F:31:23:D8)

Manual Remediation Steps:

Please ensure that all physical network connections are in place, all routers and switches are properly configured and identify any recent changes which may be causing this.

How does this alert work?

indeni monitors the critical stats of all network interfaces and ports constantly. When issues are found, such as half duplex (which, in reality, should almost never happen), indeni alerts and attempts to provide as much information as it can.

 

Get this and thousands of other checks performed algorithmically, 24/7/365. 

Want to see what indeni can find lurking in your environment? Click the pic below to try us out for free for 14 days:

Check Point Firewalls
indeni is like insurance for your entire network
[ninja_form id=10]

 

Check Point Alert of the Week: Policy installation resulted in high CPU load, cluster may failover

This is a real life sample alert from indeni

Description:

A new policy has been recently installed. During the policy installation it appears that there has been a CPU usage level of more than 70% for a period of at least 30 seconds. This may result in a failover from the currently active cluster member to a standby one. A failover during high CPU loads may result in certain network traffic being blocked temporarily.

Manual Remediation Steps:

In order to avoid the failover and the possible loss of connectivity, set the kernel parameter fwha_freeze_state_machine_timeout to 30 or 60 seconds on all cluster members.
To set a kernel parameter you can use the “fw ctl set int” command. If you would like for the change to survive a reboot, you should place the new value in the fwkern.conf file (for SPLAT or GAiA) or by using modzap (for IPSO).
indeni recommends consulting with your technical support provider prior to changing a kernel parameter.

Read SK32488 for more information.

How does this alert work?

indeni tracks the CPU usage on firewalls as well as logs when policy installations are done. Once an installation is complete, indeni looks back at the CPU usage. If it’s above 70% (by default) on a ClusterXL cluster member, an alert is issued.