Your firewall is the first line of defense against cyber attacks. No doubt firewalls are the most critical component of any security infrastructure for safeguarding your enterprise network. They prevent unauthorized access to your enterprise assets and protect your data from being compromised. When your firewall experiences outages, there is an inherent risk of exposing your environment. 

Organizations are deploying High Availability architecture with the use of active-active or active-passive clusters to minimize downtime. Even with High Availability, it is important for  organizations to take any measures necessary to avoid downtime. Proactively monitoring your security infrastructure is the best way to prevent network outages and unplanned downtime. In this blog post, we will share a few recent outages reported by our customers. By sharing these incidents, we’re hoping to create awareness and help you avoid outages in your environment. We will also discuss how Indeni is helping organizations around the globe to achieve excellent uptime. 

1. BGP peering issues breaking connectivity to the Internet

In Border Gateway Protocol, the Idle state can be indicative of BGP session issues. In state idle, the device is currently not trying to set up a BGP session with its peers. There can be a number of reasons for this. For example, there is no route towards the neighbor, or the neighbor refused an earlier connection attempt. 

In a High Availability environment, the idle state can be considered a valid state if the device is the passive or backup member in a cluster. If the device is the active member, that is a problem. Under normal conditions,  the idle state should only be temporary. When severe error conditions persist the session can remain idle indefinitely. Idle means that the BGP session is in shutdown state and the connection to the Internet is impacted. 

Solution

To address this error condition, we added a new Auto-Detect element to capture the number of routes on the active member in a cluster. If the number of routes is 0 in the active member of a cluster, an alert will be generated. If the number of routes is 0 in the passive member of a cluster, it is considered working as designed.

2. DDoS attacks causing service disruption

A distributed denial of service (DDoS) attack is a malicious attempt to disrupt an online service. This is usually done by overwhelming the target host or surrounding infrastructure with an excessive amount of Internet traffic, making the service unusable. Unfortunately, DDoS attacks are now an everyday occurrence. According to the Netscout Threat Intelligence report Issue 7, in the first half of 2021, cybercriminals launched approximately 5.4 million DDoS attacks, increasing by 11% from last year. The study also indicated  that if current trends continue, DDoS attacks are expected to reach a record of 11 million by the end of 2021. Incidentally, we have also seen increased DDoS attacks reported by our customers.

Check Point Security Gateways use several techniques to handle DoS/DDoS attacks. Rate limiting is one such technique to defend against DDoS attacks. You can configure rate limiting rules to limit the amount of traffic coming from specific sources, or sent to specific destinations. The SecureXL Penalty Box is a mechanism that performs an early drop of packets that arrive from suspicious sources. The purpose of this feature is to allow the Security Gateway to cope better under extreme traffic load, possibly caused by a DDoS attack. To ensure important traffic is not dropped inadvertently, you can configure a whitelist of source IP addresses in the SecureXL Penalty Box. The whitelist overrides which packet the SecureXL Penalty Box drops. 

Solution

To address the upsurge in DDoS attacks, we implemented several best practices and alerting capabilities to help you harden your Check Point Security Gateways and immediately notify you of incidents for your response. SecureXL DoS deny list, DoS log IP Penalty Box, DoS log drops, DoS pbox must be enabled for the penalty DoS functionality to work correctly. Indeni will alert if any one of these is disabled. Indeni will also alert you if the SecureXL Penalty box feature is blocking an IP address listed in the pbox-whitelist-v4.conf file. Indeni also generates an alert if the DoS blade penalty drop counter is 0.

3. Fwkern.conf configuration issues causing performance degradation after an upgrade

While there may be a need to change the fwkern.conf file to control the behaviour of Check Point Security Gateway, you want to ensure that the configuration is correct, as outlined in the sk26202 article. Recently, a customer reported performance CPU spikes after a hardware upgrade. For more information about this issue, review sk92810.

Solution

To address this configuration issue, we implemented a new Auto-Detect Element (ADE) to look for early symptoms before the problem becomes a bigger problem. The new ADE searches the VS0 log files for /vs0/dev/fw0 messages, specifically we look for “Connection refused” messages which indicate that something is broken with the fwkern.conf file.

Summary

Firewall outages are more frequent than you might think. We can help you minimize the impact of downtime, whether it is providing you early warnings or instantly detecting the problem so you can quickly remediate the situation. Share your pain with us, we may be able to help you avoid these outages.