How to monitor F5 devices – SNMP vs API vs SSH

F5 has many ways of interfacing with their products and when writing monitoring we had to do some research which one is more suitable in terms of performance. After all, monitoring should not harm the device it monitors. When choosing methods we looked into iControl REST, SNMP and TMSH. See below for how this test was conducted and which one won.

The best way to monitor F5 – How the test was conducted

We ran each type ~20 minutes continuously through command-runner. While running the tests the web interface was used to make sure that the web interface responsiveness was up to par.

The commands to run each test

#REST
while true; do
command-runner.sh full-command –basic-authentication user,password rest-pool-statistics.ind 10.10.10.10
done
#tmsh
while true; do
command-runner.sh full-command –ssh user,password ./show-ltm-pool-detail-raw-recursive.ind 10.10.10.10
done
#SNMP
while true; do
command-runner.sh full-command –ssh user,password ./snmp-pool-statistics.ind 10.10.10.10
done

Results

The test started out with 283 pools (with 200 additional ones created just for this test). However, when trying the tmsh command, command-runner timed out, so we had to reduce to the original 83 pools and rerun the test using rest to make it fair.

  • Test 1: REST = 283 pools
  • Test 2: Tmsh = 83 pools
  • Test 3: SNMP = 83 pools
  • Test 4: REST (take 2) = 83 pools

4 hour graph

24 hour graph for reference

REST

  • Did not produce any timeouts in the GUI in any of the two tests.
  • Always produced results.
  • Management interface only became sluggish one time during the second attempt. Most likely because of the already high swap usage created by the TMSH tests.

TMSH

TMSH produced these once in awhile:

  • When that happened you can see the gaps in the graph. It is unknown what the gap after the graph was because we was working on the snmp metrics at that time.
  • TMSH also failed to give results sometimes.
  • Forced to run with fewer metrics than rest in order to even get a result.

SNMP

  • Truncated the pool names sometimes. It is unclear why ast was always done on long names, but different lengths.
  • Did not produce any timeouts in the GUI.
  • Always produced results.
  • Did not have as many metrics as REST since the exact same metrics was not available in one command (pool state and availability is missing).
  • Management interface became a bit sluggish on and off.

Conclusion

Over all REST won the test with SNMP as second. TMSH did not even qualify as it takes up very large amounts of memory and swap which negatively affected the overall system.

Thank you to Patrik Jonsson for contributing this article.

How to select script monitoring authentication types

Considerations when selecting authentication types

Choosing an authentication method for monitoring your infrastructure devices might sound easy at first glance. After all, a monitoring script would only need read-only, right? Wrong.

Monitoring with indeni goes beyond what normal monitoring tools does. The goal of indeni is to detect problems before they occur, saving you hours of troubleshooting and root cause analysis down the road. To get early detection indeni needs access to the advanced shell. Let’s take a look at what this means on F5 devices.

Example: Selecting authentication types for F5 devices

On an F5, having access to the advanced shell means that the user in question must have administrator access. Also, iControl REST requires the user to be locally authenticated up until version 11.5.4. This means that for systems running versions up to 11.5.4 using RADIUS for authentication administrators will have to resort to the local admin account for REST calls.

On top of that if a system has configured authentication and authorization using RADIUS there is no way of setting the shell to advanced shell on any version. So yet again, administrators must resort to the local admin account in order to set the proper permissions.
We have gone above and beyond to avoid using local admin accounts by investing a lot of time running monitor commands via TMSH. However, this has turned out to cause harm to the system due to TMSH using way too much memory. So what does this mean? In order for get the most out of using indeni, administrators will have to configure authentication according to the following table:

Version
Authentication
Authorization
User
11.5.4 and earlier
Any
Any
Local admin (with SSH access)
11.6.0 and later
Remote
Remote
Local admin (with SSH access)
11.6.0 and later
Local
Local
Any account with role Administrator and shell set to Advanced Shell
11.6.0 and later
Remote
Local
Any account with role Administrator and shell set to Advanced Shell

Thank you to Patrik Jonsson for contributing this article.

Machine learning for logs, cut through the hype.

 

Splunk recently announced new machine learning capabilities in its Splunk Cloud and Splunk Enterprise 6.5 release. Does everyone have machine learning capabilities now? What exactly is machine learning? See below for key considerations for this technology approach and how indeni’s machine learning differs from Splunk.

3 things IT needs to know about machine learning


  • Machine learning algorithms have been around for decades. Most of them, especially those that are mathematically based, are not new. For example Arthur Samuel coined the term “machine learning” in 1959!
  • Machine learning works best with large sets of data. You need a substantial amount of information to determine trends, correlations, etc. Take the example of the NVIDIA self-driving car that was shown at CES this year. Only after 3000 miles of driving on highways, back roads and suburban roads was the car able to stop running over traffic cones and avoid parked cars.
  • If not constrained, Machine learning will have a very high false positive rate. To continue the analogy from above, say you are monitoring multiple types of automobiles. Comparing the device data of a semi-truck to a Tesla would be interesting, but not actionable. Say one of your rules was to alert if the engine noise exceeded 100 decibels, as you believe this level of noise indicates there is an issue with the engine. A semi-truck would generate an alert every time it turned on, whereas a Tesla would hardly say a peep. Giving your machine learning constraints (eg. compare Tesla data only with other Tesla’s) yields far more accurate results.

Moral of the story, if a vendor pitches you on “machine learning” it’s OK to be optimistic but be cautious. Here are some questions you can ask to see if the machine learning will make your team more productive:

  • How does the vendor help the algorithm focus on the important elements? How do they help their technology understand the data to reach the right conclusions?
  • How do they avoid a high rate of false positives? For example, if their machine learning algorithms find “an anomaly” what are the chances it’s a true positive?
  • How does the vendor make its alerts or findings actionable?

4 ways indeni machine learning differs from Splunk


Now that we are on the same page for machine learning, here are four ways that indeni differs from SIEM and Log Management solutions such as Splunk.

#1 indeni ingests configuration data in addition to statistics and logs of devices.

Collecting greater depths of information on devices and the software running on them allows indeni to identify issues with greater accuracy.

#2 indeni has the largest database of device knowledge.

indeni has a growing repository of known infrastructure issues and resolution steps for the largest Enterprises. This information is gathered from our customers, indeni engineers and third party experts around the globe. How does this help on a daily basis?

  • Root cause analysis: Instead of coming up with a hypothesis and then building a query in Splunk so that you can schedule alerts when the same log or event occurs, indeni has the knowledge built into its core alerting system, no scripting or queries required.
  • Troubleshooting: When you receive an alert in indeni, in addition to telling you the affected device or error code, indeni provides a human readable description, the implication of not addressing the event and steps to resolution, helping network and security operations teams prioritize focus areas and shorten the mean time to resolution.

#3 indeni connects admins and engineers across the globe

In addition to applying machine learning to the data in your environment, indeni learns from other indeni customers and applies those learnings to your indeni instance. Our users subscribe to a service called “indeni Insight,” which sends data from their environment to our central repository. The data is sanitized and contains general device characteristics and behavior information. For example what model the device is, what software is running on it, which features are enabled, the status of licenses or contracts, running metrics (CPU, memory, etc.), system logs, active users and much more. The result for administrators and engineers? It’s like leveraging the expertise of thousands of your IT operations friends at Fortune 500 companies.

#4 indeni’s algorithmic model is based on the assumption 99.9% of the time devices are working as expected.

Based on our experience as network and security professionals, we know a device malfunctions only 0.1% of the time. In addition, it is widely documented that 70% of network outages occur due to device misconfigurations. These two constraints are built into our machine learning algorithm which allow us to reduce false positives, saving our customers time and money.

At a glance: indeni vs. Splunk

SimilaritiesDifferences
  • indeni and Splunk ingest data from a variety of devices
  • indeni and Splunk machine learning are based on algorithmic models
  • indeni and Splunk machine learning correlate data
  • indeni ingests configuration data in addition to statistics and logs of network devices
  • indeni has the largest database of device knowledge
  • indeni connects admins and engineers with each other
  • indeni’s algorithmic model is based on the assumption that 99.9% of the time devices are working as expected. We help you find the .1%

Conclusion


indeni is capable of identifying specific issues, which pertain to specific types of products and even specific software builds, at a level of accuracy and actionability never seen before. With indeni customers can find health and operational issues before they happen in their infrastructure, proactively handle them and have a better life. Interested in trying indeni in your environment? Contact us or engage with one of our registered partners.

Vulnerabilities from SWEET32 in F5 Load Balancers Reveal

How to Mitigate Vulnerabilities from SWEET32 in

F5 Load Balancers

The SWEET32 vulnerability is targeting long lived SSL sessions using Triple DES in CBC mode. The attack targets the cipher itself and thus there is and will be no hotfix for this. The only way to mitigate is to either disable the 3DES-CBC ciphers or set a limit on the renegotiation size.

This guide will cover how to configure both in the load balancer, and also how to protect your management interface (only possible by changing the cipher string).

For the official F5 article, please refer to this link: https://support.f5.com/csp/#/article/K13167034

Configuring an SSL Renegotiation size limit

SSL renegotiation options determines if the BigIP will allow the client to make mid-stream reconnections or not. The option called size limit will set a limit, in megabytes, of how much data that is allowed to pass through the connection before an SSL renegotiation will be disabled (on that specific connection). Since the attack targets long lived sessions (or rather high amount of SSL data) this will mitigate the attack. F5 recommends to set the limit to 1000MB.

Possible impact of setting a renegotiation size limit

Clients with renegotiation support will need to be able to handle a failed SSL Renegotiation and instead establish a new one completing the full SSL handshake. This should not pose a problem in the vast majority of applications.

SSL session settings are regulated by SSL profiles. While you can change the parent profiles we do not recommend doing so without doing proper testing.

Procedure to set an SSL Renegotiation size limit via the GUI

  1. Log in to the F5 administration interface
  1. Navigate to Local Traffic -> Profiles -> SSL -> Client
  1. Click on the profile you wish to set the limit to
  1. In the settings, locate Renegotiation size
  1. If it’s disabled (see the picture below), click on the custom option checkbox to allow changes

  1. Change the value from Indefinite to Specify, then enter 1000

  1. Click on “Update”

Procedure to set an SSL Renegotiation size limit via TMSH

  1. Connect to your load balancer via an SSH client
  1. If not already in the tmsh, enter it
  1. To modify the size limit of a profile, issue this command and replace the profile name with the full path of the ssl profile you wish to modify:

modify /ltm profile client-ssl <profile_name> renegotiate-size 1000

Example:

modify /ltm profile client-ssl /Common/example.com renegotiate-size 1000

  1. Commit the configuration to disk by running the command “save sys config

Disabling 3DES-CBC ciphers in SSL Client profiles and the management interface

This method will remove the affected ciphers from the list of applicable ciphers during SSL connection negotiations. SSL cipher settings are regulated by SSL profiles. While you can change the parent profiles we do not recommend doing so without doing proper testing.

 

Possible impact of disabling 3DES-CBC ciphers

While most modern browsers supports wide array of ciphers this will still reduce the cipher support of the virtual servers using the modified SSL profile. Depending on your current cipher string this could or could not pose a risk to clients and the virtual servers not being able to agree on a cipher.

Please note that the following guide is just meant to show how to disable these specific ciphers and uses the system default cipher list to do so. Using the default list is not recommended. To determine which cipher list you should use, please read up on ciphers.

 

This guide is a really good start:

https://f5.com/Portals/1/Premium/Architectures/RA-SSL-Everywhere-deployment-guide.pdf

Procedure to set an SSL cipher string via the GUI

  1. Log in to the F5 administration interface
  1. Navigate to Local Traffic -> Profiles -> SSL -> Client
  1. Click on the profile you wish to alter the cipher string on
  1. In the settings, locate Ciphers
  1. If it’s disabled (see the picture below), click on the custom option checkbox to allow changes.

  1. Change the value from the current value to <CURRENTLIST>: !DHE-RSA-DES-CBC3-

SHA:!DES-CBC3-SHA:!ECDHE-RSA-DES-CBC3-SHA

  1. Click on “Update”

Procedure to set an SSL cipher string via TMSH

 

  1. Connect to your load balancer via an SSH client
  1. If not already in the tmsh, enter it
  1. Show the current cipher string with the following command:

list /ltm profile client-ssl [profile name] ciphers

Example:

list ltm profile client-ssl /Common/example.com ciphers

ltm profile client-ssl Testpartition/testthemciphers { ciphers DEFAULT

}

  1. Note your current cipher string. In the example above it was “DEFAULT”
  1. Modify the cipher string with the following command:

modify ltm profile client-ssl [profile name] ciphers <CURRENTLIST>:!DHE-RSA-DES-CBC3-SHA:!DES-CBC3-SHA:!ECDHE-RSA-DES-CBC3-SHA

Example:

modify ltm profile client-ssl /Common/example.com ciphers DEFAULT:!DHE-RSA-DES-CBC3-SHA:!DES-CBC3-SHA:!ECDHE-RSA-DES-CBC3-SHA

  1. Commit the configuration to disk by running the command “save sys config

Thank you to Patrik Jonsson for contributing this article.

Announcing indeni 5.4: New rule engine, Check Point 61000/41000 support

Welcome 5.4!

In this release we’ve included phase one of our infrastructure operations platform, added new content and as well as Check Point 41k/61k support. In addition, specific feature requests and bugfixes were included. Please reach out to our support team to get the updated release.

IMPORTANT NOTE TO ALL USERS: Starting with 5.4, the licensing mechanism is attached to the indeni instance’s unique identifier (uid) and not the IP address. This allows customers to not only change the IP of their indeni instance, but also set up cold active/standby high-availability in case the primary indeni instance is down or is cut off from the network. To set up cold active/standby, please reach out to our support team.

New content: Continue reading

Announcing indeni 5.3: more than 400 improvements!

capture

Welcome 5.3!

In this release we’ve included over 400 improvements to the underlying infrastructure and bugfixes, added new content and expanded our Palo Alto Networks firewalls’ support. Please reach out to our support team to get the updated release.

IMPORTANT NOTE TO CHECK POINT USERS: Starting with 5.3, indeni no longer uses port 8181 to communicate with the firewall. The advantages of using port 8181 prior to 5.3 are now built into the use of port 22, the standard SSH port.

NOTE: Customers who require support of a given product version prior to the main release can contact support@indeni.com and a running build will be provided.

Select new signatures: Continue reading

How to Export Palo Alto Networks Firewall Configuration to a Spreadsheet

Data connections and led lights in an industrial building grain visable in areas and colours removed from certain images to enhance them., Low aperture used to create a shallow DOF on on connections or lights
How to export Palo Alto Networks Firewalls configuration to a spreadsheet

Sometimes it becomes very important and necessary to have the configured policies, routes, and interfaces in a spreadsheet to be shared with the Design Team, the Audit team and for some other purposes. The below method can help in getting the Palo Alto Configuration in a spreadsheet as and when you require. This requires little manual effort and just a few minutes of your time. Here you go:

 

 

1. First of all, login to your Palo Alto Firewall and navigate to Device > Setup > Operations and click on Export Named Configuration Snapshot:

2. From the pop-up menu select running-config.xml, and click OK. Save the file to a desired location.

Continue reading

Comparing indeni and BackBox: In-depth intelligence vs simplicity

Safeway, a company headquartered in Rosh-Haain, Israel, has recently released BackBox version 4.5. In this new version, BackBox includes “Application level monitoring”, capable of providing “insight regarding the devices’ health, and run preemptive scans to determine upcoming problems.”. Naturally, this has caused a handful of users to ask us how does indeni and BackBox compare. This is fantastic as more and more customers are looking to stay ahead of their issues and avoid the next outage.

The Origin of BackBox’s technology

Historically, BackBox was focused on backing up devices – as many as possible. BackBox’s claim to fame was its simplicity and the fact that it could cover an impressive range of devices. Included in the software were the instructions for how to automatically backup dozens of network devices, as well as the documentation for how to restore those backups. With release 4.0, BackBox received a UI face lift as well as a re-written infrastructure in Java.

Continue reading

How to Reset Device Trust – F5 LTM Load Balancing Methods Troubleshooting ConfigSync and Device Clustering

indeni, cisco

F5 LTM Load Balancing Methods: How to Reset Device Trust.

The official F5 SOL13946 provides information on troubleshooting device clustering and configuration sync for 11v  F5 load balancers  and other products, however it is rather long winded.  This guide is designed as a quick reference when troubleshooting device clustering or config sync. An overview of the config sync process for version 9.x and 10.x units can be found in F5 SOL7024

Version 11.x

  • Communication between machines occurs in the following manner to form a device cluster:
    mcpd process on the local machine connects to the tmm process on the local machine on port 6699
  • tmm process then contacts the peer’s config sync IP on port 4353
  • Once the peer receives, they use tmm to contact mcpd over port 6699 on their local device.
  • If this process fails, it is re-attempted every 5 seconds.
  • If this process succeeds, there is a mesh between peer mcpd processes.

* local machine here refers to the self IP configured for config sync. Check it under Device Management > Devices > click on device > Device Connectivity > Config Sync, for example.

Continue reading

Cisco vs Palo Alto Networks: The Hidden Battle

Capture

In my conversations with firewall users I often hear references to the “battle of the titans” between Check Point and Palo Alto Networks. Both are leaders in the Gartner Magic Quadrant, their security technologies are often compared and the marketing slander has been seen often in all the different mediums.

As everyone is aware, PANW’s aggressive growth outpaces the growth of the firewall market. This means that a large portion of the growth is coming from the displacement of their competitors. A point PANW’s CEO, Mark McLaughlin, made recently.

However, very little attention has been given to this:

download

The sum of the percentages is greater than 100% as some customers migrated from multiple vendors to PANW.

Continue reading