How to monitor F5 devices – SNMP vs API vs SSH

F5 has many ways of interfacing with their products and when writing monitoring we had to do some research which one is more suitable in terms of performance. After all, monitoring should not harm the device it monitors. When choosing methods we looked into iControl REST, SNMP and TMSH. See below for how this test was conducted and which one won.

The best way to monitor F5 – How the test was conducted

We ran each type ~20 minutes continuously through command-runner. While running the tests the web interface was used to make sure that the web interface responsiveness was up to par.

The commands to run each test

#REST
while true; do
command-runner.sh full-command –basic-authentication user,password rest-pool-statistics.ind 10.10.10.10
done
#tmsh
while true; do
command-runner.sh full-command –ssh user,password ./show-ltm-pool-detail-raw-recursive.ind 10.10.10.10
done
#SNMP
while true; do
command-runner.sh full-command –ssh user,password ./snmp-pool-statistics.ind 10.10.10.10
done

Results

The test started out with 283 pools (with 200 additional ones created just for this test). However, when trying the tmsh command, command-runner timed out, so we had to reduce to the original 83 pools and rerun the test using rest to make it fair.

  • Test 1: REST = 283 pools
  • Test 2: Tmsh = 83 pools
  • Test 3: SNMP = 83 pools
  • Test 4: REST (take 2) = 83 pools

4 hour graph

24 hour graph for reference

REST

  • Did not produce any timeouts in the GUI in any of the two tests.
  • Always produced results.
  • Management interface only became sluggish one time during the second attempt. Most likely because of the already high swap usage created by the TMSH tests.

TMSH

TMSH produced these once in awhile:

  • When that happened you can see the gaps in the graph. It is unknown what the gap after the graph was because we was working on the snmp metrics at that time.
  • TMSH also failed to give results sometimes.
  • Forced to run with fewer metrics than rest in order to even get a result.

SNMP

  • Truncated the pool names sometimes. It is unclear why ast was always done on long names, but different lengths.
  • Did not produce any timeouts in the GUI.
  • Always produced results.
  • Did not have as many metrics as REST since the exact same metrics was not available in one command (pool state and availability is missing).
  • Management interface became a bit sluggish on and off.

Conclusion

Over all REST won the test with SNMP as second. TMSH did not even qualify as it takes up very large amounts of memory and swap which negatively affected the overall system.

Thank you to Patrik Jonsson for contributing this article.

Predictive Analytics and the Future of IT

In this world of infinite connectivity we are using data more and more to make sense of our environments. One such technology being incorporated into businesses is “Predictive Analytics”. We are already using mathematical formulas to predict certain events related to the stock market, weather, etc. With the processing power and technology available today, these algorithms have developed a fair degree of accuracy. Which leads to my next question, “Why not use ‘Predictive Analytics’ to predict IT systems / Network failure?” How about being able to anticipate network failures days before they actually happen? If you are managing a complex an IT set-up, you will want to get your hands on this technology.

What is Predictive Analytics for IT?

Predictive Analytics is a branch of data mining that uses mathematical algorithms like regression modeling techniques to describe the relationship between of various variables that contribute to the functioning of a system. Through machine learning, they assess the behavior of the variables under normal circumstances and monitor their behavior continuously to find out if there are significant abnormalities. These algorithms can be set to observe for certain behavior patterns that precede major trouble causing scenarios.

For example, predictive analytics can assign a score (probability) for each individual device or not. Institutions like insurance companies use predictive analytics to find out the relationship between various variables and the risks involved. They evaluate candidates with certain age, marital status, credit history, employment profile, etc are more prone to risky behavior than others and then decide if they want to give policies or not. Can this technology be used in IT systems?

“Monitoring” IT systems are still done the old fashion way – in silos

Various monitoring systems are in place for organizations today:

  • Network monitoring software
  • Virtualization monitoring modules
  • Servers monitoring software
  • Databases/ Applications monitoring software
  • Storage systems monitoring software

If you work in a large, complex organization you need to continuously monitor all the above management modules individually. The biggest issue with this model is as IT Systems and Network grow in complexity, the possibility human error increases and failures are only reported after they happen. The majority of  IT professionals only discover issues after the help-desk starts getting calls from the angry users that something is not functioning. Worse off, if you’re business is B2C, you could have angry customers showing their displeasure via social media and other channels.

Of course, redundancy can be set and monitored for irregularities in the system, of course these alerts are either ignored or a network outage occurs due to a totally different parameter that was overlooked, or due to incorrect threshold level settings. IT pros can easily be overwhelmed monitoring too many parameters.

 

How Predictive Analytics help forecast issues before downtime occurs in IT Systems?

When applied to an IT operations scenario, the predictive analytics system can go more in depth than existing monitoring tools to collect data about all the possible variables being monitored like cluster configurations, tracking CPU, log flows utilization, and packet drop activity. Based on this, algorithms  automatically determine the normal operating behavior of these variables and continuously analyzing live data 24/7/365 to determine if any of these variables significantly deviate from their normal behavior in a certain pattern that might indicate performance problems in the near future.

Predictive analytics accumulates as much data as possible from various sources and uses mathematical algorithms to understand the relationship between the variables in the current state. Based on this information, it can forecast what is likely to happen next, including any potential trouble causing situations. This way it tries to identify network downtime/ IT systems malfunction days before they actually occur.

The main advantage with predictive analytics is none of this data needs to be manually entered, nor is there a requirement to set manual thresholds.  Predictive Analytics systems claim to do this automatically.

Of course, the system needs to integrate with the current monitoring tools running in the organization. One way the predictive analytics systems can be tested is by feeding it with actual values of the variables (of a certain duration in the past) and monitor if it is able to predict major faults that actually happened in the past. This can, to an extent say how well a predictive analytics system can integrate within a particular environment.

Predictive Analytics can also help to forecast IT systems capacity. For example, it can predict the number of servers needed for a cloud based data center/ large organizations based on the past/ present trends of application utilization.

Of course, Predictive Analytics can never be 100% accurate and tends to have some degree of false positives. But for companies with large data centers and geographically dispersed campuses where even a small downtime in IT systems can cause huge financial or reputation losses, this technology might be worth a try? There is at least one company involved in developing Predictive Analytics for IT and network systems.

indeni Insight
indeni is an intelligent assistant that manages your network 24/7/365

How Customers Use Check Point Firewalls Around the Globe

In order to conduct the in-depth analysis of configuration and stats on network devices we collect very large amounts of data. For our customers, this data is very useful in benchmarking their network versus other networks around the world. We call this service indeni Insight.

Below is an aggregation of some of the data we’ve collected through this service. We are providing it to help the wider community consider how their network behaves as well as their future plans.

If you are interested in benchmarking your own network within an hour’s work, try indeni today. Once the system is set up reach out to support@indeni.com and we’ll do everything else.

Why does infrastructure operations still suck?

Last Friday, I met with an individual that leads a 300-person team, responsible for running the networking and computing infrastructure in 50 data centers around the globe. I asked him what he thought of his OSS stack – the set of tools his team uses to stay on top of what’s going on in their infrastructure.

He hates it.

As I want to keep this blog post PG-rated, I’ll refrain from using his adjectives, but I can tell you he’s not happy with it. It’s a clobber of open source and commercial tools. The tools required a lot of customization and a variety of extensions written over the years. At the end of the day, though, it only gives him up/down monitoring and no ability to proactively avoid the next outage. Over 70% of outages occur due to human error and misconfigurations and the tools available to him are incapable of identifying even one percent of that.

Continue reading

What We’ve Learned From Speaking With Our Customers

A month ago I shared some of our plans for 2016 and mentioned that I’d be speaking with our customers, asking them a few questions. The survey was very successful in my opinion – I spoke with dozens of customers for 30 minutes each and asked them 14 different questions. I would like to thank all of the survey participants for enduring my questions and sharing their honest feedback.

Continue reading

How to Export Palo Alto Networks Firewall Configuration to a Spreadsheet

Data connections and led lights in an industrial building grain visable in areas and colours removed from certain images to enhance them., Low aperture used to create a shallow DOF on on connections or lights
How to export Palo Alto Networks Firewalls configuration to a spreadsheet

Sometimes it becomes very important and necessary to have the configured policies, routes, and interfaces in a spreadsheet to be shared with the Design Team, the Audit team and for some other purposes. The below method can help in getting the Palo Alto Configuration in a spreadsheet as and when you require. This requires little manual effort and just a few minutes of your time. Here you go:

 

 

1. First of all, login to your Palo Alto Firewall and navigate to Device > Setup > Operations and click on Export Named Configuration Snapshot:

2. From the pop-up menu select running-config.xml, and click OK. Save the file to a desired location.

Continue reading

Gartner’s Magic Quadrant Enterprise Firewall Comparison

DISCLAIMER: indeni has no specific bias towards one manufacturer or the other, but please keep in mind indeni currently supports firewalls made by Cisco, Check Point, Fortinet, Juniper and Palo Alto Networks.

Gartner has just released its magic quadrant for Enterprise Network Firewalls. Two leaders were identified – Check Point (CHKP) and Palo Alto Networks (PANW) – congratulations to both!! You can access reprints via Check Point’s website as well as Palo Alto Networks’ website.

It is very interesting to read this report as much of it correlates highly with what we’re seeing in the market through indeni Insight as well as our own sales and marketing efforts. Kudos to Gartner, and specifically Adam Hils, Greg Young and Jeremy D’Hoinne, for doing a great job here.

Here are our insights:

  • Cisco is not labeled a leader by Gartner due to execution on the product side but we definitely see it as one of the top three by market share. Almost every customer we interact with has some Cisco ASAs, where some customers are entirely Cisco ASA based. We do see, though, that such customers’ functionality requirement from their firewalls is minimal as they either don’t put much focus on security or they augment the Cisco ASAs with other security products (Sourcefirce, Fireeye, etc.).Cisco has the largest channel and is the most established manufacturer in the market. As a result they have the most leverage and ability to get into specific customers.

 

  • Check Point is indeed one of the leaders on functionality. The set of different security functions that a Check Point firewall has is enormous. Some of these are a result of acquisitions, some developed in-house. There is a lot of effort on Check Point’s side to integrate these functions into a single management interface (and R80 is part of this). However, we do see users getting overwhelmed with the amount of functions and keeping up with their configurations. Almost every single multi-billion-dollar company we speak with, and many smaller organizations, use Check Point across at least part of their network.Price has been mentioned by customers repeatedly as an issue. Price sensitivity is less common in Fortune 500 but more common in smaller organizations or ones outside of the US (the majority of the market). Usually it is coupled with a lesser need for top-notch security. The note Gartner made regarding under-sizing appliances is something we’ve seen as well. Check Point is making efforts to deal with this with tools such as CPsizeme but it looks like undersizing is indeed occurring to reduce price. That is resulting in some frustration with customers.

 

  • Fortinet is a strong vendor in this market too. We see Fortinet a lot more in environments where there is either price-sensitivity or high performance requirements. This means that Fortune 500 (which are all US-based) tend to choose Fortinet less as they aren’t as price sensitive. We do see Fortinet quite a bit in smaller organizations as well as quite heavily outside the US (where price sensitivity is a real issue).Fortinet’s high-performance gear is a big attraction for enterprises with extremely large amounts of traffic. Their larger chassis can support unusually high amounts of traffic, however mostly when a smaller set of features are enabled. This is a great fit for data centers as the most security functions are deployed outside of the core, leaving the Fortinet chassis to focus on firewalling, switching/routing and basic security functions.

 

Download our free ultimate runbook and learn how to stop monitoring your network and start predicting issues to prevent high impact events.

 

  • Juniper has its old line of SSGs/ISGs and the newer SRX line. While we see the SSGs quite often, because in reality they very rarely fail and no one sees a reason to replace them, the SRXs should be the focus of this analysis. JunOS-running SRX are mostly deployed in smaller environments because, in our experience, SRXs are considered as a simpler firewall. Across the board, anyone who has ever used JunOS loves it. It’s easy to use and highly responsive.Customers are showing real concern around Juniper’s roadmap for security devices. While the other vendors are promoting new features increasingly, Juniper is quite silent on these. As a result, customers who are seeking security innovation are looking at alternatives. Moreover, Juniper’s SSL VPN was once the best perceived SSL VPN product, but the recent divestment is causing customers to see the end of the road for it and consider firewalls’ support for SSL VPN as a replacement.

 

  • Palo Alto Networks is the fastest growing vendor in this space. Their marketing machine is the best across the vendors we are familiar with – measured by the number of customers we interact with which are discussing Palo Alto Networks’ offering (even if they are not users yet). With a whole range of features offered, most customers are still at the firewall/App-ID/User-ID level. Wider deployment of the other features isn’t main-stream yet. Customers are generally very positive towards the additional security features provided by Palo Alto Networks’ firewalls.A very interesting situation we’ve noticed is that Palo Alto Networks’ customers love them and show far more appreciation to them as a manufacturer than others. Palo Alto Networks is putting a lot of emphasis on the end-user experience – through their online marketing, field marketing, channel, field sales and support services, in addition to the product itself – and it is paying off. This is resulting in cases where even though multiple solutions were comparable, customers chose Palo Alto Networks as they were drawn to them. Keep in mind that this is supported by a solid security product.

Throughout the report Gartner mentions issues around quality and support services provided by some of the manufacturers. In reality – all of the customers we speak with complain about this across all product lines. They feel that vendors are working day and night to push out new functionality and keep up with their competitors, while at the same time disregarding quality and making the products far more complicated to operate and keep stable. Our recommendation to the vendors is to take this note very carefully and close to heart as the current trend in quality/complexity issues is taking the entire industry in a problematic direction.

Comments are very welcome, please share your thoughts below.

Tuesday And Wednesday Are The Busiest For NetOps/SecOps Teams

Network operations and security operations teams generally work around the clock. However, there are days and times they are clearly busier. Below is a graph that analyzes all of the alerts indeni generated for our customers (those connected to indeni Insight) over the course of Mar 9th 2015 to Apr 5th 2015, according to the day of the week they were generated. At the time of writing of this blog post (Apr 7th, 2015), it was clear that Tuesday and Wednesday are the busiest days. It will be interesting to see if this changes over time, which is probably shouldn’t. To get up-to-date results, click on the image itself.

The rationale behind claiming that the days with the most alerts are the busiest, is that a big portion of alerts issued by indeni are ones that are in direct response to a configuration change. Remember – our job is to analyze configurations as they change and alert when we find possible issues in them.

The Rise of the VAR/MSP

The Managed Service Provider (MSP) market is exploding. It’s been this way for a few years now. It makes a ton of sense – MSPs can offer something organizations are lacking and for a lower cost: expertise in 24/7 operation of critical infrastructure.

Some, usually larger, enterprises are unable to use MSPs today due to security and confidentiality requirements. However, many who can, choose to use MSPs to run their infrastructure. We, at indeni, are happy to see this trend as MSPs are a great partner of ours. Take Fujitsu, for example, a leading MSP in the UK & Ireland who has converted their MSP offering to run around indeni’s technology.

One thing that we started seeing more and more in 2014 and now 2015, is the rise of the VAR/MSP. Value Added Resellers (VARs) are those who sell network equipment and professional services to large enterprises. Historically, their business was primarily around making a low margin on physical equipment sale (and maintenance) and a higher margin on professional services (design and implementation of projects).

However, much like startups and more established tech vendors have figured out, a recurring revenue stream can greatly increase the business’s profitability. Almost all such vendors today use (or are moving to) annual subscriptions to provide services instead of selling perpetual licenses. It provides them with an ability to forecast future revenues, increase margins and build a solid business.

VARs are now going the same way. They are leveraging their expertise (built through reselling equipment and providing professional services around it) to build strong MSP offerings. Several of our VARs have already begun offering this and many others are in the planning/building process. We’re very happy about this development – as indeni is becoming a core element of delivering these services. Our pricing model aligns with that very well too, so MSPs have considerable flexibility in rolling out indeni across their existing and new customers.

These are exciting times!

Up Next: Crowd-sourced SaaS

Google released something today – a tool called The Customer Journey to Online Purchase. While this just a standalone tool at the moment, I’m sure we’ll witness parts of its capabilities integrated into Google Analytics at some point in the near future.

This release, together a few more startups dotting the landscape (like RelateIQ, as described by Scott Raney of Redpoint, or indeni, on whose blog this post appears) are heralding a new generation of SaaS: the crowd-sourced SaaS. Brian Ascher of Venrock calls it Data-Driven Applications, and so does Scott Raney. However, I believe that term is confusing. Data-Driven causes too many people to think about applications that let you access and visualize data and that’s not what we’re talking about here.

The next generation of SaaS is when the service provided to customers keeps evolving over time based on the data that is stored within it. What would happen if Salesforce.com collected all the data its customers store in its service, analyzes it and uses the results to provide more value to its own customers? indeni is a SFDC customer and I’m sure our data would greatly benefit from being analysed by an automated system that has seen how the data of other SaaS companies looks like. I’m not talking Jigsaw here. I’m talking actionable insights. The kind that derive from data being turned into knowledge.

You see, SaaS is no longer just about being cheaper and easier to use, which is what Salesforce.com started with and Brian Ascher details so well in his Forbes post. That revolution is done – it’s reality now. The question a lot of people are asking themselves is what is the next revolution in software and crowd-driven SaaS is my bet. My belief in it is so big, that I’ve bet my livelihood on it.

I’d be very interested to learn about others who are working on bringing us all to the next level. As with all revolutions that are just starting – it’s very difficult to find on Google who is doing it. We’ll need to go old-school and actually use word-of-mouth. So reach my on twitter: @yonadavl