Announcing indeni 5.4: New rule engine, Check Point 61000/41000 support

Welcome 5.4!

In this release we’ve included phase one of our infrastructure operations platform, added new content and as well as Check Point 41k/61k support. In addition, specific feature requests and bugfixes were included. Please reach out to our support team to get the updated release.

IMPORTANT NOTE TO ALL USERS: Starting with 5.4, the licensing mechanism is attached to the indeni instance’s unique identifier (uid) and not the IP address. This allows customers to not only change the IP of their indeni instance, but also set up cold active/standby high-availability in case the primary indeni instance is down or is cut off from the network. To set up cold active/standby, please reach out to our support team.

New content: Continue reading

Announcing indeni 5.3: more than 400 improvements!

capture

Welcome 5.3!

In this release we’ve included over 400 improvements to the underlying infrastructure and bugfixes, added new content and expanded our Palo Alto Networks firewalls’ support. Please reach out to our support team to get the updated release.

IMPORTANT NOTE TO CHECK POINT USERS: Starting with 5.3, indeni no longer uses port 8181 to communicate with the firewall. The advantages of using port 8181 prior to 5.3 are now built into the use of port 22, the standard SSH port.

NOTE: Customers who require support of a given product version prior to the main release can contact support@indeni.com and a running build will be provided.

Select new signatures: Continue reading

Announcing indeni 5.2: Palo Alto Networks beta, improvements and bugfixes

Welcome 5.2!

In this release we’ve included many improvements to the underlying infrastructure and bugfixes, as well as kicked off the beta for our support of Palo Alto Networks firewalls. Please reach out to our support team to get the updated release. Note that between minor releases (such as 5.1 and 5.2) we make interim releases with new content and bugfixes on a weekly basis. You may have received a previous release of 5.2, which we recommend you upgrade to the newest one announced today.

New products and versions supported:

  • BETA of Palo Alto Networks firewalls running PAN-OS 6.x.x. If you are interested in joining the beta, fill out the form.
  • IK-1675: Support CP R77.30
  • IK-1840: Fortigate: Added support for FortIOS V5.2.1

NOTE: Customers who require support of a given product version prior to the main release can contact support@indeni.com and a running build will be provided.

Select new signatures:

  • IK-1677: Firewall is running with a trial license (Check Point)
  • IK-1836: Enhanced “BIG-IP node availability issue detected” (F5)
  • IK-1825: ConfigSync operational status issues (F5)
  • IK-2020: The BIG-IP system is near or out of disk space or inodes (SOL12263, SOL14403) (F5)
  • IK-2021: “Possible multicast or broadcast loop on SFP NICs detected” (F5)
  • IK-1834: “Load balancer node connection limit nearing (or reached)” (F5)
  • IK-1831: “Number of active members in pool is lower than required” (F5)
  • IK-1835: “Pool member connection limit nearing (or reached)” (F5)
  • IK-1827: “SSL transactions per second (TPS) limit nearing or reached” (F5)

Bugs fixed and minor improvements:

  • IK-1674: “A NIC has failed recently (SA#24915)”: reduced the number of log lines shown
  • IK-1518: “Cluster Members Identical Kernel Parameter Values Verification (SA#66322)”: additional dynamic parameters ignored
  • IK-1859: “DNS server resolution test failed” – eliminate false positive in Cisco devices
  • IK-1672: “Errors have been found in packets received by NIC (SA#24915)” triggered for very low packet count
  • IK-1704: “Communication with device suspended due to 2 reboots” false positive
  • IK-1712: “Hardware has reached end of support” is auto-resolving
  • IK-1683: “Hardware temperature sensor reading too high” false positive
  • IK-1391: “High storage usage has been measured” doesn’t show list of large files in Cisco devices
  • IK-1871: “HSRP cluster members differ in VLAN configuration” false positive
  • IK-1858: “License(s) have expired” false positive for CP licenses with expiration “never”
  • IS-1349: “Max SSH Session Count” remains at default
  • IK-1976: “Monitoring Suspended” creating too many alerts
  • IK-1958: “NAT cache (fwx_cache) table limit approaching or reached” false positive
  • IK-1879: “NAT connections (fwx_alloc) table limit approaching or reached” false positive
  • IK-1901: “RX traffic drastically reduced post fail over, possible ARP issue” add specific interface details
  • IK-1919: “SecureXL templates are partially disabled” false positive
  • IK-1914: “Some members of the same cluster are not being monitored” false positive
  • IK-1731: “Some proxy ARPs required by NAT are missing” – signature removed
  • IS-1000: “Some received packets have been dropped by NIC (SA#24915)” – duplicate text in e-mail alert details
  • IK-1628: “Two cluster members differ in their routing tables” failing to create alert
  • IK-1870: “Use of NTP is configured but no servers are defined” false positive
  • IK-1684: “Voltage too high or too low” false positive
  • IK-1702: “Voltage too high or too low” – don’t alert if hi/low limits are unknown
  • IS-1454: Backup: sometimes old backups are not deleted
  • IK-1501: “Proxy ARP is enabled” flapping in Cisco
  • IK-1696: GAiA R77.10: Replace use of ckp_regedit with cpinfo
  • IK-1846: ClusterXL member differences alerts are referring to the wrong cluster members
  • IK-2133: Configuration Check – “Hotfix(es) Installed” does not handle comma delimited string of HFs correctly
  • IS-1077: Connection to SecurePlatform with SSH private key fails
  • IK-1741: Correctly identify device model for CP 21700
  • IK-1742: Correctly identify device model for CP 4400
  • IK-1670: Live Configuration – all NICS are showing as Down
  • IK-1856: Hardware alert false positives from Check Point open server
  • IS-1346: Prevent “service indeni4it start” from starting the application more than one time
  • IK-1690: “Route overlap identified” – don’t alert when next-hop is the same
  • IK-1688: NIC stats alerts (e.g. packet errors) should contain the total number of packets that we compare against
  • IK-2066: SmartCenter degradation due to hanging “fw log” processes
  • IK-1993: SmartCenter backup: use “migrate export” for R75.40 and above
  • IK-2067: Reduce “sshd[xxx]: Did not receive identification string from <indeni server>” in device messages log
  • IS-1037: Update by UPD fails to restart the service
  • IK-1966: Crossbeam discovery failure
  • IS-1453: Backup Report – empty “Failed Backups” section header
  • IS-1348: Scheduled Reports delivery does not follow DST changes
  • IS-1441: F5 – wc should not show the groups common/device_trust_group and common/gtm
  • IS-1036: E-mail Alerts: remove PDFs from e-mail alerts
  • S-1019: Tools-Troubleshooting – add “cpstat os -f sensors” for Check Point firewalls
  • IS-1765: Alert Report – add alert timestamps&nbsp;
  • IS-1060: Alerts e-mails – add alert timestamp

Announcing indeni 5.1: F5© BIG-IP© support, many improvements

We’re excited to announce version 5.1. While this version has been generally available for a few months now, it has had improvements added to it over the past two months.

New product versions supported:

  • F5© BIG-IP© 11.x

New signatures:

  • The following are some of the F5-related signatures included in this release:
    • Identify node availability issues
    • Pool member connection limit nearing or reached
    • Load balancer connection limit nearing or reached
    • Number of active members in a pool lower than threshold
    • Number of SSL Transactions per Second nearing license limit
    • ConfigSync state not OK
    • Reaper process started
    • Cross check certain log lines with AskF5.com

Bugs fixed and minor improvements:

  • WC-2051: Network Health left-side widgets empty in some cases
  • IS-1365: Discovery of analyzed devices was sometimes slow due to a behavior issue with CentOS’s /dev/random
  • IS-1363: NIC details were not indexed by the Search feature in certain cases
  • IS-1346: Prevent “service indeni4it start” from starting the application more than one time
  • IS-1087: RADIUS authentication with one-time tokens resulted in lockouts
  • IK-1951: VPN debug messages contain partial information
  • IK-1924: “Coredumping setting not as desired” Profile Check – FP
  • IK-1914: “Some members of the same cluster are not being monitored” FP
  • IK-1856: Hardware alert FPs on Check Point Open Servers
  • IK-1626: SNMP monitoring – “Device clock appears to be set incorrectly” FP
  • IK-1919: “SecureXL templates are partially disabled” FP
  • IK-1871: “HSRP cluster members differ in VLAN configuration” FP
  • IK-1670: Live Configuration – all NICS are showing as Down
  • IK-1852: indeni server’s disk filled up without any storage alerts
  • IK-1847: Failed to Communicate alert: wrong details when Check Point shell is not bash
  • WC-1800: Performance of rendering of the list of devices has been improved
  • WC-2061: Network Health – scrolling alerts show acknowledged alerts
  • IS-922: Ignored items list was sometimes cleared instead of stored
  • IS-1371: Full text search improved to increase coverage and improve result sorting
  • IS-1357: “fwaccel stat” added to debug report for Check Point firewalls
  • IS-1088: Improvement to the performance of the generation of inventory reports
  • IK-1901: “RX traffic drastically reduced post fail over, possible ARP issue” add specific interface details

The “How to avoid in the future” section of a Root Cause Analysis report – any use?

You had a major network outage (like Time Warner just did). Panic, stress, sweat, people trying all kinds of crazy things. In the end, the issue is resolved and the outage is behind us. Then, comes the really fun part: doing a Root Cause Analysis (RCA).

There are a ton of templates for this, such as this one and this one. In each one, at the very end, is a section that details “how to we make sure this doesn’t happen again”. Sadly, though, in most cases, this section describes how processes will be changed, check lists will be made and extra peer reviews will be conducted. Frankly, our experience shows this rarely actually works.

Image: antkevyv / 123RF Stock Photo

Our goal is to change the way this is done. If someone were to spend the time to read every RCA ever written about a network outage and build a system that implements the recommendations detailed in that last section of the document, then issues would indeed be avoided.

We, indeni, are that someone. Of course reading RCAs manually is a bit difficult so we’ve devised more automatic ways of collecting this knowledge. With indeni, users have less RCAs to write. One of our larger customers actually told us we’ve reduced the number of RCAs they have per quarter by 93%!

So, if you’ve recently run into an annoying issue with Cisco routers and switches, or Check Point firewalls, or F5 load balancers, or anything else we support – give us a try. It only takes 45 minutes.

Announcing indeni 5.0: Trending capabilities, easier UI navigation, better performance, tons of additions

We’re excited to announce the release of version 5.0! After being used for quite a while by some of our customers, we’re ready to have the whole world enjoy what 5.0 brings.

IMPORTANT: 5.0 includes major changes to the underlying infrastructure of indeni’s engine. As a result, upgrades are done via a complete re-installation of the indeni OS and application. The upgrade maintains all of the existing monitoring definitions and alerts. Please contact indeni’s support to conduct the upgrade together.

New features:

  • The Analysis tab has been added, providing the ability to visually track critical metrics over time. These metrics are compared to the alerts issued (those orange bubbles at the bottom of the graph in the above screenshot).
  • Tabs have been re-organized in the web console to better fit our users’ task oriented activities (see more information below).
  • indeni Insight can now be configured from the web console.

New Infrastructure:

  • indeni 5 introduces the use of a new type of database in order to support the collection of data for the Analysis tab. This data includes: CPU, memory, disk space, connections and various NIC error statistics.
  • Following Lessons Learned from indeni 4 the SSH infrastructure was replaced in order to support better error handling related to connecting and communicating with the devices under monitoring.

New product versions supported:

  • IK-1675: Support CP R77.20
  • IK-951: Support FortiOS 5.0.1

NOTE: Customers who require support of a given product version prior to the main release can contact support@indeni.com and a development build will be provided.

New signatures:

  • IK-1677: Alert for Check Point device running with a Trial License
  • IK-1376: Inform when Dynamic routing protocol state changes
  • IK-1365: Notify when OSFF Topology Tree is rebuilt
  • IK-1316: evice profile – Alert if there are 2 different VTP domains in the same LAN

Changes to the Web Console

Following the feedback from our customers we have made some minor changes to the tab assignments in the Web Console in order to make it more intuitive:

The Operations Management tab incorporates the previous Home and Monitoring tabs.

  • Analysis holds the new trend graphs pane displayed above.
  • Home Dashboard is now available in the Network Health pane.
  • Signatures are now presented under Knowledge Management.

The Compliance Management tab has replaced the Device Config Management tab.

  • Configuration Checks replaces the Device Profiles pane.
  • Configuration Journal replaces Change Tracking.
  • Configuration Check Reports replaces Device Profile Compliance.

The Tools tab is a new one:

  • The Live Debug feature has been deprecated in favor of focusing on other functionality of indeni.
  • Live Configuration replaces the previous Actual Configuration pane (previously under Device Config Management).
  • Search replaces the Device Explorer pane.

The Reporting tab now holds the Inventory Report (previously under Device Config Management).

The ”indeni insight” tab was added to support enabling the new indeni Insight service (read more).

 

Bugs fixed and minor improvements:

  • Numerous performance and usability enhancements in the web console. Including:
    • WC-1996: speed up multiple Alert Acknowledge
    • WC-1642: eliminate errors caused by device deletion
    • WC-1691: Ability to select and act on multiple alerts across multiple pages
  • WC-2004: Add more details to “ClusterXL member is in a critical state”
  • WC-1969: Reports | enhance behavior of item deletion
  • WC-1962: Resolve memory leak in FF30
  • WC-1950: Improve stability of “By Management” view
  • WC-1945: Network Health | smoothen zoom behavior
  • WC-1942: “Knowledge management” subtab loading speed improvement
  • WC-1935: Signatures – columns sizes correction
  • WC-1923: Usability – increase length of search field
  • WC-1900: Improve organization of options under “Resolve” button
  • WC-1878: Make Ignored Items show in the Device’s Alert configuration
  • WC-1817: Add an ability to select more than one interface for P1 MDS or CMA
  • WC-1404: Display default thresholds for signatures
  • WC-1403: Add an asterisk (*) symbol next to every signature with changed thresholds
  • IS-1019: Tools-Troubleshooting – add “cpstat os -f sensors”
  • IS-1002: Actual Configuration – ClusterXL Mode is “unknown” in some cases
  • IS-980: Decrease occurences of alert flapping
  • IS-953: Actual Configuration performance improvement
  • IS-914: Improve the responseof the “stop monitoring” feature
  • IS-887: Make identification of a device upgrade quicker
  • IS-876: Reduce time it takes to shut down indeni
  • IS-866: Move SSH communication layer to Apache SSHD from Ganymed
  • IS-826: Improve speed of leading actual configuration
  • IS-780: Change Tracking accuracy improvements
  • IS-742: Add Alert Severity to the Subject of alert e-mails
  • IS-618: Expose API to fetch measurements + history
  • IK-1690: Route overlap identified – don’t alert when next-hop is the same
  • IK-1689: Improve accuracy of Check Point ClusterXL sync-related alerts
  • IK-1688: SA#24915 alerts (e.g. packet errors) should contain the total number of packets that we compare against
  • IK-1674: “A NIC has failed recently (SA#24915)” include concise log file data
  • IK-1672: Reduce sensitivity of “Errors have been found in packets received by NIC (SA#24915)”
  • IK-1671: “EIGRP unidirectional link identified” – improve accuracy
  • IK-1633: Cisco – improve troubleshooting when there is an issue with the privileged mode password
  • IK-1564: Improve discovery of Fortigates using banners
  • IK-1541: Reduce sensitivity of the “Proxy ARP Enabled” alert
  • IK-1250: Loopback alert – do not alert in case there is a management port
  • IK-1112: Add “vsx stat -l” and “vsx stat -v” to the debug report for VSX devices

How to Upgrade OpenSSL to a Patched Version – Heartbleed Vulnerability

As you probably have heard already, a new OpenSSL vulnerability (called “Heartbleed” ) was recently found. A bounds checking bug made it possible for an attacker to exploit the SSL heartbeat functionality to uncover sensitive data from the web server process memory. xkcd’s Randall Munroe did a wonderful job explaining it here: http://xkcd.com/1354/

While indeni does utilize SSL both as its internal communication protocol and the web application, it is NOT vulnerable to the Heartbleed bug as indeni uses Java’s implementation of SSL and not the open source OpenSSL library, in which the bug was found.

indeni provides its own operating system installation – based on the excellent CentOS Linux. Two major versions of CentOS are in use by indeni as of today: A CentOS 5.3 based indeni which is using OpenSSL 0.9.8 and is not vulnerable, and a newer CentOS 6.5 based indeni which has OpenSSL 1.0.1 which is vulnerable to Heartbleed.

However, Heartbleed attack is possible only if a HTTPS server is relying on OpenSSL to provide the SSL heartbeat functionality. indeni does not use this functionality in any way and you don’t have to do anything to protect yourself against Heartbleed attacks.

If you would like to upgrade OpenSSL to a patched version in any case, please contact our support and we will be happy to guide you through this process – support@indeni.com.