Part 7: Security

Security and System

Database Structure

Indeni stores its information locally on the hard drive on which it is installed. The database contains different types of information with two general classifications: highly confidential and confidential. The highly confidential information is stored within an encrypted file (using two types of encryption employing industry standards and best practices). The confidential information is sorted in non-encrypted files.

The database files are not accessible via the web interface and can only be retrieved by logging into the system via SSH and downloading them using standard protocols (SCP, SFTP, etc.). The SSH service is the standard sshd application, which has a long track record of being safe so long as the passwords selected by the user are strong ones. Refer to your organization’s password policies for more information on choosing a strong password.

Underlying Operating System

The operating system supplied with the system is Ubuntu 18.04 Server. By default, the set of services accessible via the network has been reduced to the absolute minimum required, further hardening the operating system. These services are:

  • SSH
  • HTTP and HTTPS (the Indeni server’s web interface, hosted inside Jetty)
  • TCP Ports 9009, 9912 used by Indeni’s Server component

Device Access Credentials Storage

The credentials used to access devices, such as the SSH Username and Password, are stored within the database described above. The username is stored in the confidential store, while the password is stored in the highly confidential store (and is encrypted). By protecting the database files, an organization is protecting this information from being compromised.

Password Security of Users Defined in the System

All users defined in the system (allowed to access the system itself via the web interface) are required to use strong passwords as defined by PCI DSS requirements 8.5.10, 8.5.12, 8.5.13, and 8.5.14. Passwords are stored as salted hashes within the encrypted database. This protects the original passwords from being recovered.

Protecting System Functions

Indeni implements authorization controls so they cannot be circumvented. Users are limited to system functions commensurate with their level of permissions.
This significantly reduces the risk of functions being carried out by unauthorized users.

Data Validation

Indeni employes a mechanism which validates all forms of user input to prevent tampering and command injection. This prevents system commands from being passed downn to the rest of the system and pevents attackers from executing malicious commands.

Protecting Analyzed Devices

The commands executed on analyzed devices (routers, firewalls, load balancers, management servers, etc.) are defined by the internal logic of the product and cannot be modified by a user. This is to limit the commands that can be executed by Indeni on analyzed devices to those which have been tested and approved by Indeni.

Indeni’s Failsafe Mechanism

Some critical devices could be sensitive to too much data collection, which can lead to performance problems. Although every effort has been made to minimize the device resource usage, the fail-safe mechanism is designed to provide additional protection and to prevent overwhelming a device under abnormal conditions.

As part of Indeni’s data collection capabilities, Indeni will regularly track the CPU and memory utilization to identify if the device is being stressed at the monitored interval. We leverage this information to inform Indeni’s task scheduling mechanism to avoid overwhelming a device. When Indeni detects that the device CPU becomes too busy, or memory becomes an issue, Indeni will significantly throttle data collection and temporarily suspend data collection until it resumes normal conditions. Indeni will continue to collect CPU and memory metrics periodically in order to resume data collection.

Indeni relies on a key-value pair in its JSON-based Knowledge scripts. A full example can be found here.

#! META
name: panos-show-system-resources
description: fetch resource utilization
type: monitoring
monitoring_interval: 1 minute
includes_resource_data: true

This is used by our Collector service, which schedules each of our scripts and executes them. When the value includes_resource_data is set to true, it informs the Collector that the script should be executed even when the analyzed device has reached critical thresholds for CPU and Memory. For scripts that should be disabled under high stress, the key-value pair is simply nonexistent. That is because the Collector service determines that the key includes_resource_data is set to false by default.

During clearly identified high stress intervals, Indeni will simply observe the CPU and memory. This allows us to analyze fundamental system resources such as CPU and Memory in order to identify when it is safe and healthy to execute the rest of the scripts again.

No Change Policy

Indeni has a very strict no change policy, meaning no changes will be made on the devices Indeni analyzes. The only writing actions Indeni executes is to write temporary files to /tmp and to initiate an additional instance of SSHD when needed.

Application Architecture

When you install the Indeni application, on a physical machine, as a virtual machine in your private cloud or as a virtual instance in a public cloud environment, you are installing multiple components:

  • Indeni base operating system – Ubuntu, a Debian-based Linux operating system maintained by Canonical. This operating system was selected for its stability, reliability, and security. For example, it is the dominant operating system in some of the world’s largest data centers, as well as the Amazon Web Services public cloud.
  • Indeni server application – a Java-based application which receives the data collected from devices by the collector (see below), stores it, analyzes it for issue and report generation and makes it available for users. The server application is also responsible for enforcing permissions on data access and modification. The server application exposes an Application Programming Interface (API) which provides access to the data stored by the server.
  • Indeni collector application – a Java-based application which connects to the network and security devices, logs in using the credentials it receives from the server and retrieves data 24/7. The collector is a lightweight application designed to be deployed in multiple locations, separate from the server or on the same machine.
  • Indeni Postgres database – All configuration data used by Indeni is stored in this database. It includes the list of devices to connect to, their credentials, the issues that were generated, rule configuration, etc. This database does not contain the actual data collected from devices.
  • Indeni in-memory metric database – Data collected by the collector is parsed into metrics, which are then sent to the server. The server has several in-memory databases used to store these metrics, analyze them for issue generation and make them accessible for reporting.

Credential Management

The Indeni collector component collects data from devices across your environment. To do so, it is required to log into these devices, often times using administrator privileges. These privileges are required in order to allow Indeni to retrieve all the data the user needs. Specific commands, such as licensing information, can be outside of the reach of non-administrator users on these devices.

To achieve this, the Indeni application must store the access credentials in a way that allows the application to use them. The credentials are encrypted for storage and decrypted on the fly when needed by the application. Hashing is not an option, as it’s only one way and would not allow the Indeni application to retrieve the credentials for use.

Technical details regarding credential storage:

  • The credentials are stored in the Postgres database in a table called “credential”. You can view the contents of this table by SSHing into the Indeni system’s console and running:
    psql indeni -c "select * from credential;"
  • The password element of the credentials is encrypted using a symmetric algorithm and key. The algorithm is AES (identified in Java as “AES/CBC/PKCS5Padding”) with a 128-bit key. We have chosen AES128 as it’s considered highly secure and difficult to brute force (see this article).
    Furthermore, following security best practices, each encrypted section uses a salt. This means that if the same password is encrypted twice, it will result in two different ciphers.
  • The key used by the AES encryption is internal to the application.

When accessing credential set information through the server API (for example, for display in the Indeni web user interface), the passwords are masked. You can see this by running the following command when SSHing into the Indeni system’s console (you may need to replace the default password “admin123!” referred to in this command to the one you have set:

curl -k "https://192.168.197.15/api/v1/credential_sets" -u 'admin:admin123!'


As you can see in the output, passwords are masked using the text “***** Hidden Password *****”.

Credentials that are entered through the user interface are immediately encrypted and saved to the Postgres database.

Your Responsibility In Securing Your Data

While the Indeni server component handles the security of access to all of the data – configurations and metrics collected – it is important that the server itself is secured by you. The following must be adhered to:

  • Change the default passwords for the following two users. A better option would be to remove both users, assuming you have created other users with the same privilege level in their place:
    • In the web user interface – change the password for the user “admin”.
    • In the machine’s console (accessible via SSH) – change the password for the user “indeni”.
  • Control who has access to the Indeni server’s hard drive. For a physical machine, this means controlling physical access. For virtual machines, this means controlling access to the hard drive files.
  • If you are using the OVA or image files provided by Indeni, then the operating system is already hardened. If not, you should consult with your information security team on how to harden the operating system which you have installed the Indeni server and collector applications on.
  • Do not run any other applications on the server(s) on which you have installed the Indeni components. Other applications may have vulnerabilities which may provide an unauthorized individual access to the data stored on the machine by the Indeni application.
  • Limit network access to the machine(s) the Indeni application components are installed on, to ensure only those individuals that should have access will be able to access the application.
  • If you have deployed the collector component on a separate machine from the server component, you should ensure there is a firewall protecting the server and allowing only the collector to reach it on TCP port 9009.
  • If you are accessing the Indeni server’s API (also on port 9009) you should follow similar guidance to the previous bullet point and ensure only hosts that should be allowed may access the server on this port.
  • When the Indeni support team reaches out to you regarding a vulnerability that was found in either the operating system or one of the components installed, apply the patch or workaround as soon as possible.

Compliance

The Indeni collector application does not collect any packet data from the devices it is connected to. This means that none of the traffic flowing through your network is accessed by the Indeni application in any way. No personally identifiable information is collected or stored at any point by the Indeni system, with the exception of the information stored within Indeni pertaining to the users accessing the application.

For all compliance purposes, please follow the instructions in the section titled “Your Responsibility In Securing The Data”.

Transport Security

Data sent between your workstation and the Indeni server is always encrypted:

  • When you are accessing the machine’s console over SSH the SSH client/server will take care of the encryption. Should your SSH client issue you that the host key has changed on the Indeni server, please halt all communications with the server and determine if a re-install took place.
  • When you are accessing the machine’s web user interface over HTTPS, the data is encrypted using the protocols agreed to between your browser and the web server included with the Indeni server application. It is recommended to install an HTTPS server certificate on the Indeni server so that you can ensure you are truly connecting to it and the connection has not been hijacked.

Data sent between the Indeni server and the Indeni collector is also encrypted using HTTPS. If you have deployed both of them on the same virtual machine, the data will not leave the machine and this will add another layer of protection. If you have deployed the collector and server components on separate machines, it is best to use a firewall to ensure that only the collector is allowed to reach the server on its API port.

Data collected by the collector from the network and security devices it connects to is protected during transport by either SSH or HTTPS, depending on how Indeni is retrieving data from the given device.

Indeni Insight

Nearly every business relies on their network to engage customers, partners, and employees. As a result, IT Ops leaders need the ability to measure their network health, in addition to the productivity of their people and technologies to achieve business objectives. Indeni Insight enables senior leaders to make smarter decisions ranging from strategic decisions (what technologies should I purchase for my data center migration?) to tactical (what version of the software should we upgrade to, or what features to enable and disable) to improve operations performance.

Architecture:
Once an hour, the Indeni server components collect data and posts it to a secure AWS S3 bucket owned by Indeni. The S3 bucket is PUT-ONLY for all Indeni customers, similar to how blind FTPs work. An AWS Lambda function, running under special permissions, has access to see the data posted in the S3 bucket. This function collects the data and inserts it to a MySQL database hosted in AWS as well. The S3 bucket, AWS lambda function, and MySQL database are all hosted within US-based regions of AWS.

This database contains all of the Insight data received from all of Indeni’s customers. To ensure the confidentiality of our customers’ information, the data is stripped of any identifying information. The working premise in the construction of the Insight database is that should the database ever be hacked, none of the information contained within it could be used to identify Indeni’s customers or provide any insight into how their network is laid out.

The following is a list of all of the data collected through Insight:

  • Instance unique identifier – such as 33a56092-bc25-11e7-abc4-cec278b6b50a. Uniquely identifies the Indeni server instance installed in your environment. The Insight database does not contain any information regarding who owns this instance.
  • Issue information:
    • Issue unique identifier – same format as the instance unique identifier.
    • Rule_Name – the rule in the Indeni server which generated the issue.
    • Headline – the headline of the issue – only the portion which is shared among all installations of Indeni (no environment-specific information).
    • Severity – one of CRITICAL/ERROR/WARNING/INFORMATIONAL
    • Device unique identifier – same format as the instance unique identifier.
    • The rule configuration which generated the issue – only its identifier and numeric parameters, no textual parameters.
    • Creation date/time
    • If the issue was archived, when that was.
    • If the issue was marked as resolved, when that was.
    • Note that the issue information does not include the issue’s description, remediation steps, or any data which could identify a device or a user.
  • Integrations information:
    • Only the types of integrations used. No information is sent pertaining to the server integrated with, credentials or anything of that nature.
  • Device information:
    • Device unique identifier – same format as the instance unique identifier.
    • Device type – such as the vendor of the device, operating system running, its version, is it part of a cluster, etc.
    • Is the device successfully connected to from Indeni, or are there any issues.
    • Note that the device’s name and IP address are not sent via Insight.
  • Rule configuration information:
    • Which rules have specific configurations and what devices/labels these are applied to.
    • For each rule, only numeric thresholds (such as CPU issue threshold) are sent via Insight. Any other type of threshold is not sent.
  • License information:
    • When a license is applied in the system, its expiration date and device count are sent to Insight. This helps ensure that users are applying their purchased licenses on time.
  • User information:
    • For each user that is defined, the user unique-identifier is sent – similar format to the Instance UID.
    • For each user-defined, their email address is also sent via Insight. The email address is not stored in the Insight database, it is solely stored in the CRM system in use by Indeni.
  • Indeni system information:
    • The version and build of the server component installed.
    • Performance metrics of various server and collector components (such as CPU, memory, and throughput).
  • Device metrics:
    • A variety of metrics that are collected from user devices are sent via Insight. These are:
      • CPU utilization
      • Memory utilization
      • Enabled features
      • License types (without serial numbers or any license identifier)
      • Uptime of device
      • Number of concurrent connections
      • Network performance metrics (received bytes/packets, transmitted bytes/packets, dropped packets, etc.)
      • Network latency between a load balancer and its server or nodes (without identifying those nodes’ IP addresses or names)
      • Network latency between a device and other devices (without identifying those devices’ IP addresses or names)
    • For those metrics which normally contain identifiable information (like IP addresses, or hostnames), the identifiable information is hashed via SHA1. This allows the Insight database to track the metric over time, without knowing the real device it pertains to.

Learn more by downloading the Indeni Insight Whitepaper.

As an Insight user, you may request a complete dump of all data collected by Indeni as part of the Insight service. Contact support@indeni.com and make an official request for the data.

BlueCat acquires Indeni to boost its industry-leading DNS, DHCP and IP address management platform to help customers proactively assess network health and prevent outages.

Learn More