Using cURL to Monitor Check Point VSX Firewalls

“If you are bad at IT, you’re going to be really bad at virtualization.”
Steve Chambers

Foreword by indeni:

Though scalable and functional, virtualization hasn’t yet stood the test of time and innately results in decreased visibility until fully socialized within the marketplace – the catch twenty-two of technology adoption. Even the best of IT professionals are going to struggle with the implications of this market shift.

In the spirit of the growing demand for VSX for Check Point, we recently asked security professionals to share with our community their knowledge regarding best practices so that we might help one another ensure the most secure, low-risk environment possible.

A very well-respected professional in his field, Irek Romaniuk, submitted the below expertise. He has provided with great detail the steps required to create scripts, increase VSX visibility, understand the resulting output, create integrations, etc. He does a very deep dive into the downtime sensitive areas of CPU consumption per VS, processor load, connection table limits, and more.

Checkpoint is showing how to use SNMP to gather information from a VSX Gateways in sk34054. According to this SecureKnowlege (SK) article, it is possible to get CPU usage per Virtual System by using the CPU counters, which are based on the Resource Control feature.

The command below is displaying CPU consumption per Virtual System (VS) using resource control.

# fw vsx resctrl -u stat

Figure 1: CPU usage per Virtual System as reported by Resource Control feature Figure 1: CPU usage per Virtual System as reported by Resource Control feature

I have more confidence in the traditional Unix ‘top’ command which usually shows higher CPU consumption per VS. For example, running ‘top’ in batch mode at the same time as the ‘resctrl’ command, shows higher CPU usage (14% on VS 3 in this example)

# while :; do clear; top -b -n 1 | grep ' fwk3' ;sleep 2; done 19519 admin ; ; 0 -20 811m 279m 48m S 14 ;2.4e output of the CLI commands. For this purpose I am doing screen scraping, starting from the bash script named ‘dev-netpro.sh’.
# !/bin/bash while true; do FWK=$(top -b -n 1 | egrep 'fwk' |awk '{print $9 $NF}') CON=$(vsx stat -l | egrep 'number' | awk '{print $3}') TIME=`date +"%T"`  DOW=$(date +%u) FWK=$(echo $FWK|tr -d 'n') CON=$(echo $CON|tr -d 'n') curl -ik -H "Accept: application/json" -H "Content-type: application/json" -d '{"WAL-VSX-02": "['"$TIME"','"$FWK"','"$CON"','"$DOW"']"}' -X POST "https://101.1.1:8880/test" sleep 2 done

This script is running on VSX, checking the processor load and connection table limits on all VSes. I can send these values by email using ‘sendmail’, but it will not help me to create live charts. I am using ‘cURL’ to post data (or rather messages) to the URL https://10.1.1.1:8880/test (where 10.1.1.1 is my server addresses), every 2 seconds. So this way I will be able to read it on my PC using browser. The ‘top’ command in batch mode is used to get CPU load and the ‘vsx stat –l’ command to obtain connection table limits.

Figure 2: Monitoring setup Figure 2: Monitoring setup

Use ‘nohup’ if you want the script to run in the background, and then ‘kill process ID’ to stop the script on VSX.

# nohup ./dev-netpro.sh & [1] 16816 # nohup: appending output to `nohup.out' [Expert@WAL-NEW-VSX-02:0]# kill -9 16816 [1]+Killednohup ./dev-netpro.sh # ps aux | grep netpro admin224010.00.0 1744 520 pts/2S+ 09:17 0:00 grep netpro

The message coming out from dev-netrpo.sh script is in the JSON (JavaScript Object Notation) format. It is easy for machines to parse and generate. See example below:

{“WAL-VSX-02″:”[13:10:38,14fwk3_dev 2fwk0_dev 2fwk5_dev, 69 40 47 24171 8577 13750 306 31 237,7]”}

  • “WAL-VSX-02” is VSX name and in square bracket there are:
    • 13:10:38 – actual time as taken from firewall
    • 14fwk3_dev 2fwk0_dev 2fwk5_dev : ‘fwk’ string is preceded by CPU load and completed by VSID. So CPU load on VS 3 was 14 and VS 5 – 2
    • 69 40 47 24171 8577 13750 306 31 237 – Connection table limits in VS ID 0 to 8, so i.e. Connection table limit on VS ID 3 is 24171
    • 7 – day of the week (7 is Sunday)

I can put as much in square brackets as I need to. It could be also interface errors and drops, policy installation times, you name it.

Figure 3: Output from bash script running on VSX Figure 3: Output from bash script running on VSX

 

Download indeni’s free white paper on the hidden impact of downtime for IT Teams

 

The output from the bash script dev-netpro.sh is received by small server running open source Express – Node.js web application framework which is event-driven JavaScript server-side environment. Any web server (Windows or Linux) using RESTful API (Application Programming Interfaces) can be used (i.e. Python Django). To learn how to build RESTful API using node and Express look here. To give you an idea how easy it is, see 5 lines of code below.;

var express = require('express'); var app = express(); app.get('/', function(req, res){ res.send('hello world'); }); app.listen(3000);

Put the above code in the file server.js and then start ‘node server.js’ on your PC (Windows/ Linux/Mac with Node.js installed), then connect to port 3000 to see ‘hello world’

Figure 4: 'hello world' in Express - Node.js framework Figure 4: ‘hello world’ in Express – Node.js framework

The API in my case takes JSON object with “WAL-VSX-02″ key in the body of https request (req) and sending it to the Websocket (wss) server running on the very same server on port 8880.

var WebSocket = require('ws'), wss = new WebSocket('wss://localhost:8880'}); router.post('/test', function (req, res, next) { var conmessage = convertmessage(req.body); res.json(req.body); //res.json() wss.send(JSON.stringify(conmessage)); });

WebSocket is a protocol providing tcp channels and making it possible to send messages in real-time. So in my case server is sending message {“WAL-VSX-02″:”[13:10:38,14fwk3_dev 2fwk0_dev 2fwk5_dev, 69 40 47 24171 8577 13750 306 31 237,7]”} to WebSocket channel in order for the PC’s browser (Chrome in my case) to connect to ;same channel and receive that message (Websocket is used very often in chat applications). In fact I don’t even need browser on PC to see message send by VSX, I can use ‘wscat’ utility (command-like WebSocket client) on Windows/Linux to connect to WebSocket server and to listen to messages

PC_with_wscat> wscat -c wss://10.1.1.1:8880  connected (press CTRL+C to quit)  < {"WAL-VSX-02":"[15:44:06,8fwk3_dev 2fwk1_dev,69 34 25 23149 9201 12890 300 10 226]"}  < {"WAL-VSX-02":"[15:44:10,20fwk3_dev,69 34 25 23036 9069 12800 306 10 223]"}

Of course I prefer browser because messages will be converted to human readable format and feed charts. So on PC client side I’m using Angular JavaScript, SmoothieChart and Angular WebSocket library, to get real-time charts. See client side code at the end of this post, it is less than one page, most of which is related to charts setup (colors, line width etc). ‘MyData’ service is copied from Angular-WebSocket library and is providing messages to controller‘monitorCtrl’ which is displaying converted data/message on the HTML web page.

Function convertmessage(req.body) is using regular expressions to extract and classify data (message) from brackets. So at the end I keep CPU load as well as connection table limits in arrays [0,0,0,14,2,0,0,0,0] or [ 69 40 47 24171 8577 13750 306 31 237] respectively. Position in the array refers to VS ID number, so you can see 14 in the 4th position for VS 3 because it counts from 0 (VS 0) to 8 (VS 8) in my case.

Figure 5: Browser with CPU load and connection limit charts Figure 5: Browser with CPU load and connection limit charts

Last part will be to setup persistence and write messages to database on the server. The good candidate for this is noSQL (non-relational) database like MongoDB. MongoDB is next generation, cross –platform, document oriented and schemaless database. MongoDB together with the other tools I used to configure server and client (Node, Express and Angular) are sometime referred as ;MEAN stack. ;It can keep up to 2 exp32 documents (messages in our case) . It is enough even if we write a message to document (message) collection every two seconds for 7 days (should be 302400 entries per one firewall in one week cycle).;MongoDB is supporting JavaScript and t JSON-like format. Document in MongoDB can looks almost like the message sent from Checkpoint VSX firewall

{ hostname : ” WAL-VSX-02″, cpu: [0,0,0,14,2,0,0,0,0], con:[ 69,40,47,24171,8577, 13750,306,31,237], time: “13:10:38”, day: 7}

To save messages from VSX as document in MongoDB I need to slightly modify code from previous page by adding Message.findOneAndUpdate, see below

router.post('/test', function (req, res, next) { … Message.findOneAndUpdate({hostname: conmessage.hostname, day: conmessage.day, time: conmessage.time}, conmessage, {upsert: true}, function(err, message) { if (err) res.send(err); }); });

If there is already message with specific hostname, time and day I only update connections and processor tables, if it is new message, all is written to collection, including time, day and hostname.

As you can see it’s possible to keep the consistent format of message all the way through firewall, server, database and client PC. Message posted by cURL in the format of {“WAL-VSX-02″:”[13:10:38,14fwk3_dev 2fwk0_dev 2fwk5_dev, 69 40 47 24171 8577 13750 306 31 237, 7 ]”} ends up on HTML page and database collection. For alerting I am using separate scripts, usually written in Node.js or Python. I schedule scripts to start i.e. every minute in Linux ‘cron’ and send email alerts if process load thresholds are exceeded. I will post code in my github repository, with link to either vagrant machine or docker container including running server

Appendix (client side code – Angular controller and service)

.controller('monitorCtrl', function ($scope, MyData) { var smoothiecpu = new SmoothieChart({maxValue:100,minValue:0}); var smoothiecon = new SmoothieChart({maxValue:50000,minValue:0}); smoothiecpu.streamTo(document.getElementById("mycanvascpu")); smoothiecon.streamTo(document.getElementById("mycanvascon")); // Data var linecpu = [new TimeSeries(),new TimeSeries(),new TimeSeries(),new TimeSeries(),new TimeSeries(), new TimeSeries(),new TimeSeries(),new TimeSeries(),new TimeSeries()]; var linecon = [new TimeSeries(),new TimeSeries(),new TimeSeries(),new TimeSeries(),new TimeSeries(), new TimeSeries(),new TimeSeries(),new TimeSeries(),new TimeSeries()]; var seriesOptions = [  … ]; // Add value to each line every second var fwk, con, message; // console.log(MyData.collection.length); setInterval(function() { if (MyData.collection.length) { message = MyData.collection[MyData.collection.length - 1]; fwk = message.cpu; //console.log(fwk); con = message.con; //console.log(con); $scope.source = message.hostname; $scope.cpu = message.time + ' - ' + fwk.join(" "); $scope.con = message.time + ' - ' + con.join(" "); for (var i=0; i < fwk.length; i++) { linecpu[i].append(new Date().getTime(), fwk[i]); linecon[i].append(new Date().getTime(), con[i]); // Add to SmoothieChart smoothiecpu.addTimeSeries(linecpu[i],seriesOptions[i]); smoothiecon.addTimeSeries(linecon[i],seriesOptions[i]); } MyData.collection.shift()} }, 1000); }) .factory('MyData', function($websocket) {  // Open a WebSocket connection  var dataStream = $websocket('wss://10.1.1.1:8880'}); // rest of the MyData service is on https://github.com/AngularClass/angular-websocket });

Afterword by indeni:

Now, imagine if you had a system that has already done this for you with hundreds of more checks, and VSX/VS visibility already pre-built in. Instead of spending the time and effort that it takes to successfully automate these scripts, you could already be avoiding potential downtime. Also, the benefit of indeni is that brilliant professionals like Irek, can share their best practices and we implement them for all customers (once thoroughly reviewed by indeni’s R&D).

Metaphorically, you have all of indeni’s customers acting as an extension of your security, networking, and ops teams – exciting. Two minds are better than one, but thousands of minds are even better than that; especially, this is the case when professionals have levels of insight similar to Irek’s.

Thank you, Irek.

Irek Romaniuk is a Senior Network Security Engineer at Commonwealth Financial Network. He has been working with Check Point firewalls for many years. If you want to contribute as well click here.