FAQ on How CRL Check Mechanism on a Check Point Gateway Works

network monitoring

A Check Point gateway must check that the certificate it received from another entity for authentication purposes has not been revoked. This is achieved by using certificate revocation lists (CRLs). In case the certificate has been issued by the Internal Certificate Authority (ICA), CRL is managed by the security management server.

This FAQ covers the specifics of CRL check implementation on Check Point gateways.

Gateway receives a certificate from another peer during IKE phase 1 negotiation. How exactly is CRL check performed?

At the beginning, the gateway checks the “CRL Distribution Points” field of the certificate it received. Then the corresponding cache is checked in the following manner:

First, the gateway checks presence and validity of the local CRL cache (cache is stored per distribution point). If CRL is present and is not expired (i.e. valid), the gateway uses this cache to check the certificate’s validity. Otherwise, the Check Point firewall sends a request to the host(s) that are listed in the [policy] section of the $FWDIR/conf/masters files in order to get the latest version of CRL.

vpn debug excerpt while trying to establish VPN tunnel with valid local cache:

(For the sake of brevity, part of debug output has been omitted)

[vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] fwValidateRevocation_e: validating using crls [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] fwFetchCRL_e_With_Reason: fetching crl for certificate dn: CN=cp-gw-02 VPN Certificate,O=gaia-sms..9m8yyi [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] X509 Certificate Version 3 Serial Number: 17643 Issuer: O=gaia-sms..9m8yyi Subject: CN=cp-gw-02 VPN Certificate,O=gaia-sms..9m8yyi [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] fwCRLCache_GetFromCache: entered. [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] fwCRL_CRLisValid: [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] thisUpdate: Tue Mar 17 07:12:24 2015 Local Time [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] nextUpdate: Tue Mar 24 07:12:24 2015 Local Time [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] now:Tue Mar 17 16:35:32 2015 Local Time [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] crl start grace period=7200 crl end grace period=1800 crl size val=0 [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] fwCRLCache_Get: dp 'http://gaia-sms:18264/ICA_CRL3.crl' [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] fwCRL_validateRevocation: validate cert: [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] Subject: CN=cp-gw-02 VPN Certificate,O=gaia-sms..9m8yyi [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] with CRL: [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] Issuer: O=gaia-sms..9m8yyi This update: Tue Mar 17 07:12:24 2015 Local Time Next update: Tue Mar 24 07:12:24 2015 Local Time Extensions: Issuing distribution points (Critical): URI: http://gaia-sms:18264/ICA_CRL3.crl DN: CN=ICA_CRL3,O=gaia-sms..9m8yyi [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] fwCert_OurValCerts: validation OK [vpnd 13409 2012104384]@cp-gw-01[17 Mar 16:35:32] vpn1ValidateCertEvent::validateCertificates: Validating certs completed with status 0. (exchange 21)

vpn debug excerpt while trying to establish VPN tunnel without valid local cache:

(For the sake of brevity, part of debug output has been omitted)

[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:57] fwValidateRevocation_e: validating using crls [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:57] fwFetchCRL_e_With_Reason: fetching crl for certificate dn: CN=cp-gw-01 VPN Certificate,O=gaia-sms..9m8yyi [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:57] X509 Certificate Version 3 Serial Number: 80237 Issuer: O=gaia-sms..9m8yyi Subject: CN=cp-gw-01 VPN Certificate,O=gaia-sms..9m8yyi [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRLCache_GetFromCache: entered. [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRLCache_GetFromCache: no cache given. [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRLCache_Get: dp 'http://gaia-sms:18264/ICA_CRL3.crl' [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRLCache_Get: dp (http://gaia-sms:18264/ICA_CRL3.crl) was not found in memory cache. [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRLCache_Get_from_dp: dp (http://gaia-sms:18264/ICA_CRL3.crl) was not found in cache (memory). [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRLCache_Get: dp (CN=ICA_CRL3,O=gaia-sms..9m8yyi) was not found in memory cache. [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwFetchCRL_e_With_Reason: CRL was not found in cache. Will fetch it async.  [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRL_convert_uri: original URI is "http://gaia-sms:18264/ICA_CRL3.crl"

How exactly does a gateway fetch the CRL from the ICA (Internal Certificate Authority)?

The gateway always fetches the CRL by the same method – HTTP GET request on TCP port 18264 (FW1_ica_services).

ICA stores multiple CRL files (for SIC certificates, for VPN certificates, etc.), files are numbered sequentially: ICA_CRL1, ICA_CRL2…

As mentioned before, the gateway determines what CRL file is needed in every situation by looking at the ‘CRL Distribution Points’ field of a certificate.

Which parameters have a CRL by default?

By default, a CRL that has been issued by the ICA is valid for one week.

A new CRL is issued upon expiration of approx. 60% of the lifetime of the current CRL (i.e. 4 days), or immediately after some certificate has been revoked.

A CRL which has been expired keeps being valid for another 30 minutes (CRL grace period).

How does a gateway determine what host it should reach in order to get a CRL?

When trying to fetch a CRL (for a certificate, issued by ICA) a gateway tries to reach all objects that are listed in the [policy] section of the $FWDIR/conf/masters file.

strace output for vpnd process while trying to fetch a CRL:

1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] resolver_gethostbyname: Performing gethostbyname for fake-smsn", 111) = 111 1752open("/etc/hosts", O_RDONLY)= 36 1752read(36, "#This file was AUTOMATICALLY GENERATEDn#Generated by /bin/hosts_xlate on Mon Mar 16 15:31:05 2015n# n#DO NOT EDITn# n10.10.73.101 cp-gw-01n127.0.0.1 localhostn::1 localhostn", 4096) = 179 1752read(36, "", 4096)= 0 1752close(36) = 0 1752connect(36, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 28) = 0 1752send(36, "21401110fake-sms341", 26, MSG_NOSIGNAL) = 26 1752write(8, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] resolver_gethostbyname6: getaddrinfo failed. reason=Name or service not knownn", 127) = 127 1752write(8, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] get_ips_or_sicnames_from_list: Failed to get the obj of fake-sms from objects.Cn", 129) = 129

vpn debug while trying to fetch a CRL:

[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] fwCRL_convert_uri: original URI is "http://gaia-sms:18264/ICA_CRL3.crl" [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] RangesMap::RangesMap: called. obj=gaia-sms. resolve_only_clust_ifs=0 is_gateway=0 MainIP=4:<10.10.72.100>, 6:<::> [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] RangesMap::RangesMap: called. obj=bogus-sms. resolve_only_clust_ifs=0 is_gateway=0 MainIP=4:<10.10.72.10>, 6:<::> [vpnd 5428 2012436160]@cp-gw-01[18 Mar 12:22:17] fwCRL_convert_uri: URI was converted to "http://10.10.72.100:18264/ICA_CRL3.crl" [vpnd 5428 2012436160]@cp-gw-01[18 Mar 12:22:17] fwCRL_convert_uri: URI was converted to "http://10.10.72.10:18264/ICA_CRL3.crl" [vpnd 5428 2012436160]@cp-gw-01[18 Mar 12:22:17] fwCRL_Internal_Fetcher: fetching from URI: http://10.10.72.100:18264/ICA_CRL3.crl [vpnd 5428 2012436160]@cp-gw-01[18 Mar 12:22:17] Snatcher constructor [vpnd 5428 2012436160]@cp-gw-01[18 Mar 12:22:17] fwCRL_Internal_Fetcher: fetching from URI: http://10.10.72.10:18264/ICA_CRL3.crl [vpnd 5428 2012436160]@cp-gw-01[18 Mar 12:22:17] vpn1ValidateCertEvent::validateCertificates: Validating certs will be async. (exchange 2)

How does a gateway resolve object names from the $FWDIR/conf/masters file?

Upon its start, vpnd process reads the $FWDIR/database/objects.C file. This file contains the description of the objects in the management server database, including their IP address, NAT settings, etc.

strace output for vpnd process upon its start:

open("/opt/CPsuite-R77/fw1/database/objects.C", O_RDONLY) = 11 brk(0xa5ed000)= 0xa5ed000 fstat64(11, {st_mode=S_IFREG|0660, st_size=1725808, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77f71000 read(11, "(nt:antispam_policy (ntt: (Global_Mail_Security_Policynttt:add_disclaimer (true)nttt:allow_mail_track (none)nttt:block_list_blocked_track (log)nttt:block_list_mode (on)nttt:customized_spam_server ()nttt:disclaimer_text ("Email secured by Check Point")nttt:email_size_scan (4096)nttt:ip_rep_mode (on)nttt:ip_rep_spam_actions (reject)nttt:ip_rep_spam_conf_level (90)nttt:ip_rep_spam_track (log)nttt:ip_rep_sus_spam_actions (reject)nttt:ip_rep_susspam_conf_level (80)nttt:ip_rep_susspam_track "..., 8192) = 8192

While attempting to resolve the object address from the $FWDIR/conf/masters file (e.g. while trying to fetch the CRL), vpnd uses the following sources: /etc/hosts file, DNS servers, and also the data that has been extracted from the objects.C file (the latter has the highest priority).

Data from the objects.C file is stored in the dynamic memory allocated by the system kernel to the vpnd process.

strace output for the vpnd process while trying to resolve the object’s IP address:

1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:09:58] resolver_gethostbyname: Performing gethostbyname for fake-smsn", 111) = 111 1752open("/etc/hosts", O_RDONLY)  = 36 1752read(36, "#This file was AUTOMATICALLY GENERATEDn#Generated by /bin/hosts_xlate on Mon Mar 16 15:31:05 2015n# n#DO NOT EDITn# n10.10.73.101 cp-gw-01n127.0.0.1 localhostn::1 localhostn", 4096) = 179 1752read(36, "", 4096)= 0 1752close(36) = 0 1752connect(36, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 28) = 0 1752  send(36, "21401110fake-sms341", 26, MSG_NOSIGNAL) = 26 1752write(8, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] resolver_gethostbyname6: getaddrinfo failed. reason=Name or service not knownn", 127) = 127 1752write(8, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] get_ips_or_sicnames_from_list: Failed to get the obj of fake-sms from objects.Cn", 129) = 129

How does the CLR cache mechanism (CRL Prefetch) work?

In order to decrease time for establishing a VPN tunnel (fetching a CRL could be rather slow process due to the large file size, which could lead to an IKE negotiation delay or even timeout), the gateway keeps a local copy of CRL.

Generally, when a local cache is present, a gateway tries to fetch a new CRL (from every host in $CPDIR/conf/masters file) every 2 hours. This parameter is configured via SmartDashboard menu Policy –> Global Properties –> SmartDashboard Customization –> Certificates and PKI properties –> prefetch_crls_duration.

Also the CRL cache is updated/flushed upon policy installation on a gateway. This behavior is configured on the same tab Policy –> Global Properties –> SmartDashboard Customization –> Certificates and PKI properties.

A CRL cache has limited lifetime – 24 hours by default (configured from Certificate Authority Properties window).

vpn debug excerpt while saving a CRL cache on a gateway:

(For the sake of brevity, part of debug output has been omitted)

[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRL_CRLisValid: [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] crl start grace period=7200 crl end grace period=1800 crl size val=0 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwFetchCRL_cb: Good CRL, successful fetch. [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRLCache_Put: Put CRL (http://gaia-sms:18264/ICA_CRL3.crl) in the memory cache - timeout 86400, crl_to 86400 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRLCache_Put: Put CRL in the FDB file till Thu Mar 19 10:10:18 2015 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] CrlDB::Key: buf= http://gaia-sms:18264/ICA_CRL3.crlinternal_ca [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] CrlDB::Key: key= 27c1ece094c15bfb48c78c960ecba984 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_new: name= /opt/CPsuite-R77/fw1/database//CrlCache version= 1 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_new: path= /opt/CPsuite-R77/fw1/database//CrlCache_1 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] FileDB_Locker::Lock: _opt_CPsuite-R77_fw1_database__CrlCache_1 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_store: save 517 bytes in '27c1ece094c15bfb48c78c960ecba984' [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRL_Hook_cb: first successful fetch. This fetch will be processed.

If a CRL cache has not been updated till its lifetime expiration, it will be deleted from a gateway.

If a CRL fetch has been successful (the gateway was able to get a CRL from one of the hosts, and hence updated its own cache) the CRL cache lifetime effectively resets back to 24 hours.

The time of the last successful refresh of a CRL cache can’t be determined from modification time of $FWDIR/database/CrlCache_1/rec_<random string> file (this is where CRL cache is stored on the file system, more on that later). Modification time of this file is updated on every CRL fetch attempt, even if it was unsuccessful.

In case you would want to find out how much time is left until the CRL cache will be deleted, vpn debug could be used.

vpn debug excerpt while the CRL is checked by a gateway:

(For the sake of brevity, part of debug output has been omitted)

[vpnd 24579 2012128960]@cp-gw-01[16 Mar 19:42:57] fwCRLCache_GetFromCache: entered. [vpnd 24579 2012128960]@cp-gw-01[16 Mar 19:42:57] fwCRL_CRLisValid: [vpnd 24579 2012128960]@cp-gw-01[16 Mar 19:42:57] crl start grace period=7200 crl end grace period=1800 crl size val=0 [vpnd 24579 2012128960]@cp-gw-01[16 Mar 19:42:57] fwCRLCache_Get: dp 'http://gaia-sms:18264/ICA_CRL3.crl' [vpnd 24579 2012128960]@cp-gw-01[16 Mar 19:42:57] fwCRLCache_Get: crl 0x6ede91a0, exp 86060 [vpnd 24579 2012128960]@cp-gw-01[16 Mar 19:42:57] fwCRLCache_Get: tempcrl (nil), exp 0

If a gateway was unable to refresh the CRL cache in a period of 24 hours, then the cache is deleted from the gateway. Also the modification time of the& $FWDIR/database/CrlCache_1/rec_<random string> file stops being updated by subsequent fetch attempts.

When there is no valid CRL cache, the gateway tries to fetch it either every hour, or when there is a necessity (e.g. when some peer is trying to establish VPN tunnel).

How is the cache stored on a gateway?

The gateway stores a local copy of the CRL in multiple places:

strace output for the vpnd process while fetching the CRL:

(For the sake of brevity, part of debug output has been omitted)

1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] nnfwFetchCRL_cb: Entering for dp: http://gaia-sms:18264/ICA_CRL3.crln", 118) = 118 1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwFetchCRL_cb: 1 X509 CRLsn", 76) = 76 1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] Issuer: O=gaia-sms..9m8yyinThis update: Tue Mar 17 07:12:24 2015 Local TimenNext update: Tue Mar 24 07:12:24 2015 Local Timenn", 175) = 175 1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwFetchCRL_cb: Good CRL, successful fetch.n", 92) = 92 1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwFetchCRL_cb: Writing CRL to web servern", 90) = 90 1752open("/opt/CPsuite-R77/fw1/conf/extender/MyCRL/3ac593becbfc9eb649937af9ca1306d5.crl", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 43 1752write(43, "02021375020134621010r6t*206H206367r11550003313102763U4n2320gaia-sms..9m8yyi27r150317041224Z27r150324041224Z0+0242333132427r140704115511Z0232207627r141118084934Z240j0h0f63U3534113774Z240X240V206"http://gaia-sms:18264/ICA_CRL3.crl24400.13102763U4n2320gaia-sms..9m8yyi12101763U43f10ICA_CRL30r6t*206H206367r1155320211t253245203302347217A]3023371Af350$F-+2017Ilr20425623225534t33220>l274235 337307371256252z[233I>266fd6270>[210274322e322356;305]275G330267306<26?377350316207]342l217,2453042603443403359334337307224H206B210 232>4365206325`221226370264375d225253230355I23132737523211a234237w~?316240;p236240323tyB$25017275252Q2008C241214234254bP216312322M355373233372wI25634030726433323336331265350`250361)323)30D5211|Hm,]2372432303312213474t301207[37384P1v325306261315p22635426435245255::o274241344256272r207264.17353305304305431021533717735233231231=363252217{236/!eM200246267|", 513) = 513 1752close(43) = 0 1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRLCache_Put: Put CRL (http://gaia-sms:18264/ICA_CRL3.crl) in the memory cache - timeout 86400, crl_to 86400n", 160) = 160 1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRLCache_Put: Put CRL in the FDB file till Thu Mar 19 10:10:18 2015nn", 120) = 120 1752write(2, “[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] CrlDB::Key: key= 27c1ece094c15bfb48c78c960ecba984nn, 100) = 100 1752write(8, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_store: save 517 bytes in '27c1ece094c15bfb48c78c960ecba984'n", 113) = 113 1752write(8, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_filename_pre: return /opt/CPsuite-R77/fw1/database//CrlCache_1/tmpXXXXXXn", 126) = 126 1752open("/opt/CPsuite-R77/fw1/database//CrlCache_1/tmpC7J1aD", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 44 1752write(2, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_filename_pre: return /opt/CPsuite-R77/fw1/database//CrlCache_1/rec_27c1ece094c15bfb48c78c960ecba984n", 153) = 153 1752rename("/opt/CPsuite-R77/fw1/database//CrlCache_1/tmpC7J1aD", "/opt/CPsuite-R77/fw1/database//CrlCache_1/rec_27c1ece094c15bfb48c78c960ecba984") = 0 1752write(8, "[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRL_Hook_cb: first successful fetch. This fetch will be processed.n", 118) = 118

• Dynamic memory of vpn process – this is from where gateway actually reads the CRL cache in order to check certificate validity.

VPND memory dump fragment:

• $FWDIR/conf/extender/MyCRL/ directory.

After fetching the CRL from the distribution point, the gateway also puts a CRL copy into the $FWDIR/conf/extender/MyCRL/ folder.

The copy of the CRL stored there is in regular format (X.509 v2 CRL) and could be viewed by standard utilities.

• $FWDIR/database/CrlCache_1/ directory.

Another CRL copy is placed in the $FWDIR/database/CrlCache_1/ folder.

The filename has the format rec_<CrlDB Key>. The administration guide states that this file is actually the local CRL cache. The truth is this file is used only for reference on the file system, you couldn’t replace it by another (e.g. newer) CRL. As mentioned before, the copy used by the gateway is stored in memory.

vpn debug excerpt while saving the CRL cache locally:

[vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRLCache_Put: Put CRL (http://gaia-sms:18264/ICA_CRL3.crl) in the memory cache - timeout 86400, crl_to 86400 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fwCRLCache_Put: Put CRL in the FDB file till Thu Mar 19 10:10:18 2015 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] CrlDB::Key: buf= http://gaia-sms:18264/ICA_CRL3.crlinternal_ca [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] CrlDB::Key: key= 27c1ece094c15bfb48c78c960ecba984 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_new: name= /opt/CPsuite-R77/fw1/database//CrlCache version= 1 [vpnd 1752 1978582720]@cp-gw-01[18 Mar 10:10:18] fdb_new: path= /opt/CPsuite-R77/fw1/database//CrlCache_1 processed.

This copy of the CRL can not be viewed by standard tools, as it’s stored in a non-regular format.

The difference is in the first four bytes of the file, which are prepended to X.509 CRL by Check Point. I presume this might be the result of some hash function.

What is the maximum size of the CRL cache?

There is a SecureKnowledge article sk19836 that states that the gateway has a 1 Mb buffer to store the CRL cache. However, it also has a recommendation not to exceed 10K size.

How to manually flush a CRL cache?

You could force a CRL cache deletion by running the command

# vpn crl_zap

The whole directory $FWDIR/database/CrlCache_1/ is deleted upon its execution, but the CRL cache is kept in memory for a few more minutes.

How to view a cached CRL from a gateway’s CLI?

The cache could be viewed by running the command

# vpn crlview -view <path-to-CRL>.

Remember, though, that you can not view the file in $FWDIR/database/CrlCache_1/ by this utility, instead you should use the one located in $FWDIR/conf/extender/MyCRL/ directory.

What happens if a gateway has failed to check the received certificate against the CRL?

In case the gateway was neither able to locate the appropriate CRL in its cache, nor fetch one from the distribution point (which is the security management server in this ICA scenario), the tunnel establishment will fail with the following message:

Hence, you should ensure the reliability of the CRL distribution point(s).

The other way to approach the temporary inaccessibility of the CRL distribution point is to increase the CRL cache validity period. In this case however you’ll have to face the tradeoff between security and impact of a CRL distribution point failure.

At last, quick troubleshooting tips to resolve CRL-related issues:

  1. Check current time and date on a gateway, CRL might not be valid yet from a gateway perspective (check NTP settings)
  2. From a gateway, check the CRL distribution point reachability by http (use wget or curl utilities) (check port 18264)
  3. If the issue has not been resolved by the first two steps, enable vpn debug and look for the word “CRL” in the output.

Maxim Bilyukov is a Technical Solutions Architect at AKON Technologies. He has been working with Check Point Firewalls for five years. If you want to contribute as well, click here.

Check Point appliances refresh: how do you compare?

We often get asked if we have data pertaining to the upgrade processes and cycles of Check Point users around the world. The short answer is, YES. The longer one, is that thanks to our indeni Insight service we get a deep view into the Check Point firewall user base. Once in a while, we publicly share the findings we’ve come to based on that data, like we did last September.

Today we’ll take a look at the appliance refresh process across our user base. Apparently the 2012 (and later) appliances are gaining a stronger foothold with almost three quarters of the Check Point firewalls indeni is connected to being these newer appliances. This is in contrast to less than half, just 10 months ago (see the September report referenced above).

This is a pretty good ratio, considering most older appliances still have until April 2017 before they reach end of support.

In our daily conversations with Check Point customers (some, who are not indeni customers, yet) we see that summer-time is being utilized to complete hardware and software upgrades. It is usually a more relaxed time and easier for the higher ups to approve maintenance windows. It is also before the holiday season, a time of change freeze for most companies.

During this process, we suggest you keep in mind that the recommended way of upgrading a Check Point firewall is through a complete rebuild, even in the case of just a software upgrade. This is better than simply backing up the firewall configuration and restoring it. It is possible because most of the interesting configurations – the security policy – are actually stored on the management server.

However, this approach can also result in issues – routes that are missing, kernel parameters that are no longer set the way they should, SecureXL settings that have been lost, etc. So be extra careful and test things thoroughly before putting the new firewalls in production, as well as after. The list of top 10  issues people run into when working with Check Point firewalls can be found here.

Happy upgrading!

v9.X To 10.2.4 Upgrade Guide: F5 LTM Load Balancing Methods

F5 LTM Load Balancing Methods: Guide on How to Upgrade to 10.2.4

Unfortunately there are still a lot of F5 ® version 9.x installations out there, and there shouldn’t be. Not only is version 9 a long way out of support, it’s really not secure in this day and age. You’re also missing out on using iHealth® which can only be used from version 10.x onwards.

Please note that you may need AskF5™ login credentials to access some of the links within this article. If you do not have an AskF5 login you can request one here.

This guide assumes an upgrade to version 10.2.4 unless otherwise stated, although the same process applies to earlier 10.x versions. Note that 10.0.x releases are no longer supported. 10.2.4 is the recommended version for this branch and is supported to 31st December 2016 (see AskF5 SOL5903).

Prerequisites:
Before considering upgrading, ensure your hardware platform will support 10.2.4 code. The following platforms support it, as confirmed by AskF5 SOL9412:

You can only upgrade directly to 10.2.4 code from 9.3.x or 9.4.x releases.  If you are running an earlier 9.x release, you must upgrade to 9.3.x or 9.4.x before continuing.

Before you begin

Assuming you meet the prerequisite requirements, consider the following:

  • YOU MUST re-activate the unit license before upgrading.  See AskF5 SOL7727 for reasoning.
  • Backup EVERYTHING: /config/bigip.conf, bigip_base.conf, bigip.license, bigip_sys.conf (if it exists), wideip.conf (if it exists), take an SCF* (recommended, but only backs up LTM configuration), take a UCS (note that UCS also saves admin/root passwords and the license, so take this after license reactivation). Save these off box.  Also save the UCS file to /shared/ (you might need to create this directory) and to /var/local/ucs.  An extra 10 minutes spent here can save agony later on! If you are running ASM, also take a backup of the policy.

*to take an SCF, use the command:

# bigpipe export <filename.scf>

for example:

# bigpipe export mybackup.scf
  • You should still read the release notes for the version you are installing.
  • If you have a HA pair, you will, of course, be working on your standby unit.
  • It is recommended to disconnect any unnecessary cables from a unit whilst upgrading (only leave connected the cables you need, such as the management cable). This will prevent any issues with the unit attempting to process traffic whilst upgrading.

Let’s upgrade

1)    Download the 10.2.4 ISO and copy* to /shared/images (create this folder if it does not already exist).
*SCP is the recommended method to copy files to/from the BigIP system. For a Windows client, check out WinSCP 

2)    Install the image2disk utility:

# im /shared/images/<filename.iso>

for example:

# im /shared/images/BIGIP-10.2.4.577.0.iso

3)    Begin the install. The full command syntax is

# image2disk --instslot=HD<slot_number> --format=<format_style> <downloaded_filename.iso>

BUT WAIT! Observe the following options first:

  • Assuming you only want version 10.x and above from now on (highly recommended!), we need to format for volumes, so use:
# image2disk --instslot=<slot_number> --format=volumes /shared/images/>filename.iso>
  • Assuming we wish to run version 10 and version 9 (not recommended, version 9.x is not supported), use:
# image2disk --instslot=<slot_number> --format=partitions /shared/images/>filename.iso>

Note that since we repartition, we will need to reinstall version 9 to a free partition after the version 10 installation completes.

  • To preserve your existing configuration, use
# image2disk --instslot=<slot_number> --nomoveconfig--format=volumes /shared/images/<filename.iso>

DO NOT use the ‘–nomoveconfig’ flag if you are running ASM. Use a backup UCS instead. Using ‘–nomoveconfig’ means: “The configuration of the target boot location (if any is present) is re-installed on the target boot location after re-imaging is complete.”

The ‘slot_number’ variable will be HD1.1 or HD1.2 for example (the inactive partition, NOT the one you are currently running in*)
* If you are not sure which partition you are currently running on, use 

# switchboot -l
  • If installing 10.2.x the flag “–instslot=<slot_number>” is NOT permitted so should be absent from the command to install software.  Reasoning: 10.2.x will always install to HD1.1 after the disk is formatted.

4)    The unit should reboot and install the new software.

5)    Once finished, you may need to do

# switchboot

and select the newly upgraded partition, then

# full_box_reboot

This depends on the specific upgrade version so may not be applicable. Install hotfixes at this point if necessary. It’s always recommended to be running the latest available hotfix.

6)    Wait for the filesystem to build and the installation to finish. This first boot after upgrade can take a while!

7)    Once the unit is up and running, load on any necessary config files.  These can either be a UCS (User Configuration Set, aka ‘Archive’) uploaded via System > Archives, or can be individual bigip.conf, bigip_base.conf, bigip.license (etc) files, which should be uploaded to the /config directory using SCP.  If you upload the individual configuration files, you’ll need to reboot again.  Check the configuration is present and correct via the GUI.  Cable the unit back up and you should be good to go.  If you are running a HA pair, you should failover during a change window and check the newly upgraded unit processes traffic correctly.  Assuming it does, repeat the upgrade process on your second unit.

8)    Beginning with version 10.0.0 of the software, a redundant system configuration must contain failover peer management addresses for each unit. If you roll forward a redundant system configuration from 9.3.x or 9.4.x, the units start up correctly, but the system logs a message every ten minutes reminding you to configure the peer management addresses. To configure the failover peer management addresses, navigate to System > High Availability > Network Failover, then specify the management IP address of the peer unit in the Peer Management Address field. Then do the same on the other unit in the configuration. Once you specify both IP addresses, the system should operate as expected. For more information, see AskF5 SOL9947.

You’re all done!

Chris Spillane is a Senior Security Analyst at NTT Com Security. He has been working with F5 Load Balancers for more than seven years. If you want to contribute as well, click here.

Find more tip on how to manage your F5 load balancers with indeni.

How to Troubleshoot Check Point Firewall VPN Connection

In this example the tunnel between GWA (Gateway A) and GWB (Gateway B) is down. Both gateways could be managed by the same management server, or different ones. Both could be Check Point Firewalls or one could be another brand.

Since at least one gateway needs to be a Check Point gateway managed by us, in this example this is GWA. GWB can either be another one of our gateways or an external one.

SmartView Monitor

Let’s see what this has to say about the tunnel. (Viewing VPN tunnels in SmartView Monitor requires a monitoring license installed on the management server, and enabled on the gateway itself).

Open the SmartView Monitor and go to “Tunnels on Gateway”:

First select GWA in the list and review if the tunnel in question is UP, DOWN or Up – Init. Up – Init means that it is trying to establish the tunnel, and will probably mean that in a few seconds the tunnel will go to DOWN state or UP state.
Now go to “Tunnels on Gateway” again and select GWB (if both gateways are managed by the same management server).

Learn how indeni enables pre-emptive maintenance of Check Point Firewalls 

One issue we could see here is for example that the tunnel is UP from GWA perspective, but DOWN from GWB perspective. If the “Permanent tunnel” is activated on the VPN community (both gateways need to be Check Point) they will exchange UDP tunnel test packages (Name: tunnel_test, UDP/18234). If GWA does not receive these packets, it will think the tunnel is down.

However we could be in a situation where packets from GWA to GWB arrive, but not in the opposite direction (GWB to GWA). We will then see that the tunnel looks to be up from one side, but not the other. The reason for this is packets lost in transit, maybe due to DDoS protections, routing on internet or other issues.

If we have a tunnel from our Check Point gateway (GWA) to a non-check point gateway (GWB) we cannot use permanent tunnels. This means that the tunnel will be down, and not appear in this list until traffic is sent in it.

So why it is down could be as simple as no traffic has been sent into the tunnel. Another issue could arise if GWB is not a Check point gateway, but the permanent tunnel is activated anyway. The tunnel will then show as down from GWAs perspective since it assumes that GWB will send the tunnel test packages.

SmartLog/SmartViewTracker

Sort traffic with GWA as source, and GWB as destination. Then also check the other way around, GWA as destination and GWB as source.

Here we could see if the PSK (pre-shared key) is incorrect for example, or if IKE packets are dropped. If the PSK is incorrect, make sure both sides have the same PSK and remember that it cannot be longer than 64 characters (longer than that and it will be cut off at 64 chars, see sk66660 on the Check Point support portal.

If the tunnel broke suddenly, check drops from the time the tunnel stopped working. There can be situations where the drop log is not shown repeatedly. The most common thing you would see here is the not so friendly error “Packet is dropped because there is no valid SA – please refer to solution sk19423 in SecureKnowledge Database for more information“. This means that the two gateways did not reach an agreement. This is due to the fact that the proposals are different between the gateways. The proposal contains for example the subnets in the encryption domain.

The most common issue in Check Point has to do with something called super netting. To understand why Check Point does this, we need to understand how a VPN tunnel works. In a VPN tunnel one Phase1 will be established and then one Phase2 per subnet pair. If you have two /24 subnets on each side of the tunnel that need to speak to each other, that is 4x Phase2. Check Point will create as few subnets as possible and therefore it will create one /23 subnet instead of 2x /24 if possible. If the other side of the tunnel has 2x /24 configured and the Check Point have one /23 in its proposal the tunnel will fail. It’s not easy to check the proposals in the Tracker or SmartLog, so for that we need to debug the VPN tunnel and check out the debug file with IKEView (see next section below).

If you get the error “invalid certificate” then the port 18264 is closed between the gateway and management server. This port is used for GWA to verify GWB’s certificate in the case that both are managed by the same management server. Then they do not use PSK.

IKEView

If we cannot establish why the tunnel fails with the above methods we need to take a better debug. You can refer to: sk63560 on the Check Point support portal. Below is a summary.

On the active gateway, run:

# vpn debug trunc

Now a debug file will be created at: $FWDIR/log/ike.elg and $FWDIR/log/ike.elg.0

Do some resets on the tunnel to get some data into this or of the tunnel is down, try to make it establish the tunnel again by sending data into the tunnel, then download the ike.elg file to your desktop and open it with IKEView (available from Check Point support site). If you do not have the monitoring license to SmartView Monitor you can use the CLI command:

# vpn tu

to reset tunnels on GWA. Select option (7) Delete all IPsec+IKE SAs for a given peer (GW) and input GWBs IP address. In this program you will see what data is being sent between the gateways, what proposals etc., to see if there is anything not matching. It is sorted on the remote gateway IP, and you can follow both what proposal GWA sends to GWB and also what GWB sends to GWA. End the debug with:

# vpn debug off # vpn debug IKEoff

Optionally delete $FWDIR/log/ike.elg* to not have old things in it the next time you troubleshoot.

zdebug

Another tool we can use is zdebug. This is a tool for checking dropped packets and reasons.
Do you wonder why it’s called zdebug? Apparently the person who wrote this program had a name starting with Z.

There are times when we can have drops which are not logged in the normal log, or the reason is not properly stated there. Then zdebug is helpful. Go to GWA and run (in expert mode):

# fw ctl zdebug drop | grep IP_OF_GWB

This will show if we have any dropped IKE packets etc. Example output:

;[cpu_5];[fw4_0];fw_log_drop_ex: Packet proto=17 REMOVED:500 -> REMOVED:500 dropped by fwpslglue_chain Reason: PSL Drop: ASPII_MT; ;[cpu_5];[fw4_0];fw_log_drop_ex: Packet proto=17 REMOVED:500 -> REMOVED:500 dropped by fwpslglue_chain Reason: PSL Drop: ASPII_MT; ;[cpu_5];[fw4_0];fw_log_drop_ex: Packet proto=17 REMOVED:500 -> REMOVED:500 dropped by fwpslglue_chain Reason: PSL Drop: ASPII_MT;

You can then search on the Check Point user center for the part “fwpslglue_chain Reason: PSL Drop: ASPII_MT” and you will in this example find issue sk90322 which explains the issue and the solution for this specific example.

If nothing else works

If nothing can be solved by the methods above and if time is critical there are some emergency measures that can be taken:

• Fail over the cluster (if it is a cluster)
• Push policy
• Delete the Community and re-create it
• Make sure you use IKE v1 in the Community

Johnathan Browall Nordström is the Team Lead of Network & Security at Betsson Group. He has been working with Check Point firewalls for more than four years. If you want to contribute as well, click here.

Palo Alto Networks Firewalls Alert Guide: Group ID Conflict Detected

This is a real life sample alert from indeni alert guide for Palo Alto Firewalls.

 

Description:

This cluster has the same Group ID as the other clusters listed below. A conflict may arise if they share a VLAN with this cluster.

Other Clusters:

buny-fw1 (10.10.24.1)

Manual Remediation Steps:

Consider changing the Group ID. For more information, see DOC-5843.

How does this alert work?

indeni automatically identifies the HA clusters in the environment and then compares the Group ID that is set on the active member of each of those clusters.

For even more alerts and in depth analysis to make your network high availability and failure proof, check out our device management solution for PAN Firewalls.

Proxy ARP Entries Removed – Check Point Firewalls Optimized Performance

This is a real life sample alert from the indeni guide to preemptive maintenance for Check Point Firewalls.

Description:

This firewall used to have (51) proxy ARP entries. They have disappeared suddenly from the output of “fw ctl arp”. Proxy ARP behavior may be impacted.

Manual Remediation Steps:

If this is due to an interface being taken down, please verify that “fw ctl arp” provides the correct output after the interface being turned back on. If it doesn’t, contact technical support.
If this is not due to an interface being taken down, we recommend you contact technical support. Please review SK98740 and SK93534.

How does this alert work?

indeni runs the “fw ctl arp” command every few minutes and identifies when there is a major change in the response.

Firewall in Maintenance Mode. Palo Alto Network Alert Guide

indeni, cisco

This is a real life sample alert from our indeni alert guide for Palo Alto Networks Firewall.

Description:

The firewall has entered maintenance mode due to an unknown reason. indeni will stop collecting data from this firewall until it exits maintenance mode.

Manual Remediation Steps:

Connect to the firewall using SSH (see DOC-5719) and determine the cause.

How does this alert work?

indeni uses a mix of SSH, API calls and SNMP to communicate with Palo Alto Networks firewalls. If it identifies that the firewall is in maintenance mode (for example, via SSH), it will alert.

F5 bigd process down

This is a real life sample alert from indeni

Description:

The F5 bigd process is down and has not restarted. Among its responsibilities, bigd runs the monitors for nodes, pool members and services. For more information, read SOL6967.

Manual Remediation Steps:

Review the logs to identify why the bigd process is down. indeni will attempt to determine the source of the issue automatically as well.

How does this alert work?

indeni tracks the status of all of the critical operating-system level processes and alerts if any of them crashes or shuts down unexpectedly.

How to Pull and View Logs Using Automation for Palo Alto Networks Firewalls

Many network monitoring tools on the market today are just good at that: monitoring. They fail to go in depth and dig deep into devices to pull the gritty data important to IT teams. We build indeni with those users in mind. Our goal is to simplify network management, not just monitor it. For example:

There are two sets of log “components” in Palo Alto Networks firewalls:

  • The easily accessible logs (for lack of better name):
  • indeni@Peanut(active)> show log > alarm Show alarm logs > appstat Show appstat logs > configShow config logs > dailythsumShow dailythsum logs > dailytrsumShow dailytrsum logs > dataShow data logs > hipmatchShow hipmatch logs > hourlythsum Show hourlythsum logs > hourlytrsum Show hourlytrsum logs > iptag Show iptag logs > mdm Show mdm logs > systemShow system logs > threatShow threat logs > thsum Show thsum logs > traffic Show traffic logs > trsum Show trsum logs > url Show url logs > useridShow userid logs > weeklythsum Show weeklythsum logs > weeklytrsum Show weeklytrsum logs > wildfireShow wildfire logs  indeni@Peanut(active)>

A different kind of logs.

indeni is now capable of accessing the SSH-only logs and analyzing those. So, if you have certain log lines you’d like to automatically collect and analyze from these files, please feel free to email us at sales@indeni.com and share your needs. We’ll be sure to include those in our software, in addition to the thousands of other log lines that are already on our list.

NERC Compliance Best Practices for Critical Infrastructure Protection (CIP) v5

We have a number of US-based energy grid operators that are leveraging indeni’s capabilities to meet the NERC CIP v5 requirements, that are soon to be upon us all (April 2016). This blog post details how indeni can help in an effort to articulate what indeni can do for you, as well as what it can’t.

Let’s start with one challenge: a recent letter from the North American Reliability Corporation (NERC) discusses the challenges of categorization and protection of network devices and externally accessible devices. It starts with identifying the challenge of knowing what devices you have and where. This is important – keep track of what you own and its responsibilities. To this end, the Inventory Report feature in indeni will help you: at any given moment, you’ll have access to an up-to-date report (in Excel format) that will list the devices you own and auxiliary information pertaining to those devices.

That same letter re-enforces the concept that 15 minutes are considered as a critical downtime period. That is – a network device that is categorized as BCA (BES Cyber Asset) and is unavailable for 15 minutes can potentially have a disastrous impact on overall system stability:
“For example, a core network switch in either an Energy Management System (EMS) at a Control Center, or located within a coordinated protection system design at a Transmission station, could be rendered unavailable or have its performance degraded to the point that the EMS or protection system could not perform its expected reliability functions, leading to a condition whereby those systems could adversely affect the reliable operation of the Bulk Electric System in real time. Such a core network switch would meet the definition of a BES Cyber Asset.”

Switch, router, firewall, load balancer or any other network device. That includes Check Point, Cisco, Fortinet, Juniper and Palo Alto Networks firewalls. That also includes F5 load balancers. All squarely within indeni’s expertise.

To drastically reduce downtime occurrences, operators must deploy a system that can anticipate issues before they occur and provide early warning. That is exactly what indeni’s core functionality includes. With indeni, operators are already staying on top of the network configuration and are averting up to 90% of their sev-one issues.

In a separate document, NERC lists the challenges energy grid operators encountered on their way to implement the requirements under CIP v5. Here, one of the items is “Implementing solutions for new inventory and change management requirements” (which is CIP‐010‐1). It was even listed as a “high interest” item.

 

We’ve discussed the inventory challenge, but what about change management? How do you ensure that:

  • The right users are configured on your network devices?
  • The TCP/UDP ports that are open on your network devices are the right ones?
  • The ports used on your switches are configured correctly (those that should be up are up, and those that should be down are down)?
  • That DNS, NTP, and similar settings are done correctly and consistently?
  • The correct operating system and software is being used? Even the specific hotfixes?

indeni’s Configuration Checks feature helps you here in conjunction with the Inventory Report feature. With indeni, you’ll be able to ensure your configurations match the requirements. If they don’t, you’ll get an alert indicating the misconfigured device and what the configuration should be. Specifically, indeni will help you meet these requirements (copied from CIP-010-01):

Develop a baseline configuration, individually or by group, which shall include the following items:

1.1.1. Operating system(s) (including version) or firmware where no independent operating system exists;
1.1.4. Any logical network accessible ports; and
1.1.5. Any security patches applied.

Monitor at least once every 35 calendar days for changes to the baseline configuration (as described in Requirement R1, Part 1.1). Document and investigate detected unauthorized changes.

The bottom line here is that you’re not alone. Ensuring compliance with NERC CIP v5 is challenging and you will need to document new processes and roll out new solutions and functionality. We’re here for you to ensure you meet the deadline and pass with flying colors.

 

See what else indeni can do for you to maintain your infrastructure in compliance. Just fill out the form below. 

[ninja_form id=16]