More than a month ago, we have received an alert from Blacklist Monitor that our main server became blacklisted on Spamhaus SBL blacklist. This was a problem for us because we do have our own mail server that sends emails to our users and suddenly, our emails were silently blocked in many systems because of this listing. And we started our investigation. Why were we blacklisted if we did not send any spam?

The monitor told us that our listing details can be found on http://www.spamhaus.org/sbl/query/SBL245859, so this was the first information to read. The information on that page was very clear:

Continue reading →

The code sample below demonstrates how to use Online Domain Tools Whois API to perform a large number of WHOIS lookups of either domain or IP records. It uses asynchronous callback method in order to minimize needs for settings of the environment where the script can be successfully run. Any PHP 5.3+ hosting should be able to run this script. The script loads WHOIS queries from the database and processes them one by one. Please read the initial comment to get more information on how to use this script and how it works.

<?php
//
// This script is an example of how to use Online Domain Tools Whois API to perform 
// a large number of WHOIS requests. It allows you to perform any amount of WHOIS queries 
// written into an input file.
//
// This script is provided "AS IS" without warranties of any kind, either express or implied.
// Everyone is allowed to use it, copy it, or create derived products from it, completely
// free of charge, provided that Online Domain Tools TOS (see the link below) is respected
// and not violated.
//
//
// Copyright (c) 2015 - Online Domain Tools
//
// Relevant links:
//  * http://whois.online-domain-tools.com/ - ODT Whois
//  * http://online-domain-tools.com/information/api - ODT API specification
//  * http://online-domain-tools.com/information/privacy-policy-and-tos - ODT TOS
//  * http://online-domain-tools.com/ - Online Domain Tools homepage
//
//
// How does it work?
//  * This script loads input queries stored in $whoisInputFile, separated by newlines.
//  * Each time it runes it removes the first line from the input file and sends it 
//    as a query to ODT API server.
//  * Asynchronous callback mode is used with ODT API so that this script is called by ODT
//    API server when the job is done.
//  * ODT uses POST request with JSON data to report the results to the callback.
//  * The results from the server are written to $whoisResultsFile. Each result is written
//    on a separate line in the format "QUERY::RESULT" (without the quotes).
//  * The script logs its progress to $logFile, so in case of any problems, check the log
//    file.
//
// How to use it?
//  * You have to have ODT account with API enabled and obtain API key and secret. See PDF 
//    ODT API specification and its section 1.
//  * Fill in your ODT API key and secret to $apiKey and $apiSecret global variables.
//  * Create a list of WHOIS queries and store them to whois-input.txt.
//    Separate queries in the input file with newlines. Each query should be either a domain 
//    or IP address.
//  * Copy whois-input.txt and this script to the same folder on a PHP hosting server.
//  * Execute the script using your browser - simply access the script's URL.
//  * Wait until the input file is empty or until log file does not grow.
//  * If whois-input.txt is empty, the work is done. Otherwise, check whois-log.txt to see
//    the problem.
//


# Settings

// Your API key & API secret
$apiKey = 'Your API key here'; // ODT-API-XXX
$apiSecret = 'Your API secret here';


// Is logging enabled
$loggingEnabled = true;
 
// Name of input file, in which queries waiting for processing are separated by newlines.
$whoisInputFile = __DIR__ . '/whois-input.txt';
 
// Name of output file with unparsed results.
$whoisResultsFile = __DIR__ . '/whois-results.txt';
 
// Name of log file.
$logFile = __DIR__ . '/whois-log.txt';
 
// URL of this script.
$thisUrl = "http" . (!empty($_SERVER['HTTPS'])?"s":"") . "://" . $_SERVER['SERVER_NAME']
         . $_SERVER['REQUEST_URI'];
 
 
ini_set('display_errors', 1);
error_reporting(E_ALL);
 


// Setup unique token
$token = uniqid();
if (isset($_GET["token"])) $token = $_GET["token"];


 
/**
 * Appends $message to file $fileName.
 *
 * @param string $fileName Name of the file to append to.
 * @param string $data Data to append.
 */
function fileAppend($fileName, $data)
{
  $fp = fopen($fileName, "a");
 
  flock($fp, LOCK_EX);
  fwrite($fp, $data);
  flock($fp, LOCK_UN);
 
  fclose($fp);
}
 
 
/**
 * Appends $message to log file with time stamp.
 *
 * @param string $message
 */
function mlog($message)
{
  global $logFile, $loggingEnabled, $token;
 
  if ($loggingEnabled)
  {
    $dateTime = new \DateTime('now', new \DateTimeZone('UTC'));
    $time = $dateTime->format('Y-m-d H:i:s');
    $line = '[' . $time . ']<' . $token . '> ' . $message . "\n";
 
    fileAppend($logFile, $line);
  }
}
 
 
/**
 * Appends $message to output file.
 *
 * @param string $message
 */
function writeOutput($message)
{
  global $whoisResultsFile;
 
  $line = $message . "\n";
  fileAppend($whoisResultsFile, $line);
}


/**
 * @param string $key ODT API key.
 * @param string $secret ODT API secret.
 * @param string $action ODT API action - see ODT API documentation.
 * @param array $postData ODT API action data - see ODT API documentation.
 * @return array [bool $success, string $data] $success is true, if the operation succeeded, 
                 in this case $data represents the returned result. If the operation failed, 
                 $success is false and $data contains error message.
 */
function odtSendRequest($key, $secret, $action, array $postData)
{
  // generate POST data string
  $postFields = http_build_query($postData);
  $dateTime = new \DateTime('now', new \DateTimeZone('UTC'));
  $time = $dateTime->format('Y-m-d H:i:s');

  $message = $key . $time . $postFields;
  $signature = hash_hmac('sha512', $message, $secret);

  // generate extra headers
  $headers = array(
    'Sign: ' . $signature,
    'Time: ' . $time,
    'Key: ' . $key
  );

  $ch = curl_init();
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_URL, 'https://secured.online-domain-tools.com/api/user/' 
                              . $action . '/');
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
  curl_setopt($ch, CURLOPT_POST, 1);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields);
  curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
  curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
  curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
  curl_setopt($ch, CURLOPT_TIMEOUT, 240);

  $data = curl_exec($ch);

  if ($data === false) 
  {
    $error = curl_error($ch);
    $result = array(false, $error);
  } else 
  {
    $result = array(true, $data);
  }
  curl_close($ch);

  return $result;
}
 

/**
 * This function is called during the inital request that starts the job.
 * It verifies that we have access rights to all files that we need.
 *
 * @return bool Returns true if succeeded, false otherwise.
 */
function initialization()
{
  $result = false;
  global $whoisInputFile, $whoisResultsFile, $logFile;

  // Clean log file.
  $res = file_put_contents($logFile, "");
  if ($res !== false)
  {
    mlog("initialization()");
 
    // Check input is readable. 
    $inputContents = file_get_contents($whoisInputFile);
    if ($inputContents !== false)
    {
    // Check input is writeable. 
      $res = file_put_contents($whoisInputFile, $inputContents);
      if ($res !== false)
      {
        // Clean output.
        $res = file_put_contents($whoisResultsFile, "");
        if ($res !== false)
        {
          $result = true;
        }
        else mlog("initialization(): ERROR: Unable to write to output file "
                . "'$whoisResultsFile'.");
      }
      else mlog("initialization(): ERROR: Unable to write to input file '$whoisInputFile'.");
    }
    else mlog("initialization(): ERROR: Unable to read input file '$whoisInputFile'.");
  }
  else mlog("initialization(): ERROR: Unable to write to log file '$logFile'.");
 
  mlog("initialization(-):" . (int)$result);
  return $result;
}
 
 
/**
 * Checks if there are any incoming data in form of JSON POST.
 * If not, does nothing and returns immediately. If yes, the data are processed.
 *
 * @return bool Returns true if succeeded, false otherwise.
 */
function processIncomingData()
{
  $result = false;
  if (!isset($_GET["query"])) 
    $result = initialization();

  mlog("processIncomingData()");
  if (isset($_GET["query"]))
  {
    $query = $_GET["query"];
    mlog("processIncomingData(): Query is '$query'.\n");
  
    // Load data from input.
    $json = file_get_contents('php://input');
    if ($json)
    {
      $jsonLen = strlen($json);
      mlog("processIncomingData(): Incoming data ($jsonLen bytes) detected "
         . "(first 256 bytes):\n" . substr($json, 0, 256) . "\n");
    
      $response = json_decode($json, true);
      if ($response !== null)
      {
        if ($response['success'] == 1)
        {
          mlog("processIncomingData(): Call succeeded.");
        }
        else 
        {
          mlog("processIncomingData(): ERROR: Call failed with error message: '"
                . $response['message'] . "'");
        }
        writeOutput($query . "::" . $json);
        $result = true;
      }
      else mlog("processIncomingData(): ERROR: Call failed. Server response is not a valid"
              . " JSON message.");
    }
    else mlog("processIncomingData(): ERROR: Call failed. No JSON POST data received.");
  }
  else mlog("processIncomingData(): No incoming data. Initialization called.");
 
  mlog("processIncomingData(-):" . (int)$result);
  return $result;
}
 
 
/**
 * Removes and returns first line from the input file.
 *
 * @return string Returns first line of the input file if succeeded, false otherwise.
 */
function removeLineFromInputFile()
{
  mlog("removeLineFromInputFile()");
 
  global $whoisInputFile;
  $result = false;
 
  $lines = file($whoisInputFile, FILE_IGNORE_NEW_LINES);
  if ($lines !== false)
  {
    if (count($lines) > 0)
    {
      $firstLine = $lines[0];
      $newLines = array_slice($lines, 1);
      $newContent = implode("\n", $newLines);
      $res = file_put_contents($whoisInputFile, $newContent);
      if ($res !== false)
      {
        $result = trim($firstLine);
        if (empty($result)) $result = false;
      } 
      else mlog("removeLineFromInputFile(): ERROR: Unable to write to file"
              . " '$whoisInputFile'.");
    } 
    else mlog("removeLineFromInputFile(): Empty input file '$whoisInputFile'.");
  } 
  else mlog("removeLineFromInputFile(): ERROR: Unable to read from file"
          . " '$whoisInputFile'.");
   
 
  mlog("removeLineFromInputFile(-):" . ($result !== false ? $result : "false"));
  return $result;
}
 
 
// Return what is expected by API.
echo "ODT: OK\n";


// If processing is OK, continue.
if (processIncomingData())
{
  $done = false;
  while (!$done) 
  {
    // Load query from input file.
    $query = removeLineFromInputFile();
    if ($query !== false)
    {
      $arg = "query=" . urlencode($query) . "&token=" . urlencode($token);
      $url = strtok($thisUrl, '?') . "?$arg";
      mlog("Callback URL set to '$url'.");


      // Send request via ODT API.
      list($isHttpRequestOk, $result) = odtSendRequest($apiKey, $apiSecret, 
      "tool/whois/query", array(
        "query" => $query,
        "asyncCallback" => $url
      ));
   
      if ($isHttpRequestOk) 
      {
        $response = json_decode($result, true);
        if ($response !== null)
        {
          if ($response['success'] == 1)
          {
            mlog("API call succeeded, waiting for callback ...");       
            $done = true;
          } else 
          {
            mlog("ERROR: API call failed with error: " . $response['message']);
            writeOutput($query . "::" . $result);
          }
        } else mlog("ERROR: API call failed. Server response is not a valid JSON message.");
      } else mlog("ERROR: HTTP request failed with error: " . $result . ".");
    }
    else 
    {
      mlog("No more queries to process, all done!");
      $done = true;
    }
  }
}
else mlog("ERROR: Processing incoming data failed, stopping execution.");

In this tutorial we will describe our IP blacklist monitoring service. You might get on its page and found out that it requires you to log in before you can set up your monitors. Maybe you are asking yourself, is this service worth registration? What should I expect? How does it look like inside? Is it complicated to use it? Hopefully, we will address all these questions in this tutorial.

Note that this tutorial is intended for basic users and those who are not registered users yet. We do not get into all the details here and some features are being skipped. If we skip something that is interesting for you, do not worry. On every page within the blacklist monitor interface, there are helper texts on the bottom that explains every control on that page. And as always, if you need any help, just contact our support and we will take care of you.

So, let’s begin.

Your Monitors

Your monitors is the very first page you will see as a logged in user. Here is how it looks (click the image for the full resolution version):

In this table you will see your blacklist monitors once you create them. At the beginning you have no monitors, so nothing interesting is to be seen here. The only useful control here is the Create New Monitor button, so let’s click it and create our very first monitor.

Create New Monitor

The form for creating monitor might seem complex, but you can use it very simply if you want to. Here is how its Parameters tab looks like:

In this picture, you can see that we have named our monitor Test Range Monitor and we have set that it is an IP range monitor. The other option for the monitor type is Single IP. What is the difference between these two types? Simply, the IP range monitors contains a range of IP addresses. In our example, we will monitor the range of 3 IP addresses 8.8.8.1 – 8.8.8.3. The Single IP type of monitor is there if you need to monitor only one IP with your monitor. The maximum range you can set here is 256 IP addresses, also known as /24 network, formerly known as C Class network.

And you know what? That’s it if you like! You do not need to fill in anything else than the values on this tab of the form. Right now, you can click the Create monitor button and your monitor will be created and it will work just fine. If this is your case, feel free to skip the rest of this section and continue with Your Monitors – Monitor Created below.

For those of you who want to tweak your monitor settings little more, let’s click the Next button, which takes us to the next tab – Rules. It looks like this:

On this tab you define monitor’s rules that will trigger an alert. You can see that one rule is defined already for you. This rule can be translated as ”Alert me if any change is detected except if the change was detected on blacklists free.v4bl.org, ip.v4bl.org, l2.apews.org, rbl.rbldns.ru.” In other words, this rule will cause that an alert is sent to you if any monitored IP address becomes listed on any blacklist except of those we ignore, or delisted from any blacklist except of those we ignore. Another type of rule you can define is Target host new listed, which is the same as Any change, except that it will send an alert only if an address becomes listed. Delisting will not trigger an alert in this case. These two options are the most common rules you should use. There is also a third option, but that is only for advanced users who are interested in receiving alerts only from a couple of blacklists.

You might wonder why it is useful to define a list of ignored blacklists? This is quite important because not all blacklists provide reliable data all the time. We have identified a couple of blacklists that are not very reliable and may cause false alerts. Ignoring them is especially useful if you are monitoring a large pool of IP addresses and you really want to be informed about real incidents. If you leave the rules on the Rules tab in their default state, you should be fine. You can also add rules and have multiple rules for your monitor, but again, this is not very useful for basic users.

So, let’s click Next and examine the Schedule tab:

The only important setting on this tab is the Period. By default, your monitor is set to monitor once per day. Each execution costs credits, so be careful when you change this. The highest monitoring frequency available is 8 hours. You can also limit the live of your monitor by changing Start time and Finish time values. By default, your monitor will start as soon as possible after you create it and will not end monitoring until you disable it, delete it, or until your account runs out of credits.

Click Next and move on to the Notifications tab:

This one is quite important, but as you can see, some values are predefined again, and as usual, if you change nothing here, you will be just fine. For some users, especially bigger companies with more users under their Online Domain Tools account, this settings might be important to change. So, what do we see here?

There are two sections – Alerts and Reports. Let’s start with alerts. Alerts are notifications that are triggered by rules as we defined them on the rules tab. What we see on our screenshot is that an alert will be sent to our user John Doe via email to his address john.doe@example.com. Note that alerts can also be sent via other methods. For example, some companies might find HTTP(S) callback notifications useful and process the alerts via their script automatically. These two fields are probably easy to understand, but how about those other two – Settings and Notify after?

The Settings value is either Do not send alerts from the initial check or Send all alerts, the former being the default value. And what that really means is that at some point every monitor has to perform the blacklist check for each of its monitored IP addresses for the first time. The Do not send alerts from the initial check value says that you are not interested to receive alerts from this very first check. Once again, this is very useful for companies that monitor many IP addresses, such as hosting companies or ISPs. It is common that almost every IP address in your range is listed on a couple of blacklists, because some blacklists simply list every IP address that does not contain a reverse DNS record. And when you manage an IP range in /24 network, it might be the case that some IP addresses are unused or are used by servers that do not need reverse DNS record defined. In this case, using Do not send alerts from the initial check will prevent to flood your mail box with two hundred of emails with alerts from the initial check. On the other hand, if you are monitoring only a single IP of your most important mail server, feel free to set this to the Send all alerts value and you will receive an alert if that IP is listed on any blacklist that is not ignored by the rules.

Notify after is the second line of defense against unreliable blacklists. Sometimes even good blacklists have their moments and report wrong results. The Notify after settings defines how many same results in a row have to be obtained from each blacklist in order to consider that value as confirmed. Until the listing status of a monitored IP is confirmed, an alert will not be sent. Let’s demonstrate this on an example. Let’s assume we have a blacklist monitor that runs daily and monitors our IP address. Let’s assume there is a blacklist X that produces the following results over a seven day period:

• 1st day: IP is listed.
• 2nd day: IP is listed.
• 3rd day: IP is NOT listed.
• 4th day: IP is listed.
• 5th day: IP is listed.
• 6th day: IP is listed.
• 7th day: IP is listed.

This scenario is real and can happen from time to time on a couple of blacklists. What happened there? Most likely we have received a wrong results on the 3rd day. If we look at the results after a couple of days, it seems evident that the IP was and is listed all the time. It was just that a system or a network problem or some other thing caused that on the 3rd day we have received a wrong result. We do not want to receive alerts because of such rare problems. And we will not receive these false alerts if our Notify after value is set to 2 or more consecutive confirmations. Because the monitor saw the OK (NOT listed) status only once, it did not become confirmed and thus did not trigger an alert. We highly recommend you to use 2 or 3 consecutive confirmations here. 2 should be just fine and will not probably reduce the false alert rate to almost zero. More than 3 would delay the alert for too long, so unless your monitor’s period is set to a higher than daily frequency, do not set this to more than 3.

Now the Report section. Reports are summaries of monitor’s operation sent to an email. You do not have to receive Reports if you do not like to. By default a Weekly reporting is being set to the account’s primary contact. We do recommend Weekly over Monthly frequency here because Monthly reports might be just too long to read, especially with IP range monitors. Some users may find Daily reports useful, so they are also available.

Monitor is Running

After we hit that Create monitor button we are back on the main page with the list of our monitors.

What we see here is our monitor and it is in the Pending state. This means that the monitor has not executed any check for any of its IP addresses yet. If we wait just a couple of minutes, this situation here will change:

We see that the status of our monitor changed to no alert. This means that the last check the monitor made did not trigger any rule, no alert was sent. OK, so let’s examine the monitor now. As you can see there are some icons in the Actions column on the right of the table. The icon on the left will navigate you to the monitor’s report page. The other icons are for disabling the monitor, editing the monitor, and deleting the monitor. We now click the first icon to see the report page.

Monitor’s Report Page

As you can see, there is a brief summary of the monitor on the top of the page and a couple of buttons that allows you to control your monitor in a same way that the icons do on the main page that we have just described. Below that we can see a simple filtering form that most of the time you will not need to use because the default view, just below the form, should contain all the data you will need for your work. So, let’s focus on the report data in that report table. At this moment, we see only two events. The first one is the information on when the monitor was created. The second one is more interesting. It is the first blacklist check for the first address. We see that the monitored IP address 8.8.8.1 was checked against 151 blacklists and was found listed on 1 blacklist. When an IP is listed on at least one list, there is a black arrow on the left of the time stamp. Click it to see the details:

The row expanded and information about blacklists that listed the monitored address was displayed. We also see the details provided by the blacklist itself. We recognize that the IP is listed on rbl.rbldns.ru blacklist, which is on the ignore list in our monitor’s rule. This is why this result could not trigger an alert. We would not receive an alert anyway, because we set that 2 consecutive confirmations are needed in order to confirm the state, so this state is not even confirmed yet. But we would not receive an alert even if that blacklist was not ignored and even if we required only 1 consecutive confirmation. Why? Because we set the Do not send alerts from the initial check setting and this would be the initial check.

Let’s now wait a couple of minutes and refresh the report page:

What happened here is that checks for the other two IP addresses that we are monitoring were checked. This is how IP range monitors work. The IP addresses are processed in a serial fashion and there is a small delay between the checks.

Alerting Monitor

We now want to see how the blacklist monitoring behaves when a monitor is alerting. In order to see that we have created a single IP monitor on one of the TOR exit nodes (a server that provides specific services to anonymous network called TOR). TOR exit nodes are listed, usually on more than one blacklists. Our monitor is called Tor Exit Node IP:

This is the main table after the initial check of our second monitor. We have set our monitor to require just 1 confirmation and to send all alerts, so even the initial check triggered an alert. Let’s examine the monitor’s report page.

Here we can see the event table with expanded row of the initial check. 151 blacklists were checked and 6 of them is listing the monitored IP. For each listing we see info / removal link that will get us to the page where we can either request delisting of our IP address or read more about the blacklist and its policies. This is the fastest way to obtain more information and get delisted.

Finally, we have received an email alert. In our email client, it looks like this:

We are informed which IP address of which monitor triggered an alert and we can clearly see the latest status of the IP address. Again, each listing is clickable and navigates us to the information or removal page of the particular blacklist. The blacklists written in red signal new listings, blacklists in bold signal confirmed listings (so the bold red that we see on our screenshot is obviously for confirmed new listings). Striked out blacklists would be those that were listed previously, but are not anymore. Blacklists not written in bold suggest unconfirmed listings.

That’s it! Thank you for reading this tutorial. We hope you will like our blacklist monitoring service and that it will help you doing your business.