Tuesday, 30 April 2013

Bible Thumpers: Y U No Check Your Pamphlets?

Got one of these in the letter box on the weekend:

Can you see it? Oh. It's a bit small. Here's a close up:

Wait. What? Oh. So that's why I could never see the kingdom. It's over there, beyond the sparkley fountains of milk and honey in... in... HAD.

Seriously? It's the primary quote on the pamphlet for Cthulhus name.
The thing that grabs peoples attention.
And you got it wrong?

I did try google maps by the way. Gave up after a bit because it was boring. And stupid.
Mainly because of the semi-useless word "HAD."
The results where quite humorous and gave results focused on Israeli restaurants in:

Ash Sharqiyah, Oman
A possible I suppose.

Al Anbar, Iraq
Hmm. Seems to stretch the idea a bit don't you think?

And surprisingly:

Casper, WY, United States

Yeah. Real place. Loads of Israeli restaurants there according to Google.
Although I would hazard a guess and say that this kingdom is not likely to be at the Ramada Plaza Riverside Hotel and Convention Center unless they prefer the decor.
And the big question is, why Israeli?
I'm babbling.

Monday, 22 April 2013

Displaying a summary of Apache2 access logs by country using Ruby

Recently I had a case where a site was being hammered by script kiddies.
Nothing compromised I might add.
First off I limited access to the site by country (Australia in this case).
Note I wasn't blocking countries, but blocking all countries except Australia.

Then I wanted to get a summary of the Apache2 access log to see where these annoying little herberts where from.
An Apache2 access log entry looks something like this: - - [22/Apr/2013:12:19:46 +1000] "GET /index.php?title=User:Coombayah HTTP/1.1" 403 409 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

Now where was that access from?
So I knocked up a *very* simple ruby script to get that data.
The code below is 100% inelegant.
It's not meant to be elegant.
It's meant to illustrate the point.
A 'real' script would have much better coding and options for dates and ranges.

Output for today would look something like this:

2013-04-22 - 128      2    0    2 2000667011 Japan      2    0    2 2000667012 Japan      2    0    2 2071807856 China       2    0    2 2071807818 China        4    0    4 2954824154 France      4    0    4 3226516386 United States     4    0    4 3633904291 United States     4    0    4 3633904315 United States       2    0    2 3659425567 China     2    0    2 3702877339 China       7    0    7  787889329 Sweden       25    0   25  773603541 Ukraine      10    0   10 1123633799 United States
  IPs with 1 access and failure: ["", "", "", 
  "", "", "", "", "", 
  "", "", "", "", "", 
  "", "", "", "", "", 
  "", "", "", "", 
  "", "", "", "", 
  "", "", "", "", 
  "", "", "", "", ""]

As you can see, pretty basic.
The principal rows show the IP address, accesses, successes, failures, IP-Value and country.

First off you need a Ip-To-Country list.
In a prior post I mentioned getting zone range files.
So go to http://software77.net/geo-ip/ and get the full list.
The file will have a lot of useful comments at the top.
Make a copy of the file excluding the comments as IpToCountry.csv.
You should see a file roughly 126,000 lines long where each line looks something like this:


The fields are 'Address From', 'Address To', 'Registrar', 'Date Assigned', 'Country2', 'Country3', 'Country'.

And now the ruby code.

First the requires:

require 'date'
require 'csv'

Since we are using *BRUTE FORCE* and not bothering to be elegant, we simply read the csv file into an array of hashes.

printf "Loading Ip to Country map\n"
ip_to_country = []
CSV.foreach('IpToCountry.csv') do |row|
  ip_to_country << { :from => row[0].to_i, :to => row[1].to_i, :country => row[6] }

The first pass now reads the access log into a hash of hashes:

unique_days = {}
f = File.new('logs/access.log', 'r')
while (line = f.gets)
  ip,tmp,tmp,dt,offset,verb,url,http,rcode,sz,tmp,browser = line.split
  d = DateTime.strptime( "#{dt} #{offset}", "[%d/%b/%Y:%H:%M:%S %Z]")
  date = d.strftime("%Y-%m-%d")
  if ! unique_days.has_key? date
    unique_days[date] = { :ip => {}, :total => 0 }
    if ! unique_days[date][:ip].has_key? ip
      unique_days[date][:ip][ip] = { :total => 0, :succeeded => 0, :failed => 0 }
    unique_days[date][:ip][ip][:total] += 1
    if rcode == '200'
      unique_days[date][:ip][ip][:succeeded] += 1
      unique_days[date][:ip][ip][:failed] += 1
    unique_days[date][:total] += 1

I have eschewed elegance here for brute force and clarity.

So. Now we have everything we need.
Now we traverse the data dumping the report out:

unique_days.each do |date,h|
  printf "#{date} - #{h[:total]}\n"
  only_one_failed = []
  h[:ip].sort.map do |k,data|
    octets = k.split('.')
    if data[:total] == 1 && data[:failed] == 1
      only_one_failed << k
    ip_value = (octets[0].to_i * 256 * 256 * 256) + (octets[1].to_i * 256 * 256) + (octets[2].to_i * 256) + (octets[3].to_i)
    country = 'Unknown'
    ip_to_country.each do |row|
      if ip_value >= row[:from] && ip_value <= row[:to]
        country = row[:country]
    printf "  %-16s %4d %4d %4d %10d %s\n", k, data[:total], data[:succeeded], data[:failed], ip_value, country
  printf "  IPs with 1 access and failure: #{only_one_failed}\n"

And you have your report.

Configuring an Apache2 instance to only allow access from a specific set of countries

Ok. This is pretty simple, but I'm documenting it here for myself.
The issue arose for me when one of our test servers started getting hammered by script kiddies from China and so on.
They didn't get in, but that wasn't the point.
The point was that the web site was for Australian users only.
Now to short circuit any comments I can say:
1) The solution had to work for several frameworks (Elgg, Joomla, etc)
2) I wasn't redirecting traffic to a special page
3) I wanted to black all access except Australian IPv4 ranges
I could have gone to MaxMind or its ilk, but that would make '1' problematic.

The first step was to locate a *correct* and *regularly updated* list of CIDR and range formatted IPv4 list.
I tried several including http://www.ipdeny.com/ipblocks/ with no luck.
Most have gaps in their lists because they use 'official' sources.
And ISP ranges are not always covered in those lists.

Eventually I found http://software77.net/geo-ip/
I downloaded the zone and CIDR lists for the countries I was interested in.
Then I modified the /etc/apache2/sites-available/my-site.something.com file to look like this:
Abbreviated for this post. :-)

<VirtualHost *:80>
  ServerAdmin     my.email@my.email.service.com
  ServerName      my-site.something.com
  ServerSignature Off
  DocumentRoot    /home/my-user/my-site/site
  LogLevel        warn
  ErrorLog        /home/my-user/my-site/logs/error.log
  CustomLog       /home/my-user/my-site/logs/access.log combined
  ScriptAlias     /cgi-bin/ /usr/lib/cgi-bin/
  <Directory /home/my-user/my-site/site/>
    Options       Indexes FollowSymLinks MultiViews
    AllowOverride All
    Order         Deny,Allow
    # See .htaccess for country limiting

Then in the /home/my-user/my-site/site/.htaccess file I added the following to the bottom:

  Order Deny,Allow
  Deny from all
  # Report generated on Mon Apr 22 01:02:36 2013
  # by http://software77.net/geo-ip/
  # Report Type  : CIDR format
  # Country      : Australia
  # ISO 3166 CC  : ALPHA-2 AU; ALPHA-3 AUS
  # Registry     : APNIC
  # Records found: 6,160 BEFORE flattening (As they appear in the database)
  # Records      : 3,999 AFTER flattening (Adjoining CIDR blocks concatenated into single blocks where possible)
  Allow from
  Allow from
  ...elided for brevity...  
  Allow from
  Allow from

The block from under the 'Deny from all' to the end is the CIDR formatted file for a country.
You have to add the 'Allow from ' in front of the address, but if you use vi that's trivial.
If you want to allow another country access, then simply add the CIDR formatted file below the first.

Then I ran '/etc/init.d/apache2 reload' and all is well.

IMPORTANT NOTE: Things change. You have to check for additions and deletions on the CIDR files on a regular basis - say, once a month to ensure that you .htaccess file is up to date.

Some readers may have noticed that I suggest downloading the zone files as well.
This is to allow you to read your apache logs and see which country that access came from.
I will be posting another entry soon that shows how to do that in Ruby.

Sunday, 21 April 2013

I've decided to stop watching debates about the existence or nature of god for a while

We foster cats and catlings.
"Wait. What?" I hear you say.
Just give me a few moments to explain.
Over the last year Ben and I have been fostering sick cats, mothers with kittens and the like.
Loads of them.
And, like 2 year olds, you have to watch them like a hawk.
They are always being mischevious, investigative and learning about the world they are in.
Since they are often sick or just kittens, we have to keep them inside for their own safety.
So they learn about the world that is the our house.

You can't concentrate on much, read a book and so on.
Some things you can do includes watching TV.
And I've been watching debates about the existence and nature of god while keeping a watchful eye on the fuzz balls.
So while you listen to Dawkins, Krause, Lennox et al, you observe the cats, their habits, their kittie-language and so on.

One thing that leaps out is the solid block that is the limit to their intelligence.
They can jump on chairs without knowing that it is a chair.
They eat food from a mechanical food chain beyond their grasp.
They cathramorphise us as their parent cats.
They can't read.
They may watch TV sometimes but have no capability of understanding any of the massive technological edifice behind those flickering images.
They are limited.
I suspect that even with evolution in play they will never be able to:
Open a can of cat food;
Read a book;
Build a car or fly a jet fighter within any reasonable time span.

So when I see a bunch of atheists, theists and deists debating I get an image in my head.
And that image is a bunch of house cats, sitting on a carpet discussing the nature of Dog.

I'm not saying that the debates are pointless and shouldn't be done.
They should continue, but on the understanding that a hundred years from now they will be debating the same thing.

So I'm laying off the debates for a while.

For cat-lovers here are some photos of the latest bunch: