Thursday, 8 January 2015

Docker: Creating ruby 2.2.0, mysql and rails images

For the purposes of this post, I'm assuming you have a private repository with the address 10.0.0.1:5000.

In this example, you will notice a distinct similarity in the build scripts.
Hmm.
Funny that.
I created a tool sleet which is like fleet but for single server installs.
So that is the essence of those build scripts.
If I have time somewhere in my aging schedule and downright ludicrous deadlines, I'll open source it.

This creates 4 images in your private repo.

- debian which is an instance of jessie with various compilers etc installed.
- ruby-2.2.0 compiled from source
- mysql-ruby-2.2.0 with the build dependencies
- rails-2.2.0 from all of the above

The purpose was to get an image that can be used for `rails-api` micro services that builds and deploys ultra-fast.
This is done by pre-installing the most commonly used gems into the `rails-2.2.0` image.

Then when a build is done that inherits from the `rails-2.2.0` image, the `bundle install` simply uses the local gems and doesn't have to go off to any external gem sources.
The bundle output shows `Using` instead of `Installing` the latter of which involves snarfing gems off the web and purhaps compiling things.


First up, here's the tree you'll be creating:

.
├── debian
│   └── jessie
│       ├── Dockerfile
│       └── build.sh
├── mysql
│   └── 2.2.0
│       ├── Dockerfile
│       └── build.sh
├── rails
│   └── 2.2.0
│       ├── Dockerfile
│       ├── build
│       │   └── Gemfile
│       └── build.sh
└── ruby
    └── 2.2.0
        ├── Dockerfile
        └── build.sh

Here are the files:

debian/jessie/Dockerfile

FROM debian:jessie

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    git curl procps \
    autoconf build-essential \
    libbz2-dev libcurl4-openssl-dev libffi-dev libssl-dev libreadline-dev libyaml-dev \
    zlib1g-dev --no-install-recommends \
  && rm -rf /var/lib/apt/lists/*

debian/jessie/build.sh

#!/bin/sh

IMAGE=debian

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo "BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""

ruby/2.2.0/Dockerfile

FROM 10.0.0.1:5000/images/debian

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

ENV RUBY_MAJOR 2.2
ENV RUBY_VERSION 2.2.0

RUN apt-get update \
  && apt-get install -y bison ruby \
  && rm -rf /var/lib/apt/lists/* \
  && mkdir -p /usr/src/ruby \
  && curl -SL "http://cache.ruby-lang.org/pub/ruby/$RUBY_MAJOR/ruby-$RUBY_VERSION.tar.bz2" | tar -xjC /usr/src/ruby --strip-components=1 \
  && cd /usr/src/ruby \
  && autoconf \
  && ./configure --disable-install-doc \
  && make -j"$(nproc)" \
  && apt-get purge -y --auto-remove bison ruby \
  && make install \
  && rm -r /usr/src/ruby

# skip installing gem documentation
RUN echo 'gem: --no-rdoc --no-ri' >> "$HOME/.gemrc"

# install things globally, for great justice
ENV GEM_HOME /usr/local/bundle
ENV PATH $GEM_HOME/bin:$PATH
RUN gem install bundler \
  && bundle config --global path "$GEM_HOME" \
  && bundle config --global bin "$GEM_HOME/bin"

# don't create ".bundle" in all our apps
ENV BUNDLE_APP_CONFIG $GEM_HOME

CMD [ "irb" ]

ruby/2.2.0/build.sh

#!/bin/bash

IMAGE=ruby-2.2.0

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo " BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""

mysql/2.2.0/Dockerfile

FROM 10.0.0.1:5000/images/ruby-2.2.0

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

ENV RUBY_MAJOR 2.2
ENV RUBY_VERSION 2.2.0

RUN buildDeps='libmysqlclient-dev'; \
      set -x \
      && apt-get update && apt-get install -y $buildDeps --no-install-recommends

CMD [ "irb" ]

mysql/2.2.0/build.sh

#!/bin/bash

IMAGE=ruby-2.2.0

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo " BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""

rails/2.2.0/Dockerfile

FROM 10.0.0.1:5000/images/debian-mysql-ruby-2.2.0

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

ENV HOME /build
ENV RUBY_MAJOR 2.2
ENV RUBY_VERSION 2.2.0
ENV GEM_HOME /usr/local/bundle
ENV PATH $GEM_HOME/bin:$PATH

ADD ./build /build

WORKDIR /build

RUN bundle install --jobs 8

CMD [ "irb" ]
rails/2.2.0/build/Gemfile

source 'http://your.private.gem.server:9900'

# ---------------------------------------------------------------------------
# Always these
# ---------------------------------------------------------------------------
gem 'rails', '4.2.0'
gem 'mysql2'

gem 'thin'
gem 'foreman'

gem 'jsonapi-resources' # and/or roar

gem 'db_populate', git: 'https://github.com/ffmike/db-populate.git'

gem 'rack-cors'
gem 'activeresource'

gem 'jbuilder', '~> 2.0'
gem 'typhoeus'
gem 'etcd'
gem 'hutch'
gem 'elasticsearch'
gem 'searchkick'

gem 'therubyracer'
gem 'oj'

# ---------------------------------------------------------------------------
# Unused
# ---------------------------------------------------------------------------
# gem 'jquery-rails'
# gem 'turbolinks'
# gem 'bcrypt', '~> 3.1.7' # Use ActiveModel has_secure_password
# gem 'sass-rails', '~> 4.0.3'
# gem 'uglifier', '>= 1.3.0'
# gem 'coffee-rails', '~> 4.0.0'
rails/2.2.0/build.sh
#!/bin/bash

IMAGE=rails-2.2.0

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo " BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""
Enjoy. YMMV.

Docker: Run/Administer an instance of elasticsearch on your boot2docker vm

First ssh into your boot2docker vm, then run these commands:

docker pull dockerfile/elasticsearch
mkdir elastic_search
cat > elastic_search/elasticsearch.yml <<-EOF
path:
  logs: /data/log
  data: /data/data

http.cors.enabled: true
EOF
docker run -d -p 9200:9200 -p 9300:9300 \
  -v ${HOME}/elastic_search:/data \
  --name ElasticSearch \
  dockerfile/elasticsearch \
  /elasticsearch/bin/elasticsearch \
  -Des.config=/data/elasticsearch.yml
docker@boot2docker:~$ docker ps -a
CONTAINER ID  IMAGE COMMAND  CREATED  STATUS  PORTS  NAMES
[whatever]    dockerfile/elasticsearch:latest "/elasticsearch/bin/ 24 seconds ago Up 23 seconds 0.0.0.0:9200->9200/tcp,0.0.0.0:9300->9300/tcp ElasticSearch

Now from your Mac browse to http://192.168.59.103:9200 and you should get a status result.

Now run elasticsearch-head on your Mac (not the vm):

cd /some/tools/folder/of/your/choice
git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head
npm install
grunt server

Now browse to http://localhost:9100 and:
1) set the connect box to http://192.168.59.103:9200 and
2) click connect

You now have ES running on your boot2docker instance and can access it from your Mac.

Docker: Private Registry push yields "Error: Invalid registry endpoint" and "insecure-registry"

For some time I've been using boot2docker on my mac.
This was version 1.3.1.
I also have a private registry which for the purposes of this post I've called 10.0.0.1:5000.

Some time ago I accidently brew updated and got docker 1.4.1.
So I couldn't access the running boot2docker daemon.
For some reason that seemed quite reasonable at the time, I didn't restart the boot2docker vm.
Go figure.

So I had to do this kind of thing:

(
export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
export DOCKER_TLS_VERIFY=1
/usr/local/Cellar/docker/1.3.2/bin/docker push 10.0.0.1:5000/some_image
)

Then today I accidently started Kitematic.
Which upgraded the boot2docker vm and restarted it.
And blammo I could no longer push to the private repository.

FATAL[0002] Error: Invalid registry endpoint https://10.0.0.1:5000/v1/: \
  Get https://10.0.0.1:5000/v1/_ping: EOF. \
  If this private registry supports only HTTP or HTTPS with an unknown CA certificate, \
  please add `--insecure-registry 10.0.0.1:5000` to the daemon's arguments. \
  In the case of HTTPS, if you have access to the registry's CA certificate, \
  no need for the flag; simply place the CA certificate at \
  /etc/docker/certs.d/10.0.0.1:5000/ca.crt

Bugger.
Dope slap.
Noob.

So the fix is:

1) ssh into your boot2docker vm and
2) sudo vi /var/lib/boot2docker/profile and
3) Add 'EXTRA_ARGS="--insecure-registry 10.0.0.1:5000"' to it, :wq and
4) sudo /etc/init.d/docker restart
5) And just use the /usr/local/bin/docker and not the Cellar version.

YMMV.

Docker: Delete "dangling" images

After using docker for a while one notices a distinct slow down in performance of your host.

Some digging revealed that "dangling" images are the culprit.

While being a little simplistic, these are images that are no longer associated with any running container.

I dare you to go onto your docker host (even boot2docker) and do this:

docker images -q --filter "dangling=true"

My guess is that you'll see hundreds (or thousands if you've been busy) of those.

Here's the trick to getting rid of those large incontinent beasts that are leaving great piles of steaming images in your host:

docker images -q --filter "dangling=true" | xargs docker rmi

You may see a few false positives, but "hey!" at least the trash has been taken out.

Might be worth cronning it...

YMMV.

Get next free port on Linux with Ruby

Get next free port:

require 'socket'
(1..10).each do |n|
socket = Socket.new(:INET, :STREAM, 0)
socket.bind(Addrinfo.tcp("127.0.0.1", 0))
port = socket.local_address.ip_port
puts "port -> #{port}"
socket.close
end

command line:

port=$(ruby -e "require 'socket'; \
s = Socket.new(:INET, :STREAM, 0); \
s.bind(Addrinfo.tcp('127.0.0.1', 0)); \
port = s.local_address.ip_port; \
s.close; \
p port")

Sunday, 28 December 2014

A very Diggy Diggy Minecraft and Gaming Christmas!

Well.
Not just MineCraft.
Here's some photos:

Our decorations include Minecraft items!


And Daleks and K9 and some paper figures of us:

Ben got a metal Millenium Falcon to make. Warning: You will need snippers, a tweezer and the eyesight of an eagle.


He got me a Minecraft GiftPack of books.
Aimed at kids, but really cool.
I particularly liked that on page 2 of every book is a page about staying safe online.
Sigh. Vanilla Minecraft. Awesome.


Also some Minecraft paper figurines.
(The video is Wil Wheaton and others playing Ticket to Ride)


I got him a baby StrandBeest!
It really works and walks and creeps you out with its organic look when it's moving.


And would you believe it, we bought each other a copy of the KJV Manga Bible!
See Moses using his Ninja Skills!
Curiously the one on the left covers the whole old testament in great detail while the one on the right skips most of it to concentrate on the new testament.
Both leave out a shed load of books from both testaments, but they are both very cool.


Oh. And here's Ticket to Ride!
We played it straight away and love it.


Monday, 8 December 2014

"Minecraft Colada" Parody of Escape (The PiƱa Colada Song)"

I was gaming with my lady, been on our console too long.
Replaying all those old missions, felt just like donkey kong
So while she installed some patches, I browsed a forum instead
And in a thread about questing, there was this post that I read...

[chorus]
If you like mining for diamonds and hunting sheep in the rain
If you'll forge your own armour and you'll eat zombie brains
If you kill endermen at midnight... and you wear a cool cape
I've the game that you've looked for... play minecraft and escape

I didn't show it my lady, I know that sounds kind of mean...
But she seemed happy with the console, I didn't want to break up her routine
And so i logged into the forum, and hit reply on the post.
I wanted to try this new game, was even willing to host.

[chorus]
Yes I like mining or diamonds, and hunting sheep in the rain
Zombies, skelons, and spiders... all of these will be slain
I've got to log in by tomorrow night, for a gaming escape
We'll use a mod pack from yogscast, while it's still up to date.

And so I spawned on the server, and started punching a tree
I saw my new gaming partner... as she killed squids in the sea
It was my own gaming lady and she said, "Aw, it's you."
Then as she put down a workbench, she said "I never knew."

[chorus]
That you like mining for diamonds and hunting sheep in the rain
And the crafting of armour, and the farming of grains.
if you kill endermen at midnight... and you wear a cool cape
Your the player I've looked for... play with me and escape

[chorus]
If you like mining for diamonds and hunting sheep in the rain
If you'll forge your own armour and you'll eat zombie brains
if you kill endermen at midnight... and you wear a cool cape
I've the game that you've looked for... play minecraft and escape

[chorus]
Yes I like mining or diamonds, and hunting sheep in the rain
Zombies, skelons, and spiders... all of these will be slain
I've got to log in by tomorrow night, for a gaming escape

...

Tuesday, 21 October 2014

Minecraft Yogscast Complete Pack - How to make BRONZE!

Annoying as buggery.
Want to play with BEES!
But you need bronze.
And wouldn't you know it if you pop 3 copper ingots around a tin ingot in a crafting bench as described in the wiki it does nothing.
Nada.
Nil.
Zip.
I thought it might be that the ingots were Mekanism, but I tried others.

So here's the trick.

1) Get a stack of ingots and go to your crusher.
2) Crush them to dust.
3) Go to your metallurgic infuser.
4) Pop the dust in the left hand "power" bar (purple box).
(If you have redstone or coal in it, click the 'dump' button')
5) Pop your stack of copper ingots in the input (red box).
6) Out come bronze ingots.


Thursday, 16 October 2014

Monday, 6 October 2014

Just published my first book!

Well well well.

Many people have asked me to curate all the humorous posts I've done over the years and publish them.

So I did.

244 pages of them.

From 2005 to today. Ok I lost a few due to hardware failure, but I made up for it by adding some humorous photos and of course: Kittens!

To whet your appetite here is a sample of the contents:


  • Blackpool. Gack...
  • Superman and other heroes...
  • Marcel, Speed Cameras and Rat Throwing as a Sport
  • Greek Gods, Washed Hedgehogs and Bluetooth
  • Dead Flies and Calculations
  • 12 steps for vampires
  • Obesity is the key to the survival of man!
  • When Sticking a Screw Driver in your Face is Preferable to Going to the Dentist and More Movie Reviews
  • Cthulhu Eggs and What's really funny
  • I am not “The Internet with Boobs”
  • Monastic Speeders, Pants around Ankles and Bikes
  • Rawhead Rex, Maniac Nurses and a Steam Powered Cat Cannon
  • Is everything shit or is it just me? Oh and H.P.Lovecraft
  • Rats, an old house and worlds craziest chases
  • Deep Fried Squirrel with Garlic Mayonnaise!
  • Mansquito (TV 2005) O.M.G. Bad Sci-Fi Reaches new Lows
  • There's a dead cockroach next to my ashtray
  • Portal for Programmers - Still Have a Job
  • Mambo Number 5 for Programmers - Rebuild Number 5
  • This is an ex-iPhone...

So go to https://leanpub.com/randomactsofsentience and support your local nutter! That's me if you haven't figured it out.

I'll be starting my next book shortly.