Migrating from Perforce to Bazaar

I've a big fan of Perforce for ages.

Recently Perforce changed the licensing from 20 free users to 5. This broke my backups.  Time to switch to something FOSS.
I decided to give Bazaar a crack, on paper it ticks all the boxes.

Below is how I exported from Perforce (with history) in to a new Bazaar repo and setup triggers/hooks to run builds after code is submitted.

Export repos from Perforce

Due to the nature of my repo layout, I have I need to export each of my projects individually.  This gives me the chance to migrate one project at a time.

Took a while to decipher the docs for P4 exporting to git fast-import format, its a bit broken, but command below enabled and export and import including change history.

sudo aptitude install bzr
sudo aptitude install perforce-p4python python-fastimport
mkdir -p ~/.bazaar/plugins
cd ~/.bazaar/plugins
bzr branch lp:bzrp4
bzr branch lp:bzr-fastimport
# rename the repo
mv bzr-fastimport fastimport

Then you need to set your normal P4_xx environment and p4 login

N.B. the documented bzr fast-export-from-p4 does not seem to work.

With taht setup the following got me a local bzr repo (cairoruler is a C project I decided to test with)

~/.bazaar/plugins/bzrp4/git_p4.py export //depot/cairoruler@all > cairoruler.fi
bzr fast-import cairoruler.fi cairoruler.bzr

This generates a directory ./cairoruler.bzr/p4/master.remote .  Careful the ./cairoruler.bzr looks like a repo (it has .bzr) but its ./cairoruler.bzr/p4/master.remote that has the newly imported workspace.

The p4 export code is based on the git export tool, hence the odd name git_p4.py.

So far this is only working on my local HD.

Setting up the bzr server

I created a full OS in a Linux container, any Linux VM or baremetal server will do.
LXC gives me a playground like a VM but faster to build.

lxc-create -t ubuntu -n bzf -f lxc-config.conf

Then install in the container bzr with just

apt-get install bzr

In theory thats all you need to do to use bzr over SSH.

There are other server mechanisms but, be warned, bzr default protocols are not secure.
bzr serve --directory=/srv/bzr/repo
should not be exposed on the Internet.  You need to use SSH or do your own SSH tunneling.

Since I'm running a LXC container with ssh used for other stuff I have to NAT a port (9922) on my LXC Host to port 22 on the LXC Guest.

iptables -t nat -A PREROUTING -p tcp -i eth0 -d $PUBLIC_IP_ADDRESS --dport 9922 -j DNAT --to $BZR_IP_ADDRESS:22
iptables -A FORWARD -p tcp -d $BZR_IP_ADDRESS --dport 9922 -j ACCEPT

Thus BZR ULRs look like this.

bzr co bzr+ssh://me@myhost:9922/home/bzr/myrepo myrepo

Uploading to the server

Bazaar uses normal Linux user auth so first thing you'll want to do is create users and groups for uploads and share SSH keys.
This does not give you any fine grained read/write control to each project, tan pis.

There is no special command for setting a default remote location in bzr just push

bzr push bzr+ssh://me@myhost:9922/home/bzr/myrepo/cairoruler

Porting Triggers to Hooks

Perforce triggers are one-liners that run a command when you submit code.

Bzr has no triggers but it has Hooks.  Its all pretty hacky I'm afraid.

You have to write hooks as Python libraries and install them in you bzr deployment.  N.B. they are not part of backup/restore.

Most hooks run on the client.
In order to write a hook that runs on the server to build code when its submitted you have to write a single global post_change_branch_tip(). This is the same hook for all branches and submits, so the first thing the code has to do is work out which vbranch has changed.
When the hook is called it is passed the variable ChangeBranchTipParams, which is very poorly documented.
To debug you can't user python's print function, if you do you mess up the bzr protocol, so you need to write debug to a file.

After a couple of hours of hacking I got this

/usr/lib/python2.7/dist-packages/bzrlib/plugins/linci_hook.py


"""A linci hook.
"""

from bzrlib import branch
import subprocess
import datetime

#
# params == bzrlib.branch.ChangeBranchTipParams
#   params.branch    The branch being changed. An object of type bzrlib.branch.BzrBranch7
#   params.branch.base is a "URL" filtered-139959297573392:///var/bzr/root/cairoruler/ from which we can extract the url
#   params.old_revno    Revision number before the change.
#   params.new_revno    Revision number after the change.
#   params.old_revid    Tip revision id before the change.
#   params.new_revid    revision id after the change
#
def post_change_branch_tip(params):
    #
    # post_change_branch_tip is called under various circumstances, only fire if the rev no has changed.
    #
    if params.old_revno < params.new_revno:
        file = open("/var/log/bzr-comitlog", "a")
        file.write("%s %s\n" % (datetime.datetime.now(), params.branch.base))
        path = params.branch.base.split('/')
        branch_name = path[len(path) - 1]
        if not branch_name:
            branch_name = path[len(path) - 2]
        if not branch_name:
            return
        if branch_name in ['cairoruler']:
            file.write(datetime.datetime.now() + " executing: ci.sh " + branch_name + "\n")
            file.close()
            subprocess.call('/home/bzr/bin/ci.sh ' + branch_name, shell=True)

#
# register hook, 3rd parameter is what is shown by `bzr hooks`
#
branch.Branch.hooks.install_named_hook('post_change_branch_tip', post_change_branch_tip, 'linci post change hook')

Install this with

python /usr/lib/python2.7/dist-packages/bzrlib/plugins/linci_hook.py

Remember the script runs with the Linux permissions of the user that did the submit.

So now

bzr push

Sends latest changes and my linci server builds then and publishes a .deb as it did with Perforce.

Now I need a Sunday without a hangover to port the rest of my projects.  It should be a scriptable process.


References:
https://launchpad.net/bzr-fastimport

http://doc.bazaar.canonical.com/bzr-0.12/server.htm
https://launchpad.net/bzrp4
http://people.canonical.com/~mwh/bzrlibapi/bzrlib.branch.html

Plus some help from folks on #bzr channel on irc.freenode.net

How to setup a debrepo

My aim here is to setup a directory full of .deb files that can be installed with

apt-get install whatever

Not a full Debian repo, nor a "distribution", just a few of my own packages. Given how simple it is to create .deb packages its strange setting up a repo was so complicated. Reading Debian repo management docs complicated matters, I don't recommend it if you are setting up a simple repo.

N.B. google "Launchpad PPA" if you don't need to host the debs yourself and your code is open source. Save yourself some hassle.

I presume you do have a webserver setup to host the debs.

These are tools I needed to install on the server.

aptitude install dpkg-dev dpkg-sig gnupg apt-utils

I needed to generate a gpg key pair  gpg --gen-key is the command but its interactive so useless in a shell script.  It has a very weird syntax but here it is.

echo '
Key-Type: RSA
Key-Length: 1024
Key-Usage: sign
Name-Real: dpkg1
Expire-Date: 0
%commit
%echo done
' | gpg --gen-key --batch


dpgk1 is the name I've chosen for my package signing key, it can be any string e.g. your name.

It takes ages to run, on a quiet server/container with no disk activity it may never finish since its waiting for entropy from /dev/random.  Worst case run that command on a laptop/physical box, and export and move the keys back to the server.

Show the keys

gpg --list-keys

If you need to move the keys.

gpg -a --export-secret-key dpkg1 > secret.gpg
gpg -a --export dpkg1            > public.gpg
scp secret.gpg public.gpg root@wherever:
ssh root@wherever '
gpg --import -v -v ./secret.gpg
gpg --import -v -v ./public.gpg
'

The security conscious should delete secret.gpg now.


With the keys imported you need to copy debs to a folder that is being published by nginx/apache or similar.  And then create the repository indexes.
The index files I needed were all of Packages Packages.gz Release and Release.gpg, supposedly Packages is not needed but I could not get downloads to work without it.  That may be a bug in the tool versions I'm using.

cd /var/www/mydebrepo
gpg -a --export dpkg1 > public.gpg
rm -f Release Release.gpg Packages Packages.gz
dpkg-scanpackages . /dev/null > Packages
gzip -k Packages
apt-ftparchive release . > Release
gpg --yes -abs -u dpkg1 -o Release.gpg Release

Then sign all the debs.

dpkg-sig -k dpkg1 -s builder *.deb

That should be it for the serverside.

Setting up the client is a multi-step process

import the pgp public key
edit sources.list
apt-get update

I added a README.md file to the repo so I don't forget the process.

The syntax of sources.list is confusing, I don't understands why I have to put ./ and Ubuntu put wily release, it works but it looks different to all other lines in the file.



echo "

Add the following to your `/etc/apt/sources.list` to use this repository.

  deb http://download.tp23.org/download/deb/ ./

And run this to import the key

  wget -q http://download.tp23.org/download/deb/public.gpg -O - | sudo apt-key add -

If you still have problem try rebuilding the apt lists cache

  sudo rm -fr /var/lib/apt/lists/*
  sudo apt-get update

" > README.md

Seems deleting the indexes and rebuilding them is needed every time a file is added or removed. 

Doing that in such a way as not to interrupt clients use of the repo is left as an exercise for the reader. 

Installing Linux from Linux

I often be underwhelmed by Linux's usb disk creator app, both the KDE and GTK versions.

• It crashes.
• It requires a FAT32 partition ?!?!
• It boots to a noddy try me out application.
• Its an annoyance on the way to installing Linux.

I've done netboot installs, just to avoid the USB creator stuff. Netboot is cool if you have a few PCs to install but a headache if you only have one to do.

This trick I've just discovered saves you from all that, you install Linux from Linux directly, no downloading ISOs nor reboots.  You can install Linux on to any HD that is connected to your current system.

In this case I wanted to install a full Linux system onto a 128GB USB3 drive I'd just purchased for the princley sum of 39€ & 80 cents.

This is the process, in my case, all done from a running Ubuntu system installing Ubuntu.

Create the partitions as you like them

Use whatever tools you prefer.
Personally I like to keep it simple and have everything on one partition.You have so many options for the tools to use to setup partitions in Linux.  Unlike when you install from a pen drive where you get only which one comes with the installer.

Rather than GUI tools I prefer simply...

fdisk /dev/sdb
mkfs.ext4 /dev/sdb1

since this is what I'm familiar with.

Install a base Linux to the partition

mount /dev/sdb1 /mnt/installer
apt-get install debootstrap
debootstrap trusty /mnt/installer

Where "trusty" is a string I had to hunt around the internet to find out.

This page helped http://packages.ubuntu.com/search?keywords=debootstrap

debootstrap installs a minimal Linux to the HD. Just enough for apt-get to be useful. No kernel or bootloader at this stage.  I notice debootstrap what lxc uses to setup a base container.

Create a chroot and use familiar tools to install the system

mount --bind /dev /mnt/installer/dev
mount --bind /dev/pts /mnt/installer/dev/pts
mount -t proc proc /mnt/installer/proc
mount -t sysfs sys /mnt/installer/sys
chroot /mnt/installer


Now your in a chroot where apt-get works and you can setup the system as you like it.  The following for a base Ubuntu desktop.  With this method you can install a simpler system if you prefer.

apt-get update
apt-get install language-pack-en-base
apt-get upgrade

dpkg-reconfigure tzdata
apt-get install grub-pc linux-image
apt-get install ubuntu-desktop ubuntu-standard

N.B. grub pops up a question asking where you want to install it, be careful not to mess with the system you are running on.  The chroot is not a "jail" you can still do damage as root from where you are.

In the chroot you can also add a user (useradd), set the root password (passwd), change hostname (vi /etc/hostname) and/or do whatever setup you feel like before you boot the system's kernel for the first time.

That's it. You now have Linux installed on the drive.  Without having to shutdown the current system.

You can boot from this Disk now or, as in my case unmount the drive and use it to boot a different machine.

This blog post is pretty much copied from halfway down this page.

https://help.ubuntu.com/community/Installation/FromLinux#Without_CD


N.B. after setting up a system, before logging in for the first time. I'd recommend trying to remove all the advertising nonsense from the CLI.

apt-get remove unity-scope-musicstores
apt-get remove unity-scope-video-remote
apt-get remove unity-scope-yelp
apt-get remove unity-scope-home
apt-get remove unity-scope-gdrive

To ditch that Amazon advertising that Ubuntu installs by default is now rather difficult you have to get rid of the whole webapps thing which is a shame.

apt-get remove unity-webapps-service
apt-get remove unity-webapps-qml

And of course you can now triumphantly

apt-get remove usb-creator-gtk

The Pomodoro Technique is for post WWW kids with attention deficit.

The Pomodoro Technique has gained a fair bit of traction over the years as a way to mark time and get things done.  Its pretty simple and pretty cool because its pretty simple.

However, the recommend time slice of 25 minutes strikes me as very little.

A grown adult getting paid for a living to work at a job ought really to be able to get their head down for an hour to get stuff done. I'm sure I didn't have problem concentrating for an hour when I was kid. We had 3 hour exams, hour long lessons and I've never had a problem sitting though a film.

I reckon its a problem that we consume media in smaller and smaller chunks these days just as Fahrenheit 451 predicted.  I have noticed my attention span becoming a problem.  I used to attribute it to smoking; the nicotine monster kicks in well before an hour is up if you let it.  I presumed that after giving up smoking I was left with this short little span of attention and the rest of my life was going to be hard.

Then I tried the pomodoro technique and set the timer to an hour.  Turned out it was not that hard.  It took 4 or 5 attempts to get comfortable with an hours continuous work.

I've stuck to an hour since and the evils of 451 are disappearing. I can keep my head in a book as long as I like and spend a lot less time on arsebook.  I recommend anyone sold on the pomodoro technique to increase the span as much as they can, or soon enough, Amazon will remote wipe your book collection and that knock on the door will be the fire brigade.

Generation X

I've recently had a lot of XML generating to do in Java code.  A boring job. Its an integration piece between two companies to swap some data.  The XML is as simple as it can be, each data item is needed. The data is fetched from a database, a webservices and the filesystem.  Very little of the XML is static.

The code is not elegant.  Using a templating system like JSP or freemarker would not be a good fit: all the data is pulled from somewhere in Java code.  There would be more code than template.

Writing this code in Java using the standard org.w3c.dom API results in a lot of boilerplate code.  For each new element you have to doc.createElement() and then reference the parent node and insert the element  someElem.appendChild() then set the text content and or attributes.

I've used Dom4j and JDom before and they are nicer APIs but still nothing revolutionary.


I woke up in the middle of the night last night with a really cool fix to this problem in my head.

XPaths are a very expressive way of searching in XML, the idea was to use XPath like expressions for generating new Elements in the XML.


Still not sure about the name for this little library, for now its "XGen" and the paths are "xGenPaths".

The following xGenPath will create the expected XML output.

/html/body/div#container/table.table/tbody/tr[5]

In a oneliner!
   
This is much much more concise than a series of create, append lines like this

Element bodyElem = createElement("body");
htmlElem.appendChild(bodyElement);
Element divElem = createElement("div");
bodyElem.appendChild(divElement);
divElem.setAttribute("id", "container");
...

I set about development, after one false start trying to implement subclass Java's Node and NodeList. I've come up with a very neat little library in less than a day.  All the objects are org.w3c.dom objects and the code is separated to helper functions.
Its very easy to mix and match between w3c APIs and the helper functions.
The ouput is an org.w3c.dom.Document.

Its simple and elegant.

The flow of typical code is like this

XGen xGen = XGenFactory.newInstance();

// New document with an xGenPath to start you off
xGen.newDocument("/html{lang=en}/head/title").setTextContent(TITLE);

// select with XPaths and create with xGenPaths
xGen.select("//head").create("link{rel=stylesheet}")


select() and create() return org.w3c.dom.NodeList instances containing the tail nodes just created or modified, and with a few bells and whistles includeing the ability to mutate all the items in the nodelist with familiar methods.

select("//div").setAttribute("foo", "baz");

creates the foo attribute on all the divs in the document and returns the list of  nodes.

This enables chaining together statements to create elements and attributes and you end up with some very concise code to build up an XML doc.

XGen xGen = XGenFactory.newInstance();
xGen.newDocument("/xml")
    .create("div/ul/li[3]")
    .setTextContent("foo")
    .setAttribute("abc", "123")
    .create("a/span")
    .setAttribute("class", "grey");
xGen.serialize(System.out);



The create method also contains an each() method for generating specific content for each node created with an xGenPath.

final int[]i = new int[1];
XGen xGen = XGenFactory.newInstance();
xGen.newDocument("/xml")
    .create("div/ul/li[3]").setTextContent("a", "b", "c")
    .each(new NodeMutator() {
        public Node each(Node node) {
            ((Element)node).setAttribute("id", "123-" + i[0]++);
            return node;
        }
    });
xGen.serialize(System.out);

which results in

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<xml>
  <div>
    <ul>
      <li id="123-0">a</li>
      <li id="123-1">b</li>
      <li id="123-2">c</li>
    </ul>
  </div>
</xml>

Pretty slick if I do say so myself and crying out for lamda functions.

I'm downloading JDK-8 as we speak, slight worried there is no Icedtea JDK8 yet, but this library is going to be some much smoother with lambdas.


get it here

https://github.com/teknopaul/generation-x

Simple Jenkins Job Prioritizing with Local Slaves

If you are familiar with Jenkins the title is probably all you need to read from this post.

Yesterday I hit upon a neat solution to the problem of scheduling many small jobs and a few big jobs on the same Jenkins server.

The problem is as follows...

We use Jenkins for all our build jobs but also for process automation and executing scripts. Jenkins provides a nice generic UI for executing parametrized scripts.

We have one Jenkins master server that compiles code and then installs the code.  This is the only server with root SSH access to all the servers in all our clusters, which it needs to perform installations.

We use the Jenkins master also for automated operations, starting and stopping the systems taking backups etc because the master node has the required root SSH access to all servers.  Our QAs and OPs get a job called START_SYSTEM which they can run and get a green light without having to know the details of which scripts on which hosts need to run to boot the system.

Running more than 2 build jobs on the master node grinds the server to a halt. To prevent too many concurrent jobs we limit the number of executors on master to 2.

The problem arises because if there are 2 compile jobs executing this prevents QA from being able to run operations jobs, the jobs get held in the build queue.


The option of setting up physical slave server is complicated because we would have to add the details of the host to the SSH config of every server in all clusters. Then I hit upon the simple solution of creating a Jenkins slave on the same physical server as the master.


Hence: Simple Jenkins Job Prioritizing with Local Slaves.

Now the master Jenkins node has 2 executors that limit concurrent compilation jobs and we have 20 executors available on the same physical server with the same SSH permissions for smaller jobs.

It was very simple to set up and it seems there is no practical difference running jobs on a local slave to the master executor queue.

For now a "big jobs" queue and a "lightweight jobs" queue is sufficient, I can see in the future we will want to divide up jobs further.  As a side effect of this set up Jenkins admins get a neat way to disable all build jobs without impacting the operations jobs.

Extreme standups

The daily stand-up meeting is a great invention, despite my distrust of scrum I'm sold on stand-ups.  Getting everyone together and time-boxing a short meeting is a good idea.  The stand-up scope limitation of last 24 / next 24 and impediments keeps stand ups focused and helps keep to the time-box.

However as a project moves on I find a 09:00 stand-up starts to be a drain. I'm not the 9 to 5 type. I prefer long sessions of hacking followed by the occasional day at the sea side.  The obligation to be somewhere at 09:00 is bearable when a new project is kicking off but soon starts to be a finger pushing down on the wrong side of the work/life balance.

When a team is up and running with tasks well understood, stand-ups are uneventful. Last 24 as expected, next 24 per the sprint plan.  Impediments occur, but they dont get sat on until the next meeting. Stand-ups become a status check (eurgh!) followed by "impediments, none".  In the project where we coined the phrase "extreme stand-ups" we were using IRC and had sufficiently well distributed tasks and shared knowledge that there was rarely a problem that only one person could solve. A question fired to IRC got quickly resolved by whoever was first available.  I love the async nature of IRC, it sits there in the background and does not interrupt your flow. This is important when your trying to focus on a complicated task.  When you reach natural breaks you check IRC, not when someone else has a question.  IRC is not a ringing phone or a flashing IM, its real-time but it does not pester you.  We soon realised that IRC was a fine solution for stand-ups too.


The extreme stand-up protocol is as follows...

• Create a dedicated IRC channel for stand-ups.
• Before the stand-up, each team members prepares their last24/next24/impediments in a text editor.
• At the allotted time, everyone switches to the stand-up channel and copy/pastes their contribution.
• Everyone reads everyone else's contributions, impediments are addressed outside this channel.
• If you know you aren't going to make it, you can post early, the previous day even.
• Start promptly, there no need to wait for everyone to show up.

There are various benefits to the extreme stand-up.

Primarily its async, you don’t have to hang up a phone call to attend.  This saves stress.  The same result is achieved if you post the night before then miss the bus in the morning than if you turn up exactly on time.

Time is saved, as if stand-ups weren’t efficient enough, we got them down to 120 seconds. Because you don't type your comments live the tendency to waffle is reduced.  We started to deliver short statements that conveyed the right level of detail.  Reading an extreme stand-up is more efficient than attending a verbal one.

A history is generated automatically, you can view the channel history at your leisure.  The channel becomes a diary of what the team actually spent time on as a team rather than what was planned.  That's useful documentation and it comes with no effort.

Chickens (as in pigs and chickens) can drop in or get a copy of the minutes. Because of the pub/sub nature of IRC, chickens don’t have to be explicitly invited. The meeting is written and formally structured so there's a lot less chance of a chicken taking over the meeting.  Chickens taking over meetings is something that tends to happen when word gets about that the whole team is in one place at one time, once per day.

By copy/pasting each contribution is delivered as an atomic commit.  This avoids the common problem of IM chats where more than on thread is active at the same time.

If you miss the stand-up for some reason its no biggie, your colleagues read your contribution first the next day and often it makes very little difference if you did task a on Wednesday or Thursday.  If there were dependencies you'd be communicating that already.


Scrum-no-buts stand-ups.

The current project I'm on has not accepted extreme stand-ups (yet).

We don’t have IRC we have to use Skype :( booo. Skype has a much higher probability of turning into a tedious conf call than IRC. Skye calls regularly drop out voice or video or loose one person completely. The technology itself is often the cause of slipping the time-box.

We have now hit the issue where some team members don't necessarily join the stand-ups. Either because they have other things to do at that time or because they don’t want to dedicate more than 15 mins.

Perhaps its time to kick some scrum-butts and force the issue, extreme stand-ups worked a charm with my previous team.