How to setup a debrepo

My aim here is to setup a directory full of .deb files that can be installed with

apt-get install whatever

Not a full Debian repo, nor a "distribution", just a few of my own packages. Given how simple it is to create .deb packages its strange setting up a repo was so complicated. Reading Debian repo management docs complicated matters, I don't recommend it if you are setting up a simple repo.

N.B. google "Launchpad PPA" if you don't need to host the debs yourself and your code is open source. Save yourself some hassle.

I presume you do have a webserver setup to host the debs.

These are tools I needed to install on the server.

aptitude install dpkg-dev dpkg-sig gnupg apt-utils

I needed to generate a gpg key pair  gpg --gen-key is the command but its interactive so useless in a shell script.  It has a very weird syntax but here it is.

echo '
Key-Type: RSA
Key-Length: 1024
Key-Usage: sign
Name-Real: dpkg1
Expire-Date: 0
%commit
%echo done
' | gpg --gen-key --batch


dpgk1 is the name I've chosen for my package signing key, it can be any string e.g. your name.

It takes ages to run, on a quiet server/container with no disk activity it may never finish since its waiting for entropy from /dev/random.  Worst case run that command on a laptop/physical box, and export and move the keys back to the server.

Show the keys

gpg --list-keys

If you need to move the keys.

gpg -a --export-secret-key dpkg1 > secret.gpg
gpg -a --export dpkg1            > public.gpg
scp secret.gpg public.gpg root@wherever:
ssh root@wherever '
gpg --import -v -v ./secret.gpg
gpg --import -v -v ./public.gpg
'

The security conscious should delete secret.gpg now.


With the keys imported you need to copy debs to a folder that is being published by nginx/apache or similar.  And then create the repository indexes.
The index files I needed were all of Packages Packages.gz Release and Release.gpg, supposedly Packages is not needed but I could not get downloads to work without it.  That may be a bug in the tool versions I'm using.

cd /var/www/mydebrepo
gpg -a --export dpkg1 > public.gpg
rm -f Release Release.gpg Packages Packages.gz
dpkg-scanpackages . /dev/null > Packages
gzip -k Packages
apt-ftparchive release . > Release
gpg --yes -abs -u dpkg1 -o Release.gpg Release

Then sign all the debs.

dpkg-sig -k dpkg1 -s builder *.deb

That should be it for the serverside.

Setting up the client is a multi-step process

import the pgp public key
edit sources.list
apt-get update

I added a README.md file to the repo so I don't forget the process.

The syntax of sources.list is confusing, I don't understands why I have to put ./ and Ubuntu put wily release, it works but it looks different to all other lines in the file.



echo "

Add the following to your `/etc/apt/sources.list` to use this repository.

  deb http://download.tp23.org/download/deb/ ./

And run this to import the key

  wget -q http://download.tp23.org/download/deb/public.gpg -O - | sudo apt-key add -

If you still have problem try rebuilding the apt lists cache

  sudo rm -fr /var/lib/apt/lists/*
  sudo apt-get update

" > README.md

Seems deleting the indexes and rebuilding them is needed every time a file is added or removed. 

Doing that in such a way as not to interrupt clients use of the repo is left as an exercise for the reader. 

Installing Linux from Linux

I often be underwhelmed by Linux's usb disk creator app, both the KDE and GTK versions.

• It crashes.
• It requires a FAT32 partition ?!?!
• It boots to a noddy try me out application.
• Its an annoyance on the way to installing Linux.

I've done netboot installs, just to avoid the USB creator stuff. Netboot is cool if you have a few PCs to install but a headache if you only have one to do.

This trick I've just discovered saves you from all that, you install Linux from Linux directly, no downloading ISOs nor reboots.  You can install Linux on to any HD that is connected to your current system.

In this case I wanted to install a full Linux system onto a 128GB USB3 drive I'd just purchased for the princley sum of 39€ & 80 cents.

This is the process, in my case, all done from a running Ubuntu system installing Ubuntu.

Create the partitions as you like them

Use whatever tools you prefer.
Personally I like to keep it simple and have everything on one partition.You have so many options for the tools to use to setup partitions in Linux.  Unlike when you install from a pen drive where you get only which one comes with the installer.

Rather than GUI tools I prefer simply...

fdisk /dev/sdb
mkfs.ext4 /dev/sdb1

since this is what I'm familiar with.

Install a base Linux to the partition

mount /dev/sdb1 /mnt/installer
apt-get install debootstrap
debootstrap trusty /mnt/installer

Where "trusty" is a string I had to hunt around the internet to find out.

This page helped http://packages.ubuntu.com/search?keywords=debootstrap

debootstrap installs a minimal Linux to the HD. Just enough for apt-get to be useful. No kernel or bootloader at this stage.  I notice debootstrap what lxc uses to setup a base container.

Create a chroot and use familiar tools to install the system

mount --bind /dev /mnt/installer/dev
mount --bind /dev/pts /mnt/installer/dev/pts
mount -t proc proc /mnt/installer/proc
mount -t sysfs sys /mnt/installer/sys
chroot /mnt/installer


Now your in a chroot where apt-get works and you can setup the system as you like it.  The following for a base Ubuntu desktop.  With this method you can install a simpler system if you prefer.

apt-get update
apt-get install language-pack-en-base
apt-get upgrade

dpkg-reconfigure tzdata
apt-get install grub-pc linux-image
apt-get install ubuntu-desktop ubuntu-standard

N.B. grub pops up a question asking where you want to install it, be careful not to mess with the system you are running on.  The chroot is not a "jail" you can still do damage as root from where you are.

In the chroot you can also add a user (useradd), set the root password (passwd), change hostname (vi /etc/hostname) and/or do whatever setup you feel like before you boot the system's kernel for the first time.

That's it. You now have Linux installed on the drive.  Without having to shutdown the current system.

You can boot from this Disk now or, as in my case unmount the drive and use it to boot a different machine.

This blog post is pretty much copied from halfway down this page.

https://help.ubuntu.com/community/Installation/FromLinux#Without_CD


N.B. after setting up a system, before logging in for the first time. I'd recommend trying to remove all the advertising nonsense from the CLI.

apt-get remove unity-scope-musicstores
apt-get remove unity-scope-video-remote
apt-get remove unity-scope-yelp
apt-get remove unity-scope-home
apt-get remove unity-scope-gdrive

To ditch that Amazon advertising that Ubuntu installs by default is now rather difficult you have to get rid of the whole webapps thing which is a shame.

apt-get remove unity-webapps-service
apt-get remove unity-webapps-qml

And of course you can now triumphantly

apt-get remove usb-creator-gtk

The Pomodoro Technique is for post WWW kids with attention deficit.

The Pomodoro Technique has gained a fair bit of traction over the years as a way to mark time and get things done.  Its pretty simple and pretty cool because its pretty simple.

However, the recommend time slice of 25 minutes strikes me as very little.

A grown adult getting paid for a living to work at a job ought really to be able to get their head down for an hour to get stuff done. I'm sure I didn't have problem concentrating for an hour when I was kid. We had 3 hour exams, hour long lessons and I've never had a problem sitting though a film.

I reckon its a problem that we consume media in smaller and smaller chunks these days just as Fahrenheit 451 predicted.  I have noticed my attention span becoming a problem.  I used to attribute it to smoking; the nicotine monster kicks in well before an hour is up if you let it.  I presumed that after giving up smoking I was left with this short little span of attention and the rest of my life was going to be hard.

Then I tried the pomodoro technique and set the timer to an hour.  Turned out it was not that hard.  It took 4 or 5 attempts to get comfortable with an hours continuous work.

I've stuck to an hour since and the evils of 451 are disappearing. I can keep my head in a book as long as I like and spend a lot less time on arsebook.  I recommend anyone sold on the pomodoro technique to increase the span as much as they can, or soon enough, Amazon will remote wipe your book collection and that knock on the door will be the fire brigade.

Generation X

I've recently had a lot of XML generating to do in Java code.  A boring job. Its an integration piece between two companies to swap some data.  The XML is as simple as it can be, each data item is needed. The data is fetched from a database, a webservices and the filesystem.  Very little of the XML is static.

The code is not elegant.  Using a templating system like JSP or freemarker would not be a good fit: all the data is pulled from somewhere in Java code.  There would be more code than template.

Writing this code in Java using the standard org.w3c.dom API results in a lot of boilerplate code.  For each new element you have to doc.createElement() and then reference the parent node and insert the element  someElem.appendChild() then set the text content and or attributes.

I've used Dom4j and JDom before and they are nicer APIs but still nothing revolutionary.


I woke up in the middle of the night last night with a really cool fix to this problem in my head.

XPaths are a very expressive way of searching in XML, the idea was to use XPath like expressions for generating new Elements in the XML.


Still not sure about the name for this little library, for now its "XGen" and the paths are "xGenPaths".

The following xGenPath will create the expected XML output.

/html/body/div#container/table.table/tbody/tr[5]

In a oneliner!
   
This is much much more concise than a series of create, append lines like this

Element bodyElem = createElement("body");
htmlElem.appendChild(bodyElement);
Element divElem = createElement("div");
bodyElem.appendChild(divElement);
divElem.setAttribute("id", "container");
...

I set about development, after one false start trying to implement subclass Java's Node and NodeList. I've come up with a very neat little library in less than a day.  All the objects are org.w3c.dom objects and the code is separated to helper functions.
Its very easy to mix and match between w3c APIs and the helper functions.
The ouput is an org.w3c.dom.Document.

Its simple and elegant.

The flow of typical code is like this

XGen xGen = XGenFactory.newInstance();

// New document with an xGenPath to start you off
xGen.newDocument("/html{lang=en}/head/title").setTextContent(TITLE);

// select with XPaths and create with xGenPaths
xGen.select("//head").create("link{rel=stylesheet}")


select() and create() return org.w3c.dom.NodeList instances containing the tail nodes just created or modified, and with a few bells and whistles includeing the ability to mutate all the items in the nodelist with familiar methods.

select("//div").setAttribute("foo", "baz");

creates the foo attribute on all the divs in the document and returns the list of  nodes.

This enables chaining together statements to create elements and attributes and you end up with some very concise code to build up an XML doc.

XGen xGen = XGenFactory.newInstance();
xGen.newDocument("/xml")
    .create("div/ul/li[3]")
    .setTextContent("foo")
    .setAttribute("abc", "123")
    .create("a/span")
    .setAttribute("class", "grey");
xGen.serialize(System.out);



The create method also contains an each() method for generating specific content for each node created with an xGenPath.

final int[]i = new int[1];
XGen xGen = XGenFactory.newInstance();
xGen.newDocument("/xml")
    .create("div/ul/li[3]").setTextContent("a", "b", "c")
    .each(new NodeMutator() {
        public Node each(Node node) {
            ((Element)node).setAttribute("id", "123-" + i[0]++);
            return node;
        }
    });
xGen.serialize(System.out);

which results in

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<xml>
  <div>
    <ul>
      <li id="123-0">a</li>
      <li id="123-1">b</li>
      <li id="123-2">c</li>
    </ul>
  </div>
</xml>

Pretty slick if I do say so myself and crying out for lamda functions.

I'm downloading JDK-8 as we speak, slight worried there is no Icedtea JDK8 yet, but this library is going to be some much smoother with lambdas.


get it here

https://github.com/teknopaul/generation-x

Simple Jenkins Job Prioritizing with Local Slaves

If you are familiar with Jenkins the title is probably all you need to read from this post.

Yesterday I hit upon a neat solution to the problem of scheduling many small jobs and a few big jobs on the same Jenkins server.

The problem is as follows...

We use Jenkins for all our build jobs but also for process automation and executing scripts. Jenkins provides a nice generic UI for executing parametrized scripts.

We have one Jenkins master server that compiles code and then installs the code.  This is the only server with root SSH access to all the servers in all our clusters, which it needs to perform installations.

We use the Jenkins master also for automated operations, starting and stopping the systems taking backups etc because the master node has the required root SSH access to all servers.  Our QAs and OPs get a job called START_SYSTEM which they can run and get a green light without having to know the details of which scripts on which hosts need to run to boot the system.

Running more than 2 build jobs on the master node grinds the server to a halt. To prevent too many concurrent jobs we limit the number of executors on master to 2.

The problem arises because if there are 2 compile jobs executing this prevents QA from being able to run operations jobs, the jobs get held in the build queue.


The option of setting up physical slave server is complicated because we would have to add the details of the host to the SSH config of every server in all clusters. Then I hit upon the simple solution of creating a Jenkins slave on the same physical server as the master.


Hence: Simple Jenkins Job Prioritizing with Local Slaves.

Now the master Jenkins node has 2 executors that limit concurrent compilation jobs and we have 20 executors available on the same physical server with the same SSH permissions for smaller jobs.

It was very simple to set up and it seems there is no practical difference running jobs on a local slave to the master executor queue.

For now a "big jobs" queue and a "lightweight jobs" queue is sufficient, I can see in the future we will want to divide up jobs further.  As a side effect of this set up Jenkins admins get a neat way to disable all build jobs without impacting the operations jobs.

Extreme standups

The daily stand-up meeting is a great invention, despite my distrust of scrum I'm sold on stand-ups.  Getting everyone together and time-boxing a short meeting is a good idea.  The stand-up scope limitation of last 24 / next 24 and impediments keeps stand ups focused and helps keep to the time-box.

However as a project moves on I find a 09:00 stand-up starts to be a drain. I'm not the 9 to 5 type. I prefer long sessions of hacking followed by the occasional day at the sea side.  The obligation to be somewhere at 09:00 is bearable when a new project is kicking off but soon starts to be a finger pushing down on the wrong side of the work/life balance.

When a team is up and running with tasks well understood, stand-ups are uneventful. Last 24 as expected, next 24 per the sprint plan.  Impediments occur, but they dont get sat on until the next meeting. Stand-ups become a status check (eurgh!) followed by "impediments, none".  In the project where we coined the phrase "extreme stand-ups" we were using IRC and had sufficiently well distributed tasks and shared knowledge that there was rarely a problem that only one person could solve. A question fired to IRC got quickly resolved by whoever was first available.  I love the async nature of IRC, it sits there in the background and does not interrupt your flow. This is important when your trying to focus on a complicated task.  When you reach natural breaks you check IRC, not when someone else has a question.  IRC is not a ringing phone or a flashing IM, its real-time but it does not pester you.  We soon realised that IRC was a fine solution for stand-ups too.


The extreme stand-up protocol is as follows...

• Create a dedicated IRC channel for stand-ups.
• Before the stand-up, each team members prepares their last24/next24/impediments in a text editor.
• At the allotted time, everyone switches to the stand-up channel and copy/pastes their contribution.
• Everyone reads everyone else's contributions, impediments are addressed outside this channel.
• If you know you aren't going to make it, you can post early, the previous day even.
• Start promptly, there no need to wait for everyone to show up.

There are various benefits to the extreme stand-up.

Primarily its async, you don’t have to hang up a phone call to attend.  This saves stress.  The same result is achieved if you post the night before then miss the bus in the morning than if you turn up exactly on time.

Time is saved, as if stand-ups weren’t efficient enough, we got them down to 120 seconds. Because you don't type your comments live the tendency to waffle is reduced.  We started to deliver short statements that conveyed the right level of detail.  Reading an extreme stand-up is more efficient than attending a verbal one.

A history is generated automatically, you can view the channel history at your leisure.  The channel becomes a diary of what the team actually spent time on as a team rather than what was planned.  That's useful documentation and it comes with no effort.

Chickens (as in pigs and chickens) can drop in or get a copy of the minutes. Because of the pub/sub nature of IRC, chickens don’t have to be explicitly invited. The meeting is written and formally structured so there's a lot less chance of a chicken taking over the meeting.  Chickens taking over meetings is something that tends to happen when word gets about that the whole team is in one place at one time, once per day.

By copy/pasting each contribution is delivered as an atomic commit.  This avoids the common problem of IM chats where more than on thread is active at the same time.

If you miss the stand-up for some reason its no biggie, your colleagues read your contribution first the next day and often it makes very little difference if you did task a on Wednesday or Thursday.  If there were dependencies you'd be communicating that already.


Scrum-no-buts stand-ups.

The current project I'm on has not accepted extreme stand-ups (yet).

We don’t have IRC we have to use Skype :( booo. Skype has a much higher probability of turning into a tedious conf call than IRC. Skye calls regularly drop out voice or video or loose one person completely. The technology itself is often the cause of slipping the time-box.

We have now hit the issue where some team members don't necessarily join the stand-ups. Either because they have other things to do at that time or because they don’t want to dedicate more than 15 mins.

Perhaps its time to kick some scrum-butts and force the issue, extreme stand-ups worked a charm with my previous team.

Rapid Application Development with Nodejs and Nodetoy

There is nothing I like more than bashing out code fast and not having to be concerned about documentation or customer requirements or other such necessary evils.

While I'm a Java hacker by day, JavaScript is the code I like to write. Its fun, its single thread is an enjoyable constraint that forces you to think a little as you code.  Multi-threaded JavaScript would be like playing football but being allowed to use your hands.

This month I've been permitted by my pay masters to run with a couple of little projects written in nodejs. Not doing big scalable web sockets things, just quickly developing web apps that resolve specific problems, a REST API, GUI that's shared across a few peeps in the team.

Step up nodetoy...

 

Nodetoy is a little project I wrote a while back when nodejs was only 0.5 versions old.

Nodetoy provides...

• A webserver for HTML and JS/CSS.
• Apache/nginx style Server Side Includes (SSI) for templating.
• A JSON/REST CRUD backend that saves flat files to the file system.

No code or config required, you turn it on and it start serving your web pages and fetching and storing JSON data. It has no security, no user/pass, no DDoS protection or anything like that, its a toy server for Intranet use, rapid prototyping, web/desktop app development with the browser as a GUI or just playing around with REST.

JSON layer is REST layer is so trivial its a joy to use.

• POST a blob of JSON to /data/foo.json to persist data.
• GET /data/foo.json to read data.
• DELETE /data/foo.json to delete data.
• GET /data/ returns a list of the .json files.

Thats it, there is no more to the backed.

Included in the download is and old version of jQuery, you probably want to upgrade that, a little tester web page so you can get started GETing and POSTing without having to write any JavaScript yet and a little TODO app in case you need some boiler plate code to start with.


I like everything about this project.
It took no time to write, it was done as part of a series of introductions to nodejs we did in the Riereta hacklab in Barcelona.
It does one thing and it does one thing well, a Linux mantra.
It makes use of the right patterns in the right places.
Features are nicely encapsulated, which is easy with node.
It has tools instead of docs.
I'm quite proud of the logo, it conveys the simplicity.
It has no build process, its source code is both ready to run and ready to be hacked.

Originally it had no dependencies other than node, because it was written back when there weren't many modules and before npm was included with node.  Since then its been added to npm and various modules have been separated out for re-use. You can still  clone the github repo to get up and running.

I've not touched the codebase for a while but when I got an opportunity to give it a run for its money recently I was pleasantly surprised.

With nodetoy is very simple to quickly build webapps that are fun to write, starting with the HTML.

It has a very clear separation of front end and back end, and both are easily hackable. Because the back end is so limited your forced to keep it simple, which for RAD development soon pays off.

There is no code in the BE that is tied to the front end, its the antitheses of GWT.  I write my FE code in jQuery but any ol' AJAX API would do.

The older I get the more convinced I become of the fact that SSI is the only templating that is needed, if you want to manipulate the DOM do so in the browser.

One advantage of developing the BE in nodejs is that its implicitly "discoverable" since everything is delivered as code the application is the source code, there is no build process just hack away and it boots so fast that [ctrl+c] [up] [enter] is actually an acceptable way to work.

Of course being a grownup language with node you also have unit testing, mocking, lint for validating code code and a million and one bells and whistles that you would expect with a professional development environment and that you don't get if you do prototyping with CGI and bash. Despite all the features node remains fun and easy to code. You can bounce between working in unit tests on the CLI to running your web server for real, literally, in a second.

In the last project I knocked-up the JSON crud from nodetoy was sufficient to not need to make any BE changes. This enabled me to focus on doing dev purely in Firefox and notepad++.  which brings me neatly to the next part of RAD development.

jQuery / Bootstrap...

If your a pro HTML designer I'm sure you'll cringe, bootstrap is used by 1% of the web, its far from original, but the fact is GUIs knocked up with bootstrap are very fast to write and always end up looking decent.  Arseing around with browser incompatibilities is not a fun part of development. I write HTML for Bootstrap in FireFox and it always workes in Chrome and Konqueor and the time I had to support IE8 for one of my GUIs it all just worked, thank you very much bootstrap.

Bootstrap doesn't focus much on providing lots of features, it strikes me its a coders CSS tool.  It tries to have as little impact on your HTML as possible and lets you focus on your code.  The 960 grid system and responsive grid for me have proven easily enough options for layout control. Sticking to a tight grid format always helps to make pages look neat and tidy.  Shame I didn't know about it when I wrote nodetoy's test.html page.

Bootstrap does not require jQuery but it seems to be a natural pairing.

jQuery is a coder friendly toolkit too. While it has a ton of plugins its core focus is on helping you write code.  Clearly no one wants to have to write another date picker, however its not the widgets that makes jQuery cool, its the code you end up writing. It is succinct.  The selector logic is as good as your HTML naming conventions are.  jQuery does nothing unexpected, it does exactly what you ask it to and no more. jQuery does a wonderful job of making my code cross platform. I have never had a problem with a bit of jQuery code not working where it should.  You say "jump" jQuery says "yes sir".

Comparing to the good ol' lamp stack or tomcat + myslq I have to say that my  nodetoy stack appeals to me more and more.

Pure HTML5 wins over JSP/PHP the less code you write generating HTML the better.  Pulling request specific pre-generated HTML from server-side is soo 1999.

JavaScript every where, is a very convenient way to work. Mixing data types, formatting and conversions are annoyances to a coder. If your back end, front end, data store and data transfer format are all talking the same language you cut out a lot of crap from your code.  You'll hear me rant about JSON and how much I dislike it's use in Java & Python but when doing RAD web GUIs its clearly the best fit for the job.

Schema free Backends are much more convenient that relational databases. For prototyping the benefit is immediate, and no Hibernate does not count.

If I needed to scale up to DB for some reason a file system did not suffice for the job or the complete lack of locking or versioning becomes a worry I'd drop in CouchDB and keep the data in JSON.  Going back to defining the data's detailed attributes before you write the code seems so backwards.  With schema free development you want to add new attribute to your object you just
obj.newAtt = foo
and thats it. No DDL, no size limits no datatype considerations or data conversion considerations, if foo was a string well it stays a string and if it was boolean it stays a boolean and if it was undefined good job you didn't bother to create a column for it isn't it.

Bottom line is that I've been able to turn around little dynamic webapps in around 4 hours. 

KISS principal has been working for me of late.