Rapid Application Development with Nodejs and Nodetoy

There is nothing I like more than bashing out code fast and not having to be concerned about documentation or customer requirements or other such necessary evils.

While I'm a Java hacker by day, JavaScript is the code I like to write. Its fun, its single thread is an enjoyable constraint that forces you to think a little as you code.  Multi-threaded JavaScript would be like playing football but being allowed to use your hands.

This month I've been permitted by my pay masters to run with a couple of little projects written in nodejs. Not doing big scalable web sockets things, just quickly developing web apps that resolve specific problems, a REST API, GUI that's shared across a few peeps in the team.

Step up nodetoy...

 

Nodetoy is a little project I wrote a while back when nodejs was only 0.5 versions old.

Nodetoy provides...

• A webserver for HTML and JS/CSS.
• Apache/nginx style Server Side Includes (SSI) for templating.
• A JSON/REST CRUD backend that saves flat files to the file system.

No code or config required, you turn it on and it start serving your web pages and fetching and storing JSON data. It has no security, no user/pass, no DDoS protection or anything like that, its a toy server for Intranet use, rapid prototyping, web/desktop app development with the browser as a GUI or just playing around with REST.

JSON layer is REST layer is so trivial its a joy to use.

• POST a blob of JSON to /data/foo.json to persist data.
• GET /data/foo.json to read data.
• DELETE /data/foo.json to delete data.
• GET /data/ returns a list of the .json files.

Thats it, there is no more to the backed.

Included in the download is and old version of jQuery, you probably want to upgrade that, a little tester web page so you can get started GETing and POSTing without having to write any JavaScript yet and a little TODO app in case you need some boiler plate code to start with.


I like everything about this project.
It took no time to write, it was done as part of a series of introductions to nodejs we did in the Riereta hacklab in Barcelona.
It does one thing and it does one thing well, a Linux mantra.
It makes use of the right patterns in the right places.
Features are nicely encapsulated, which is easy with node.
It has tools instead of docs.
I'm quite proud of the logo, it conveys the simplicity.
It has no build process, its source code is both ready to run and ready to be hacked.

Originally it had no dependencies other than node, because it was written back when there weren't many modules and before npm was included with node.  Since then its been added to npm and various modules have been separated out for re-use. You can still  clone the github repo to get up and running.

I've not touched the codebase for a while but when I got an opportunity to give it a run for its money recently I was pleasantly surprised.

With nodetoy is very simple to quickly build webapps that are fun to write, starting with the HTML.

It has a very clear separation of front end and back end, and both are easily hackable. Because the back end is so limited your forced to keep it simple, which for RAD development soon pays off.

There is no code in the BE that is tied to the front end, its the antitheses of GWT.  I write my FE code in jQuery but any ol' AJAX API would do.

The older I get the more convinced I become of the fact that SSI is the only templating that is needed, if you want to manipulate the DOM do so in the browser.

One advantage of developing the BE in nodejs is that its implicitly "discoverable" since everything is delivered as code the application is the source code, there is no build process just hack away and it boots so fast that [ctrl+c] [up] [enter] is actually an acceptable way to work.

Of course being a grownup language with node you also have unit testing, mocking, lint for validating code code and a million and one bells and whistles that you would expect with a professional development environment and that you don't get if you do prototyping with CGI and bash. Despite all the features node remains fun and easy to code. You can bounce between working in unit tests on the CLI to running your web server for real, literally, in a second.

In the last project I knocked-up the JSON crud from nodetoy was sufficient to not need to make any BE changes. This enabled me to focus on doing dev purely in Firefox and notepad++.  which brings me neatly to the next part of RAD development.

jQuery / Bootstrap...

If your a pro HTML designer I'm sure you'll cringe, bootstrap is used by 1% of the web, its far from original, but the fact is GUIs knocked up with bootstrap are very fast to write and always end up looking decent.  Arseing around with browser incompatibilities is not a fun part of development. I write HTML for Bootstrap in FireFox and it always workes in Chrome and Konqueor and the time I had to support IE8 for one of my GUIs it all just worked, thank you very much bootstrap.

Bootstrap doesn't focus much on providing lots of features, it strikes me its a coders CSS tool.  It tries to have as little impact on your HTML as possible and lets you focus on your code.  The 960 grid system and responsive grid for me have proven easily enough options for layout control. Sticking to a tight grid format always helps to make pages look neat and tidy.  Shame I didn't know about it when I wrote nodetoy's test.html page.

Bootstrap does not require jQuery but it seems to be a natural pairing.

jQuery is a coder friendly toolkit too. While it has a ton of plugins its core focus is on helping you write code.  Clearly no one wants to have to write another date picker, however its not the widgets that makes jQuery cool, its the code you end up writing. It is succinct.  The selector logic is as good as your HTML naming conventions are.  jQuery does nothing unexpected, it does exactly what you ask it to and no more. jQuery does a wonderful job of making my code cross platform. I have never had a problem with a bit of jQuery code not working where it should.  You say "jump" jQuery says "yes sir".

Comparing to the good ol' lamp stack or tomcat + myslq I have to say that my  nodetoy stack appeals to me more and more.

Pure HTML5 wins over JSP/PHP the less code you write generating HTML the better.  Pulling request specific pre-generated HTML from server-side is soo 1999.

JavaScript every where, is a very convenient way to work. Mixing data types, formatting and conversions are annoyances to a coder. If your back end, front end, data store and data transfer format are all talking the same language you cut out a lot of crap from your code.  You'll hear me rant about JSON and how much I dislike it's use in Java & Python but when doing RAD web GUIs its clearly the best fit for the job.

Schema free Backends are much more convenient that relational databases. For prototyping the benefit is immediate, and no Hibernate does not count.

If I needed to scale up to DB for some reason a file system did not suffice for the job or the complete lack of locking or versioning becomes a worry I'd drop in CouchDB and keep the data in JSON.  Going back to defining the data's detailed attributes before you write the code seems so backwards.  With schema free development you want to add new attribute to your object you just
obj.newAtt = foo
and thats it. No DDL, no size limits no datatype considerations or data conversion considerations, if foo was a string well it stays a string and if it was boolean it stays a boolean and if it was undefined good job you didn't bother to create a column for it isn't it.

Bottom line is that I've been able to turn around little dynamic webapps in around 4 hours. 

KISS principal has been working for me of late.

MidiOverIP

I've started to play around a bit with Linux as a music studio of late.

Some stuff is up to scratch, some stuff is better and some stuff is way behind.  Its the commercial stuff that is behind, or rather way ahead on Windows but in the case of music making this means VST plugins are a pain on Linux.  No VST mean no instruments which makes sound quite dull.

I recently purchased a midi keyboard  which comes with a bunch of analog soft-synths in VST format which I quite fancy using. The plugin failed to load in the VSTi plugin in LMMS. There was no claim from the manufacturer that they might work. With that fail I needed to hook up my LMMS / Hydrogen / Jack setup up running on my main PC to a Vista box I have that can run the VST plugins.

I started by spending a day shopping for midi hardware before coming up with nothing under 100€ and needing two of them.  This struck me as too much for an ameture tekno hack like myself. On the way back from the shops I resolved to write some midi over IP software since it was not going to be hard, I figured. It is not a tricky job midi over IP you need to sent 4 bytes of digital data across the wire and Ethernet speeds nowadays are easily fast enough.

Before I got started I checked what was out there on the net to see if there was anything that would just work out of the box. It seems there is progress.

Mac has midi over IP built in to the operating system which is handy for mac users.

Linux has a couple of options that work fine, multimidicast is nice and simple and does exactly what I wanted it too.  Trouble is, the Windows side is a pay tool to be found at http://nerds.de, not free software.

There are plenty of options if you fancy paying up real money but we are talking about 4 bytes of un-modified data over the wire so I thought Id rectify the Lin-Win interop issues myself. multimidicast is designed to be compatible with ipMidi from nerds.de so that was my starting point.

I thought I'd try to just port multimidicast code to Windows, but in the end that was not going to happen. The ALSA part (the midi side of things) is obvioulsy Linux specific thats half the code anyway. The network side should be posix compatible but writing C in for Windows you always end up getting sucked in to proprietory stuff there is really no point trying to write portable code IMHO. It was only going to be 1000 lines of code anyway, so I started with a blank page.  A blank page has the advantage of meaning I can apply GPLv3 to the code I write.

Windows MultiMedia has some failrly straight-forward midi APIs with some really shitty documentation, as usual and no example code. Once you get it working it all looks pretty simple. In the world of open source you get used to the fact that you have top quality example code for everything you might like to do. Not so with Windows C, its a dark art. It took me too long to find out that the header I needed was called mmsystem.h and the DLL is called winmm.dll and the linking should be done with -lwinmm.
For winsock I have to include winsock2.h Ws2tcpip.h mswsoc.h the dll is WINSOCK.DLL and you link with -lwsock32.
Perhaps there is some logic to Microsoft naming conventions, but it seems there just is no convention at all. Case is significant in C code and MS libraries are all over the shop so it is impossible to take educated guesses at where to start googling.

After spending a while day with firefox open and textpad to write the C I found a few good articles, notably, nothing hosted at MSDN. With some example code I had enough to search MSDN and from there could bash out the first few lines of code.  MSDN docs there are perhaps sufficient if you have plenty of C on Windows experience, but I dont. After about half a days scribbling I had something up and running that would read packets from the LAN and send them to midi. I wasted a lot of time on the fact that Linux APIs return error codes and Windows return random numbers, again no conventions, and WSAGetLastError() is where the real error is to be found.  Turns out I was listening to multicast about 2 hours before I new I was, since the return code was not as I expected it to be. Winsock is based on Berkley sockets, but that base was taken a long time ago and the similarites are now lost and just confuse more than they help.

Next step was to hook up the midi to a "real" soft-synth, and here we enter frustrating terriroy. It turns out to talk to the instrument a piece of the puzzle is missing, you need a virtual midi device and this is a software driver.  Presumably one that does very little, but a driver none the less and that means kernel code. I've been down this route before and its a pain in the arse, essentially Microsoft reserve the right to tell you what you are doing is not allowed and to get your driver work in you have to have it signed, that costs money.  You can not write you own code on your own PC in the world of Microsoft they still own your hardware.  Slightly scared I would hit a brick wall again I scoured the net for virtual midi software and came up with, thankfully, a few options.  The rtsMidi tool that is a mac compatible midi over IP thing works in Windows and has a virtual midi driver you can download for free. Nerds.de also have a virtual midi device that you can download for free and the basic version does not have a 60 minutes usage limit. This enabled me to skip the driver writing and plug in my VST instruments onto the network. A bit more fiddling and another 100 lines of frustrating code and LMMS can trigger my VST soft synths, nice.

Since I'd got this far, I figured it made sense to get the app to work in the other direction, taking midi from a real or virtual device and broadcasting it onto the network. This was as frustrating a job too.  Mostly because to read midi you can only have ONE line of code in you APP that opens the device, not two. This is wierd since it is farily mormal to have multiple sources and sinks in the world of middi. Having one APP with a main loop listenting for midi events and another for listening for network events meant two threads.  In Linux since ALSA uses the same APIs that the network does you only need one. Threading is C is a dark art too it seems. There are APIs that just dont work. You find this out by googling, MSDN makes no mention of the fact that the seemingly handly looking method called CreateThread will break your C app if you use it. The much clearer named _beginthreadex is the method you want. This was the first function I have found all day that starts with an underscore; wins points for originality I suppose.
The Windos midi APIs are split down the middle with IN and OUT very seprate so despite having two threads there was nothing shared and no need to synchronize.  This is good since I don't fancy writing thread safe code with guess work. To write thread safe code you need to know that what you are doing is correct. You can't test it, it could run for days before blowing up just by chance.

At the end of the day I have got somewhere, I now have Linux and Windows talking happily over midi. I can plug the hardware device in either PC and bang the keys and make that trigger whatever I like. Latency in Linux is exceptionally low, not so in Windows. Tomorrow's job is to get Windows midi to play the note when I tell it to not 100ms after.

Code is here, there is an exe there too with a README.  The code is fairly ledgibubble if you feel like modding it.

Unhelpful Old Git

OK so Git is cool and its hip.

But Git help is actively an un-helpful, even spiteful, little bitch with man pages that insult more than the Git himself does.

You need to read the manual before you can read the help, some help manages to confuse simple concepts, and other help manages to skip over things that are not obvious but are important.

I could rant on this for ages but lets keep this simple, t his is the output from git help,  with my annotations and some highlights from man git-* which is all in the same vein.

The most commonly used git commands are:

   add        Add file contents to the index  // define "index" the word means many things,
                                              // why "Add file contents" and not "add a file"

   bisect     Find by binary search the change that introduced a bug   
        // does "by binary search" really help

   branch     List, create, or delete branches  // ok

   checkout   Checkout a branch or paths to the working tree
    // what is a "working tree" its not a thing that has "branches" is it perhaps :)
    // A working tree is a local working copy of a branch with perhaps a few changes,
    // IMHO the word tree just does not help
    // so a bit more help from...  

git help checkout

    "Updates files in the working tree to match the version in the index or the specified tree.
    If no paths are given, git checkout will also update HEAD to set the specified branch as the
    current branch." 

    // ?? confusing
    // presumably git checkout does what svn checkout does, simply

svn help checkout | head -1
checkout (co): checkout a working copy from a repository.

    // If it does not do that, I have not worked out from the help what it does do. 
    // When I got to "tree-ish" I gave up on the man page, WTF is a "tree-ish"

   clone      Clone a repository into a new directory          
    // Should this not say something like
    //  "Make a local copy of a remote remote repository and create a working copy"
    // so lets looks at the man page...

Clones a repository into a newly created directory, creates remote-tracking branches for each branch in the cloned repository (visible using git branch -r), and creates and checks out an initial branch that is forked from the cloned repository’s currently active branch. After the clone, a plain git fetch without arguments will update all the remote-tracking branches, and a git pull without arguments will in addition merge the remote master branch into the current master branch, if any. This default configuration is achieved by creating references to the remote branch heads under refs/remotes/origin and by initializing remote.origin.url and remote.origin.fetch configuration variables.

    // OH right, thanks man :)  I particularly like

"creates and checks out an initial branch that is forked from the cloned repository’s currently active branch."

    // instead of

"creates a working copy"

   commit     Record changes to the repository 
   // perhaps a mention of "local" here, since that is a distinguishing feature of git
   // perhaps some mention that the committed changes come from the "index" not the "working tree"

   diff       Show changes between commits, commit and working tree, etc   
   // "etc" really helps :)  I once bought a bottle of cheap wine that said

"Great with meat, cheese, fish, etc"

   fetch      Download objects and refs from another repository
   // lets mention "remote" here
   // and what is an object and what is a "ref"? we are trying to version control files, are we not?

   grep       Print lines matching a pattern
   // what does this add that grep -r does not do, or is this just grep -r ?
   // the only difference I can find is that git grep hangs after one page says END
   // and you have to Ctrl+c to continue

   init       Create an empty git repository or reinitialize an existing one 
   // shh don't mention the local thing

   log        Show commit logs     // finally something simple and accurate

   merge      Join two or more development histories together         
   // I have not worked out how to merge in git yet, supposedly git is good at merging
   // fuct-if-i-know-how(tm)  not sure what a development history is, a branch perhaps?

   mv         Move or rename a file, a directory, or a symlink     

   pull       Fetch from and merge with another repository or a local branch 
   // some commas please, this help should read
   // "pull   something you should never do, see fetch and merge"

   push       Update remote refs along with associated objects     
   // You what? This means nothing to me, and git push is important.
   // If you didn't know push synchronizes local commits with the remote repository
   // you would not guess from the help

   rebase     Forward-port local commits to the updated upstream head  
   // eh?

   reset      Reset current HEAD to the specified state                
   // what?
   rm         Remove files from the working tree and from the index    
   // This would be OK provided we agree on "working tree" and "index"

   show       Show various types of objects               
   // This "help" just fills a line with text, replace with "lorum ipsum   status     Show the working tree status               
    // This is a key command that should be used all the time,
    // and that description does not indicate what git status does.
    // git status lists all the files in your working copy that you have changed or added

   tag        Create, list, delete or verify a tag object signed with GPG  
    // Presumably this is a "label" surely "tag or label" is better than "tag object".
    // What meaning does object convey here?

You may not agree with all my comments above, and most are facetious, I admit, but I think you must agree that the help is unhelpful and perhaps we need to set up some charity organisation to help pay for the git's reform and re-introduction into polite society.

git-scm.org does not have a bug reporting mechanism, doh!

I'm not just complaining, I would gladly contribute better help for git, just as soon as I've worked out what the feck the little git is doing.

I realise that git tries its hardest to copy nothing from other SCM systems, but perhaps they should read the man from SVN and takes some tips.

If you have to use GIT I suggest you use google "git introduction" and go and read a tutorial or any piece of non-official documentation before you start trying to interpret the help.  Git is  sufficiently different to all other SCMs to merit a bit of research before making the jump.

There are lots and lots of web pages attempting to fill the void that is git help.

like this for example

http://help.github.com/git-cheat-sheets/