If you are familiar with Jenkins the title is probably all you need to read from this post.
Yesterday I hit upon a neat solution to the problem of scheduling many small jobs and a few big jobs on the same Jenkins server.
The problem is as follows...
We use Jenkins for all our build jobs but also for process automation and executing scripts. Jenkins provides a nice generic UI for executing parametrized scripts.
We have one Jenkins master server that compiles code and then installs the code. This is the only server with root SSH access to all the servers in all our clusters, which it needs to perform installations.
We use the Jenkins master also for automated operations, starting and stopping the systems taking backups etc because the master node has the required root SSH access to all servers. Our QAs and OPs get a job called START_SYSTEM which they can run and get a green light without having to know the details of which scripts on which hosts need to run to boot the system.
Running more than 2 build jobs on the master node grinds the server to a halt. To prevent too many concurrent jobs we limit the number of executors on master to 2.
The problem arises because if there are 2 compile jobs executing this prevents QA from being able to run operations jobs, the jobs get held in the build queue.
The option of setting up physical slave server is complicated because we would have to add the details of the host to the SSH config of every server in all clusters. Then I hit upon the simple solution of creating a Jenkins slave on the same physical server as the master.
Hence: Simple Jenkins Job Prioritizing with Local Slaves.
Now the master Jenkins node has 2 executors that limit concurrent compilation jobs and we have 20 executors available on the same physical server with the same SSH permissions for smaller jobs.
It was very simple to set up and it seems there is no practical difference running jobs on a local slave to the master executor queue.
For now a "big jobs" queue and a "lightweight jobs" queue is sufficient, I can see in the future we will want to divide up jobs further. As a side effect of this set up Jenkins admins get a neat way to disable all build jobs without impacting the operations jobs.
