There’re many CI/CD tools outside, some of them are available only as SaaS (and many are free for Open Source projects but some of them are closed source), others are open source and you can install them wherever you want. So: why another CI/CD tool?

At Sorint.lab we used many different CI/CD tools for years, our Open Source projects usually use free SaaS tools, internally and from our customer we used Open Source tools installed on premise. So, in the years, we got a great deal of knowledge on many CI/CD tools and learned many of their pros and cons. In the end we always struggled to achieve some features that we considered very important for such kind of tools. That’s why we created our own tool: Agola.

What are the requirements we tried to satisfy while designing and writing Agola?

It has been a while since I blogged about my hand wired atreus keyboard. In this year I spent some time testing new layouts and experimenting with different keyboards and I think it’s now time to start describing this. As the post title says the quest is infinite but I reached a good milestone.

First some background: I’m primarily an open source software engineer and a system engineer and architect so I spend most of my time writing code and using the shell. I use (only) GNU/Linux from 1999 and my workflow is heavily based on the terminal (and TMUX) and text editors (primarily VIM). The keyboard is my main interface with the computer and I heavily use shortcuts and so modifier keys.

When I was using classic keyboards layout I always hated different things of them:

• Staggered rows: one stupid inheritance from the typewriters. It should have been ditched some decades ago. It makes touch typing a really bad experience due to the asymmetric keys arrangement. I believe that one of the cause people don’t do touch type in these days is caused by them.
• Modifier keys position: having to stretch your pinkies or do some strange wrist contortions to press them was very stressing and was slowing me down a lot given the number of time I have to use them.

EDIT (26 July 2017) Updated the post with a better solution (since I was wrong and also etcd 3.2 accepts peerURLs containing domain names).

When using an etcd cluster to store important key value data you’ll probably prefer data persistency over availability. If more than half of your etcd cluster members goes down you’ll prefer to wait for them to return back accepting a loss of availability instead of recreating a new etcd cluster restoring a backup of your data that will probably be an old version of the data at the moment of the disaster.

As an example in stolon the stolon cluster data is saved inside a store like etcd or consul. Restoring the stolon cluster data from a backup could lead to bad behaviors since the stolon cluster state (that contains different information and the primary one is which postgres is the master/primary) won’t be in sync wiTh the real stolon cluster state).

Stolon was architected to be seamlessy deployed inside a k8s cluster so it becomes logical to also deploy the store (etcd or consul) inside k8s.

Today there’re different ways to deploy an etcd cluster inside k8s but, as I’m going to explain in this post, these don’t meet the above requirement:

• etcd stateful set. For example this helm chart.
• etcd-operator

Why these above options fail to meet our requirements?

I always felt traditional keyboards not really good for touch typing since the staggered letters felt very innatural and my wrists were forced to be angled causing some stress on them.

Since I use the mouse as less as possible preferring all the possible keyboard shortcuts, trying to reach CTRL/ALT/SUPER without moving my hands from the home row position is not possible and time consuming. Hitting “esc” or the various punctuation required a lot of stretch of the pinkies. Also moving from the VIM “hjkl” to the default arrows requires too much movement. Additionally I always wanted to be able to configure the keyboard layout with my preferences using open source firmwares like tmk or qmk so I finally decided to build my own keyboard.

So I started looking for some more ergonomical keyboards. These were my primary requisites:

• Use better key placement than staggered rows
• Be able to use my thumbs not only for hitting a long spacebar
• Use multiple layers to avoid moving my hands as much as possible and so have less keys
• Be able to use it as primary keyboard also when moving.

Some of the most famous DIY keyboards that could partly satisfy my needs were the planck keyboard and the infinity ergodox keyboard. The planck has an “ortholinear” key placement while the ergodox has staggered columns and a thumb cluster. At a first sight the ergodox looked perfect for my needs with the exception of the portability.

Then I found the atreus and I was fashinated by its layout since it seemed to fit all my needs and, being my first self built keyboard, it looked quite easy and cheaper than other models.

I honestly don’t like the woodish case and I decided to build it using acrylic and, since the matrix is quite simple and I have a bit of experience at soldering, I don’t see the need for a pcb but choosed to manually solder it.

Finished look

This is the (almost) final (needs some more cleanup) result:

Note: A simplified but complete way to see and test all of this in action is available on github.

Since we are always promoting new methodologies and technologies to our customers, the last years were full of projects based on IaaS (public and private) managed using Infrastructure as Code. Today, with the experience we gained in these years using different solution and technologies, I’d like to describe one way to manage an immutable infrastructure.

To make the below description the simplest I’m going to skip all the various parts related to application containers and their orchestration (that we are deeply using) (since also this is immutable infrastructure) and their way to do upgrade of stateless services (rolling vs blue/green deployments) but I’m going to just focus on the IaaS immutable infrastructure and in particular on how to handle rolling upgrades of stateful services.

I’m not going to explain here (it deserves a dedicated post) all the rationales that brought us to this solution and why some technologies over other (and this will probably change in future).

First a brief list of the primary requirements that this solution tries to satisfy:

• We want to be able to create from scratch in minutes complete test environments as much as possible similar to the production environment. In this way we can run reliable integrations tests for every pull request (but also making the developers run the tests by themself).
• We want to avoid the management problems and the uncertain of the upgrade consistency of our infrastructures using classic provisioning.
• We want to create our server images and test the new infrastructure creation and upgrade before doing this in production.
• We have to manage high available stateful services (databases etc…) that needs to be upgraded without downtime.
• Manage independent block of the full infrastructure and handle the dependencies between them.

To achieve this primary requirements we ended up creating an IaaS Immutable infrastructure using packer for image creation and terraform for infrastructure creation/upgrade all integrated to achieve rolling upgrades of stateful services. Everything fully testable.

At sorintlab, in the old days and with customers with “old school” infrastructures we had different ways to make an high available PostgreSQL. We used heartbeat, heartbeat+crm, rhcs, coreosync+pacemaker with a shared storage and a correctly (!!!) configured fencing/stonith.

When moving our infrastructures and customers to cloud based infrastructures (being it a private or a public one), the way to design the infrastructures drastically (and fortunately) changed. Finally our customers were forced to think about the various parts of the infrastructure like something that can break in every moment and that’s also ephemeral. This has always been the case on every infrastructure, but previously the customers tried to ignore this fact and prefered to pay tons of money for something claimed to be “unbreakable”.

In the cloud the “stateful” layers like databases are the most difficult the get right. You can completely offload it to your cloud provider DBaaS offer (like Amazon RDS) or you can implement it by yourself inside the IaaS.

We are more and more focusing on noSQL databases that already provide, given their nature, a shared nothing, sharded and replicated architecture.

But sometimes a relational database is still the best choice or is needed by some products (for example some Hadoop components need a PostgreSQL instance).

In the last days, I started working on an oauth2 and openid connect server (thanks to osin). As the project is starting as a corporate Single Sign On based on an ldap, the users needs to know when their password is expiring and be able to change it during the login phase. So some ldap client features were needed: Password policies (fedora 389 and other netscape derivatives are using this draft rfc while openldap ppolicy overlay uses this other draft rfc Password Modify Extended operation (rfc 3062) StartTLS I noticed that there were a lot of go ldap clients forks.