How to Mock Your Browser’s Timezone With Jasmine and MomentJS

Last week, I’ve been working at adding a distributed countdown to my Online Planning Poker App. As our team works from Paris and Beirut, I wanted to unit test that it would work well through different timezones. I found a surprisingly simple solution.

What Google told me

I first searched Google to see how to do it. I found 2 answers that looked promising :

Known results for such a simple situation were disappointing !

What I ended up with

After a good deal of dabbling around, I eventually found a pretty simple solution using Jasmine and Moment Timezone :

1
2
3
jasmine.clock().install();
...
jasmine.clock().mockDate(moment.tz("2017-03-23 10:00:00", "Europe/Paris").toDate())

Obviously, the drawback is that it implies setting both the timezone and the time. This should be ok in most of unit tests though, but might be an issue in some cases.

Almost 15 Years of Using Design by Contract

I first read about Design By Contract in 2002, in Object Oriented Software Construction 2. As soon as I read it, I was convinced, today, I still believe it’s a great and fundamental technique. That’s why, I almost never write a contract ! Let me explain.

Phase 1 : DbC ignorance

I started to code professionally in 2001. This was a time where design and quality software meant Rational Rose (a UML design and code generation tool) while I, at the contrary, was just Cow Boy Coding my way out of any problem I was given.

I wasn’t really doing Object Oriented programming, but rather imperative programming, using objects as structs, getters, setters, and classes as a way to organize the code … In this context, my design skills were improving slowly, and I was at the risk of falling in love with a local-optimum practice that would prevent me from growing further.

That’s why I started to read books such as the Gang Of Four Design Patterns, or OOSC2.

Phase 2 : DbC enlightenment

The cover of the Object Oriented Software Construction 2

Reading this book was a profound experience to me. My programming changed fundamentally before and after reading it. The chapter about contracts, taught me what objects are.

One the one hand, Pre and Post conditions can be used in any kind of programming and are just a kind of C assert macro on steroids. Class invariant, on the other hand, is a completely different kind of beast. The invariant of a class is a predicate about an instance of this class that should always be true. For example : field X should never be null, or the value of field N should always be greater than 0.

In some way, grasping the concept of invariant is close to understanding what a class is.

Phase 3 : DbC everywhere

That’s when I started to write contracts everywhere. I was writing C++ code at the time, and my code must have looked something like that :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
class MonkeyWrench
{
    bool _isStarted;
    std::vector<Part>* _movingParts;

protected:

    virtual void invariant() const
    {
        assert(_isStarted == (_movingParts != NULL));
    }

public:

    MonkeyWrench()
    {
        this->_isStarted = false;
        this->_movingParts = NULL;

        invariant();
    }

    bool isStarted() const
    {
        return this->isStarted();
    }

    void start()
    {
        assert(!this->isStarted());
        invariant();

        this->_movingParts = ...

        invariant();
        assert(this->isStarted());
    }

    const std::vector<Part>& movingParts() const
    {
        assert(this->isStarted());
        invariant();

        return *this->_movingParts;
    }
    ...
};

I definitely over-abused contracts, it made the code unreadable. Plus sometimes, I was using excessively long and intricate assertions which made the problem even worse.

Hopefully, overusing contracts also taught me a lot in a short time. Here are some of the lessons I learned :

  • DbC is not very well supported, it’s never something built in the language, and edge cases like inheriting an invariant or conditions can become messy pretty fast.
  • Checking for intricate contracts at every method call can be pretty slow.
  • Checking everything beforehand is not always the simplest thing to do, at times, throwing an exception on failure just does a better job.
  • It happened that removing the contract made the code do just what I wanted. It’s easy to write unnecessary strict contracts.
  • Command Query Separation Principle is great ! Having ‘const’ or ‘pure’ queries that don’t change anything make writing contracts a lot simpler.
  • Preconditions on queries are painful. When possible, returning a sensible ‘null value’ works better, nothing is worse than getting an error when trying to call a const query from the interactive debugger.
  • Finally, the more immutable a class is, the simpler the invariant. With a lot of mutable fields, you might resort to have the invariant check that fields are synchronized as expected. If fields are immutable, this simply vanishes.

Phase 4 : DbC hangover

At the same time I discovered all these small subtleties about contracts, I fell upon Martin Fowler’s book Refactoring, improving the design of existing code and started to use Unit Tests extensively. This lead me to the following conclusions :

  • Tests are more efficient at producing quality software
  • Contracts can be an hindrance when trying to do baby steps refactorings as described in Martin Fowler’s book

On top of that, as DbC is not natively supported by languages, no documentation is generated, meaning that most of the time, the callers still have to look into the code. As a result, I was using contracts less and less often.

Phase 5 : DbC Zen

Looking back, I might not be writing a lot of asserts in my code, but I am still thinking in terms of contracts all the time. In fact, there are a ton of ways to use DbC without writing assertions :

  • Use as much immutability as possible. An immutable class does not need to check its invariant all the time, just throwing from the constructor if arguments are not valid is enough.
  • Use conventions as much as possible, for example, constructor arguments should be set for all the life of the object (cf Growing Object Oriented Software Guided by Tests which describes the different ways to inject something in an object)
  • Looking back at my DbC assertions, most relate to null values. Again conventions work better ! At work, we simply forbid passing null values around. If something can be null, it means it’s optional, Java has an Optional<T> class for just that (I’m pretty sure it is possible to do something even better with C++ templates). In this case, if the contract is broken, NullReferenceException will eventually be our assertion.
  • Replace as many pre & post conditions with invariants on the callee, the arguments or the return objects as possible. It makes sense as it’s just making sure we are using ‘valid’ objects everywhere. Again, if these objects are immutable, it makes the whole thing even simpler !
  • To take further benefit of the invariant of immutable objects, introduce new types. For example, instead of changing an object’s state through a command with associated involved contracts, split the class in 2 and make the method a query returning an immutable object, potentially making the initial class immutable as well. Remember, immutable classes mean almost no assertions !
  • Use your language. Ex, instead of asserting that 2 lists remain of the same length, refactor to a list of pairs ! (I know that’s an obvious example, but you get the point)
  • If you are using a statically typed language, use types ! For example, I remember at one project I worked on, we had an bug involving a duration : somewhere in the code milliseconds got mistaken for seconds … We fixed that by replacing the integer by TimeSpan all over the place. Again, that’s so obvious !
  • Eventually, when all else fails, or when it’s just too much overhead, use the simple asserts provided by your language or common libraries.

To come back at the previous code section, this how it could be written without assertions :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class MovingMonkeyWrench
{
    const std::vector<Part> _parts;

public:
    MovingMonkeyWrench() : _parts(...) {}

    const std::vector<Part>& parts() const
    {
        return this->_parts;
    }
    ...
};

class MonkeyWrench
{
public:
    MovingMonkeyWrench start() const
    {
        return MovingMonkeyWrench();
    }
    ...
};

Details are omitted, but it’s easy to see how shorter the code is.

Conclusion

When applying all the techniques above, you’ll see that cases for explicit assertions are rare. Less assertions also workarounds the issues coming from the poor support for DbC : no documentation and intricate cases.

In the end, assertions made my code more ‘functional’. I’m not alone to have done the same journey, and if you are interested you should read Eric Evans’ DDD book where he presents things like immutable value objects and specification objects.

My Ultimate Jira Personal Kanban

A few years ago, I wrote about how I started to use Jira as my personal Kanban board at work. A lot of things have changed since then, which brought me to update my board and make it even more productive !

The context

During the almost 18 months since I wrote this first post, a lot of things have changed in my daily work (hopefully : I’m not doing the same thing again and again !). Essentially, I got involved in more projects, some of which involve people from all around the company and some of which don’t require any code to be written. For example, I’m now engaged in our Agile Community of Practice, where I sometimes contribute content.

Here are the consequences on my work :

  • I have more tasks to deal with, not necessarily more work, but still more tasks
  • I have more sources of tasks : tasks can come from any of the projects I am involved in
  • I have more tasks depending on other people, and that are in a WAITING state meanwhile

I had to adapt my personal Kanban to this new workload.

The changes

As I explained in the previous description of my Jira Personal Kanban, I am using a custom project and Kanban board to aggregate all my tasks from various projects, in order to see everything in a single unique place. Here are the changes I’ve made since, so if you haven’t yet, it might be a good idea to read that previous version first.

Quick filters

In his post Maker’s Schedule, Manager’s Schedule Paul Graham explained the challenge of having a lot of non-programming work to do everyday for programmers. He then advises to use slots for different activities during the day, in order to keep uninterrupted chunks of time to do creative work. To apply this technique, I reserved ‘Unbookable except for X’ slots in my calendar everyday.

I had previously been using Swim-lanes to track work from different projects. This turned out not to scale very well to more projects : it made the board messy, and I kept being distracted by all these other tasks. I ditched all the Swim-lanes (not exactly, I kept one for urgent issues only). Instead of Swim-lanes for tracking projects, I now use Quick Filters. I created such filters as With Project X and Without Project X. During the day, when I want to focus on Project X, I use quick filters to only show tasks related to it.

Quick filters screen capture

Day markers

I have a daily routine of checking what’s on my plate and deciding what I’d like to achieve during the day (picking the right time to do this is an art in itself). In order to keep track of this, I use special day marker tasks : as ^^^ TODAY ^^^, ^^^ TOMORROW ^^^ and ^^^ THIS WEEK ^^^. This tasks are always in my TODO column, and will never be completed. I move them around to mark what I expect to finish at different time horizon. Ex : everything above ^^^ TODAY ^^^ should be finished before the end of the day.

Again, this helps me to focus on today’s activities, and to do just enough prioritization.

Day marker tasks screen capture

One last thing here, you’ll have noticed the Epic for these special tasks. It’s a way to identify them in JQL queries.

WAITING flag

Quite often, you have tasks waiting for someone else. That’s surely not the most efficient situation, but once you leave the comfort of your focused dev team, handoffs are often the norm (at least until the lean principles spread in every part of the business). Status of waiting tasks is worth checking regularly, but very certainly not many times per day !

Again, leaving them in my board created useless distraction. I have now taken the habit of renaming the tasks I’m waiting for with a [WAITING] ... prefix. On top of that, I created 2 quick filters WAITING and Without WAITING to quickly check and then forget about waiting tasks.

Waiting tasks screen capture

Watching tasks I’m not the assignee of

On some occasions, we might be two of us working on the same task, or I might want to keep an eye on a task to know if something is needed from me. As there is only a single possible assignee in Jira, I changed my global filters to also include tasks with a custom label pbourgau-watch. Any time I want to add a task in my board, I just add this label to it.

Screen capture of a task description I'm not the assignee of

Getting the Lean reports back

In order not to have too many old tasks in my board, I used to filter out old items in the global filter. This did the job, but at the cost of invalidating the lean reports (cumulative flow and control charts). In order to get these back, I removed this constraint from the global filter, and created yet another quick filter Without Old which I almost always keep on.

Control chart screen capture

Scripts

Global Filter

1
2
3
4
5
6
7
project in (POP, POPABTODO, "Development Engineering Program", COPA)
AND type != Epic
AND (Assignee = pbourgau OR
    Co-Assignees in (pbourgau) OR
    mentors in (pbourgau) OR
    labels in (pbourgau-watch))
ORDER BY Rank ASC

Quick Filters

1
2
3
4
5
6
7
8
9
10
11
12
13
14
-- With "Project X" + Day marker tasks (Epic link ...) + tasks containing "BRANDING"
project = "Project X" or "Epic Link" = POPABTODO-410 or summary ~ "BRANDING"

-- Without "Project X"
project != "Project X" and summary  !~ "BRANDING"

-- Without Old
status not in (DONE,CLOSED) OR updated >= -14d

-- WAITING
summary ~ 'WAITING'

-- Without WAITING
summary !~ 'WAITING'

Things that did not change

I still use a WIP limit on the In Progress column, display the Epic in on the cards and special use custom color coding for tasks :

1
2
3
4
5
-- Tasks with an imminent due date become red
duedate <= 1d or priority = "1-Very High"

-- Tasks with a due date are orange
duedate is not EMPTY

The result

Overall, this is how my board looks like :

Full board screen capture

I guess I’m a kind of personal productivity geek … but I believe it’s a skill of utter importance for developers, especially when they get a bit of experience and are not fed ready made tasks to do.

How to Subscribe to an ActionCable Channel on a Specific Page With Custom Data ?

In my spare time, I’m writing a Planning Poker App. As a reminder, planning poker is a group estimation technique designed to eliminate influence bias. Participants keeps their estimates secret until everyone unveils them at the same time (See Wikipedia for more details).

The driving idea behind my app is for team members to connect together and share a view of the current vote happening in their team. Each team has an animator, who is responsible to start new votes. This is the aspect I’ve been working on during the last few days. I want all team members to be notified that a new vote started by displaying a countdown on their page.

I am building the app with Rails 5 but I did not have a clear idea of what technology to use to build this feature. After some googling, I found that ActionCable provides just the kind of broadcasting I am looking for (Have a look at the ActionCable Rails guide for more details).

A Specific Page

The Rails guide is pretty clear, as usual I would say, but all the examples show subscriptions at any page load. As explained above, I only want participants to subscribe to their own team’s votes : until they have joined a team, it is not possible to subscribe to a particular channel.

As my app is currently behaving, once identified, participants get to a specific team page. I wanted to use this page as the starting point to my subscription. After some more googling about page specific JavaScript in Rails, I found this page from Brandon Hilkert that explains how to do this cleanly. The idea is to add the controller and action names to the body tag, and to filter out js code at page load. This is what I ended up doing :

First, I adapted the app layout to keep track of the controller and action names in the HTML body :

1
2
3
4
5
6
7
8
<!-- app/layouts/application.html.erb -->
<!DOCTYPE html>
<html>
  ...
  <body class="<%= controller_name %> <%= action_name %>">
    ...
  </body>
</html>

Then I replaced the default channel subscription with a function :

1
2
3
4
5
6
7
8
# app/assets/javascripts/channels/team.coffee
window.App.Channels ||= {}
window.App.Channels.Team ||= {}

App.Channels.Team.subscribe = ->
  App.cable.subscriptions.create "TeamChannel",
    received: (data) ->
      # Do something with this data

As a reminder, here is what the server side channel would look like :

1
2
3
4
5
class TeamChannel < ApplicationCable::Channel
  def subscribed
    stream_from "team_channel"
  end
end

Finally, I called this subscribe function from some page specific Javascript :

1
2
3
4
5
# app/assets/team_members.coffee
$(document).on "turbolinks:load", ->
  return unless $(".team_members.show").length > 0

  App.Channels.Team.subscribe()

That’s it. By playing around in your browser’s js console, you should be able to test it.

Custom Data

That’s just half of the story. The code above subscribes on a specific page, but it does not specify any particular team channel to subscribe to. This means that all participants would receive notifications from all teams !

In his article about unobtrusive JavaScript in Rails, Brandon Hilkert also suggests using HTML data attributes to pass parameters to the a JavaScript button event handler. There’s no button in our case, but we can still use the same technique. Let’s add data specific attributes to the HTML body.

To subscribe to specific team channel, the plan is to add the team name to the HTML body tag through a data attribute, then to capture and use this team name when subscribing.

Again, let’s enhance the layout :

1
2
3
4
5
6
7
8
<!-- app/layouts/application.html.erb -->
<!DOCTYPE html>
<html>
  ...
  <body class="<%= controller_name %> <%= action_name %>" <%= yield :extra_body_attributes %> >
    ...
  </body>
</html>

I had to adapt my views. In the team members show view (the one doing the subscription), I added an extra data attribute for the team name :

1
2
3
4
<!-- app/views/team_members/show.html.erb -->
<% provide(:extra_body_attributes, raw("data-team-name=\"#{@team.name}\"")) %>

...

With this done, it is possible to capture the team name from the page load event and feed it to the subscribe method :

1
2
3
4
5
# app/assets/team_members.coffee
$(document).on "turbolinks:load", ->
  return unless $(".team_members.show").length > 0

  App.Channels.Team.subscribe($('body').data('team-name'))

I then used the team name to subscribe to a specific channel :

1
2
3
4
5
6
7
8
# app/assets/javascripts/channels/team.coffee
window.App.Channels ||= {}
window.App.Channels.Team ||= {}

App.Channels.Team.subscribe = (teamName) ->
  App.cable.subscriptions.create {channel: "TeamChannel", team_name: teamName},
    received: (data) ->
      # Do something with this data

The last piece is to actually start a specific channel :

1
2
3
4
5
class TeamChannel < ApplicationCable::Channel
  def subscribed
    stream_from "team_channel_#{params[:team_name]}"
  end
end

Same as before, hack a bit with your browser’s console, you should be able to check that it’s working.

Last thoughts

This is not exhaustive, depending on your situation, there might be other things you’ll need to do, like unsubscriptions for example.

I’d also like to give a word of feedback about ActionCable after this first look at it. Overall, it worked great both in development and production. Everything seemed to work almost out of the box … Except testing : I did not manage to write robust unit tests around it. There is pull request for that that should be merged in Rails 5.~ sometimes soon. For the moment, I’m sticking to large scale cucumber tests.

How I Finally Use Docker on Small Open Source Side Projects

A few months ago, I started Philou’s Planning Poker, an open source side project to run planning poker estimate sessions remotely. The main technology is Rails, and I’d been planning to use Docker as much as possible as a way to learn it. Indeed, I learned that Docker is no Silver Bullet !

The Docker love phase

At first everything seemed great about Docker. I’d used it on toy projects and it proved great to quickly setup cheap and fast virtual machines. I even created the Rubybox project on Github to clone new ruby VMs in a matter of seconds. I also used Docker to host my Octopress environment to write this blog. As a long time Linux user, my dev machines have repeatedly been suffering from pollution : after some time, they get plagued with all the stuff I installed to do my various dev experiments, and at some point, re-install seems easier than cleaning up all the mess. If I could use containers for all my projects, Docker would be a cure for this.

Going through all these successes, when I started my planning poker app, I decided to go all into Docker, development, CI and deployment. You can read the log of how I did that in these posts. Fast forward a bit of searching, experimenting and deploying, all was setup : my dev env was in containers, my CI was running in containers in CircleCI and the app was pushed to containers on DgitalOcean.

Reality strikes back

At first, everything seemed to be working fine. Even if there were a few glitches that I would have to fix down the road like :

  • Whenever I wanted to update my app’s dependencies, I had to run bundle update twice, and not incrementally. Surely, I would manage to fix that with a bit of time
  • Obviously, the CI was slower, because it had to build the containers before deploying them to Docker Hub, but that was the price to pay in order to know exactly what was running on the server … right ?
  • And … Guard notifications did not appear on my desktop. I was accessing my dev env through ssh, so I would have to fix that, just a few hours and it should be working

After a while, I got used to my work environment and became almost as productive as I used to be … but you know, shit happens !

  • I had to install PhantomJS on my CI, and if that comes out of the box on TravisCI, you’re all alone in your own containers. Installing this on the Debian container proved unnecessarily complex, but I figured it out
  • Then all of a sudden, my CI started to break … You can read a summary of what I did to fix it here. Long story short : I had forgotten to clean up old docker images, and after enough deployments, the server ran out of space, and that corrupted the docker cache somehow. I eventually re-installed and upgraded the deployment VM. That made me lose quite some time though.
  • Finally, as I started to play with ActionCable, I could not get the web-socket notifications through my dev host. There must be some settings and configuration to make this work, for sure, but it’s supposed to work out of the box.

Eventually, this last issue convinced me to change my setup. All these usages of Docker where definitely worth it from a learning point of view, but as my focus moved to actually building the app, it was time to take pragmatic decisions.

My use of Docker now

There were 2 main ideas driving my changes to my dev env for this open source side project :

  1. Use the thing most people do
  2. Use commercially supported services & tools

These should avoid losing my time instead of being productive. My setup is now almost boring ! To summarize I now use TravisCI, Heroku, and rbenv on my physical machine. I kept Docker where it really shines : all the local servers required for development are managed by Docker Compose. Here is my docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
db:
  image: postgres:9.4.5
  volumes:
    - planning-poker-postgres:/var/lib/postgresql/data
  ports:
    - "5432:5432"

redis:
  image: redis:3.2-alpine
  volumes:
    - planning-poker-redis:/var/lib/redis/data
  ports:
    - "6379:6379"

This saves me from installing Postgresql or Redis on my dev machine, and I can start all the services required for app with a single docker-compose up command !

My future uses of Docker

More generally, in the near future, here is when I’ll use docker

  • As I just said, to manage local servers
  • To boot quick and cheap VMs (check rubybox)
  • To handle CI and deployment of large or non-standard systems, where Docker can provide a lot of benefits in terms of price, scaling or configurability

Docker came from the deployment world, and this is where it is so great. As of today though, even if it is usable as dev VM, it is still not up to a standard dev machine. Despite that, all the issues I ran into could be fixed, and I’m pretty sure they’ll be some day.

Developer ! Are You Losing Your Rat Race ?

A rat race is an endless, self-defeating, or pointless pursuit. It conjures up the image of the futile efforts of a lab rat trying to escape while running around a maze or in a wheel.

Are we building our own maze self defeating landscape by our exacerbated focus on technology ? Let me explain.

The context

As Marc Andreessen famously said “Software is eating the world”, which means that there is more and more demand for software. At the same time, giant countries like China, India, Russia or Brazil are producing more and more master’s degrees every year. This also means more and more software engineers. The consequence is that there has never been so many new technologies emerging than these days. The software landscape his huge, growing and complex.

That’s great for progress, but it’s a puzzle for hiring. In this chaotic environment, years of experience with a particular technology is something that remains easy to measure, that’s why employers (and developers) tend to use keywords to cast for a job.

The effects

As a result, developers tend to pick a few technologies to become masters at, to put them on their CV and get job offers. There’s a danger with specializing on a particular technology : eventually, it will become deprecated, in this keyword driven world, it’s almost like if you’ll have to start from zero again. Even if a specialization is wide enough now, as time goes on and more and more technologies are created, any area of expertise will become a tiny spot in all the landscape. One might think this is only an issue for old guys that did not stay up to date … I strongly believe this is wrong, it happened to all past technologies, I don’t see why today’s latest .js framework wouldn’t be legacy stuff one day.

One could think that sticking to a good employer is a good fix against that. It is … for some time ! Sticking to an company actually means betting on this company. What would happen if it went out of business, or through difficult times and you’re asked to leave ? When you reach the job market after so long with a single employer, you’ll be a de-facto specialist, on proprietary stuff that no one is interested about.

Finally, you might work hard not to specialize, but it’s going to be a lot more difficult to get a job as a generalist, only a few shops actually hire this way.

To summarize, we are forced into specialization, which is great in the short term, but risky in the long run.

1€ advice

So what can we do about this ? Obviously, we cannot change the world … The only ones we can act on are ourselves !

Learning

In our fast moving tech world, learning remains key ! But instead of trying to keep up with all the cool new techs that are invented every day, we should study fundamental skills, and only learn just enough specific skills to get the job done. To me fundamental skills are all the things you’ll apply whatever the language and technology you are using, for example :

  • design
  • architecture (whatever that is …)
  • clean code
  • refactoring
  • legacy code
  • testing
  • tooling
  • mentoring & coaching
  • programming paradigms (functional, dynamic, static, imperative, OO, concurrent …)
  • process flow
  • communication
  • product definition
  • concurrency
  • performance

I wrote this post that explains how I did learn some of these skills (by no mean would I say that this is the only way). Good mastery of these skills should be enough to quickly get up to speed in any project you are involved. This other article In 2017, learn every language, which I found through the excellent hackernewsletter, explains how this is possible.

Unfortunately, knowing is not enough …

Selling

How do you convince others that you are up to the job in a particular technology ? Unfortunately, I don’t have a definitive answer yet …

Regularly, people try to coin a word to describe the competent generalist developer : polyglot, full stack, craftsman … If it’s good enough, it usually gets taken over quite fast by the industry and just becomes yet another buzzword (the only exception being eXtreme Programming, but who would like to hire and eXtreme Programmer ?).

In Soft Skills, John Somnez says the trick is to explain to people that you might not have experience in a technology ‘yet’. This might work, if your resume gets through, which is not sure.

Here’s my try : the next time I’ll polish my resume, I’ll try to put forward my fundamental skills first, for example with 5 stars self-assessments. Only after will I add something like “By the way, I could work with tech X Y Z …”.

Independence

Being your own boss could be a solution in the long term. I recently listened to The End of Jobs in which the author explains that entrepreneurship is an accessible alternative these days, and that like any skill, it’s learnable. The catch is that there are no schools, no diplomas, and that it seems a lot riskier in the short run. Despite that, he makes the point that the skills you’ll learn makes it quite safe in the long run !

Questions

I feel like my post asks more questions than it provides answers :–). Honestly, I’d really love to read other people’s opinions and ideas. What are your tricks to market yourself on new technologies ? As a community, what could we do to fight our planned obsolescence ? Do you think I’m totally wrong and that the problem does not exist ? What do you think ?

How I Fixed ‘Devicemapper’ Error When Deploying My Docker App

A few months ago, I started continuously deploying my latest side project to a Digital Ocean box. If you are interested, here is the full story of how I did it. All was going pretty well until last week, when the builds unexpectedly started to fail. I wasn’t getting the same error at every build, but it was always the Docker deployment that failed. Here are the kind of errors I got :

1
2
3
4
5
6
7
8
# At first, it could not connect to the db container
PG::ConnectionBad: could not translate host name "db" to address: Name or service not known

# Then I started to have weird EOF errors
docker stderr: failed to register layer: ApplyLayer exit status 1 stdout:  stderr: unexpected EOF

# Eventually, I got some devicemapper errors
docker stderr: failed to register layer: devicemapper: Error running deviceCreate (createSnapDevice) dm_task_run failed

You can read the full error logs here.

That’s what happens when you go cheap !

After searching the internet a bit, I found this issue which made me understand that my server had ran out of disk space because of old versions of my docker images. I tried to remove them, but the commands were failing. After some more search, I found this other issue and came to the conclusion that there was no solution except resetting docker completely. Hopefully, Digital Ocean has a button for rebuilding the VM.

Once the VM was rebuilt, the first thing that I did was to try to connect from my shell on my local machine. I had to clean up my known host file, but that was simple enough.

1
nano ~/.ssh/known_hosts

Once this was done, I just followed the steps I had documented in my previous blog post

Was I all done ?

Almost … I ran into another kind of errors this time. Processes kept getting killed on my VM.

1
2
3
4
5
6
INFO [cc536697] Running /usr/bin/env docker-compose -f docker-compose.production.yml run app bundle exec rake db:migrate as root@104.131.47.10
rake aborted!
SSHKit::Runner::ExecuteError: Exception while executing as root@104.131.47.10: docker-compose exit status: 137
docker-compose stdout: Nothing written
docker-compose stderr: Starting root_db_1
bash: line 1: 18576 Killed

After some more Google searching, I discovered that this time, the VM was running out of memory ! The fast fix was to upgrade the VM (at the extra cost of 5$ / month).

After increasing the memory (and disk space) of the VM, deployment went like a charm. Others have fixed the same issue for free by adding a swap partition to the VM.

The end of the story

I wasted quite some time on this, but it taught me some lessons :

  1. I should have taken care of cleaning up the old images and containers, at least manually, at best automatically
  2. I should write a script to provision a new server
  3. The cheap options always come at a cost
  4. For an open source side project like this one, it might be a better strategy to only use Docker to setup my dev env, and use free services like Travis-ci and Heroku for production
  5. Doing everything myself is not a good recipe to getting things done … I well past time I leave my developer hat for an entrepreneur cap
  6. In order to keep learning and experimenting, focused 20h sessions of deliberate practice might be the most time effective solution

5 Minutes Hack to Speed Up RSpec in Rails 5 Using In-memory SQLite

Here is the story : you have a Rails 5 app that uses RSpec, but your RSpec suite is getting slower and slower to run. You’ve already considered some solutions :

  • Use SQLite in memory for your test env.
1
2
3
test:
  adapter: sqlite3
  database: ":memory:"

That’s the most straightforward thing to do, but unfortunately, if you are sharing your test env with Cucumber, you might want to use a production like DB with Cucumber (PostgreSQL or whatever). So unless you are ready to setup a new env for cucumber (which I tried and don’t recommend) you’re stuck.

  • Use mocks. That’s surely going to work, it’s going to make your test hell of a lot faster ! It will also make your tests a lot more fragile and more expensive to maintain … If you want to read more about why I think mocks are a bad idea, just have a look at these posts.

The hack

Here is a third alternative, I’ve already written about it, but here it comes updated and tested for Rails 5 :

  1. Don’t change anything to your config/database.yml
  2. Obviously, you’ll need to add sqlite3 to your Gemfile
  3. At the beginning of your spec/rails_helper.rb, replace
1
2
3
# Checks for pending migration and applies them before tests are run.
# If you are not using ActiveRecord, you can remove this line.
ActiveRecord::Migration.maintain_test_schema!

with

1
2
3
4
5
# In order to keep the same RAILS_ENV for rspec and cucumber, and to make rspec
# faster, patch the connection to use sqlite in memory when running rspec
ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: ':memory:')
ActiveRecord::Schema.verbose = false
load "#{Rails.root.to_s}/db/schema.rb"

That’s it ! Run your specs … not bad for a 5 minutes investment !

Rails 5.1 (2017-03-29 Edit)

My fresh hack started to fail on Rails 5.1 ! If schema.rb is generated with the Postgres adapter, it is now incompatible with this injected Sqlite adapter. Here is a patch that removes the glitches :

1
2
3
4
5
6
7
# In order to keep the same RAILS_ENV for rspec and cucumber, and to make rspec
# faster, patch the connection to use sqlite in memory when running rspec
ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: ':memory:')
ActiveRecord::Schema.verbose = false
# load db agnostic schema by default. Needed to remove the ", id: :serial" from
# the table definitions to make it load on sqlite
eval(`cat #{Rails.root.to_s}/db/schema.rb | sed 's/,[^:]*: :serial\//g'`)

I admit this is getting a bit crappy, and I don’t know how long it is going to work …

One more thing …

If you need even more speed, you can now run your specs in parallel in different processes ! Each in-memory SQLite DB is bound to its process, so unlike a real PostgreSQL dev DB, you won’t get any conflicts between your tests ;–)

A Plain English Introduction to Paxos Protocol

A few weeks ago, I had to have a look at the distributed consensus protocol Paxos. Even though I know its purpose and I’ve built and used distributed systems and databases in the past, Paxos remains mind boggling at first !

The hard way

The best overall description I found is this answer by Vineet Gupta on Quora. After turning my head around it for a while, I finally gained the instinctive understanding which comes when you ‘get’ something.

As a way to both help others to understand Paxos faster and to burn all this in my own memory, I though it would he a good idea to illustrate it as a story (I was inspired by A plain English introduction to CAP Theorem which I found really instructive; I also later discovered that the original Paxos paper itself related the protocol using the metaphor of a parliament).

Once upon a time …

… there were 3 brothers and sisters, Kath, Joe & Tom, living happily. They lived far away, and it was not easy for them to meet and spend some time together. Neither did they have phone or internet, for this was a long time ago. All they had to discuss and share news was good old mail …

Unfortunately, one day, the worst happened : their parents die. All 3 are informed by a letter from the notary, telling them that they need to sell the family house in order to pay for their inherited debts. It also advises to use Paxos to agree on a price (Note : I never said the story was going to be chronologically sound !).

The happy end

As the oldest in the family, Kath decides to take things in hand, and starts the whole thing. She knows Paxos consists of 2 phases : ‘prepare’ and ‘accept’.

Prepare Phase

Kath sends a signed and dated price value proposal to her brothers, by mail.

Joe and Tom both receive the letter from Kath, they think the price is fair. In order to send their agreements back to Kath, they make a copy of the proposition, mark it as agreed, date it, sign it, and send it back.

Accept Phase

Joe lives a bit further away from Kath than Tom does, so correspondence between Kath and Tom is usually faster. Kath indeed receives the agreement from Tom first, she knows she can go on with the protocol straight away, because Paxos relies on majority, and not unanimity. In his letter, Tom agreed to the same price she proposed, so she just picks this one as the final price to agree on.

She sends new letters, called accept letters this time, to her brothers to finalize the agreement. In these letters, she specifies the price that they are agreeing on, plus the date at which it was first suggested (see Prepare Phase). When Tom and Joe receive the accept letter, they simply need to check that the time and price of the proposal to make sure it is what they agreed on, before sending back their final accept letter.

At the time when Kath receives the accept letters from her brothers, everyone knows that the price has been agreed.

Cover of the book "The Fairy Tales of the Grimm Brothers"

After

She then informs the notary on the agreed price. This one sends an information letter to the Kath, Tom & Joe. The house is sold pretty quickly, leaving the family out of financial problems for the rest of their lives …

Shit happens

That story went unexpectedly well ! Let’s see different variations about what would happen in real life.

Joe is particularly slow to answer

Joe has never been good at paperwork … he’s always out partying and having fun, and he does not want to bother answering letters. When Joe receives the prepare letter from Kath, he does not reply straightaway but leaves it on his desk to handle later. Meanwhile, Tom answers as soon as he got the letter. As mentioned before, Paxos relies on majority, as soon as Kath gets Tom’s answer, she can continue to the next phase. In fact, the accept phase also relies on majority, so she can continue to the end of the protocol if Tom continues to answer.

In this case, Joe would receive the accept letter before he sent his answer to the prepare letter, and would know that the consensus is moving on without him. He can try to catch up or not, but the consensus can be reach without him.

Tom wants to speed things up by becoming the master

Tom has always been the hurried brother. He does not like when things linger forever but prefers things to be done quickly. As soon as he receives the letter from the notary, he starts waiting impatiently for the prepare letter from his sister. Kath, on her part, takes a lot of time to settle on a price. Not knowing what is going on, Tom decides to take action, and to takes on the master role : he sends his own copies of the prepare letters. While these letters are in the mail, Kath finally settles on a price, and sends hers.

Joe gets Tom’s proposal first. Thinking that it’s a change in the plan, he responds straight away by signing the proposal and taking a copy for himself. The following day, he receives Kath’s proposal ! He’s a bit surprised, but hopefully, Paxos tells him exactly what to do in this situation. By agreeing to Tom’s proposal, he made a promise to stick to it whatever happens later. Here the date on Kath’s proposal is later than on Tom’s, so Joe is going to answer to Kath that he agrees but to to Tom’s proposal, of which he’ll join a copy.

After receiving the Joe’s agreement on his proposal, Tom has the majority, and should be able to end the protocol.

What about Kath ?

She should have received Tom’s proposal, and rejected it, because she had already proposed a later value. That will not prevent Tom to reach a consensus.

She should have received Joe’s agreement to Tom’s proposal. The same way, she might as well have received Tom’s agreement to his own proposal as an answer to hers. She’d get the majority of agreements, so she might then want to push on. For the accept letter, she must pick a value that has been accepted, in this case, it’s Tom’s proposed value ! Everything ends as expected as she’ll reach the same price as Tom.

Tom wants a higher price an becomes the master

Imagine Tom is obsessed about money ! When he receives Kath’s proposal, he’s outraged ! Believing the house has a lot more value than the proposed price, he sets on to act as a master in Paxos and sends his own proposal letters to his brother and sister.

Unfortunately, when they receive his proposal, they have already agreed to Kath’s older proposal, so they send him back a copy of it as an agreement. Having received agreements to Kath’s value only, he cannot push forward his value. Whether he continues his Paxos or not does not really matter, as he would reach the same value as Kath would.

River flood split between brothers and Kath

There’s a wide river that separates Kath from Joe and Tom. While they were trying to reach consensus, the river flood, cutting all communication between the brothers and their sister. Kath might abort the consensus as she won’t be able to get answers from the majority. On their side, Joe or Tom can takeover the consensus, take on the master role, and still reach a price, as they form a majority. As soon as the river would settle, the messages would arrive to both parties, eventually informing Kath that a price was accepted.

Lots of others

You can imagine zillions of ways in which the consensus between Kath, Joe and Tom could go wrong. For example :

  • Mail is so slow that Kath sends new proposals
  • One letter gets lost and arrives after Kath made a new proposal
  • Kath is struck by a lightning

Go ahead and execute Paxos step by step on all of them, you’ll see that whatever happens, Kath, Joe and Tom will reach a price.

More Formally

Now that you have an instinctive understanding of Paxos, I encourage you to read out the full explanation I found on Quora. Here is a extract with the protocol part :

Protocol Steps:

1) Prepare Phase:

  • A node chooses to become the Leader and selects a sequence number x and value v to create a proposal P1(x, v). It sends this proposal to the acceptors and waits till a majority responds.

  • An Acceptor on receiving the proposal P1(x, v1) does the following:

    • If this is the first proposal to which the Acceptor is going to agree, reply ‘agree’ – this is now a promise that the Acceptor would reject all future proposal requests < x
    • If there are already proposals to which the Acceptor has agreed: compare x to the highest seq number proposal it has already agreed to, say P2(y, v2)
      • If x < y, reply ‘reject’ along with y
      • If x > y, reply ‘agree’ along with P2(y, v2)

2) Accept Phase

  • If a majority of Acceptors fail to reply or reply ‘reject’, the Leader abandons the proposal and may start again.

  • If a majority of Acceptors reply ‘agree’, the Leader will also receive the values of proposals they have already accepted. The Leader picks any of these values (or if no values have been accepted yet, uses its own) and sends a ‘accept request’ message with the proposal number and value.

  • When an Acceptor receives a ‘accept request’ message, it sends an ‘accept’ only if the following two conditions are met, otherwise it sends a ‘reject’:

    • Value is same as any of the previously accepted proposals
    • Seq number is the highest proposal number the Acceptor has agreed to
  • If the Leader does not receive an ‘accept’ message from a majority, abandon the proposal and start again. However if the Leader does receive an ‘accept’ from a majority, the protocol can be considered terminated. As an optimization, the Leader may send ‘commit’ to the other nodes.

And here are the key concepts to map my story to this formal description of Paxos.

Story Paxos
proposal letter (and copy of) P(x,v)
Date (and time) sequence number

At the time of slow mail based communication, using the date and time down to the second is enough to build up unique sequence numbers. In our current time of digital messages, it’s another story, typical Paxos implementation assigns a different and disjoint infinite set of integers for every participant, it does not exactly follow ‘time’, but it’s enough for the algorithm to work.

What Happens to Non-Enthusiast Programmers in the Long Run ?

A few months ago, after receiving good feedback from my regular readers, I posted my latest article Is There Any Room for the Not-Passionate Developer ? on Hackernews and Reddit. I got a huge number of visits, a lot more than I typically get !

I also got a lot more comments, some nice, some tough, some agreeable and some challenging !

First, a summary

In this previous article, I wanted to contrast the different views about work/life balance in the software industry.

Some, like agile gurus and companies like Basecamp, and studies, strongly advocate for sane work hours. They explain that it results in greater productivity and healthy life.

On the other hand, the software field is always bubbling with novelty, and keeping up to date with technologies is by itself a challenge that takes time. For some companies, which might already be fighting for their survival against competition, it is almost impossible to grant some extra training time to their employees. The problem becomes particularly difficult when engineers get older, become parents and cannot afford to spend some extra time learning the latest JavaScript framework.

As a conclusion, I said that for most of us, it’s really difficult to remain a developer in the long run without the grit that only passion for programming brings. I encourage you to read it for more details.

What I learned from the comments

First of all, thanks a lot for all these, they were very valuable, they forced me to think even more about the issue.

People have been burnt !

The word ‘passion’ in particular, triggered engaged comments. As some pointed out, ‘enthusiast’ or ‘professional’ should be favored. It seems that some companies have asked their employees for unquestionable passion for their business (and not for engineering or programming) at the cost of the people’s own lives. As a commenter said, a lot of shops do not integrate the absolute necessity for their programmers to learn continuously in their business model. It made me kind of sad to feel once more this state of our industry.

As a result, people are weary of any statement of ‘passion’ in the workplace, and would prefer to be seen as very skilled professional, dedicated to keeping their skills up to date.

The particular question of France

I received some comments from all over the world, but my observations came from where I work : in France. Here, all in all, we have at least 7 weeks of paid leaves per year. It’s a lot more than in other parts of the world. I think it’s around 2 weeks in the US (other sources point the same fact). Imagine two companies, one from France, and one from the US. The one in the US can invest 5 weeks per year in exploratory learning (which can result in good things for both the business and the employee) while still producing as much as the french one.

Obviously, there are other parameters to take into account for overall productivity like hours per day, the effects of holidays or long hours on creativity, or funding … but here are some facts about software engineering in France :

  • 20% time policy, hackathons and other exploratory learning are extremely rare (I’ve seen it once in 15 years)
  • It’s slowly getting better, but if you remain a programmer in your thirties, you’re seen as a loser
  • France has no software powerhouse like Microsoft, Google, Apple …

This lead me to this open question : What’s the effect of the 7 weeks of paid leaves on the french software industry ?

By no means will I try to give an answer, I just don’t know. Plus, for those who might be wondering : I love my 7 weeks of holidays !

The conclusion I came to

Yet, I can try to draw a conclusion at the individual level. In France, if you’re not really enthusiastic about programming, you won’t put the extra effort off-the-job to learn the latest technologies. Within a few years, you’ll be ‘deprecated’, which will leave you with mainly 2 options :

  • become a manager
  • stick to your current codebase (and become completely dependent of your employer)

To me, the sad truth is that if you want to make a career as a professional developer in France, you’d better be ready to spend some of your free time practicing !