How I Fixed 'Devicemapper' Error When Deploying My Docker App

A few months ago, I started continuously deploying my latest side project to a Digital Ocean box. If you are interested, here is the full story of how I did it. All was going pretty well until last week, when the builds unexpectedly started to fail. I wasn’t getting the same error at every build, but it was always the Docker deployment that failed. Here are the kind of errors I got :

1
2
3
4
5
6
7
8
# At first, it could not connect to the db container
PG::ConnectionBad: could not translate host name "db" to address: Name or service not known

# Then I started to have weird EOF errors
docker stderr: failed to register layer: ApplyLayer exit status 1 stdout:  stderr: unexpected EOF

# Eventually, I got some devicemapper errors
docker stderr: failed to register layer: devicemapper: Error running deviceCreate (createSnapDevice) dm_task_run failed

You can read the full error logs here.

That’s what happens when you go cheap !

After searching the internet a bit, I found this issue which made me understand that my server had ran out of disk space because of old versions of my docker images. I tried to remove them, but the commands were failing. After some more search, I found this other issue and came to the conclusion that there was no solution except resetting docker completely. Hopefully, Digital Ocean has a button for rebuilding the VM.

Once the VM was rebuilt, the first thing that I did was to try to connect from my shell on my local machine. I had to clean up my known host file, but that was simple enough.

1
nano ~/.ssh/known_hosts

Once this was done, I just followed the steps I had documented in my previous blog post

Was I all done ?

Almost … I ran into another kind of errors this time. Processes kept getting killed on my VM.

1
2
3
4
5
6
INFO [cc536697] Running /usr/bin/env docker-compose -f docker-compose.production.yml run app bundle exec rake db:migrate as root@104.131.47.10
rake aborted!
SSHKit::Runner::ExecuteError: Exception while executing as root@104.131.47.10: docker-compose exit status: 137
docker-compose stdout: Nothing written
docker-compose stderr: Starting root_db_1
bash: line 1: 18576 Killed

After some more Google searching, I discovered that this time, the VM was running out of memory ! The fast fix was to upgrade the VM (at the extra cost of 5$ / month).

After increasing the memory (and disk space) of the VM, deployment went like a charm. Others have fixed the same issue for free by adding a swap partition to the VM.

The end of the story

I wasted quite some time on this, but it taught me some lessons :

  1. I should have taken care of cleaning up the old images and containers, at least manually, at best automatically
  2. I should write a script to provision a new server
  3. The cheap options always come at a cost
  4. For an open source side project like this one, it might be a better strategy to only use Docker to setup my dev env, and use free services like Travis-ci and Heroku for production
  5. Doing everything myself is not a good recipe to getting things done … I well past time I leave my developer hat for an entrepreneur cap
  6. In order to keep learning and experimenting, focused 20h sessions of deliberate practice might be the most time effective solution

Comments