A few months ago, I started continuously deploying my latest side project to a Digital Ocean box. If you are interested, here is the full story of how I did it. All was going pretty well until last week, when the builds unexpectedly started to fail. I wasn’t getting the same error at every build, but it was always the Docker deployment that failed. Here are the kind of errors I got :
1 2 3 4 5 6 7 8
You can read the full error logs here.
That’s what happens when you go cheap !
After searching the internet a bit, I found this issue which made me understand that my server had ran out of disk space because of old versions of my docker images. I tried to remove them, but the commands were failing. After some more search, I found this other issue and came to the conclusion that there was no solution except resetting docker completely. Hopefully, Digital Ocean has a button for rebuilding the VM.
Once the VM was rebuilt, the first thing that I did was to try to connect from my shell on my local machine. I had to clean up my known host file, but that was simple enough.
Once this was done, I just followed the steps I had documented in my previous blog post
Was I all done ?
Almost … I ran into another kind of errors this time. Processes kept getting killed on my VM.
1 2 3 4 5 6
After some more Google searching, I discovered that this time, the VM was running out of memory ! The fast fix was to upgrade the VM (at the extra cost of 5$ / month).
After increasing the memory (and disk space) of the VM, deployment went like a charm. Others have fixed the same issue for free by adding a swap partition to the VM.
The end of the story
I wasted quite some time on this, but it taught me some lessons :
- I should have taken care of cleaning up the old images and containers, at least manually, at best automatically
- I should write a script to provision a new server
- The cheap options always come at a cost
- For an open source side project like this one, it might be a better strategy to only use Docker to setup my dev env, and use free services like Travis-ci and Heroku for production
- Doing everything myself is not a good recipe to getting things done … I well past time I leave my developer hat for an entrepreneur cap
- In order to keep learning and experimenting, focused 20h sessions of deliberate practice might be the most time effective solution