Serverless — Moving from virtual machines to app engine for deployment and improving the process in the way
This series of posts is about how I went from deploying everything on a single VM to using separate managed services. While I was migrating, I also had another goal — improving my process and having a cleaner codebase.
The Problem
Traditionally, I used to develop the applications on my local machine and when it was time to deploy them, I tried to replicate my local setup on a virtual machine. I used to spawn EC2 instances and configure everything either manually or using an orchestration tool like Ansible. Soon, I realized that I am repeating a certain set of tasks every time I had to deploy — building and deploying a frontend and backend, making changes to the database, etc. In a multi-environment case where I had a production setup, a staging setup and my local development setup, all these tasks had to be done again and again. This is a pain for a single person and drains your energy every time you have to do it. Of course, I could write scripts (and I did that too) but then I had to write scripts for every project I worked on. Things are slightly different every time.
I had these major problems:
- Repeating tasks when deployment
- Deployment in multi-environment setup and keeping everything isolated
- Avoid writing scripts for each project (to automate the first two)
- Slow response time (since everything was on a machine sitting somewhere around the globe)
The Plan
CI and CD
I needed a system which would deploy automatically once I push my changes. CI for merging my feature branches and CD for deployment on sandbox and production servers. There should not be any human intervention after I push my changes to Git.
A Strong Test Suite
To achieve this, I needed a strong test suite. Earlier I used to plan to write tests but never wrote them. This needed to be changed. It’s a mindset change and so to force myself to do it, I moved to test-driven development (TDD) and started by setting up a test suite for my project before I actually started writing code.
Migration Framework for the database
I also used a relational database (MySQL), so maintaining and migrating the schema was another challenge. I needed some sort of migration framework which would run all the migrations when the code is deployed. This should happen automatically as well and on all the environments.
Local, Sandbox and Production
I need to take care of deployment environments in the code. Usually, I first used to write code and develop on my local setup and when I was happy with what I have done, I would deploy (not every project I work goes live, but some do). Code used to be tightly coupled with my setup — database connections, filesystem path, and all the other configurations. I needed to think about multi-environment set up from day 0. De-coupling this becomes messy and I could never have modular clean code.
With all these, I was ready to start. I evaluated AWS Elastic Beanstalk and Google App Engine and selected the Google ecosystem (just because it was cheaper even though offered fewer features as compared to AWS). In the next series of posts, I’ll walk through my complete setup while I was developing a ToDo application.
- Deploying my Flask application with Google App Engine and setting up triggers for continuous deployment
- Setting up the test suite
- Isolating production, sandbox and local
- Database Migration Framework and connecting with Cloud Database
- Frontend and deployment on Cloud CDN
- Integrating other services — Cloud Storage, Firestore, etc
