March 29, 2024
Let’s be honest, the African startup world has seen its fair share of crashes lately — and I’m not talking about app outages (although those were a thing for us too). I’m talking about companies running out of steam and shutting down. Now, I won’t name names, but some pretty well-funded players, companies with much larger funding than Safeboda, have not been able to make it.
So, what’s the common thread here? In my experience, it usually boils down to cash flow. You need to have money to pay the bills, and some startups get a big funding round and think they’re invincible. But startup leaders should know that without managing your cash burn the funds can eventually run out.
At Safeboda, profitability is a team effort, but from the tech team’s perspective, let me tell you what we did to be super efficient with our cash and keep the burn rate low. This, I believe, is a big reason we’re still up and running.
From Crash City to Uptime Paradise
When I joined Safeboda, things were rough to say the least. The app would crash for hours at a time, and our team was running on fumes trying to fix it. We were wading through a sea of 2.5 million lines of code, basically playing whack-a-mole with bugs.
Our motto? Fix it fast, get customers back online, then perform the root cause analysis to figure out why it broke. Makes sense — downtime means lost revenue and unhappy customers. But after fixing the immediate issue, the root cause analysis felt impossible. It was like plugging holes in a leaky dam; another one would spring up just as quickly.
This, I think, is a classic startup trap. You get funded, the team grows, the codebase explodes, and quality control gets thrown out the window. The easy solution? Throw more resources at the problem. Bigger servers, more storage, you name it. This “vertical scaling” might work temporarily, but it’s not sustainable. It burns cash like crazy. Especially for a startup CTO with limited funds, burning through cash like that is a recipe for disaster.
So, we decided to ditch the vertical scaling bandaid and took a gamble on a new approach: microservices architecture. Now, Safeboda isn’t exactly a tiny startup anymore — we have hundreds of thousands of loyal customers. I say this because I want to caution that for a brand new company with a small codebase, a microservices approach could very well be overkill. Monolithic is the way to go until you can really validate your model.
For Safeboda’s size and scalability woes, though, breaking down the Safeboda backend into smaller, independent services felt like the right call.
The Great Untangling — Untangling the Monolithic Monster
Our monolithic database was a beast — a tangled mess of millions of daily incoming ride events and who knows how many tables! It held everything: user data, ride history, driver information, wallet data, configurations, you name it it was there. Think of it as a disorganized filing cabinet that everyone threw everything into. Finding anything specific was an archaeological dig, and troubleshooting issues felt never ending.
So, how did we untangle this mess? We started by identifying natural boundaries in the data. Functional areas like user management, pricing, third party payment integration, wallet management, and ride processing became the cornerstones for our new data model. We also studied our usage patterns and identified complex and frequently accessed data sets. The more frequently used data became the building blocks for our microservices.
Keeping Our Heads Above Water in a Decentralized World
Breaking the monolith into microservices felt liberating at first, but decentralization brought its own set of challenges. Data consistency became a huge problem we had to tackle. Imagine each microservice having its own version of the truth — a recipe for disaster. To maintain data integrity, I had to be willing to embrace the idea of eventual consistency, which ensures that all data copies will eventually converge. It wasn’t perfect — there could be a slight lag between updates — but for our use case, it offered a good balance between performance and consistency.
Another challenge was handling transactions that spanned multiple services. We had to orchestrate these transactions carefully to ensure that either all changes succeeded or none did. A great example is trying to make a payment using a third party system like Mpesa. We can’t take your money from Mpesa but not ensure your payment is applied on our systems. Think of it like a complicated relay race — if one runner drops the baton, the whole race is disqualified. We utilized message queues and other distributed architecture patterns to manage these complex workflows.
Data duplication was another sneaky monster we had to wrestle with. Microservices can lead to redundant data copies if not managed properly. We had to really make tough decisions as to what our source data system of reference would be for a lot of corner cases. It isn’t always sunshine and rainbows, but with some careful planning and clever engineering, we have greatly reduced data inconsistency.
Lessons Learned from a painful process
Looking back, if we could do it all again, we would have invested even more time in upfront planning and communication. Microservices can introduce complexity, so ensuring all teams (engineering, product, etc.) were on the same page was crucial. Having a more well-defined migration strategy and clear communication channels could have saved us a ton of headaches down the road.
For other startups considering a similar move, my advice is this: don’t be afraid to break things down, but do it strategically. Utilize data analysis tools, and domain driven design to identify natural boundaries and leverage proven techniques like eventual consistency and message queues. Remember, clear communication and a well-defined plan are your secret weapons in the battle against monolithic mayhem. The microservices path may not be easy, but the rewards — scalability, maintainability, and cost savings — are definitely worth the fight.
The Payoff — Stability, Savings, and Avoiding Rioting Drivers
So what are the results three years into the journey? Those six-hour app outages? Gone (thankfully, no more driver riots!). We still have hiccups and unplanned downtime — that’s the tech life — but our core system uptime is consistently meeting our established levels of quality we track with our OKR’s. But the real win here goes beyond stability. This architecture switch saved us a cool 30% on infrastructure costs. You might think having a bunch of mini-services would be more expensive — each one needing its own database, additional code, and other infrastructure components. But here’s the magic:
Bye-bye Big Databases (and Big Bills): One giant XXL database? Expensive. Lots of smaller micro to small service-specific databases? Much cheaper. This way, we can allocate resources exactly where they’re needed. Our KYC database that is used by our users one time doesn’t need the same horsepower as our daily ride management system, so we can keep its database lean. No more overprovisioning and wasting money.
Tech à la Carte: With microservices, we can choose the right database technology for the job. Some services benefit from caching solutions like Redis, others need the power of Elasticsearch, and some are perfectly happy with a good old-fashioned Postgres database. This lets us leverage cost-effective options wherever possible.
Scaling Smart, Not Just Big: Say goodbye to vertical scaling’s one-size-fits-all approach. With containerized microservices, we can run lots of small, efficient services. If a service gets overloaded, we have systems in place to automatically scale it horizontally (think adding more servers) instead of just throwing more power at it vertically. This targeted scaling saves us a ton of money in the long run.
Look, the benefits of microservices are well-known in the tech world. But I wanted to share our real-life journey to show you that this approach can be a game-changer for startups. It’s not just about managing cash burn (although that’s a pretty big deal!). By breaking free from the monolith, we’ve gained agility, scalability, and development speed. This translates to a faster time to market, the ability to experiment more readily, and ultimately, a more competitive edge.
There’s no magic number for when to make the switch to microservices, but if you’re feeling the strain of a monolithic codebase, or if your infrastructure costs are keeping you up at night, it’s definitely worth considering. A deep dive into your resource allocation could reveal some serious savings opportunities — and unlock a whole new level of efficiency for your startup.
The tech world is constantly evolving, and embracing new architectures like microservices can be the key to staying ahead of the curve. Remember, the road to a successful startup is paved with innovation and adaptation. Don’t be afraid to break things down — sometimes, to truly thrive, you need to start from scratch (or at least, from a bunch of well-designed microservices).
March 29, 2024