Choosing a Tech Stack for Your Startup - Part 1: Client Apps

When I started FastBar in 2014, one of the first major decisions I needed to make was: which tech stack we were going to use? If you're starting a tech company, chances are you'll need to make this decision as well. Hopefully my decision process and lessons learnt can help you as well. 

This is the first in a 2 part series - in part 1, we'll deal with the client side of our app. I'll explore the history and decision making process behind the technologies I chose, talk about where we are at today, along with lessons learnt.

 By way of quick background, FastBar provides a cashless payment system for live events - think concerts, music festivals, beer and wine events etc… When an attendee arrives at an event, they get a wristband that's connected to their credit card. At the bar, they tap to pay in less than a second, and at the end of the event we automatically close them out with their card on file and send them an electronic receipt. I've previously posted some more details on the beginning of FastBar and why I started it.

 Fundamentally, FastBar is a payment system for events and is composed of 3 key components: a Registration App, Point of Sale (POS) App and the backend website and API. Since I first started FastBar, the product has evolved significantly, and the technology stack along with it.

High Level Components

 From a high-level perspective, here's what FastBar looks like today:

FastBar - High Level Components.png
  • Registration App - this is an app used onsite at events by event staff to swipe credit cards and issue RFID (or more technically, NFC) wristbands to attendees. It's also used for a variety of other maintenance tasks required onsite at events, like adding a new wristband to a attendee's account, disabling existing wristbands, loading cash onto wristbands etc…

  • POS App - used at events by event staff to sell stuff to attendees - whether that’s beverages, food or merchandise. Event staff enter what the attendee wants to purchase, then the attendee taps their wristband against an RFID/NFC reader connected to the POS to complete their transaction

  • The backend Web App and API have 4 key areas:

    • Attendee area: this allows attendees to check their tab during and after the event, request an email receipt, and register themselves online before they arrive at the event

    • Event organizer area: otherwise known as our Event Control Center, this is used by event organizers to setup and manage all aspects of their event, including configuring products, pricing, menus and view all kinds of reports

    • Admin area: used by FastBar staff to administer the system

    • API: used by our Registration and POS Apps, along with some 3rd party systems that integrate with FastBar

In this post, we’re going to focus on the client side technologies:

FastBar Components - Client Hilighted.png

FastBar's Early Development and Client-Side Tech Choices

The first app we built was the POS app. The prototype was created at Startup Weekend, and we went with iOS primarily because one of the guys on the team had a background in iOS development and was familiar with it. Skillset dictated our choice.

Android vs iOS

After Startup Weekend, when we got serious about moving FastBar forward and turning it into a real company, we needed to decide what platform we were going to use for the POS app. We went with Objective-C on iOS primarily because iOS was the most prevalent and consistent tablet platform on the market.

At the time (mid 2014), Android commanded a higher share of the global market for tablets and mobile devices 48% vs 33% for iOS:

However, when you narrow that to just tablets, iOS was significantly in front at 71%:

Narrow that down even further to North America, the primary market we cared about, and iOS is even more dominant with nearly 77%

Incidentally, here's how the tablet market share in North America has changed in the past 5 years:

In other words, not much.

Not only was iOS dominating in market share, but it was far less fragmented. As of June 2014, there were only 7 different iOS devices, all manufactured by the same company, Apple:

  • iPad

  • iPad 2

  • iPad 3rd gen

  • iPad Mini

  • iPad 4th gen

  • iPad Air

  • iPad Mini 2

The only real difference between these devices from a development perspective was the screen size and resolution. Building apps to run on all of these devices was straightforward.

By contrast, the Android market was highly fragmented with hundreds or probably thousands of devices available from tons of different manufacturers. Trying to build an application that will work consistently across all Android devices was a lot harder.

Given iOS was dominant and consistent, this meant:

  1. Plenty of dev - There were plenty of developers around who knew how to build apps for iOS

  2. Customer familiarity - our potential customers often times already had iOS devices. It would be easier for us to convince them to use a platform they were already familiar with rather than introduce something new into their environment

  3. Large rental market - there was a large market for renting iPads, meaning we didn’t need to incur a lot of up-front capital expense, instead, we could own a handful of devices and rent the additional ones we needed for an event

  4. Lots of accessories - there was a lot of choice in accessories for iPads, like point of sale stands. Less so for Android devices

From a pricing standpoint, although it was possible to find really cheap Android tablets (around $50-80), by the time you went for something that was higher quality and similarly speced to the iPad, you were looking at a similar price point.

With the hardware decision made, we continued to evolve the first prototype and stuck with Objective-C as the language of choice. Later in 2014 Apple release Swift, so we started to build some parts of the app in Swift as well.

The First Live Event

In August 2014 we executed our first live event where we processed payments. This made use of our POS App, and to register attendees we used a web-based registration app running on a laptop with a USB connected RFID reader. We quickly realized that the web-based app running on a PC was not a good option for us. We needed to be mobile, and a phone form factor was going to be a much better solution. 

At that point we set out to build the first version of our Registration app, again using Objective-C and Swift running on iOS.

Native or Not?

In early 2015 we decided we needed 3rd app, an Attendee app, that would be used by attendees to check their tabs during the event. Since this app was attendee facing it needed to be deployed on their own devices and we had no control over what hardware they'd be using (unlike the POS and Registration apps), that meant it needed to be multi-platform. Building for Android and iOS would allow us to cover well over 90% of the market, which was good enough for us as a startup. Back at the time Windows Phone still kind of existed, but was on the decline, and quite frankly just not relevant for us.

We had a decision to make: do we go native and build separate apps for each platform, or use a use a cross-platform technology that would allow us to target both Android and iOS. Both options had their pros and cons. If we went native, we'd have full access to the underlying platform, and be able to create the best experience possible for users. However, we'd be maintaining 2 different code bases, which is tough, especially for a resourced-strapped startup. If we went cross-platform, we'd have a single code base to maintain so development would be quicker. On the flip side, cross-platform technology wasn't as evolved back then as it is now, couple that with the face that no matter what, you will never be able to create the most optimized experience using a cross-platform technology compared to native tech. By definition, you're catering for the lowest common denominator.

Betting on Xamarin

We decided that the benefits of some kind of cross-platform tech far outweighed the cons, and our requirements were fairly basic and something that could fit within the realm of what cross-platform tech was good for, ie a fairly standard "business" app which primarily required forms and data synchronization. In our case, we also needed to interact with local hardware.

We decided to give Xamarin a shot. If you're not familiar with it, Xamarin allows you to build native applications for iOS (using Xamarin.iOS) or Android (using Xamarin.Android) using C# and .NET. The cool thing about Xamarin is that if you factor your application correctly, you can re-use any code that is non-platform specific. For example, a lot of our app dealt with network communication, data synchronization to the server and data manipulation locally (ie reading from and writing to our local SQLite database). Essentially all of that could be re-used across both iOS and Android without changes. In addition, in mid-2014 Xamarin released a technology called Xamarin Forms which allows you to build UI screens in a cross platform manner. This meant we could write a screen once, and it would run on iOS and Android without us having to build separate screens for both platforms. Xamarin Forms is also smart enough to render controls in a platform specific manner, for example, when we define a date picker on the screen, it will look like a native iOS date picker when running on iOS, and a native Android date picker when running on Android.

Essentially, Xamarin allowed us to build an app that was both native and cross platform - it's kind of like the best of both worlds.

I was initially cautious of Xamarin - I'd had some previous negative experiences when using an earlier version of Xamarin a couple of years prior, but one of the guys on our team had used it recently, was familiar with it, and assured us it had evolved significantly in the last couple of years.

In addition, it would allow us to use the same language on the client apps that we were using on the backend, C# (more on that in part 2 of this blog post), and that meant it was easier for developers who'd be working on the client and server to transition between codebases given the language was the same.

Our bet on Xamarin proved to be a good one. We rolled out the Attendee app on both Android and iOS simultaneously. It took us around 20% extra time to release apps for both iOS and Android vs what it would have taken for us to build for a single platform if we were doing it natively.

First Major Rebuild for Registration and POS

Towards the end of 2015, we decided it was time for our first major re-build. We were constantly evolving the code bases for both the POS and Registration Apps, but were starting to find it was harder for us to add new features and harder to find and fix bugs. Both of these apps had evolved from prototype code, and were never designed to last as long as they had. We were in a position where were just couldn't move as fast as we wanted to.

Our experiment with Xamarin for the Attendee app had been a success, so we felt confident picking Xamarin as our primary development platform moving forward. The Registration app was the highest priority for us to rebuild, as it needed some significant changes. We decided to tackle it first, with a view to building as many re-usable components as possible for use in the POS, when we got around to rebuilding that as well.

We rolled out the first version of our new Registration app in early 2016 and continued to evolve it throughout the year, eventually sunsetting the legacy Registration app when the new one had proven itself at live events and had subsumed all features of the legacy version.

Towards the end of 2016, we started work on the fully re-built version of the POS app, which initially rolled out early 2017. Similarly, we maintained the legacy version of the POS until the re-built version was battle-tested and ready for prime time.

Side note: Sunsetting the Attendee App

Also towards the end of 2016 we decided to sunset the Attendee app. We always knew that we could never force attendees to download an app at an event - it's a poor user experience to require this, and also impractical in an event environment where connectivity could be scarce or nonexistent, ie 5000 people each trying to download and 80Mb app with little to no connectivity is a nonstarter.

The key features that an attendee needed were straightforward, so we decided to focus on just delivering those features via a mobile friendly website and SMS, which was a simple and more universally accessible solution.

Code Reuse Across Multiple Apps

We found that reusing code across both the Registration and POS app was tougher than we originally thought. It wasn't until the Registration app was re-built and we started work on the new POS app that we realized a lot of decisions we made in the implementation of the Registration app were not conducive to reuse in another app, so some refactoring was required.

Code reuse is an interesting beast - on one hand we wanted to move quickly on the development of the Registration app and didn’t want to invest too much time in over-complicating the design or over genericizing things that might need to be reused one day, or might not. That violates the YAGNI principle: Ya Aint Gunna Need It. On the other hand it doesn't make sense to duplicate logic and components in two different apps, which just leads to inconsistency from the user experience and is harder to maintain and debug . That violates the DRY principle: Don't Repeat Yourself.

It's a tough balance to find the right level. Over time, we got better at determining what should be reused across an app and how to structure that code, and what we should keep separate for each app. It's something that we're constantly monitoring as we build new features.

Side note: Microsoft's Acquisition of Xamarin

In February 2016 Microsoft acquired Xamarin - I felt that was a good sign in terms of the longevity of the technology. Since then, I think Xamarin has continued to develop nicely, and looking back, I'm happy with the bet we made on Xamarin.

Where We're at Today

Today, FastBar has 2 client apps: the Registration app:

And the POS app:

They're both built in C# using Xamarin, and both target iOS only (at the moment).

Almost all screens are built using Xamarin Forms, except for 1 screen in the POS, the main screen:

IMG_0102.PNG

(Note: it is possible to have screens built in Xamarin Forms and screen built in Xamarin iOS or Android exist side by side in the one app)

Originally, this screen was built in Xamarin Forms, but we found the responsiveness when tapping the screen wasn't quite where we wanted it to be. Sometimes, when tapping the screen in quick succession taps would be missed and those items would not end up in the shopping cart. We decided to build natively in Xamarin iOS to achieve the best possible performance. It seems slightly quicker and more responsive than it's Xamarin Forms counterpart. Having said that, I'm sure Xamarin Forms has improved, and nowadays it might be as performant as the native implementation. What we have now works well, so there is no need for us to change it.

Since both apps are quite similar, both visually and from an underlying logic standpoint and how they interact with the server, we have a high-degree of code and UI screen re-use across both of the apps.

As of the date of this post, we don’t yet target Android, although it is reasonably trivial for us to do so.

We don’t have an Attendee app any more, it’s not really necessary, although may choose to re-introduce one in the future. We may also add an Event Organizer app that will allow Event Organizers to manage their events when the event is in operation. Today they can fully manage their event from our mobile friendly website, but a app running locally would allow us to provide a better experience, especially operating at event environments with poor internet connectivity.

Lessons Learnt

Optimize for Productivity

When choosing what technology you want to use for your startup, optimize for productivity.

Let's say there are 2 technology stacks: A and B. Your team has background and experience in A and thinks it will take 12 weeks to build your minimal viable product (MVP). Your well-meaning, but technologically religious friend, let's call him Bob, says "You're crazy, why would you choose A, B is so much better. I could build it in 8 weeks using B!".

Maybe Bob is right.

Maybe B is way more productive than A.

Maybe, given 2 capable teams, building this solution out using A will take 12 weeks and B will take 8 weeks.

But probably it won't.

And it will almost certainly will take your team, familiar with A, longer than 8 weeks to build something in B, because they need to get up to speed with the tech.

When thinking about productivity, definitely factor in things like language productivity, frameworks available etc… and also be sure to factor in the experience and skill set for your team. Choose what's going to be most productive for you.

Choose Something Popular

Quite frankly, it doesn't really matter what technology stack you choose, within reason.

For the most part, a good team is going to be able to build out your solution in Objective-C or Swift (native iOS) or Java (native Android) or Xamarin or React Native or Ionic or Phone Gap (cross-platform-ish technologies). Just make sure you choose something that is popular, meaning:

  • It is used by many applications

  • You can easily find and hire developers who understand it

  • It is actively maintained

Popular frameworks have companies, or more commonly, an open source community, behind them who keep building new features, keep fixing bugs, keep plugging security holes etc… all on your behalf! You don’t want to take this burden on by yourself, you have better things to do. Choose a technology that's popular.

Choose the Simpliest Thing That Works

Follow the old acronym, KISS: Keep It Simple Stupid.

Engineers like to engineer. It's in their nature. This means it very easy for them to over-engineer a solution. Choose the simplest solution that will solve your problem, and work with it for as long as possible.

For us, we ended up sunsetting one of our apps, the Attendee app, when we realized it was not the simplest solution possible. It was extra code to maintain that we didn’t really need, it was not something we could force on users, therefore we needed a more universally accessible solution, like a mobile friendly website. Given we needed the website anyway, did we really need the app? It was unnecessary, so we killed it.

Now, we may bring it back at some point in the future if it becomes necessary or advantageous, but for the moment, the mobile web solution we have is simple, it's easy, and it works. Choose the simplest thing that works.

Favor Cross-Platform Tech

If you're building for multiple platforms, do yourself a favor and use some cross-platform tech.

If you know for certain that only care about Android and your team has experience in Java-based Android development, awesome, refer to "Optimize for Productivity" above and go for it.

Likewise, if you're only building for iOS and you've got the experience in Swift or Objective-C, good for you, fire up Xcode and get crackin'.

But if you're looking to build apps for multiple platforms, like iOS and Android, do yourself a favor and go with some kind of cross platform tech so you can get as much code reuse as possible. We went with Xamarin and it has worked well for us, but whatever you choose, you don’t want to be implementing the same features 2 different ways in 2 different platforms, dealing with 2 different sets of bugs to fix  etc… That's a nightmare, and you've got better things to do.

Also, if you think there is a high probability that you'll need to go for multiple platforms in the future, do yourself a favor and choose some cross platform tech to begin with. It will probably be just as quick to build your app as it would be to do it natively, and now you're got the added flexibility that you can add that other platform if and when you need to later on. Favor cross-platform tech.


Stay tuned for part 2 of this series where we'll explore server side technology choices…

Starting FastBar

In 2014 I started a company called FastBar with the goal to solve the problem of long bar lines at events.

At the time, I had a side business, Bonza Bash, throwing large social events in Seattle. We'd host big Halloween and New Year's Eve parties, along with summer parties and some other events throughout the year. These events would range in size from 1,000-3,500 or so people. Dealing with a large volume of people like that, there was one key problem we faced over and over again: long bar lines… and everyone hated them.

For attendees, long lines compromised their experience at the event. People don’t go to a Halloween party to wait in line for 20 minutes to grab a beer - they're there to enjoy the entertainment, to spend time with their friends and to make new ones - perhaps lifelong friends, or perhaps just for the night.

Venues don't like long lines either. Long lines mean they're losing out on sales and it's harming their reputation. No venue wants to be known as the place where it takes 20 minutes to get anything at the bar.

And finally, for us as the event organizer, it sucked as well. We'd get complaints from attendees who were understandably upset, and typically we'd be signed up for a food and beverage minimum with the venue. If sales didn't hit a certain level, we'd be responsible for making up the difference, and when the event is already running on thin margins, that's a real issue.

Everyone hates long bar lines at events

We'd tried everything we could to solve the problem:

  • Cash only - this was generally quicker than credit cards at the bar, and would work even when internet connectivity sucked, which was most of the time. However, there were many challenges with using cash as the only payment mechanism. For instance, people were starting to carry less and less cash with them, we'd need to hire extra security to deal with all of the cash drops and securely transporting cash, we'd usually need to dedicated 1/3rd of the labor behind the bar to managing the cash, ie 2 bartenders per 1 cashier. We even had times when the ATM would run out of money, leaving people unable to get more cash, and unable to buy anything.

  • Credit card machines - old school credit card machines would be expensive to rent, and would require a phone line (more common 5 years ago, less common these days), or reliable WiFi for payment processing, none of which were typically available to us at events. Phone-based or iPad based point of sale systems like Square were starting to gain in popularity, but they too needed a reliable WiFi connection, and couldn’t deal with working offline. In our event environments, internet connectivity would range from non-existent to poor. Typically, accepting a credit card would take around 45-90 seconds, if the device worked at all

  • Drink tickets - these were quite efficient at the bar, but had their own issues. Attendees could quickly exchange the ticket for a drink, streamlining the process at the bar, however, we'd need different color drink tickets for different drinks (eg green = beer, blue = wine, white = spirits), we'd need to group all of the prices together to try and make it simpler for attendees and bartenders (doesn't make sense for anyone to be dealing with 12 different colors of drink tickets), and they were a total hassle for the attendee. You need to line up, buy the tickets you thought you needed, and then go to the bar. If you get to the front of the line and had a green ticket, but wanted a wine which needed a blue ticket, you'd have to get back in the drink ticket line, then back in the bar line. If you overbought drink tickets, the venue usually would not provide a refund, so you're wasting money. If you under bought drink tickets, you'd need to go back to the drink ticket line to get more tickets, then line up at the bar. It was a crappy experience

  • Adding more staff - this was a common mitigation strategy, but it increased the cost pretty substantially and sometimes there just wasn't any more room to physically add more bartenders behind a given bar

None of these "solutions" (I use the term loosely) were any good. There had to be a better way.

Digitizing the drink ticket

Drink tickets were the closest to a viable solution, so I started thinking about how I could digitize them. What is there was a way I could create some kind of electronic wristband and have the drink tickets pre-loaded on there? Could I eliminate cash and credit card handling at the bar and make the payment process super quick so the only thing the bartenders needed to worry about was making drinks?

This idea bubbled around in the back of my head for a few years. Somewhere around early 2014 I read an article about the Disney Magic Band. Disney World in Orlando had rolled out this new technology: after you booked your holiday, you'd receive a wristband in the mail, you'd then use that wristband to check-in to your hotel, open your hotel door, gain access to the park and pay for stuff while inside. Not only that, but it also acted like a localized GPS within the park so Disney knew where guests were at any time, and how they moved through the park. This enabled cool scenarios like quickly tracking down lost kids, or if you'd booked into Space Mountain at 1pm, but were walking near it at say 11am and it was empty, Disney could send a push notification to your phone to see if you wanted to go to Space Mountain right then, instead of waiting until 1pm.

At the time, I read Disney spent around $1.5B on implementing this, so I'm sure that figure has gone up substantially over the past 5 years. This got me thinking - what if there was a way to create a similar concept, but heavily focus it on payments, and make the technology cheap and broadly accessible to events all around the world.

I started asking some of my other friends who also produced events if they had similar problems with long lines at the bar and poor mechanisms for payment. They did. I described the solution I was envisaging and got their feedback. They loved it, and based on feedback, I started to refine the idea.

Startup Weekend

In March 2014 I decided to pitch this idea at Startup Weekend. I wanted to work on the idea for a weekend to see if it had legs, and determine if I should bother devoting any more time to it. I remember standing there in the pitch line, waiting for my turn to pitch, holding a large piece of paper and a marker. Everyone who was pitching needed to write the name of their "company" on the piece of paper, which people would later vote on by placing sticky notes on said large piece of paper. While I was standing there a dozen or so people deep waiting to pitch, the name "FastBar" popped into my head, so I wrote it down.

Long story short, I pitched the idea, built a team and we worked on it for the weekend. When it came time for the final pitches on Sunday, we came in 2nd of around a dozen teams.

Startup Next

After Startup Weekend, I continued to work on the idea with a couple of guys from the team. We went through a 6 week program called Startup Next run by Dave Parker. It was an awesome program, really cheap (I think it was around $200 either for the team or per person - regardless, a steal for what we got out of it), super valuable and ultimately provided us with some great education that helped us navigate the early stages of starting a company.

9 Mile Labs and the Official Start of FastBar

After Next, we applied for an accelerator in Seattle, 9 Mile Labs. Around this time, we had to decide if we were all serious about pursuing FastBar full time or not. The guys who I was working with at the time decided it wasn't for them, so we parted ways, and I continued on.

I ended up getting into 9 Mile Labs as a solo founder, a whole other topic I'll save for a future post, and FastBar was born.

The Second Post Has Been a Long Time Coming

I wrote my first blog post back in 2014. Ugh - that was a while ago.

At the time, there was a loose intention to write 1 post a week or month or something like that, I don’t recall the specifics. Clearly, that fell by the wayside and never happened.

Starting a regular blog has been something on my radar for a while. Back when I was at Microsoft (2000-2007) and blogging started to become a thing, my peers, my manager and customers were constantly prodding me to stat a blog. It was never a priority for me, so it never happened.

These days people often ask me for advice around starting a company, choosing technology or any other number of random topics I might be somewhat well-versed on. I'd respond, sometimes with a copious amount of detail as to what I thought and why, and they'd say something to the effect of "you should make this into a blog post". So here we are.

First order of business - selecting a platform. As a technologist, before I choose a technology, I like to investigate it, play around with it, kick the tires a bit. Then select what I think is the best tool for the job. Over the years I've spun up many websites for myself and for various businesses, both my own and others. I've tried Wordpress either hosting it myself (on various platforms like GoDaddy or Azure) or the hosted version, Squarespace, various other pieces of open source blogging software, and rolling my own.

After considering all of these options for this site, I decided to go with Squarespace. It was a pretty easy decision. Last year I re-built FastBar's public facing website using Squarespace. It was a pretty good experience, it's simple and easy to use, templates look good and most importantly it means I don’t have to worry about things like backup, upgrading the underlying software and plugins, renewing my SSL certificates etc…

There are some pitfalls, but overall, I think the pros, primarily convenience, outweigh the cons, namely things missing basic features like version control, less flexibility compared to self-hosted Wordpress and the fact that I could find a cheaper hosting solution myself.

Troy Hunt says it well:

Hosting websites is like having kids; they’re continually attacked by nasty things, they need ongoing care and as good an idea as they may seem at the time, they end up costing you a heap more than you planned. And no, I don’t care that [whatever your favourite is] only costs 3 cents a month because that’s not what matters; time is the commodity that’s most valuable to me now. It’s not just time in terms of hours actually spent, it’s needing to be ready to patch any nasties, managing (and testing) backups, installing updates so you can leverage new features and so on and so forth. Let’s be overly optimistic about it and say all that only takes 2 hours a month - what’s that worth to you? Go on, put a dollar figure on it and consider what you could charge for the time plus what it’s worth to simply not have to think about it
— https://www.troyhunt.com/its-a-new-blog/

So, for now, here we are. Squarespace, please take my $12 a month and make it so I never again have to create and manage my own backups, renew my SSL certificates and update my own plugins - for this site as least.

Let's see how long this blogging thing lasts…

Why the cloud can't be trusted

Late the night of June 19th I was chatting with one of my developers and he mentioned that our hosted source control provider, Codespaces, was down. I checked the website, and sure enough, nothing. At first, it didn’t seem like anything new, from time to time Codespaces would go down (more often than I'd like) but it would eventually come back up and everything would be back to normal.

I figured I'd check their Twitter feed to see if there was any details on what was happening and an ETA for it to be back online:

The Codespaces website they were pointing to was down so I couldn't get any more information there, but some quick googling revealed that the nightmare cloud security scenario had just occurred to my hosted source code provider. From reading various articles online it became clear that all or most of Codespaces data had been deleted. Ouch.

An attacker orchestrated a DDOS attack, gained control of Codespaces' AWS account and tried to extort money. When Codespaces didn't pay, the attacker started deleting assets in the AWS account. That includes all the machine images, EBS volumes, backups, customer source code repositories and snapshots. Almost everything was deleted, essentially wiping out the business along with all of their customers precious data.

Fortunately I was one of the lucky ones. By chance, my repository was hosted on one of their older nodes that survived the attack, and I was able to get them to send me a link to download a dump of the repository. It was an intense 48 hours while I contemplated next steps in case my source code repository containing 3.5 years worth of development history was lost.

As a side note, I recall an email from Codespaces some time ago with an offer to upgrade to their new SVN hosting. It didn’t seem super important and I never got around to it. I'm normally all about the upgrades, but that's one upgrade I'm very glad I didn’t do :)

During the 48 hours when I was contemplating the worst case scenario for Shindigg, I realized that it wasn't actually that bad. The latest version of the code is available locally on our development and build machines so at worst we'd lose our history, but still have the latest version. Although being able to look back in time at source code history isn't something that you need everyday, it's important to have when you need it. Losing history is a hassle, but not crippling.

So, what can we learn from this?

Lesson 1 - Do Not Trust Your Data To Any Single Cloud Provider

The first and biggest lesson is that you should never trust your data to any one provider. No matter who the provider is or what kind of redundancy they have in place, never trust your data to a single provider.

Whether you're an individual, a business using hosted services, or a business providing hosted services to others (and probably using someone else's hosted services in the process), you've got to take responsibility for your own backups.

I made a mistake trusting our source code repository data to Codespaces and relying on them to properly operate their business and back-up their data, which they failed to do. Our most important data, the latest version of the source, was replicated in several places, so it was safe, but the repository (and therefore history) was vulnerable.

I fixed that by choosing a new hosted source code control provider that has a feature to automatically create backups and send them to my S3 account, thus giving me redundancy across 2 different providers. If my source code control provider gets wiped out, I've got a backup on a completely different system.

At Shindigg, we use a combination of self-hosting and a variety of other services including Azure, Cloudinary, Stripe, Mandrill etc… The most important data for us is our core database, and it gets backed up every 10 minutes and those backups are immediately uploaded offsite to a completely different provider. Which brings me to my next point…

Lesson 2 - Offsite Means At a Different Cloud Provider

Codespaces claimed to have offsite backups. Technically, the data was probably replicated to different AWS data centers, but it was still all in AWS, and it was all accessible through a single AWS account. That's not really "offsite" when it comes to the cloud.

In the cloud, offsite doesn't just mean relying on the cloud provider to have the data in a different physical location - you need to have critical data backing up at least to a separate account but really to a completely different provider.

At a minimum what Codespaces should have done is have their backups pushing to a separate AWS account, but ideally send backups to Azure, Rackspace or any other cloud storage provider. That way when their main AWS account was compromised, they wouldn't have lost everything. It may have taken them a couple of days to recover from back-ups which would have been bad, but having almost all of your customer data and therefore your business irrevocably wiped out is significantly worse. Which brings me to lesson 3...

Lesson 3 - Think About The Worst Case Scenario

Spend some time thinking about the worst case scenario and taking steps to mitigate the risk - what happens if a part of your infrastructure gets compromised? Security is all about layers.

For most small-mid size websites, the reality is a determined attacker can probably get in if they want to. Nothing is 100% secure, but you need to be taking reasonable steps to protect yourself and harden your site against attack. Even large websites or companies with huge teams dedicated to security get compromised from time to time (Google, Adobe, Target etc...).

Ask yourself what happens if someone does get in to part of your system? What's the worst that can happen? If for any one component of your infrastructure the answer is "we're completely screwed" then take steps to fix it and make it harder for your business to get wiped out. The answer needs to be more like "if A happened and B happened and C happened and D happened and E happened and F happened, then we'd be totally screwed".

Different applications require different degrees of redundancy and security, and building all of this costs time and money. You need to figure out what makes sense for your business.

If Codespaces had have asked themselves this simple question, they could have taken steps to mitigate the risk by having a backup at a different provider.

Incidentally, Codespaces could have suffered the same loss of data through operator error (ie accidentally deleting something from the AWS account), massive AWS failure or a disgruntled employee. That's far too many ways to completely wipe out their business.

Years ago at university, I remember a lecturer talking about how the digitization or information makes it not only more accessible, but easier to destroy. He used the analogy of a pallet full of papers vs a CD-ROM (back before the days of flash storage :)). Destroying a stack of papers takes effort. Sure, you could rip them, burn them, shred then, blow them up. But any of those options required a decent amount of time, effort or equipment to execute. Compare that with destroying the same amount of data on a CD, which you could just snap with your hands in less than 1 second. The cloud exacerbates this problem. Now it's possible to destroy a pallet load of CD's worth of data in seconds.

Lesson 4 - Follow Best Practices

There are plenty of resources on best practices for security. It's hard to know where to begin when analyzing Codespace's security failures, but one thing that likely wasn't in place and should have been was multi-factor authentication. AWS has it, Azure has it, Google has it, Facebook has it, Twitter has it. Pretty much all major sites have it these days and encourage people to use it even for personal accounts.

If thousands of customers are trusting you with their data, do yourself a favor and enable it. It probably would have saved Codespaces' ass.

Conclusion

Just because your data is in the cloud doesn't mean it's safe. As an individual or a business, never trust your data to a single provider, make sure you've got backups in multiple places. As a business, think about the worst case scenario and make sure it's not too easy to have your data (and consequently your business) wiped out. Security can be complicated and expensive, but there are a lot of simple, cheap things you can do to make your app more secure. If Codespaces had followed these, they'd still be in business.