How to architect data flow and storage in a new startup
You may think you have a good idea, but it’s data’s job to tell you otherwise.
Ironically, in the startup world today where everyone knows they should make “data-driven decisions”, few founders actually setup their data architecture and flow to give them the data they need on a daily basis to make smart choices, let alone facilitate growth.
DATA? YOU MEAN IN OUR DATABASE?
Haha, funny. But seriously, you’d be surprised how many new founders forget that all their emails, analytics, project tasks, documents, invoices, code, notes, chat messages, tweets, articles, posts, etc… are “data” that — especially as it relates to customers — should be smartly organized, tracked, and used to conduct experiments in order to better serve customers.
However, most organizations below the enterprise level (well, many of them too) have a chaotic mess when it comes to all this data. At at certain stage, most growing companies need to hire expensive third-party consultants to take a hammer to their systems and help them “grow up” into software that everyone else uses (you know the names) and setup smarter data flow.
SO, UM, SHOULD I STILL USE A DATABASE?
If you’re used to setting up your own API server and hooking it up to PostgreSQL or MongoDB or whatnot, go ahead and do that. Just keep in mind that this is only one part of your information architecture. I’ve written up previously an article to help Rails developers architect their model structure which is still relevant for technical founders thinking through that particular step.
In addition, most companies also need to choose:
- Email (internal, transactional, and marketing)
- Support Desk
- Live Chat
- Analytics
- CRM
- CMS
- Financials (Payment Gateway, Invoicing, Bookkeeping)
- etc…(every company uses a bunch of unique tools to house and control data)
So the decision of whether or not you need to build your own API server and database will depend on your ability to properly integrate with all these other services. Don’t forget that.
Enterprise-level companies have attempted to solve a lot of these problems by using Enterprise Resource Planning (ERP) solutions, but the rest of the world is left to cobble together their own systems.
GOTCHA, OK, WHAT WOULD YOU RECOMMEND FOR THE BASICS?
In general it’s best to use free/low-cost solutions that lots of other people use so there are plenty of integrations available (sounds obvious, right?).
Use Google for your web analytics, documents, file storage, and team email. Don’t re-create Google Tag Manager and Google Analytics. Instead, spend the time needed to get up to speed on the latest features and how to use them.
Take advantage of as many Web Services that Amazon provides as you can that are applicable for your business (yes, they do transactional email; yes they host DNS, etc…)
Unless you are very confident about your ability to build your own lightweight solutions in early days, use Intercom for your CRM, support desk, live chat, and automated marketing.
Unless your bookkeeper tells you otherwise, use Quickbooks Online and Mailchimp.
If you can get away with it (in the early days, especially), use a backend-as-a-service like Firebase, AWS Mobile Hub, or Brightwork.
You get the idea. There will be many more services applicable for your business needs.
INTEGRATIONS: SMARTLY TYING ALL YOUR DATA TOGETHER
Finally, before your first private beta users start using your application, it’s time to decide how to coordinate your data flow and storage across all the third-party and customer services you’ve decided to build/use.
Services like Zapier and Workato can be used to move data around your architecture based on custom events and triggers (this is why using common services is a good idea; the Zapier’s of the world have already built the integrations for you).
No matter how you swing it, make sure that there is one single-source-of-truth for your customer data, and that you send a daily aggregate email of your funnels and KPIs so you can make smart product and growth decisions based on what the data is telling you.
Author’s note: this is the 15th post in a series of articles outlining a framework for startup operations that my partners and I at Prota Ventures have developed. We’re building a SaaS product on top of this framework to help founders spin up new ventures and get connected to co-founders + advisors/investors along the way. Jump into the private beta queue here. Also, feel free to subscribe to my newsletter and I’ll let you know when I get new content up. Thanks!