I have been engaged in many discussions with founders and teams on the pros and cons of having some components of Business Intelligence in-house and outsourcing other components of Business Intelligence

Throughout my career, I have been a strong supporter of the fact that the value of Business Intelligence for the company is in the Data, not in the technology and infrastructure to manage it. Ironically enough, most of the companies actually share a similar infrastructure, making it pointless to try to outsmart your competitors on how you manage your data.

It is the decisions the company makes using the data, that creates the competitive advantage, not the underlying technology of managing that data.

How can business intelligence turn into a competitive advantage?

After all, the true competitive advantage begins in your data is in one place, accessible to all your teams and your data is complete, timely and accurate.

Most of the companies I have worked with have the same portfolio of tools that they use to solve various needs connected with the Business Intelligence:

• Event tracking technologies (Snowplow, Heap, Mixpanel, Kissmetrics)

• Data Visualisation (Tableau, Metabase, Looker, Chart.io)

• Data Warehouse | Data Lake (Redshift, Snowflake)

• ETL and Data Modeling (Airflow, Talend)

• Integrations (StitchData, Tray, Panoply)

I regularly encounter founders and CTOs who insist on keeping some (or all) of the Business Intelligence built and maintained in-house. While I understand the peace of mind that comes with knowing that some elements of Data Analytics are owned by the enterprise, unfortunately, this security-blanket mindset often comes with a hefty cost that is not frequently a one-line-item on the cashflow statement. This cost is buried across teams and individuals who internally support Data Management technology in-house. Individuals who could be investing their time into the core product or services of the company.

My strongest argument to founders who want to build everything or some of Data Management in-house has always been the same:

If your in-house team can build a better BI solution than these providers who actually have specialised teams focusing only on building the best version of that BI solution, you should stop what you are doing and become the provider of that BI solution.

Breaking IT Down your Business Intelligence

Let’s take an example of an Event Tracking Technology. Putting aside the complexity of defining an event taxonomy, the sheer amount of effort and know-how needed to deliver an event from the application to the event collector, validate the event, normalize the data and load it into the event repository extends far beyond the capabilities of the average tech team of a company that is not specializing in event tracking.

Let’s look at some other Business Intelligence modules and tasks associated with making them work:

This is by far not the most exhaustive list of the tasks…

They need to be done before the company gets into extracting any business value out of the data! Because some of these responsibilities spread across multiple departments, it is hard to put a value on them because the standard way of thinking is

IT has to maintain our Tech Stack anyhow, might as well add Business Intelligence technology there

See also: Do you really need to BUILD a Business Intelligence system in house?

So, how much is the money drain?

It is not unusual for a startup of about 50 people to hire 2 or 3 developers to maintain some of these infrastructure requirements. Usually, there is one person hired into the BI team to undertake the tasks that are operationally close to the BI team, such as the management of Integrations. Another specialist is often hired into the IT team to build, maintain and scale stuff like notification management, disaster recovery, container and code repositories, and logs.

On top of the human capital costs, there are infrastructure costs of actually running the tech. Server instance, log repos, monitoring pipelines, jenkins, messaging service and so on.

If we ballpark programmer salary at €70K/year and infrastructure costs at €3K/month, even at the most conservative estimates of needing to have 2 extra full-time employees the cost is already creeping up to €200 000/year. In reality, though it usually ends up being around €500 000/year. That is half-a-million you will not spend on Advertising or building your product/service. In monthly costs, this is between 16 000 and 40 000 per month. Getting external specialized providers, like SageData, to do all the heavy lifting, while maintaining ownership of your data, will run you anywhere from 200/month to a maximum of 5000 per month. So even with the most expensive plans from the external providers, it is still a lot cheaper than maintaining and dealing with all the technology in-house.

Wrapping up my thoughts

Please do not get the impression that I am against the enterprise owning their data. I think data is the new oil and it is the most valuable asset a company could have. But short of owning and taking care of your Data Warehouse, why bother with all the stuff around it?

Companies no longer maintain their own email servers and build email clients. We rent them from the cloud providers to keep the company resources focused on the core competencies.

Why not adopt the same approach to data management? As long as the data makes its way into your Data Warehouse and your data is Timely, Accurate, and Complete, why burden yourself with having to pay for maintaining the processes to get the data there? Do we care about the process or the end result?

Companies should not be afraid of outsourcing the most problematic and time-consuming parts of maintaining a Business Intelligence stack to an expert provider who can manage the processes better, faster and cheaper because this is all the provider does day in and day out. Companies who specialize in building and maintaining the BI systems for multiple clients often have the best practices fine-tuned, in addition to economies of scale and focus. The best part of all, the client does not have to give up their data or their independence because the client still retains ownership of all their data and their DWH. With high turnover in the IT industry, it is good to have a consistent and reliable partner company responsible for the BI infrastructure for many years to come.