The True Cost of Business Intelligence

Man's face calling to talk about business intelligence

I have been engaged in many discussions with founders and teams on the pros and cons of having some components of Business Intelligence in-house and outsourcing other components of Business Intelligence

Even a small startup can be wasting upwards of €200 000 per year on managing data. Money they can re-invest into advertising or product! ????.

Throughout my career, I have been a strong supporter of the fact that the value of Business Intelligence for the company is in the Data, not in the technology and infrastructure to manage it. Ironically enough, most of the companies actually share a similar infrastructure, making it pointless to try to outsmart your competitors on how you manage your data.

It is the decisions the company makes using the data, that creates the competitive advantage, not the underlying technology of managing that data.

How can business intelligence turn into a competitive advantage?

After all, the true competitive advantage begins in your data is in one place, accessible to all your teams and your data is complete, timely and accurate.

Most of the companies I have worked with have the same portfolio of tools that they use to solve various needs connected with the Business Intelligence:

  • Event tracking technologies (Snowplow, Heap, Mixpanel, Kissmetrics)

  • Data Visualisation (Tableau, Metabase, Looker, Chart.io)

  • Data Warehouse | Data Lake (Redshift, Snowflake)

  • ETL and Data Modeling (Airflow, Talend)

  • Integrations (StitchData, Tray, Panoply)

I regularly encounter founders and CTOs who insist on keeping some (or all) of the Business Intelligence built and maintained in-house. While I understand the peace of mind that comes with knowing that some elements of Data Analytics are owned by the enterprise, unfortunately, this security-blanket mindset often comes with a hefty cost that is not frequently a one-line-item on the cashflow statement. This cost is buried across teams and individuals who internally support Data Management technology in-house. Individuals who could be investing their time into the core product or services of the company.

My strongest argument to founders who want to build everything or some of Data Management in-house has always been the same:

If your in-house team can build a better BI solution than these providers who actually have specialised teams focusing only on building the best version of that BI solution, you should stop what you are doing and become the provider of that BI solution.

Breaking IT Down your Business Intelligence

Let’s take an example of an Event Tracking Technology. Putting aside the complexity of defining an event taxonomy, the sheer amount of effort and know-how needed to deliver an event from the application to the event collector, validate the event, normalize the data and load it into the event repository extends far beyond the capabilities of the average tech team of a company that is not specializing in event tracking.

Let’s look at some other Business Intelligence modules and tasks associated with making them work:

BI Components + Operational ItemsWho will do this?
INTEGRATIONS 
Customization of available integrationsDevOps
Building completely custom integrationsProgrammers
Historical data reloadBI Team
Monitoring executions and automatic re-run on failureDevOps
Monitoring data deliveryBI Team
Upgrades to match source API specificationsDevOps
Throttling data pull based on quota requirementsDevOps
Creating and approving Application (ex. FB, Google)DevOps
Failure notification processDevOps
  
ELT / ETL PIPELINE 
Manage Code repository (Git, setup and support)IT (DevOps or SysAdmin)
Manage CI/CD Pipeline for DEV and PRODIT (DevOps or SysAdmin)
Monitoring, Scaling, and Load ManagementIT (DevOps or SysAdmin)
Backup and Disaster RecoveryDatabase Admin
Failure notification processHead of BI
UpgradingIT (DevOps or SysAdmin)
TerraformIT (DevOps or SysAdmin)
  
EVENT TRACKING 
Creating and maintaining collectorIT (DevOps or SysAdmin)
Security and encryption of the eventsIT (DevOps or SysAdmin)
Disaster RecoveryIT (DevOps or SysAdmin)
Load management and scalingIT (DevOps or SysAdmin)
Quick collector-to-database event deliveryIT (DevOps or SysAdmin)
Event validation processIT (DevOps or SysAdmin)
Pipeline monitoring, log managementIT (DevOps or SysAdmin)
Shredding of events and contextsIT (DevOps or SysAdmin)
Creating tracking code for Web and Mobile events in-line with best practicesBI Team
Event validation before production releaseBI Team
Linking KPIs with Tracking RequirementsBI Team
Managing Event LibraryBI Team
Bad events analysisBI Team
Failure notification processIT (DevOps or SysAdmin)
TerraformIT (DevOps or SysAdmin)
  
DATA WAREHOUSE 
Workload ManagementDatabase Admin
Performance Monitoring and Identification of Problematic CodeBI Team
Optimizing Slow QueriesBI Team
Backup and Disaster RecoveryBI Team
Access and Security ManagementBI Team
Data Warehouse Architecture and DesignHead of BI/BI Team
Workload management and query prioritizationDatabase Admin
Failure notification processIT (DevOps or SysAdmin)
Synchronize DEV and PROD clustersIT (DevOps or SysAdmin)
TerraformIT (DevOps or SysAdmin)
  
DATA VISUALIZATION / REPORTING 
Backup and Disaster RecoveryDatabase Admin
Container Monitoring and Load Management (DEV and PROD)IT (DevOps or SysAdmin)
Failure notification processIT (DevOps or SysAdmin)
TerraformIT (DevOps or SysAdmin)

This is by far not the most exhaustive list of the tasks…

They need to be done before the company gets into extracting any business value out of the data! Because some of these responsibilities spread across multiple departments, it is hard to put a value on them because the standard way of thinking is

IT has to maintain our Tech Stack anyhow, might as well add Business Intelligence technology there

 

See also: Do you really need to BUILD a Business Intelligence system in house?

So, how much is the money drain?

It is not unusual for a startup of about 50 people to hire 2 or 3 developers to maintain some of these infrastructure requirements. Usually, there is one person hired into the BI team to undertake the tasks that are operationally close to the BI team, such as the management of Integrations. Another specialist is often hired into the IT team to build, maintain and scale stuff like notification management, disaster recovery, container and code repositories, and logs.

On top of the human capital costs, there are infrastructure costs of actually running the tech. Server instance, log repos, monitoring pipelines, jenkins, messaging service and so on.

If we ballpark programmer salary at €70K/year and infrastructure costs at €3K/month, even at the most conservative estimates of needing to have 2 extra full-time employees the cost is already creeping up to €200 000/year. In reality, though it usually ends up being around €500 000/year. That is half-a-million you will not spend on Advertising or building your product/service. In monthly costs, this is between 16 000 and 40 000 per month. Getting external specialized providers, like SageData, to do all the heavy lifting, while maintaining ownership of your data, will run you anywhere from 200/month to a maximum of 5000 per month. So even with the most expensive plans from the external providers, it is still a lot cheaper than maintaining and dealing with all the technology in-house.

Wrapping up my thoughts

Please do not get the impression that I am against the enterprise owning their data. I think data is the new oil and it is the most valuable asset a company could have. But short of owning and taking care of your Data Warehouse, why bother with all the stuff around it?

Companies no longer maintain their own email servers and build email clients. We rent them from the cloud providers to keep the company resources focused on the core competencies.

Why not adopt the same approach to data management? As long as the data makes its way into your Data Warehouse and your data is Timely, Accurate, and Complete, why burden yourself with having to pay for maintaining the processes to get the data there? Do we care about the process or the end result?

Companies should not be afraid of outsourcing the most problematic and time-consuming parts of maintaining a Business Intelligence stack to an expert provider who can manage the processes better, faster and cheaper because this is all the provider does day in and day out. Companies who specialize in building and maintaining the BI systems for multiple clients often have the best practices fine-tuned, in addition to economies of scale and focus. The best part of all, the client does not have to give up their data or their independence because the client still retains ownership of all their data and their DWH. With high turnover in the IT industry, it is good to have a consistent and reliable partner company responsible for the BI infrastructure for many years to come.

Need help with Data? Give us a call!
Comments are closed.