Table of Contents
Many organizations understand the value of external data, yet struggle with external data procurement. To remain competitive, organizations need a strategy for purchasing external data in order to derive the maximum value from it.
Our 2022 State of External Data report aimed to discover how organizations use external data today and understand the challenges they face when building their data strategy.
We asked respondents to report:
- What type of external (third-party) data does their organization purchase, and from where?
- What is challenging about sourcing external data?
- What business and analytics initiatives do they use external data for?
- What do they need to get the most value from the third-party/external data they purchase?
This blog will highlight the survey’s key findings and the processes and skills organizations have to find, ingest, onboard, and integrate external data.
Who we surveyed
We surveyed 223 data leaders from companies across the United States in various industries about using external data and some of their challenges. Of these, over a third said they worked in analytics and BI (39%) and data management (36%).
Most respondents (64%) held either director or executive-level positions in their organization and said their annual external data spend spiked between $100,000 and $500,000 (18%), and again at over $1 million (13%).
The survey also revealed respondents primarily purchased company data (52%), with the next most common data types being demographic (44%) and financial data (41%). At the same time, 44% of respondents revealed they purchase external data from five or more paid or public sources.
Key findings
- Organizations source data from multiple providers. Since nearly half of survey respondents said they get their external data from five or more sources, this may represent significant budget and associated overhead costs. The financial and time cost of the procurement process, legal negotiation of terms and conditions, and IT security teams maintaining and monitoring usage of the data to ensure compliance all add up.
- A formal strategy is needed. Of all the respondents, only 40% said their organization has a plan to source and integrate external data. A successful data and analytics strategy for organizations means turning data into actionable insights that drive decision-making within the organization. Forward-thinking organizations understand that having the correct data to fit their analytics is vital for success, and this includes both internal and external data.
- Looking for technology solutions to overcome challenges. By far, the main challenge to using external data was difficult data preparation and integration. However, new technologies, powered by AI and ML capabilities, are increasingly tackling this challenge. Today, external data platforms do much of the heavy lifting when it comes to entity resolution and integration of external with internal data.
Data procurement challenges
A company’s ability to compete is increasingly driven by how well it can leverage data and apply analytics. Companies can build more accurate predictive models to improve decision-making processes by enriching their internal data with external data. In our survey, just 40% of survey respondents said they have a formal external data sourcing strategy in place.
While this is an improvement over last year’s result of 28%, it’s still clear that most organizations tend to find data on an ad hoc basis. They might have general guidelines with no consistent strategy, or outsource data procurement altogether.
With the volume of big data available, it is hard to know what to look for and which data sets will add value. Overlooking external data is a missed opportunity as it provides essential context not always captured in internal data, so successfully leveraging external data creates a competitive advantage.
Our survey respondents highlighted several challenges in sourcing external data: regulatory constraints (34%) and difficulty understanding the ROI of some datasets (34%).
Additionally, 63% of survey respondents report the most common barrier to using external data is data prep and integration, but lack of strategies to address usage and risk (34%) and lack of process for data usage (32%) also rank in the top three usage inhibitors.
Any purchased datasets from third-party vendors must be managed carefully to comply with data usage policies such as the EU’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA). Organizations must ensure that data vendors are providing compliant data. This is also further complicated when managing multiple external data sources and vendors.
Even though most companies understand that augmenting their internal data with alternative datasets will provide better insights, it can be hard to know why it is important. It is tough to make the case to spend the money on external data without knowing exactly what to purchase and how to use it.
Where external data is most commonly used
As for why organizations purchase external data, findings from the survey revealed leading usages in business intelligence (69%), data science (56%), and visualization and reporting (55%).
The most common use cases for external data include financial risk analysis (47%), customer segmentation (43%), fraud detection (36%), and demand forecasting (35%). As the value of external data becomes more apparent and is used to create competitive advantages, we expect the number of use cases to increase significantly as organizations seek out more ways to leverage it across their key business initiatives.
External data and getting the most value
Fifty-two percent of the respondents said they need to easily match and integrate internal and external data to maximize the value. They also need to ensure data quality and accuracy (49%) and a better way to locate the right type of data (46%).
Most of the time, collecting data is operational collateral. So there is little value in data beyond validating decisions already made or confirming that “they’re going in the right direction.” Data leaders need to find a way to prove, tangibly, that embracing data and investing in better ways to use it has true value.
The key takeaway here is that organizations need to seek new technologies to address the challenges with data preparation and integration and expand their use of external data.
One way to do that would be to leverage an external data platform. External data platforms work by automating your connections to thousands of pre-vetted data sources. Not only have the datasets been curated for quality and reliability, they form a single, collective catalog, so you don’t have to pay for access to each one separately.
The datasets are inter-compatible, meaning that you can treat them as a single resource, lifting out just the details you need to enhance and augment your existing datasets or combining them into brand new datasets.
An external data platform should remove as many of the roadblocks associated with finding and purchasing data sources as possible. It should also simplify your data pipelines and get your predictive model development off to a great start. That’s where the true value lies.
To learn more, download the full 2022 State of External Data report here.
About Explorium
Explorium offers a first of its kind end-to-end automated external data platform for advanced analytics and machine learning. Our unique all-in-one platform automatically connects and matches internal enterprise data with thousands of relevant external data sets to accelerate your ML investment, ROI, and help solve complex problems. The Explorium platform empowers data scientists and business leaders to drive decision-making by eliminating the barrier to acquire and integrate the right external data and dramatically decreasing the time to superior predictive power. Visit explorium.ai to learn more.
References:
- Bernard Marr, “Why every business needs a data and analytics strategy”, Bernard Marr & Co., accessed Feb 22, 2022. https://bernardmarr.com/why-every-business-needs-a-data-and-analytics-strategy/
- Explorium, “The 5 Biggest Challenges of Sourcing External Data”, Dec 13, 2021. https://www.explorium.ai/blog/the-5-biggest-challenges-of-sourcing-external-data/
- Explorium, “Where’s the ROI? Demonstrating the Value of Data”, May 26, 2020. https://www.explorium.ai/blog/wheres-the-roi-demonstrating-the-value-of-data/
- Explorium, “Why External Data Platforms Are the Next Step for Alternative Data Seekers”, April 7, 2021. https://www.explorium.ai/blog/why-external-data-platforms-are-the-next-step-for-alternative-data-seekers/
