Does information visualization provide
sufficient return on investment (ROI)? Let us examine the investment value—the
estimated financial worth—of providing end-user data visualization. For an
organization desiring to depict complex or large data sets in various pictorial
or graphical formats for 10,000 users, a commercial visualization product
subscription model in which each seat license may cost $50 per month equates to
$600,000 per year, not including the labor fee to prepare, implement and
maintain the tool.
Consuming the information on a
proliferating number of endpoint mobile devices could further increase costs to
more than $1 million, a nontrivial amount in any organization’s budget. The
potential outlays escalate when spanned across the federal enterprise.
Data visualization value, whether
expressed directly from straightforward monetized return or subjectively
derived from intangible benefits, needs to be assessed quantitatively to
determine the economic return to permit a comparison with expected losses and
gains from other organizational investments. Without an operationally relevant
ROI performance metric, any project expense could be justified to counterweigh
the risk of loss.
Extracting value from data typically
focuses on the larger and more expensive issues of management and use of big
data, where it is assumed that information visualization is a derived
byproduct. Yet when tallied as a separate line of investment, the intended
scope of graphically depicted data may not provide enough justification for the
production cost and potential difficulties. Consequently, as with any
significant investment, the chief information officer and the chief financial
officer should conduct a timely review of the data-visualization business case
for a quantifiable performance measure of success or failure. For example, a
good ROI likely would not involve spending more than $1 million on data
visualization to save $200,000.
Investments in data visualization must
compete with other organizational priorities. Determining the ROI is a
challenging exercise because it requires that the organization quantifiably
measure not only the quality of the tool’s functional characteristics (whether
it is accessible, accurate and well designed) but also utilization of the
produced information— what can be achieved with better, data-driven management
decisions? Too often investment decisions are made, and ROI is not measured,
because it is considered unrealistic to expect a quantified measurement of less
tangible benefits. The abstract goal of loosely defined long-term benefits then
underpins the business case: greater business and customer insight, faster
decision-to-answer time, or faster response to customers and markets. However, reducing
uncertainty for intangible investments is possible, as indicated by Douglas
Hubbard’s Rule of Five (How to Measure Anything: Finding the Intangibles in
Business, published by John Wiley & Sons, Inc., 2010 and 2014). This was
applied in the investment risk simulation example in my article, “How to
Improve Communication of Information Technology Investments Risks,” in the
November–December 2017 issue of Defense AT&L magazine. Subject-matter
expert (SME) knowledge, supplemented with historical and industry statistics,
may be a reliable source for accurate numerical value metrics.
Most organizations produce or consume data
for leadership to monitor performance and answer such basic questions as: “Are
we accomplishing our objectives . . . Are we using our resources in the most
effective manner ... Are we learning ways to improve our performance?” Some
outcomes are relatively straightforward, such as “certifying compliance within
a numeric benchmark for system defects that either did or did not decline over
time.” For example, the Internal Revenue Service investment in the Return Review
Program (RRP) fraud detection system—replacing the Electronic Fraud Detection
System that dated from 1994—either does or does not help prevent, detect and
resolve criminal and civil noncompliance. A successful system should result in
greater success with more revenue returned to the U.S. Treasury to offset the
RRP cost.
But it is more difficult to pinpoint how
the data results would reduce risk or improve organizational performance in
essential planning, organizing, directing and controlling operations—i.e.,
identifying the specific business decision problem, the root issue, and how the
data visualization investment would help. The answer would then define the
metric created to evaluate visualization product cost against expected business
results: ROI = Investment Gain/ Cost of Investment.
To select the best tool for the job,
management must first precisely determine how visualization would support
users’ efforts to distinguish between evidence-based reality and
unsubstantiated intuitive understanding. The tool must present raw abstract
data in a manner that is meaningful to users for improving understanding,
discovery, patterns, measurement, analysis, confirmation, effective ness,
speed, efficiency, productivity, decision making, and in reducing redundancy.
Classic approaches for extracting information from data include descriptive,
predictive and prescriptive analytics. The most common is descriptive analysis,
used as a lag metric to review what has already occurred. Predictive analysis
also uses existing data as the basis for a forecast model. Prescriptive builds
on predictive analytics, going a step further by offering greater calculated
insight into possible outcomes for selected courses of action leading to better
decision making. Data visualization of these approaches range from classic bar
and pie charts to complex illustrations.
The approach selected must align with the
organization’s senior leader expectations or else the experiment will be short
lived. The organization may already possess visualization tools that can be
leveraged at little or no additional cost. If the organization is just getting
started, a proof of concept pilot approach may be best, initiating a seminal
demonstration that can be progressively refined until an effective management
tool emerges. The beginning point could be basic metrics to more accurately
measure and assess success associated with the organizational goals, objectives
and performance plan. Basic example performance measurement of services,
products and processes includes:
• Cost Benefit =
(program cost avoided or cost incurred) / (total program cost)
• Productivity =
(number of units processed by an employee) / (total employee hours)
• Training
Effectiveness = (number of phishing email clicked) / (total phishing attempts)
Performance metrics enable quantitative
analysis of whether the tool investment produces sufficient monetary value,
fundamentally a risk decision about business outlays. One common method for
quantifying risk is: Annualized Loss Expectancy (ALE) = Single Loss Expectancy
(SLE) x Annualized Rate of Occurrence (ARO). For example, if the average cost
of a phishing and social engineering attack is $1.6 million (M) for a midsize
company and the likelihood of a targeted user clicking on the malicious attachment
or link is 0.02736, then the risk value = ($1.6M x 0.02736) = $43,776. After
weighing the organization’s cyber defenses and history of cyber-attacks, the
business-investment decision makers could better determine if investing in
employee anti-phishing training and training data visualization is a reasonable
risk-reduction expenditure. After the visualization tool has been purchased and
deployed, the value of the insights revealed by the analytics must at that
point be substantiated through organizational actions—i.e., cause and effect
linkage leading to input/process/output adjustments. As a means of generating
business intelligence, the organization is then able to weigh the tool’s value,
which should be equal to or greater than the production cost. Generally, a more
complex visualization results in higher tool cost. The journey from feasibility
determination to requirements refinement and then to operational maturity,
should be undertaken with the understanding that the initial investment may not
be supported by the magnitude of the early results, but total improvement over
time should be greater than total outlay.
In conclusion, managing and mining vast
amounts of complex data typically results in the need to view information in
ways that are measurably meaningful and actionable to the organization. Added
benefits include selective sharing, on-demand viewing and more informed
decisions. Information visualization tools range from low cost Microsoft Excel
charts to more powerful applications capable of producing relationship and
pattern analysis, forecasts, scorecards and performance dashboards from large
unstructured data. Organization leaders can then shift from reacting to lag
measures towards proactive actions based upon predictive data presentation.
Data visualization has a potentially
significant cost that must be balanced against the payback benefits rather than
simply bundled into a data management package. Selecting the best tool for the
organization should include basic cost-benefit analysis based upon a
performance metric of the value of the decisions made from the information
provided.
Published
in Defense AT&L: July-August 2018 (https://www.dau.mil/library/defense-atl/blog/The-Value-of-Data-Visualization)
No comments:
Post a Comment