According to the second annual Teradata survey, revealed at the 2003 Teradata Partners User Group Conference & Expo in Seattle, Washington, executives are having a harder time making decisions. It seems there’s too much data, too many decisions to make, and not enough time to make them.
Conducted this past July, the survey queried 158 executives of major U.S. corporations with revenues of $500 million or higher. These vice presidents, presidents and CEOs represented banking, insurance, technology, manufacturing, travel, transportation, telecommunications, Internet services, professional services and government.
73% of the respondents said that the number of decisions they must make daily has increased, in many cases tripling over last year’s double and triple numbers. And the decisions they have to make are becoming more complex, said 56% of that group. What worries these executives most is the danger of making poor decisions. Over 75% of the surveyed executives said that poor decision-making would affect profits, revenue, reputation and corporate morale.
The solution, said these executives, was to have a clear corporate vision to use as the foundation for making decisions and a system that would integrate and share information across the company.
The vision responsibility belongs to the CEO. New solutions, though, are emerging that can help integrate and share information gathered by all departments within a company to help executives execute the vision. Of course, this means there’s a new buzzword to describe this solution, and it is Business Intelligence (or BI). A couple of the tools of BI go by the names of data warehousing and data mining. If you’ve dealt with the Internet or other IT functions, you’ve probably heard these terms.
Data warehousing is the function of storing huge amounts of data (gigabytes are common, with talk of terabytes next and petabytes on the horizon), usually on some type of computer disk. Data mining is the process of extracting that data from the warehouse for analysis.
In the last decade, significant strides have been made in data mining. It has evolved into an iterative process of discovering and interpreting previously unknown patterns in data to solve business problems, and rating the success of the resulting strategies with a probability. Data mining has been used in the scientific and academic worlds since the 1980s. It is now poised to move into the business world to take over where reports and ad hoc query tools leave off.
These data analysis tools are the next step up from online analytical processing and online transaction processing.
The Conference introduced new tools to Teradata’s business partners and customers as well as explored various technical and business issues associated with data gathering and analysis.
In the session on Managing Cost and Enhancing Value, Kevin Strange, vp and research director at Gartner focused on corporations’ need to eliminate the high cost of redundancy. Every company with data storage tends to replicate the same data multiple times and store them in individual silos or “data marts.” Because of this, claims Strange, “through 2006 more than 50% of organizations will spend an order of magnitude more on implementation than is necessary due to a lack of coordination of various business intelligence initiatives including CRM.”
Mr. Strange focused on executives’ need to look beyond ROI to TVO, or total value opportunity. “Don’t follow a model blindly,” he said. “Understand your market place better than your competitors.”
He went on to talk about the trend of real-time data feeds. Apparently the idea of getting data in “real time” and analyzing it and using it to make decisions is viewed as the next great way to obtain strategic advantage over competitors. However, obtaining and analyzing data in real-time may turn out to be more expensive than it is worth. Strange cautioned the audience that only about 29% of data mining applications within a corporation will require access to real-time data, which are data gathered in hours rather than days or weeks. “Through 2006,” he said, “a minority of business intelligence solutions will use real-time data feeds.” Strange told the audience to take the trend of real-time information seriously, but deploy carefully and only if necessary.
What’s coming, as a result of data mining, is the daily close. By 2010, claimed Strange, the daily close will be a reality for all large enterprises. Between 2006 and 2007, we’ll see the development of the business intelligence network.
There will be major areas of cost in order to establish these goals. These areas include the cost of the data warehouse platform, servers, disks and the staffing needed to install and use all of the equipment. Extracting data, because it is labor intensive, will be one of a corporation’s larger cost areas. Other factors will contribute to the high cost of data warehousing. “Business fragmentation will grow because companies will buy staggering numbers of disparate and unrelated business intelligence technologies,” continued Strange. And ERP systems with data warehouses inside are a question of when and how much, not if.
Despite these tools’ abilities to offer better information to use in decision-making, there will be several technical challenges will cause a few setbacks, and crimp several budgets. First of all, the ability of disks to store more data is doubling about every 9 months, said Stephen Brobst, chief technology officer at NCR Teradata, in his session on The Future of Data Warehousing.
As he put it, Moore’s Law is not in danger of reaching its limit, and instead will continue for another 20 years. The problem is that the ability of data storage systems to access all that data have not increased in speed at the same rate as the storage capacity. You can store more data but you can’t get to it any faster. The risk is greater wear and tear on the disks. There are some solutions to this problem, which involve various ways of partitioning data for greater accessibility. Brobst noted that what is outpacing Moore’s Law is the appetite for data.
Which leads to a caution. It’s a trendy notion that more data and analyzed data will lead to greater profits and revenue, and will push companies ahead of their competition. However, data and analyzed data may be no better at improving performance and profits than any other system tool. One of the messages throughout the conference was, “Why do you want these data? What’s the business strategy?” If vendors are already cautioning executives against implementing this tool without a clear reason to do so, then it’s already in danger of becoming the latest strategy that failed to deliver on its promises. Data mining can lead to significant cost reductions for a company. But a clear strategy for its use is key.
The next area companies must deal with regarding all these data they are collecting is ensuring the privacy of the “donors” (i.e. you and me) of that data and ensuring that the data are always accurate. This is a brand new area of discussion. The potential for misuse, let alone mistakes, is feared by many to be huge. Some legislation, in anticipation of potential misuse, is starting to come out. The financial industry has the Sarbanes-Oxley Act, the health industry the HIPAA Act, and there are others of various merit and enforcement.
Executives need to realize that while having access to all this data may provide a strategic benefit over the competition, the public has fears about who’s collecting and using it for what means. Enron and other corporate malfeasance have not built up trust. And policies need to be established and reviewed on a regular basis.
As Dr. Richard Hackathorn noted in his presentation, “Ethics in Business Intelligence: A Practical Treatment,” you can design a data warehouse to do a lot of things. But too few executives ask the question of should you design the warehouse to do a specific task that risks the privacy and security of a customer? And, just because you can legally do something does not mean it’s ethical to do it.
Most business decisions are ethically examined from the position of ignorance, indifference and apathy. But there are other bases for making ethical judgements, including the Golden Rule, common sense, and societal good. Other bases, such as materialistic goals, have less to recommend them.
The right to know versus the right to privacy are due for a head-on collision, and executives must be aware of this and have policies in place to deal with the ramifications. Preparation is the key, claims Dr. Hackathorn. It will be crucial to the bottom line for companies to be open and honest about the issues and to make ethical discussions acceptable within the company, and more importantly, within society.
Teradata, a division of NCR, introduced a range of new products in data mining and data warehousing. For more information on product offerings and on the conference, visit www.teradata.com.
