The architecture of a business intelligence system, is depicted in three major components.

⎯ Data sources

In the first stage, it is necessary to gather and integrate the data stored in the various primary and secondary sources, which are heterogeneous in origin and type. 

The sources consist for the most part of data belonging to operational systems but may also include unstructured documents, such as emails and data received from external providers.

⎯ Data warehouses and data marts 

Using extraction and transformation tools known as extract, transform, load (ETL), the data originating from the different sources are stored in databases intended to support business intelligence analyses.

⎯ Businessintelligencemethodologies 

Data are finally extracted and used to feed mathematical models and analysis methodologies intended to support decision-makers.In a business intelligence system, several decision support applications may be implemented, most of which will be described in the following chapters: 

• multidimensional cube analysis; 

• exploratory data analysis; 

• time series analysis; 

• inductive learning models for data mining; 

• optimization models. 

⎯ Data exploration 

At the third level of the pyramid, we find the tools for performing a passive business intelligence analysis, which consists of query and reporting systems, as well as statistical methods. 

These are referred to as passive methodologies because decision-makers are requested to generate prior hypotheses or define data extraction criteria, and then use the analysis tools to find answers and confirm their original insight. 

 For instance, consider the sales manager of a company who notices that revenues in a given geographic area have dropped for a specific group of customers. 

Hence, she might want to bear out her hypothesis by using extraction and visualization tools and then apply a statistical test to verify that her conclusions are adequately supported by data. 

 ⎯ Data mining 

The fourth level includes active business intelligence methodologies, whose purpose is the extraction of information and knowledge from data. These include mathematical models for pattern recognition, machine learning, and data mining techniques. 

Unlike the tools described at the previous level of the pyramid, the models of an active kind do not require decision-makers to formulate any prior hypothesis to be later verified. Their purpose is instead to expand the decision makers’ knowledge. 

⎯ Optimization 

By moving up one level in the pyramid we find optimization models that allow us to determine the best solution out of a set of alternative actions, which is usually fairly extensive and sometimes even infinite. 

⎯ Decisions 

Finally, the top of the pyramid corresponds to the choice and the actual adoption of a specific decision, and in some way represents the natural conclusion of the decision-making process. 

Even when business intelligence methodologies are available and successfully adopted, the choice of a decision pertains to the decision-makers, who may also take advantage of informal and unstructured information available to adapt and modify the recommendations and the conclusions achieved through the use of mathematical models.

As we progress from the bottom to the top of the pyramid, business intelligence systems offer increasingly more advanced support tools of an active type. 

Even roles and competencies change. At the bottom, the required competencies are provided for the most part by the information systems specialists within the organization, usually referred to as database administrators. Analysts and experts in mathematical and statistical models are responsible for the intermediate phases. 

Finally, the activities of decision-makers responsible for the application domain appear dominant at the top. As described above, business intelligence systems address the needs of different types of complex organizations, including agencies of public administration and associations. 

However, if we restrict our attention to enterprises, business intelligence methodologies can be found mainly within three departments of a company, as depicted in logistics and production accounting and control. 

 ⎯ A business intelligence system provides decision-makers with information and knowledge extracted from data, through the application of mathematical models and algorithms. 

In some instances, this activity may reduce to calculations of totals and percentages, graphically represented by simple histograms, whereas more elaborate analyses require the development of advanced optimization and learning models. 

⎯ In general terms, the adoption of a business intelligence system tends to promote a scientific and rational approach to the management of enterprises and complex organizations. Even the use of a spreadsheet to estimate the effects on the budget of fluctuations in interest rates, despite its simplicity, forces decision-makers to generate a mental representation of the process of the financial flow. 

⎯ Classical scientific disciplines, such as physics, have always resorted to mathematical models for the abstract representation of real systems. Other disciplines, such as operations research, have instead exploited the application of scientific methods and mathematical models to the study of artificial systems, for example,  public and private organizations. Part II of this book will describe the main mathematical models used in business intelligence architectures and decision support systems, as well as the corresponding solution methods, while PartIIIwill illustrates several related applications. 

⎯ The rational approach typical of a business intelligence analysis can be summarized schematically in the following main characteristics. 

⎯ First, the objectives of the analysis are identified and the performance indicators that will be used to evaluate alternative options are defined. ⎯ Mathematical models are then developed by exploiting the relationships among system control variables, parameters, and evaluation metrics. ⎯ Finally, what-if analyses are carried out to evaluate the effects on the performance determined by variations in the control variables and changes in the parameters. 

⎯ Although their primary objective is to enhance the effectiveness of the decision-making process, the adoption of mathematical models also affords other advantages, which can be appreciated particularly in the long term. First, the development of an abstract model forces decision-makers to focus on the main features of the analyzed domain, thus inducing a deeper understanding of the phenomenon under investigation. Furthermore, the knowledge about the domain acquired when building a mathematical model can be more easily transferred in the long run to other individuals within the same organization, thus allowing sharper preservation of knowledge in comparison to empirical decision-making processes. Finally, a mathematical model developed for a specific decision-making task is so general and flexible that in most cases it can be applied to other ensuing situations to solve problems of similar type.

 • Several public and private enterprises and organizations have developed in recent years formal and systematic mechanisms to gather, store and share their wealth of knowledge, which is now perceived as an invaluable intangible asset. 

• The activity of providing support to knowledge workers through the integration of decision-making processes and enabling information technologies is usually referred to as knowledge management. Similarity 

• It is apparent that business intelligence and knowledge management share some degree of similarity in their objectives. The main purpose of both disciplines is to develop environments that can support knowledge workers in decision-making processes and complex problem-solving activities. 

• To draw a boundary between the two approaches, we may observe that knowledge management methodologies primarily focus on the treatment of information that is usually unstructured, at times implicit, contained mostly in documents, conversations, and past experience. Dissimilarity 

• Conversely, business intelligence systems are based on structured information, most often of a quantitative nature and usually organized in a database. 

 • However, this distinction is a somewhat fuzzy one: for example, the ability to analyze emails and web pages through text mining methods progressively induces business intelligence systems to deal with unstructured information.

 Data 

⎯ As observed above, a vast amount of data has been accumulated within the information systems of public and private organizations. 

These data originate partly from internal transactions of an administrative, logistical and commercial nature and partly from external sources.

 ⎯ However, even if they have been gathered and stored in a systematic and structured way, these data cannot be used directly for decision-making purposes. 

They need to be processed by means of appropriate extraction tools and analytical methods capable of transforming them into information and knowledge that can be subsequently used by decision-makers.

⎯ The difference between data, information, and knowledge can be better understood through the following remarks.

 ⎯ Data. Generally, data represent a structured codification of single primary entities, as well as of transactions involving two or more primary entities. 

For example, for a retailer data refer to primary entities such as customers, points of sale, and sale items, while sales receipts represent the commercial transactions.

Information 

Knowledge

The main purpose of business intelligence systemsisto provides knowledge workers with tools and methodologies that allow them to make effective and timelydecisions.

Effective decisions

Timely decisions

Effective and timely decisions

Principal component analysis (PCA) is the most widely known technique of attribute reduction by means of projection. 

Generally speaking, the purpose of this method is to obtain a projective transformation that replaces a subset of the original numerical attributes with a lower number of new attributes obtained as their linear combination, without this change causing a loss of information. 

Experience shows that a transformation of the attributes may lead to many instances of better accuracy in the learning models subsequently developed.

Before applying the principal component method, it is expedient to standardize the data, so as to obtain for all the attributes the same range of values, usually represented by the interval [−1, 1]. Moreover, the mean of each attribute aj is made equal to 0 by applying the transformation Let X denote the matrix resulting from applying the transformation (6.6) to the

X be the covariance matrix of the attributes (for original data, and let V = X )

a definition of the covariance and variance matrices, see Section 7.3.1). If the correlation matrix is used to develop the principal component analysis method

instead of the covariance matrix, the transformation (6.6) is not required. Starting from the n attributes in the original dataset, represented by the matrix X, the principal component method derives n orthogonal vectors, namely the principal components, which constitute a new basis of the space Rn. 

Principal components are better suited than the original attributes to explain fluctuations in the data, in the sense that usually a subset consisting of q principal components, with q<n, has an information content that is almost equivalent to that of the original dataset. As a consequence, the original data are projected into a lower-dimensional space of dimension q having the same explanatory capability.

Principal components are generated in sequence by means of an iterative algorithm. The first component is determined by solving an appropriate optimization problem, in order to explain the highest percentage of variation in the data. 

At each iteration, the next principal component is selected, among those vectors that are orthogonal to all components already determined, as the one which explains the maximum percentage of variance not yet explained by the previously generated components. At the end of the procedure, the principal components are ranked in non-increasing order with respect to the amount of variance that they are able to explain.

The purpose of feature selection, also called feature reduction, is to eliminate from the dataset a subset of variables that are not deemed relevant for the purpose of the data mining activities. One of the most critical aspects in a learning process is the choice of the combination of predictive variables more suited to accurately explain the investigated phenomenon.

Feature reduction has several potential advantages. Due to the presence of fewer columns, learning algorithms can be run more quickly on the reduced dataset than on the original one. Moreover, the models generated after the elimination from the dataset of uninfluential attributes are often more accurate and easier to understand.

Feature selection methods can be classified into three main categories: filter methods, wrapper methods, and embedded methods.

Filter methods. Filter methods select the relevant attributes before moving on to the subsequent learning phase and are therefore independent of the specific algorithm being used. 

The attributes deemed most significant are selected for learning, while the rest are excluded. Several alternative statistical metrics have been proposed to assess the predictive capability and relevance of a group of attributes. 

Generally, these are monotone metrics in that their value increases or decreases according to the number of attributes considered. The simplest filter method to apply for supervised learning involves the assessment of every single attribute based on its level of correlation with the target. Consequently, this leads to the selection of the attributes that appear mostly correlated with the target.

Wrapper methods. If the purpose of the data mining investigation is classification or regression, and consequently performances are assessed mainly in terms of accuracy, the selection of predictive variables should be based not only on the level of relevance of every single attribute but also on the specific learning algorithm being utilized.

Wrapper methods are able to meet this need since they assess a group of variables using the same classification or regression algorithm used to predict the value of the target variable. 

Each time, the algorithm uses a different subset of attributes for learning, identified by a search engine that works on the entire set of all possible combinations of variables, and selects the set of attributes that guarantees the best result in terms of accuracy. 

Wrapper methods are usually burdensome from a computational standpoint since the assessment of every possible combination identified by the search engine requires one to deal with the entire training phase of the learning algorithm. An example of the use of a wrapper method for attribute selection is given in Section 8.5 in the context of multiple linear regression models.

Embedded methods. For the embedded methods, the attribute selection process lies inside the learning algorithm, so that the selection of the optimal set of attributes is directly made during the phase of model generation. Classification trees, described in Chapter 10, are an example of embedded methods. At each tree node, they use an evaluation function that estimates the predictive value of a single attribute or a linear combination of variables. In this way, the relevant attributes are automatically selected and they determine the rule for splitting the records in the corresponding node.

Filter methods are the best choice when dealing with very large datasets, whose observations are described by a large number of attributes. In these cases, the application of wrapper methods is inappropriate due to very long computation times.

Moreover, filter methods are flexible and in principle can be associated with any learning algorithm. However, when the size of the problem at hand is moderate, it is preferable to turn to the wrapper or embedded methods which afford in most cases accuracy levels that are higher compared to filter methods.

As described above, wrapper methods select the attributes according to a search scheme that inspects in sequence several subsets of attributes and applies the learning algorithm to each subset in order to assess the resulting accuracy of the corresponding model. 

If a dataset contains n attributes, there are 2n possible subsets and therefore an exhaustive search procedure would require excessive computation times even for moderate values of n. 

As a consequence, the procedure for selecting the attributes for wrapper methods is usually of a heuristic nature, based in most cases on a greedy logic which evaluates for each attribute a relevance indicator adequately defined and then selects the attributes based on their level of relevance.

In particular, three distinct myopic search schemes can be followed: forward, backward and forward-backward search.

Forward. According to the forward search scheme, also referred to as bottom-up search, the exploration starts with an empty set of attributes and subsequently introduces the attributes one at a time based on the ranking induced by the relevance indicator. The algorithm stops when the relevance index of all the attributes still excluded is lower than a prefixed threshold.

Backward. The backward search scheme also referred to as top-down search, begins the exploration by selecting all the attributes and then eliminates them one at a time based on the preferred relevance indicator. The algorithm stops when the relevance index of all the attributes still included in the model is higher than a prefixed threshold.

Forward-backward. The forward-backward method represents a trade-off between the previous schemes, in the sense that at each step the best attribute among those excluded is introduced and the worst attribute among those included is eliminated. Also, in this case, threshold values for the included and excluded attributes determine the stopping criterion.

The various wrapper methods differ in the choice of the relevance measure as well as the threshold preset values for the stopping rule of the algorithm.

 A further reduction in the size of the original dataset can be achieved by extracting a sample of observations that is significant from a statistical standpoint. This type of reduction is based on classical inferential reasoning.

It is therefore necessary to determine the size of the sample that guarantees the level of accuracy required by the subsequent learning algorithms and to define an adequate sampling procedure. Sampling may be simple or stratified depending on whether one wishes to preserve in the sample the percentages of the original dataset with respect to a categorical attribute that is considered critical.

Generally speaking, a sample comprising a few thousand observations is adequate to train most learning models. 

It is also useful to set up several independent samples, each of a predetermined size, to which learning algorithms should be applied. In this way, computation times increase linearly with the number of samples determined, and it is possible to compare the different models generated, in order to assess the robustness of each model and the quality of the knowledge extracted from data against the random fluctuations existing in the sample. 

It is obvious that the conclusions obtained can be regarded as robust when the models and the rules generated remain relatively stable as the sample set used for training varies.

When dealing with a small dataset, the transformations described above are usually adequate to prepare input data for a data mining analysis.

However, when facing a large dataset it is also appropriate to reduce its size, in order to make learning algorithms more efficient, without sacrificing the quality of the results obtained.

There are three main criteria to determine whether a data reduction technique should be used: efficiency, accuracy, and simplicity of the models generated.

Efficiency. The application of learning algorithms to a dataset smaller than the original one usually means a shorter computation time. I

f the complexity of the algorithm is a superlinear function, as is the case for most known methods, the improvement in efficiency resulting from a reduction in the dataset size may be dramatic. As described in Chapter 5, within the data mining process it is customary to run several alternative learning algorithms in order to identify the most accurate model. 

Therefore, a reduction in processing times allows the analyses to be carried out more quickly.

Accuracy. In most applications, the accuracy of the models generated represents a critical success factor, and it is, therefore, the main criterion followed in order to select one class of learning methods over another. 

As a consequence, data reduction techniques should not significantly compromise the accuracy of the model generated. As shown below, it may also be the case that some data reduction techniques, based on attribute selection, will lead to models with a higher generalization capability on future records.

Simplicity. In some data mining applications, concerned more with interpretation than with prediction, it is important that the models generated be easily translated into simple rules that can be understood by experts in the application domain. 

As a trade-off for achieving simpler rules, decision-makers are sometimes willing to allow a slight decrease in inaccuracy. Data reduction often represents an effective technique for deriving models that are more easily interpretable.

Since it is difficult to develop a data reduction technique that represents the optimal solution for all the criteria described, the analyst will aim for a suitable trade-off among all the requirements outlined.

Data reduction can be pursued in three distinct directions, described below: a reduction in the number of observations through sampling, a reduction in the number of attributes through selection and projection, and a reduction in the number of values through discretization and aggregation.

 The aim of standardization techniques is to replace the values of an attribute with values obtained through an appropriate transformation. However, there are situations in which more complex transformations are used to generate new attributes that represent a set of additional columns in the matrix X representing the dataset D. Transformations of this kind are usually referred to as feature extraction. For example, suppose that a set of attributes indicate the spending of each customer over consecutive time intervals. It is then possible to define new variables capable of capturing the trends in the data through differences or ratios between spending amounts of contiguous periods.

In other instances, the transformations may take even more complex forms, such as Fourier transforms, wavelets, and kernel functions. The use of such methods will be explained within the classification methods called support vector machines.

Attribute extraction may also consist of the creation of new variables that summarize within themselves the relevant information contained in a subset of the original attributes. For example, in the context of image recognition, one is often interested in identifying the existence of a face within a digitalized photograph. There are different indicators intended for the synthesis of each piece of information contained in a group of adjacent pixels, which make it easier for classification algorithms to detect faces.

 In most data mining analyses, it is appropriate to apply a few transformations to the dataset in order to improve the accuracy of the learning models subsequently developed. Indeed, outlier correction techniques are examples of transformations of the original data that facilitate subsequent learning phases. The principal component method, described in Section 6.3.3, can also be regarded as a data transformation process.

 The term noise refers to a random perturbation within the values of a numerical attribute, usually resulting in noticeable anomalies.

 First, the outliers in a dataset need to be identified, so that subsequently either they can be corrected and regularized or entire records containing them are eliminated. 

In this section, we will describe a few simple techniques for identifying and regularizing data affected by noise, while in Chapter 7 we will describe in greater detail the tools from exploratory data analysis used to detect outliers.

 The easiest way to identify outliers is based on the statistical concept of dispersion. The sample mean ¯μj and the sample variance ¯σ2 j of the numerical attribute aj are calculated. If the attribute follows a distribution that is not too far from normal, the values falling outside an appropriate interval centered around the mean value ¯μj are identified as outliers, by virtue of the central limit theorem. More precisely, with a confidence of 100(1 − α)% (approximately 96% for α = 0.05), it is possible to consider as outliers those values that fall outside the interval

(μ¯ j − zα/2σ¯j ,μ¯ j + zα/2σ¯j ),

where zα/2 is the α/2 quantile of the standard normal distribution. This technique is simple to use, although it has the drawback of relying on the critical assumption that the distribution of the values of the attribute is bell-shaped and roughly normal. However, by applying Chebyshev’s theorem, described in Chapter 7, it is possible to obtain analogous bounds independent of the distribution, with intervals that are only slightly less stringent. Once the outliers have been identified, it is possible to correct them with values that are deemed more plausible or to remove an entire record containing them.

An alternative technique, illustrated in Figure 6.1, is based on the distance between observations and the use of clustering methods. Once the clusters have been identified, representing sets of records having a mutual distance that is less than the distance from the records included in other groups, the observations that are not placed in any of the clusters are identified as outliers. Clustering techniques offer the advantage of simultaneously considering several attributes, while methods based on dispersion can only take into account every single attribute separately.

A variant of clustering methods, also based on the distances between the observations, detects the outliers through two parametric values, p and d, to be assigned by the user. An observation xi is identified as an outlier if at least a percentage p of the observations in the dataset is found at a distance greater than d from xi.

The above techniques can be combined with the opinion of experts in order to identify actual outliers with respect to regular observations, even though these fall outside the intervals where regular records are expected to lie. In marketing applications, in particular, it is appropriate to consult with experts before adopting corrective measures in the case of anomalous observations.

Unlike the above methods, aimed at identifying and correcting every single anomaly, there exist also regularization techniques that automatically correct anomalous data. For example, simple or multiple regression models predict the value of the attribute aj that one wishes to regularize based on other variables existing in the dataset. 

Once the regression model has been developed, and the corresponding confidence interval around the prediction curve has been calculated, it is possible to substitute the value computed along the prediction curve for the values of the attribute aj that fall outside the interval.

A further automatic regularization technique, described in Section 6.3.4, relies on data discretization and grouping based on the proximity of the values of the attribute aj.

To partially correct incomplete data one may adopt several techniques.

Elimination. It is possible to discard all records for which the values of one or more attributes are missing. In the case of a supervised data mining analysis, it is essential to eliminate a record if the value of the target attribute is missing. A policy based on systematic elimination of records may be ineffective when the distribution of missing values varies in an irregular way across the different attributes, since one may run the risk of incurring a substantial loss of information.

Inspection. Alternatively, one may opt for an inspection of each missing value, carried out by experts in the application domain, in order to obtain recommendations on possible substitute values. Obviously, this approach suffers from a high degree of arbitrariness and subjectivity and is rather burdensome and time-consuming for large datasets. On the other hand, experience indicates that it is one of the most accurate corrective actions if skillfully exercised.

Identification. As a third possibility, a conventional value might be used to encode and identify missing values, making it unnecessary to remove entire records from the given dataset. For example, for a continuous attribute that assumes only positive values, it is possible to assign the value {−1} to all missing data. By the same token, for a categorical attribute, one might replace missing values with a new value that differs from all those assumed by the attribute.

Substitution. Several criteria exist for the automatic replacement of missing data, although most of them appear somehow arbitrary.

For instance, missing values of an attribute may be replaced with the mean of the attribute calculated for the remaining observations.

This technique can only be applied to numerical attributes, but it will clearly be ineffective in the case of an asymmetric distribution of values. In a supervised analysis, it is also possible to replace missing values by calculating the mean of the attribute only for those records having the same target class. Finally, the maximum likelihood value, estimated using regression models or Bayesian methods, can be used as a replacement for missing values.

However, estimate procedures can become rather complex and time-consuming for a large dataset with a high percentage of missing data.

The quality of input data may prove unsatisfactory due to incompleteness, noise, and inconsistency.

Incompleteness. Some records may contain missing values corresponding to one or more attributes, and there may be a variety of reasons for this. It may be that some data were not recorded at the source in a systematic way, or that they were not available when the transactions associated with a record took place.

 In other instances, data may be missing because of malfunctioning recording devices. It is also possible that some data were deliberately removed during previous stages of the gathering process because they were deemed incorrect. 

Incompleteness may also derive from a failure to transfer data from the operational databases to a data mart used for a specific business intelligence analysis. Noise. 

Data may contain erroneous or anomalous values, which are usually referred to as outliers. Other possible causes of noise are to be sought in malfunctioning devices for data measurement, recording, and transmission. The presence of data expressed in heterogeneous measurement units, which therefore require conversion, may in turn cause anomalies and inaccuracies. 

Inconsistency. Sometimes data contain discrepancies due to changes in the coding system used for their representation and therefore may appear inconsistent. For example, the coding of the products manufactured by a company may be subject to a revision taking effect on a given date, without the data recorded in previous periods being subject to the necessary transformations in order to adapt them to the revised encoding scheme. 

The purpose of data validation techniques is to identify and implement corrective actions in case of incomplete and inconsistent data or data affected by noise.

Business intelligence systems and mathematical models for decision-making can achieve accurate and effective results only when the input data are highly reliable. 

However, the data extracted from the available primary sources and gathered into a data mart may have several anomalies which analysts must identify and correct. 

This chapter deals with the activities involved in the creation of a high-quality dataset for subsequent use for business intelligence and data mining analysis. Several techniques can be employed to reach this goal: data validation, to identify and remove anomalies and inconsistencies; data integration and transformation, to improve the accuracy and efficiency of learning algorithms; data size reduction and discretization, to obtain a dataset with a lower number of attributes and records but which is as informative as the original dataset.