Now since there is enormous increase in data sets, it is necessary to contain them so that the computational cost is bearable. By doing so it is possible to have more number of experiments instead of gathering more information from limited number of experiments, refitting the models instead of changing conditions, allow for cross validation, bagging, boosting, bootstrapping like intensive methods, and using more sophisticated statistical methods on compressed data sets. This will help making use of the computational resources more efficiently. Imbalanced data sets are common nowadays. Data imbalances are of two types namely, marginal imbalance such as data sets for predicting click-through rates on online advertising, and conditional imbalance like email spam filtering where mistakes are uncommon. The data sets can have such imbalances. The approach to make the data balanced is by subsampling the original data that enriches rare events. This article proposes a data reduction scheme called local-control sampling to be used for fitting logistic regression models. This approach requires one parallelizable scan over the full data set that yields a potentially much smaller subsample containing almost half of the information as in the original data set. (20 refs.)
展开▼
