Houses_feat_set2.Rmd

---
# Get YAML keywords from myYAML_ref.Rmd
title: "UofWashington:MachineLearning:House Prices:: price regression:: Houses_feat_set2"
author: "bdanalytics"

# Choose one:
output:
    html_document:
        keep_md: yes
        pandoc_args: ["+RTS", "-K64M", "-RTS"]

# output:
#   pdf_document:
#     fig_width: 8
#     highlight: zenburn
#     #keep_md: yes
#     keep_tex: yes
#     number_sections: yes
#     toc: yes
---

**  **    
**Date: `r format(Sys.time(), "(%a) %b %d, %Y")`**    

# Introduction:  

Data: Washington Sate:Kings County:Seattle House Prices 
Source: 
    Training:   home_data.csv  
    New:        <obsNewFileName>  
Time period: 

```{r set_global_options_wd, echo=FALSE}
# Rename 
#   indep_vars_vctr -> indep_vars ; create a function ?
setwd("~/Documents/Work/Courses/Coursera/uwashington/uwashington-machinelearning/notebooks")
# print(names(sys.frame(1)))
# print(getSrcFilename(function (x) { x }))
#print(sys.frame(1)$file)
# stop(here")
```

# Synopsis:

Based on analysis utilizing <> techniques, <conclusion heading>:  

Summary of key steps & error improvement stats:

### Prediction Accuracy Enhancement Options:
- transform.data chunk:
    - derive features from multiple features
    
- manage.missing.data chunk:
    - Not fill missing vars
    - Fill missing numerics with a different algorithm
    - Fill missing chars with data based on clusters 
    
### ![](<filename>.png)

## Potential next steps include:
- Organization:
    - Categorize by chunk
    - Priority criteria:
        0. Ease of change
        1. Impacts report
        2. Cleans innards
        3. Bug report
        
- all chunks:
    - at chunk-end rm(!glb_<var>)
    
- manage.missing.data chunk:
    - cleaner way to manage re-splitting of training vs. new entity

- extract.features chunk:
    - Add n-grams for glbFeatsText
        - "RTextTools", "tau", "RWeka", and "textcat" packages
    
- fit.models chunk:
    - Prediction accuracy scatter graph:
    -   Add tiles (raw vs. PCA)
    -   Use shiny for drop-down of "important" features
    -   Use plot.ly for interactive plots ?
    
    - Change .fit suffix of model metrics to .mdl if it's data independent (e.g. AIC, Adj.R.Squared - is it truly data independent ?, etc.)
    - create a custom model for rpart that has minbucket as a tuning parameter
    - varImp for randomForest crashes in caret version:6.0.41 -> submit bug report

- Probability handling for multinomials vs. desired binomial outcome
-   ROCR currently supports only evaluation of binary classification tasks (version 1.0.7)
-   extensions toward multiclass classification are scheduled for the next release

- fit.all.training chunk:
    - myplot_prediction_classification: displays 'x' instead of '+' when there are no prediction errors 
- Compare glb_sel_mdl vs. glb_fin_mdl:
    - varImp
    - Prediction differences (shd be minimal ?)

- Move glb_analytics_diag_plots to mydsutils.R: (+) Easier to debug (-) Too many glb vars used
- Add print(ggplot.petrinet(glb_analytics_pn) + coord_flip()) at the end of every major chunk
- Parameterize glb_analytics_pn
- Move glb_impute_missing_data to mydsutils.R: (-) Too many glb vars used; glb_<>_df reassigned
- Do non-glm methods handle interaction terms ?
- f-score computation for classifiers should be summation across outcomes (not just the desired one ?)
- Add accuracy computation to glb_dmy_mdl in predict.data.new chunk
- Why does splitting fit.data.training.all chunk into separate chunks add an overhead of ~30 secs ? It's not rbind b/c other chunks have lower elapsed time. Is it the number of plots ?
- Incorporate code chunks in print_sessionInfo
- Test against 
    - projects in github.com/bdanalytics
    - lectures in jhu-datascience track

# Analysis: 
```{r set_global_options}
rm(list = ls())
set.seed(12345)
options(stringsAsFactors = FALSE)
source("~/Dropbox/datascience/R/myscript.R")
source("~/Dropbox/datascience/R/mydsutils.R")
source("~/Dropbox/datascience/R/myplot.R")
source("~/Dropbox/datascience/R/mypetrinet.R")
source("~/Dropbox/datascience/R/myplclust.R")
source("~/Dropbox/datascience/R/mytm.R")
# Gather all package requirements here
suppressPackageStartupMessages(require(doMC))
glbCores <- 6 # of cores on machine - 2
registerDoMC(glbCores) 

suppressPackageStartupMessages(require(caret))
require(plyr)
require(dplyr)
#source("dbgcaret.R")
#packageVersion("snow")
#require(sos); findFn("cosine", maxPages=2, sortby="MaxScore")

# Analysis control global variables
# Inputs
#   url/name = "<pointer>"; sep = choose from c(NULL, "\t")
glbObsTrnFile <- list(name = "home_data.csv") 

glbObsNewFile <- #list(url = "<obsNewFileName>") # default OR
    #list(splitSpecs = list(method = NULL #select from c(NULL, "condition", "sample", "copy")
    #                      ,nRatio = 0.3 # > 0 && < 1 if method == "sample" 
    #                      ,seed = 123 # any integer or glbObsTrnPartitionSeed if method == "sample" 
    #                      )
    #    )                   
    list(splitSpecs = list(method = "sample", nRatio = 0.2, seed = 0))

glbInpMerge <- NULL #: default
#     list(fnames = c("<fname1>", "<fname2>")) # files will be concatenated

glb_is_separate_newobs_dataset <- FALSE    # or TRUE
    glb_split_entity_newobs_datasets <- TRUE  # FALSE not supported - use "copy" for glbObsNewFile$splitSpecs$method # select from c(FALSE, TRUE)
    glb_split_newdata_condition <- NULL # or "is.na(<var>)"; "<var> <condition_operator> <value>"

glbObsDropCondition <- NULL # : default
#   enclose in single-quotes b/c condition might include double qoutes
#       use | & ; NOT || &&    
#   '<condition>' 
    # 'grepl("^First Draft Video:", glbObsAll$Headline)'
#nrow(do.call("subset",list(glbObsAll, parse(text=paste0("!(", glbObsDropCondition, ")")))))
    
glb_obs_repartition_train_condition <- NULL # : default
#    "<condition>" 

glb_max_fitobs <- NULL # or any integer
glbObsTrnPartitionSeed <- 123 # or any integer
                         
glb_is_regression <- TRUE; glb_is_classification <- !glb_is_regression; 
    glb_is_binomial <- NULL # or TRUE or FALSE

glb_rsp_var_raw <- "price"

# for classification, the response variable has to be a factor
glb_rsp_var <- glb_rsp_var_raw # or "price.fctr"

# if the response factor is based on numbers/logicals e.g (0/1 OR TRUE/FALSE vs. "A"/"B"), 
#   or contains spaces (e.g. "Not in Labor Force")
#   caret predict(..., type="prob") crashes
glb_map_rsp_raw_to_var <- NULL 
# function(raw) {
#     return(raw ^ 0.5)
#     return(log(raw))
#     return(log(1 + raw))
#     return(log10(raw)) 
#     return(exp(-raw / 2))
#     ret_vals <- rep_len(NA, length(raw)); ret_vals[!is.na(raw)] <- ifelse(raw[!is.na(raw)] == 1, "Y", "N"); return(relevel(as.factor(ret_vals), ref="N"))
#     as.factor(paste0("B", raw))
#     as.factor(gsub(" ", "\\.", raw))    
#     }

#if glb_rsp_var_raw is numeric:
#print(summary(glbObsAll[, glb_rsp_var_raw]))
#glb_map_rsp_raw_to_var(tst <- c(NA, as.numeric(summary(glbObsAll[, glb_rsp_var_raw])))) 

#if glb_rsp_var_raw is character:
#print(table(glbObsAll[, glb_rsp_var_raw]))
#glb_map_rsp_raw_to_var(tst <- c(NA, names(table(glbObsAll[, glb_rsp_var_raw])))) 

glb_map_rsp_var_to_raw <- NULL 
# function(var) {
#     return(var ^ 2.0)
#     return(exp(var))
#     return(10 ^ var) 
#     return(-log(var) * 2)
#     as.numeric(var)
#     gsub("\\.", " ", levels(var)[as.numeric(var)])
#     c("<=50K", " >50K")[as.numeric(var)]
#     c(FALSE, TRUE)[as.numeric(var)]
# }
# glb_map_rsp_var_to_raw(glb_map_rsp_raw_to_var(tst))

if ((glb_rsp_var != glb_rsp_var_raw) && is.null(glb_map_rsp_raw_to_var))
    stop("glb_map_rsp_raw_to_var function expected")

# List info gathered for various columns
# <col_name>:   <description>; <notes>
# 'condition', # condition of house				
# 'grade', # measure of quality of construction				
# 'waterfront', # waterfront property				
# 'view', # type of view				
# 'sqft_above', # square feet above ground				
# 'sqft_basement', # square feet in basement				
# 'yr_built', # the year built				
# 'yr_renovated', # the year renovated				
# 'lat', 'long', # the lat-long of the parcel				
# 'sqft_living15', # average sq.ft. of 15 nearest neighbors 				
# 'sqft_lot15', # average lot size of 15 nearest neighbors 

# currently does not handle more than 1 column; consider concatenating multiple columns
# If glbFeatsId == NULL, ".rownames <- as.numeric(row.names())" is the default
glbFeatsId <- "id.date" # choose from c(NULL : default, "<id_feat>") 
glbFeatsCategory <- "sqft.living.cut.fctr" # choose from c(NULL : default, "<category_feat>")

# User-specified exclusions
glbFeatsExclude <- c(NULL
#   Feats that shd be excluded due to known causation by prediction variable
# , "<feat1", "<feat2>"
#   Feats that are linear combinations (alias in glm)
#   Feature-engineering phase -> start by excluding all features except id & category & work each one in
    ,"id","date"
    ,".pos","sqft.living.cut.fctr"
    #,"sqft_living"
    #,"bedrooms","bathrooms","sqft_lot","floors","zipcode"
    #,"condition","grade","waterfront","view","sqft_above","sqft_basement","yr_built","yr_renovated"
    #   ,"lat","long","sqft_living15","sqft_lot15"
                    ) 
if (glb_rsp_var_raw != glb_rsp_var)
    glbFeatsExclude <- union(glbFeatsExclude, glb_rsp_var_raw)                    

glbFeatsInteractionOnly <- list()
#glbFeatsInteractionOnly[["<child_feat>"]] <- "<parent_feat>"

glb_drop_vars <- c(NULL
                # , "<feat1>", "<feat2>"
                )

glb_map_vars <- NULL # or c("<var1>", "<var2>")
glb_map_urls <- list();
# glb_map_urls[["<var1>"]] <- "<var1.url>"

glb_assign_pairs_lst <- NULL; 
# glb_assign_pairs_lst[["<var1>"]] <- list(from=c(NA),
#                                            to=c("NA.my"))
glb_assign_vars <- names(glb_assign_pairs_lst)

# Derived features; Use this mechanism to cleanse data ??? Cons: Data duplication ???
glbFeatsDerive <- list();

# glbFeatsDerive[["<feat.my.sfx>"]] <- list(
#     mapfn = function(<arg1>, <arg2>) { return(function(<arg1>, <arg2>)) } 
#   , args = c("<arg1>", "<arg2>"))

    # character
#     mapfn = function(Week) { return(substr(Week, 1, 10)) }

#     mapfn = function(descriptor) { return(plyr::revalue(descriptor, c(
#         "ABANDONED BUILDING"  = "OTHER",
#         "**"                  = "**"
#                                           ))) }

#     mapfn = function(description) { mod_raw <- description;
    # This is here because it does not work if it's in txt_map_filename
#         mod_raw <- gsub(paste0(c("\n", "\211", "\235", "\317", "\333"), collapse = "|"), " ", mod_raw)
    # Don't parse for "." because of ".com"; use customized gsub for that text
#         mod_raw <- gsub("(\\w)(!|\\*|,|-|/)(\\w)", "\\1\\2 \\3", mod_raw);
    # Some state acrnoyms need context for separation e.g. 
    #   LA/L.A. could either be "Louisiana" or "LosAngeles"
        # modRaw <- gsub("\\bL\\.A\\.( |,|')", "LosAngeles\\1", modRaw);
    #   OK/O.K. could either be "Oklahoma" or "Okay"
#         modRaw <- gsub("\\bACA OK\\b", "ACA OKay", modRaw); 
#         modRaw <- gsub("\\bNow O\\.K\\.\\b", "Now OKay", modRaw);        
    #   PR/P.R. could either be "PuertoRico" or "Public Relations"        
        # modRaw <- gsub("\\bP\\.R\\. Campaign", "PublicRelations Campaign", modRaw);        
    #   VA/V.A. could either be "Virginia" or "VeteransAdministration"        
        # modRaw <- gsub("\\bthe V\\.A\\.\\:", "the VeteranAffairs:", modRaw);
    #   
    # Custom mods

#         return(mod_raw) }

    # numeric
# Create feature based on record position/id in data   
glbFeatsDerive[[".pos"]] <- list(
    mapfn = function(.rnorm) { return(1:length(.rnorm)) }       
    , args = c(".rnorm"))    

glbFeatsDerive[["id.date"]] <- list(
    mapfn = function(id, date) { return(paste(as.character(id), as.character(date), sep = "#")) }       
    , args = c("id", "date"))    

glbFeatsDerive[["sqft.living.cut.fctr"]] <- list(
    mapfn = function(sqft_living) { return(cut(sqft_living, breaks = c(0, 1427, 1910, 2550, 14000))) }
    ,args = c("sqft_living"))

# Add logs of numerics that are not distributed normally
#   Derive & keep multiple transformations of the same feature, if normality is hard to achieve with just one transformation
#   Right skew: logp1; sqrt; ^ 1/3; logp1(logp1); log10; exp(-<feat>/constant)
# glbFeatsDerive[["WordCount.log1p"]] <- list(
#     mapfn = function(WordCount) { return(log1p(WordCount)) } 
#   , args = c("WordCount"))
# glbFeatsDerive[["WordCount.root2"]] <- list(
#     mapfn = function(WordCount) { return(WordCount ^ (1/2)) } 
#   , args = c("WordCount"))
# glbFeatsDerive[["WordCount.nexp"]] <- list(
#     mapfn = function(WordCount) { return(exp(-WordCount)) } 
#   , args = c("WordCount"))
#print(summary(glbObsAll$WordCount))
#print(summary(mapfn(glbObsAll$WordCount)))
    
#     mapfn = function(HOSPI.COST) { return(cut(HOSPI.COST, 5, breaks = c(0, 100000, 200000, 300000, 900000), labels = NULL)) }     
#     mapfn = function(Rasmussen)  { return(ifelse(sign(Rasmussen) >= 0, 1, 0)) } 
#     mapfn = function(startprice) { return(startprice ^ (1/2)) }       
#     mapfn = function(startprice) { return(log(startprice)) }   
#     mapfn = function(startprice) { return(exp(-startprice / 20)) }
#     mapfn = function(startprice) { return(scale(log(startprice))) }     
#     mapfn = function(startprice) { return(sign(sprice.predict.diff) * (abs(sprice.predict.diff) ^ (1/10))) }        

    # factor      
#     mapfn = function(PropR) { return(as.factor(ifelse(PropR >= 0.5, "Y", "N"))) }
#     mapfn = function(productline, description) { as.factor(gsub(" ", "", productline)) }
#     mapfn = function(purpose) { return(relevel(as.factor(purpose), ref="all_other")) }
#     mapfn = function(raw) { tfr_raw <- as.character(cut(raw, 5)); 
#                             tfr_raw[is.na(tfr_raw)] <- "NA.my";
#                             return(as.factor(tfr_raw)) }
#     mapfn = function(startprice.log10) { return(cut(startprice.log10, 3)) }
#     mapfn = function(startprice.log10) { return(cut(sprice.predict.diff, c(-1000, -100, -10, -1, 0, 1, 10, 100, 1000))) }    

#     , args = c("<arg1>"))
    
    # multiple args    
#     mapfn = function(id, date) { return(paste(as.character(id), as.character(date), sep = "#")) }        
#     mapfn = function(PTS, oppPTS) { return(PTS - oppPTS) }
#     mapfn = function(startprice.log10.predict, startprice) {
#                  return(spdiff <- (10 ^ startprice.log10.predict) - startprice) } 
#     mapfn = function(productline, description) { as.factor(
#         paste(gsub(" ", "", productline), as.numeric(nchar(description) > 0), sep = "*")) }

# # If glbObsAll is not sorted in the desired manner
#     mapfn=function(Week) { return(coredata(lag(zoo(orderBy(~Week, glbObsAll)$ILI), -2, na.pad=TRUE))) }
#     mapfn=function(ILI) { return(coredata(lag(zoo(ILI), -2, na.pad=TRUE))) }
#     mapfn=function(ILI.2.lag) { return(log(ILI.2.lag)) }

# glbFeatsDerive[["<var1>"]] <- glbFeatsDerive[["<var2>"]]

glb_derive_vars <- names(glbFeatsDerive)

# tst <- "descr.my"; args_lst <- NULL; for (arg in glbFeatsDerive[[tst]]$args) args_lst[[arg]] <- glbObsAll[, arg]; print(head(args_lst[[arg]])); print(head(drv_vals <- do.call(glbFeatsDerive[[tst]]$mapfn, args_lst))); 
# print(which_ix <- which(args_lst[[arg]] == 0.75)); print(drv_vals[which_ix]); 

glbFeatsDateTime <- list()
# glbFeatsDateTime[["<DateTimeFeat>"]] <- 
#     c(format = "%Y-%m-%d %H:%M:%S", timezone = "America/New_York", impute.na = TRUE, 
#       last.ctg = TRUE, poly.ctg = TRUE)

glbFeatsPrice <- NULL # or c("<price_var>")

glbFeatsText <- list()
Sys.setlocale("LC_ALL", "C") # For english

#glbFeatsText[["<TextFeature>"]] <- list(NULL,
#   ,names = myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL, 
#       <comma-separated-screened-names>
#   ))))
#   ,rareWords = myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL, 
#       <comma-separated-nonSCOWL-words>
#   ))))
#)

# Text Processing Step: custom modifications not present in txt_munge -> use glbFeatsDerive
# Text Processing Step: universal modifications
glb_txt_munge_filenames_pfx <- "<projectId>_mytxt_"

# Text Processing Step: tolower
# Text Processing Step: myreplacePunctuation
# Text Processing Step: removeWords
glb_txt_stop_words <- list()
# Remember to use unstemmed words
if (length(glbFeatsText) > 0) {
    require(tm)
    require(stringr)

    glb_txt_stop_words[["<txt_var>"]] <- sort(myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
        # Remove any words from stopwords            
#         , setdiff(myreplacePunctuation(stopwords("english")), c("<keep_wrd1>", <keep_wrd2>"))
                                
        # Remove salutations
        ,"mr","mrs","dr","Rev"                                

        # Remove misc
        #,"th" # Happy [[:digit::]]+th birthday 

        # Remove terms present in Trn only or New only; search for "Partition post-stem"
        #   ,<comma-separated-terms>        

        # cor.y.train == NA
#         ,unlist(strsplit(paste(c(NULL
#           ,"<comma-separated-terms>"
#         ), collapse=",")

        # freq == 1; keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>

        # chisq.pval high (e.g. == 1); keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>

        # nzv.freqRatio high (e.g. >= glbFeatsNzvFreqMax); keep c("<comma-separated-terms-to-keep>")
            # ,<comma-separated-terms>        
                                            )))))
}
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^man", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
#glbObsAll[glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, 4866] > 0, c(glb_rsp_var, txtFeat)]

# To identify terms with a specific freq
#paste0(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], freq == 1)$term), collapse = ",")
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], freq <= 2)$term), collapse = ",")
#subset(glb_post_stem_words_terms_df_lst[[txtFeat]], term %in% c("zinger"))

# To identify terms with a specific freq & 
#   are not stemmed together later OR is value of color.fctr (e.g. gold)
#paste0(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], (freq == 1) & !(term %in% c("blacked","blemish","blocked","blocks","buying","cables","careful","carefully","changed","changing","chargers","cleanly","cleared","connect","connects","connected","contains","cosmetics","default","defaulting","defective","definitely","describe","described","devices","displays","drop","drops","engravement","excellant","excellently","feels","fix","flawlessly","frame","framing","gentle","gold","guarantee","guarantees","handled","handling","having","install","iphone","iphones","keeped","keeps","known","lights","line","lining","liquid","liquidation","looking","lots","manuals","manufacture","minis","most","mostly","network","networks","noted","opening","operated","performance","performs","person","personalized","photograph","physically","placed","places","powering","pre","previously","products","protection","purchasing","returned","rotate","rotation","running","sales","second","seconds","shipped","shuts","sides","skin","skinned","sticker","storing","thats","theres","touching","unusable","update","updates","upgrade","weeks","wrapped","verified","verify") ))$term), collapse = ",")

#print(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (freq <= 2)))
#glbObsAll[which(terms_mtrx[, 229] > 0), glbFeatsText]

# To identify terms with cor.y == NA
#orderBy(~-freq+term, subset(glb_post_stop_words_terms_df_lst[[txtFeat]], is.na(cor.y)))
#paste(sort(subset(glb_post_stop_words_terms_df_lst[[txtFeat]], is.na(cor.y))[, "term"]), collapse=",")
#orderBy(~-freq+term, subset(glb_post_stem_words_terms_df_lst[[txtFeat]], is.na(cor.y)))

# To identify terms with low cor.y.abs
#head(orderBy(~cor.y.abs+freq+term, subset(glb_post_stem_words_terms_df_lst[[txtFeat]], !is.na(cor.y))), 5)

# To identify terms with high chisq.pval
#subset(glb_post_stem_words_terms_df_lst[[txtFeat]], chisq.pval > 0.99)
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (chisq.pval > 0.99) & (freq <= 10))$term), collapse=",")
#paste0(sort(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (chisq.pval > 0.9))$term), collapse=",")
#head(orderBy(~-chisq.pval+freq+term, glb_post_stem_words_terms_df_lst[[txtFeat]]), 5)
#glbObsAll[glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, 68] > 0, glbFeatsText]
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^m", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])

# To identify terms with high nzv.freqRatio
#summary(glb_post_stem_words_terms_df_lst[[txtFeat]]$nzv.freqRatio)
#paste0(sort(setdiff(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], (nzv.freqRatio >= glbFeatsNzvFreqMax) & (freq < 10) & (chisq.pval >= 0.05))$term, c( "128gb","3g","4g","gold","ipad1","ipad3","ipad4","ipadair2","ipadmini2","manufactur","spacegray","sprint","tmobil","verizon","wifion"))), collapse=",")

# To identify obs with a txt term
#tail(orderBy(~-freq+term, glb_post_stop_words_terms_df_lst[[txtFeat]]), 20)
#mydspObs(list(descr.my.contains="non"), cols=c("color", "carrier", "cellular", "storage"))
#grep("ever", dimnames(terms_stop_mtrx)$Terms)
#which(terms_stop_mtrx[, grep("ipad", dimnames(terms_stop_mtrx)$Terms)] > 0)
#glbObsAll[which(terms_stop_mtrx[, grep("16", dimnames(terms_stop_mtrx)$Terms)[1]] > 0), c(glbFeatsCategory, "storage", txtFeat)]

# Text Processing Step: screen for names # Move to glbFeatsText specs section in order of text processing steps
# glbFeatsText[["<txtFeat>"]]$names <- myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
#         # Person names for names screening
#         ,<comma-separated-list>
#         
#         # Company names
#         ,<comma-separated-list>
#                     
#         # Product names
#         ,<comma-separated-list>
#     ))))

# glbFeatsText[["<txtFeat>"]]$rareWords <- myreplacePunctuation(str_to_lower(gsub(" ", "", c(NULL
#         # Words not in SCOWL db
#         ,<comma-separated-list>
#     ))))

# To identify char vectors post glbFeatsTextMap
#grep("six(.*)hour", glb_txt_chr_lst[[txtFeat]], ignore.case = TRUE, value = TRUE)
#grep("[S|s]ix(.*)[H|h]our", glb_txt_chr_lst[[txtFeat]], value = TRUE)

# To identify whether terms shd be synonyms
#orderBy(~term, glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^moder", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ])
# term_row_df <- glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^came$", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ]
# 
# cor(glb_post_stop_words_terms_mtrx_lst[[txtFeat]][glbObsAll$.lcn == "Fit", term_row_df$pos], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")

# To identify which stopped words are "close" to a txt term
#sort(cluster_vars)

# Text Processing Step: stemDocument
# To identify stemmed txt terms
#glb_post_stop_words_terms_df_lst[[txtFeat]][grep("^la$", glb_post_stop_words_terms_df_lst[[txtFeat]]$term), ]
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^con", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
#glbObsAll[which(terms_stem_mtrx[, grep("use", dimnames(terms_stem_mtrx)$Terms)[[1]]] > 0), c(glbFeatsId, "productline", txtFeat)]
#glbObsAll[which(TfIdf_stem_mtrx[, 191] > 0), c(glbFeatsId, glbFeatsCategory, txtFeat)]
#glbObsAll[which(glb_post_stop_words_terms_mtrx_lst[[txtFeat]][, 6165] > 0), c(glbFeatsId, glbFeatsCategory, txtFeat)]
#which(glbObsAll$UniqueID %in% c(11915, 11926, 12198))

# Text Processing Step: mycombineSynonyms
#   To identify which terms are associated with not -> combine "could not" & "couldn't"
#findAssocs(glb_full_DTM_lst[[txtFeat]], "not", 0.05)
#   To identify which synonyms should be combined
#orderBy(~term, glb_post_stem_words_terms_df_lst[[txtFeat]][grep("^c", glb_post_stem_words_terms_df_lst[[txtFeat]]$term), ])
chk_comb_cor <- function(syn_lst) {
#     cor(terms_stem_mtrx[glbObsAll$.src == "Train", grep("^(damag|dent|ding)$", dimnames(terms_stem_mtrx)[[2]])], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
    print(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], term %in% syn_lst$syns))
    print(subset(get_corpus_terms(tm_map(glbFeatsTextCorpus[[txtFeat]], mycombineSynonyms, list(syn_lst), lazy=FALSE)), term == syn_lst$word))
#     cor(terms_stop_mtrx[glbObsAll$.src == "Train", grep("^(damage|dent|ding)$", dimnames(terms_stop_mtrx)[[2]])], glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
#     cor(rowSums(terms_stop_mtrx[glbObsAll$.src == "Train", grep("^(damage|dent|ding)$", dimnames(terms_stop_mtrx)[[2]])]), glbObsTrn[, glb_rsp_var], use="pairwise.complete.obs")
}
#chk_comb_cor(syn_lst=list(word="cabl",  syns=c("cabl", "cord")))
#chk_comb_cor(syn_lst=list(word="damag",  syns=c("damag", "dent", "ding")))
#chk_comb_cor(syn_lst=list(word="dent",  syns=c("dent", "ding")))
#chk_comb_cor(syn_lst=list(word="use",  syns=c("use", "usag")))

glbFeatsTextSynonyms <- list()
# list parsed to collect glbFeatsText[[<txtFeat>]]$vldTerms
# glbFeatsTextSynonyms[["Hdln.my"]] <- list(NULL
#     # people in places
#     , list(word = "australia", syns = c("australia", "australian"))
#     , list(word = "italy", syns = c("italy", "Italian"))
#     , list(word = "newyork", syns = c("newyork", "newyorker"))    
#     , list(word = "Pakistan", syns = c("Pakistan", "Pakistani"))    
#     , list(word = "peru", syns = c("peru", "peruvian"))
#     , list(word = "qatar", syns = c("qatar", "qatari"))
#     , list(word = "scotland", syns = c("scotland", "scotish"))
#     , list(word = "Shanghai", syns = c("Shanghai", "Shanzhai"))    
#     , list(word = "venezuela", syns = c("venezuela", "venezuelan"))    
# 
#     # companies - needs to be data dependent 
#     #   - e.g. ensure BNP in this experiment/feat always refers to BNPParibas
#         
#     # general synonyms
#     , list(word = "Create", syns = c("Create","Creator")) 
#     , list(word = "cute", syns = c("cute","cutest"))     
#     , list(word = "Disappear", syns = c("Disappear","Fadeout"))     
#     , list(word = "teach", syns = c("teach", "taught"))     
#     , list(word = "theater",  syns = c("theater", "theatre", "theatres")) 
#     , list(word = "understand",  syns = c("understand", "understood"))    
#     , list(word = "weak",  syns = c("weak", "weaken", "weaker", "weakest"))
#     , list(word = "wealth",  syns = c("wealth", "wealthi"))    
#     
#     # custom synonyms (phrases)
#     
#     # custom synonyms (names)
#                                       )
#glbFeatsTextSynonyms[["<txtFeat>"]] <- list(NULL
#     , list(word="<stem1>",  syns=c("<stem1>", "<stem1_2>"))
#                                       )

for (txtFeat in names(glbFeatsTextSynonyms))
    for (entryIx in 1:length(glbFeatsTextSynonyms[[txtFeat]])) {
        glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$word <-
            str_to_lower(glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$word)
        glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$syns <-
            str_to_lower(glbFeatsTextSynonyms[[txtFeat]][[entryIx]]$syns)        
    }        

glbFeatsTextSeed <- 181
# tm options include: check tm::weightSMART 
glb_txt_terms_control <- list( # Gather model performance & run-time stats
                    # weighting = function(x) weightSMART(x, spec = "nnn")
                    # weighting = function(x) weightSMART(x, spec = "lnn")
                    # weighting = function(x) weightSMART(x, spec = "ann")
                    # weighting = function(x) weightSMART(x, spec = "bnn")
                    # weighting = function(x) weightSMART(x, spec = "Lnn")
                    # 
                    weighting = function(x) weightSMART(x, spec = "ltn") # default
                    # weighting = function(x) weightSMART(x, spec = "lpn")                    
                    # 
                    # weighting = function(x) weightSMART(x, spec = "ltc")                    
                    # 
                    # weighting = weightBin 
                    # weighting = weightTf 
                    # weighting = weightTfIdf # : default
                # termFreq selection criteria across obs: tm default: list(global=c(1, Inf))
                    , bounds = list(global = c(1, Inf)) 
                # wordLengths selection criteria: tm default: c(3, Inf)
                    , wordLengths = c(1, Inf) 
                              ) 

glb_txt_cor_var <- glb_rsp_var # : default # or c(<feat>)

# select one from c("union.top.val.cor", "top.cor", "top.val", default: "top.chisq", "sparse")
glbFeatsTextFilter <- "top.chisq" 
glbFeatsTextTermsMax <- rep(10, length(glbFeatsText)) # :default
names(glbFeatsTextTermsMax) <- names(glbFeatsText)

# Text Processing Step: extractAssoc
glbFeatsTextAssocCor <- rep(1, length(glbFeatsText)) # :default 
names(glbFeatsTextAssocCor) <- names(glbFeatsText)

# Remember to use stemmed terms
glb_important_terms <- list()

# Text Processing Step: extractPatterns (ngrams)
glbFeatsTextPatterns <- list()
#glbFeatsTextPatterns[[<txtFeat>>]] <- list()
#glbFeatsTextPatterns[[<txtFeat>>]] <- c(metropolitan.diary.colon = "Metropolitan Diary:")

# Have to set it even if it is not used
# Properties:
#   numrows(glb_feats_df) << numrows(glbObsFit
#   Select terms that appear in at least 0.2 * O(FP/FN(glbObsOOB)) ???
#       numrows(glbObsOOB) = 1.1 * numrows(glbObsNew) ???
glb_sprs_thresholds <- NULL # or c(<txtFeat1> = 0.988, <txtFeat2> = 0.970, <txtFeat3> = 0.970)

glbFctrMaxUniqVals <- 20 # default: 20
glb_impute_na_data <- FALSE # or TRUE
glb_mice_complete.seed <- 144 # or any integer

glb_cluster <- FALSE # : default or TRUE
glb_cluster.seed <- 189 # or any integer
glb_cluster_entropy_var <- NULL # c(glb_rsp_var, as.factor(cut(glb_rsp_var, 3)), default: NULL)
glbFeatsTextClusterVarsExclude <- FALSE # default FALSE

glb_interaction_only_feats <- NULL # : default or c(<parent_feat> = "<child_feat>")

glbFeatsNzvFreqMax <- 19 # 19 : caret default
glbFeatsNzvUniqMin <- 10 # 10 : caret default

glbRFESizes <- list()
#glbRFESizes[["mdlFamily"]] <- c(4, 8, 16, 32, 64, 67, 68, 69) # Accuracy@69/70 = 0.8258

glbObsFitOutliers <- list()
# If outliers.n >= 10; consider concatenation of interaction vars
# glbObsFitOutliers[["<mdlFamily>"]] <- c(NULL
    # is.na(.rstudent)
    # is.na(.dffits)
    # .hatvalues >= 0.99        
    # -38,167,642 < minmax(.rstudent) < 49,649,823    
#     , <comma-separated-<glbFeatsId>>
#                                     )
glbObsTrnOutliers <- list()

# influence.measures: car::outlier; rstudent; dffits; hatvalues; dfbeta; dfbetas
#mdlId <- "RFE.X.glm"; obs_df <- fitobs_df
#mdlId <- "Final.glm"; obs_df <- trnobs_df
#mdlId <- "CSM2.X.glm"; obs_df <- fitobs_df
#print(outliers <- car::outlierTest(glb_models_lst[[mdlId]]$finalModel))
#mdlIdFamily <- paste0(head(unlist(str_split(mdlId, "\\.")), -1), collapse="."); obs_df <- dplyr::filter_(obs_df, interp(~(!(var %in% glbObsFitOutliers[[mdlIdFamily]])), var = as.name(glbFeatsId))); model_diags_df <- cbind(obs_df, data.frame(.rstudent=stats::rstudent(glb_models_lst[[mdlId]]$finalModel)), data.frame(.dffits=stats::dffits(glb_models_lst[[mdlId]]$finalModel)), data.frame(.hatvalues=stats::hatvalues(glb_models_lst[[mdlId]]$finalModel)));print(summary(model_diags_df[, c(".rstudent",".dffits",".hatvalues")])); table(cut(model_diags_df$.hatvalues, breaks=c(0.00, 0.98, 0.99, 1.00)))

#print(subset(model_diags_df, is.na(.rstudent))[, glbFeatsId])
#print(subset(model_diags_df, is.na(.dffits))[, glbFeatsId])
#print(model_diags_df[which.min(model_diags_df$.dffits), ])
#print(subset(model_diags_df, .hatvalues > 0.99)[, glbFeatsId])
#dffits_df <- merge(dffits_df, outliers_df, by="row.names", all.x=TRUE); row.names(dffits_df) <- dffits_df$Row.names; dffits_df <- subset(dffits_df, select=-Row.names)
#dffits_df <- merge(dffits_df, glbObsFit, by="row.names", all.x=TRUE); row.names(dffits_df) <- dffits_df$Row.names; dffits_df <- subset(dffits_df, select=-Row.names)
#subset(dffits_df, !is.na(.Bonf.p))

#mdlId <- "CSM.X.glm"; vars <- myextract_actual_feats(row.names(orderBy(reformulate(c("-", paste0(mdlId, ".imp"))), myget_feats_imp(glb_models_lst[[mdlId]])))); 
#model_diags_df <- glb_get_predictions(model_diags_df, mdlId, glb_rsp_var)
#obs_ix <- row.names(model_diags_df) %in% names(outliers$rstudent)[1]
#obs_ix <- which(is.na(model_diags_df$.rstudent))
#obs_ix <- which(is.na(model_diags_df$.dffits))
#myplot_parcoord(obs_df=model_diags_df[, c(glbFeatsId, glbFeatsCategory, ".rstudent", ".dffits", ".hatvalues", glb_rsp_var, paste0(glb_rsp_var, mdlId), vars[1:min(20, length(vars))])], obs_ix=obs_ix, id_var=glbFeatsId, category_var=glbFeatsCategory)

#model_diags_df[row.names(model_diags_df) %in% names(outliers$rstudent)[c(1:2)], ]
#ctgry_diags_df <- model_diags_df[model_diags_df[, glbFeatsCategory] %in% c("Unknown#0"), ]
#myplot_parcoord(obs_df=ctgry_diags_df[, c(glbFeatsId, glbFeatsCategory, ".rstudent", ".dffits", ".hatvalues", glb_rsp_var, "startprice.log10.predict.RFE.X.glmnet", indep_vars[1:20])], obs_ix=row.names(ctgry_diags_df) %in% names(outliers$rstudent)[1], id_var=glbFeatsId, category_var=glbFeatsCategory)
#table(glbObsFit[model_diags_df[, glbFeatsCategory] %in% c("iPad1#1"), "startprice.log10.cut.fctr"])
#glbObsFit[model_diags_df[, glbFeatsCategory] %in% c("iPad1#1"), c(glbFeatsId, "startprice")]

# No outliers & .dffits == NaN
#myplot_parcoord(obs_df=model_diags_df[, c(glbFeatsId, glbFeatsCategory, glb_rsp_var, "startprice.log10.predict.RFE.X.glmnet", indep_vars[1:10])], obs_ix=seq(1:nrow(model_diags_df))[is.na(model_diags_df$.dffits)], id_var=glbFeatsId, category_var=glbFeatsCategory)

# Modify mdlId to (build & extract) "<FamilyId>#<Fit|Trn>#<caretMethod>#<preProc1.preProc2>#<samplingMethod>"
glb_models_lst <- list(); glb_models_df <- data.frame()
# Regression
if (glb_is_regression) {
    glbMdlMethods <- c(NULL
        # deterministic
            #, "lm", # same as glm
            , "glm", "bayesglm", "glmnet"
            , "rpart"
        # non-deterministic
            , "gbm", "rf" 
        # Unknown
            , "nnet" , "avNNet" # runs 25 models per cv sample for tunelength=5
            , "svmLinear", "svmLinear2"
            , "svmPoly" # runs 75 models per cv sample for tunelength=5
            , "svmRadial" 
            , "earth"
            , "bagEarth" # Takes a long time
        )
} else
# Classification - Add ada (auto feature selection)
    if (glb_is_binomial)
        glbMdlMethods <- c(NULL
        # deterministic                     
            , "bagEarth" # Takes a long time        
            , "glm", "bayesglm", "glmnet"
            , "nnet"
            , "rpart"
        # non-deterministic        
            , "gbm"
            , "avNNet" # runs 25 models per cv sample for tunelength=5      
            , "rf"
        # Unknown
            , "lda", "lda2"
                # svm models crash when predict is called -> internal to kernlab it should call predict without .outcome
            , "svmLinear", "svmLinear2"
            , "svmPoly" # runs 75 models per cv sample for tunelength=5
            , "svmRadial" 
            , "earth"
        ) else
        glbMdlMethods <- c(NULL
        # deterministic
            ,"glmnet"
        # non-deterministic 
            ,"rf"       
        # Unknown
            ,"gbm","rpart"
        )

glbMdlFamilies <- list(); glb_mdl_feats_lst <- list()
# family: Choose from c("RFE.X", "CSM.X", "All.X", "Best.Interact")
#   methods: Choose from c(NULL, <method>, glbMdlMethods) 
#glbMdlFamilies[["RFE.X"]] <- c("glmnet", "glm") # non-NULL vector is mandatory
glbMdlFamilies[["All.X"]] <- c("glmnet", "glm")  # non-NULL vector is mandatory
#glbMdlFamilies[["Best.Interact"]] <- "glmnet" # non-NULL vector is mandatory

# Check if interaction features make RFE better
# glbMdlFamilies[["CSM.X"]] <- setdiff(glbMdlMethods, c("lda", "lda2")) # crashing due to category:.clusterid ??? #c("glmnet", "glm") # non-NULL list is mandatory
# glb_mdl_feats_lst[["CSM.X"]] <- c(NULL
#     , <comma-separated-features-vector>
#                                   )
# dAFeats.CSM.X %<d-% c(NULL
#     # Interaction feats up to varImp(RFE.X.glmnet) >= 50
#     , <comma-separated-features-vector>
#     , setdiff(myextract_actual_feats(predictors(rfe_fit_results)), c(NULL
#                , <comma-separated-features-vector>
#                                                                       ))    
#                                   )
# glb_mdl_feats_lst[["CSM.X"]] <- "%<d-% dAFeats.CSM.X"

glbMdlFamilies[["Final"]] <- c(NULL) # NULL vector acceptable

glbMdlAllowParallel <- list()
#glbMdlAllowParallel[["<mdlId>"]] <- FALSE
glbMdlAllowParallel[["Max.cor.Y##rcv#rpart"]] <- FALSE
glbMdlAllowParallel[["All.X##rcv#glm"]] <- FALSE

# Check if tuning parameters make fit better; make it mdlFamily customizable ?
glbMdlTuneParams <- data.frame()
# When glmnet crashes at model$grid with error: ???
glmnetTuneParams <- rbind(data.frame()
                        ,data.frame(parameter = "alpha",  vals = "0.100 0.325 0.550 0.775 1.000")
                        ,data.frame(parameter = "lambda", vals = "9.342e-02")    
                        )
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams,
#                                cbind(data.frame(mdlId = "<mdlId>"),
#                                      glmnetTuneParams))

    #avNNet    
    #   size=[1] 3 5 7 9; decay=[0] 1e-04 0.001  0.01   0.1; bag=[FALSE]; RMSE=1.3300906 

    #bagEarth
    #   degree=1 [2] 3; nprune=64 128 256 512 [1024]; RMSE=0.6486663 (up)
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "bagEarth", parameter = "nprune", vals = "256")
#     ,data.frame(method = "bagEarth", parameter = "degree", vals = "2")    
# ))

    #earth 
    #   degree=[1]; nprune=2  [9] 17 25 33; RMSE=0.1334478
    
    #gbm 
    #   shrinkage=0.05 [0.10] 0.15 0.20 0.25; n.trees=100 150 200 [250] 300; interaction.depth=[1] 2 3 4 5; n.minobsinnode=[10]; RMSE=0.2008313     
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "gbm", parameter = "shrinkage", min = 0.05, max = 0.25, by = 0.05)
#     ,data.frame(method = "gbm", parameter = "n.trees", min = 100, max = 300, by = 50)
#     ,data.frame(method = "gbm", parameter = "interaction.depth", min = 1, max = 5, by = 1)
#     ,data.frame(method = "gbm", parameter = "n.minobsinnode", min = 10, max = 10, by = 10)
#     #seq(from=0.05,  to=0.25, by=0.05)
# ))

    #glmnet
    #   alpha=0.100 [0.325] 0.550 0.775 1.000; lambda=0.0005232693 0.0024288010 0.0112734954 [0.0523269304] 0.2428800957; RMSE=0.6164891
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "glmnet", parameter = "alpha", vals = "0.550 0.775 0.8875 0.94375 1.000")
#     ,data.frame(method = "glmnet", parameter = "lambda", vals = "9.858855e-05 0.0001971771 0.0009152152 0.0042480525 0.0197177130")    
# ))

    #nnet    
    #   size=3 5 [7] 9 11; decay=0.0001 0.001 0.01 [0.1] 0.2; RMSE=0.9287422
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "nnet", parameter = "size", vals = "3 5 7 9 11")
#     ,data.frame(method = "nnet", parameter = "decay", vals = "0.0001 0.0010 0.0100 0.1000 0.2000")    
# ))

    #rf # Don't bother; results are not deterministic
    #       mtry=2  35  68 [101] 134; RMSE=0.1339974
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "rf", parameter = "mtry", vals = "2 5 9 13 17")
# ))

    #rpart 
    #   cp=0.020 [0.025] 0.030 0.035 0.040; RMSE=0.1770237
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()    
#     ,data.frame(method = "rpart", parameter = "cp", vals = "0.004347826 0.008695652 0.017391304 0.021739130 0.034782609")
# ))
    
    #svmLinear
    #   C=0.01 0.05 [0.10] 0.50 1.00 2.00 3.00 4.00; RMSE=0.1271318; 0.1296718
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "svmLinear", parameter = "C", vals = "0.01 0.05 0.1 0.5 1")
# ))

    #svmLinear2    
    #   cost=0.0625 0.1250 [0.25] 0.50 1.00; RMSE=0.1276354 
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method = "svmLinear2", parameter = "cost", vals = "0.0625 0.125 0.25 0.5 1")
# ))

    #svmPoly    
    #   degree=[1] 2 3 4 5; scale=0.01 0.05 [0.1] 0.5 1; C=0.50 1.00 [2.00] 3.00 4.00; RMSE=0.1276130
# glbMdlTuneParams <- myrbind_df(glbMdlTuneParams, rbind(data.frame()
#     ,data.frame(method="svmPoly", parameter="degree", min=1, max=5, by=1) #seq(1, 5, 1)
#     ,data.frame(method="svmPoly", parameter="scale", vals="0.01, 0.05, 0.1, 0.5, 1")
#     ,data.frame(method="svmPoly", parameter="C", vals="0.50, 1.00, 2.00, 3.00, 4.00")    
# ))

    #svmRadial
    #   sigma=[0.08674323]; C=0.25 0.50 1.00 [2.00] 4.00; RMSE=0.1614957
    
#glb2Sav(); all.equal(sav_models_df, glb_models_df)
    
glb_preproc_methods <- NULL
#     c("YeoJohnson", "center.scale", "range", "pca", "ica", "spatialSign")

# Baseline prediction model feature(s)
glb_Baseline_mdl_var <- NULL # or c("<feat>")

glbMdlMetric_terms <- NULL # or matrix(c(
#                               0,1,2,3,4,
#                               2,0,1,2,3,
#                               4,2,0,1,2,
#                               6,4,2,0,1,
#                               8,6,4,2,0
#                           ), byrow=TRUE, nrow=5)
glbMdlMetricSummary <- NULL # or "<metric_name>"
glbMdlMetricMaximize <- NULL # or FALSE (TRUE is not the default for both classification & regression) 
glbMdlMetricSummaryFn <- NULL # or function(data, lev=NULL, model=NULL) {
#     confusion_mtrx <- t(as.matrix(confusionMatrix(data$pred, data$obs)))
#     #print(confusion_mtrx)
#     #print(confusion_mtrx * glbMdlMetric_terms)
#     metric <- sum(confusion_mtrx * glbMdlMetric_terms) / nrow(data)
#     names(metric) <- glbMdlMetricSummary
#     return(metric)
# }

glbMdlCheckRcv <- FALSE # Turn it on when needed; otherwise takes long time
glb_rcv_n_folds <- 3 # or NULL
glb_rcv_n_repeats <- 3 # or NULL

glb_clf_proba_threshold <- NULL # 0.5

# Model selection criteria
if (glb_is_regression)
    glbMdlMetricsEval <- c("min.RMSE.OOB", "max.R.sq.OOB", "max.Adj.R.sq.fit", "min.RMSE.fit")
    #glbMdlMetricsEval <- c("min.RMSE.fit", "max.R.sq.fit", "max.Adj.R.sq.fit")    
if (glb_is_classification) {
    if (glb_is_binomial)
        glbMdlMetricsEval <- 
            c("max.Accuracy.OOB", "max.AUCROCR.OOB", "max.AUCpROC.OOB", "min.aic.fit", "max.Accuracy.fit") else        
        glbMdlMetricsEval <- c("max.Accuracy.OOB", "max.Kappa.OOB")
}

# select from NULL [no ensemble models], "auto" [all models better than MFO or Baseline], c(mdl_ids in glb_models_lst) [Typically top-rated models in auto]
glb_mdl_ensemble <- NULL
#     "%<d-% setdiff(mygetEnsembleAutoMdlIds(), 'CSM.X.rf')" 
#     c(<comma-separated-mdlIds>
#      )

# Only for classifications; for regressions remove "(.*)\\.prob" form the regex
# tmp_fitobs_df <- glbObsFit[, grep(paste0("^", gsub(".", "\\.", mygetPredictIds$value, fixed = TRUE), "CSM\\.X\\.(.*)\\.prob"), names(glbObsFit), value = TRUE)]; cor_mtrx <- cor(tmp_fitobs_df); cor_vctr <- sort(cor_mtrx[row.names(orderBy(~-Overall, varImp(glb_models_lst[["Ensemble.repeatedcv.glmnet"]])$imp))[1], ]); summary(cor_vctr); cor_vctr
#ntv.glm <- glm(reformulate(indep_vars, glb_rsp_var), family = "binomial", data = glbObsFit)
#step.glm <- step(ntv.glm)

glb_sel_mdl_id <- "All.X##rcv#glmnet" #select from c(NULL, "All.X##rcv#glmnet", "RFE.X##rcv#glmnet", <mdlId>)
glb_fin_mdl_id <- NULL #select from c(NULL, glb_sel_mdl_id)

glb_dsp_cols <- c(glbFeatsId, glbFeatsCategory, glb_rsp_var
#               List critical cols excl. glbFeatsId, glbFeatsCategory & glb_rsp_var
                  )

# Output specs
glbOutDataVizFname <- NULL # choose from c(NULL, "<projectId>_obsall.csv")
glb_out_obs <- NULL # select from c(NULL : default to "new", "all", "new", "trn")
glb_out_vars_lst <- list()
# glbFeatsId will be the first output column, by default

if (glb_is_classification && glb_is_binomial) {
    glb_out_vars_lst[["Probability1"]] <- 
        "%<d-% mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$prob" 
} else {
    glb_out_vars_lst[[glb_rsp_var]] <- 
        "%<d-% mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$value"
}    
# glb_out_vars_lst[[glb_rsp_var_raw]] <- glb_rsp_var_raw
# glb_out_vars_lst[[paste0(head(unlist(strsplit(mygetPredictIds$value, "")), -1), collapse = "")]] <-

glbOutStackFnames <- NULL #: default
    # c("ebayipads_txt_assoc1_out_bid1_stack.csv") # manual stack
    # c("ebayipads_finmdl_bid1_out_nnet_1.csv") # universal stack
glb_out_pfx <- "Houses_feat_set2_"
glb_save_envir <- FALSE # or TRUE

# Depict process
glb_analytics_pn <- petrinet(name = "glb_analytics_pn",
                        trans_df = data.frame(id = 1:6,
    name = c("data.training.all","data.new",
           "model.selected","model.final",
           "data.training.all.prediction","data.new.prediction"),
    x=c(   -5,-5,-15,-25,-25,-35),
    y=c(   -5, 5,  0,  0, -5,  5)
                        ),
                        places_df=data.frame(id=1:4,
    name=c("bgn","fit.data.training.all","predict.data.new","end"),
    x=c(   -0,   -20,                    -30,               -40),
    y=c(    0,     0,                      0,                 0),
    M0=c(   3,     0,                      0,                 0)
                        ),
                        arcs_df = data.frame(
    begin = c("bgn","bgn","bgn",        
            "data.training.all","model.selected","fit.data.training.all",
            "fit.data.training.all","model.final",    
            "data.new","predict.data.new",
            "data.training.all.prediction","data.new.prediction"),
    end   = c("data.training.all","data.new","model.selected",
            "fit.data.training.all","fit.data.training.all","model.final",
            "data.training.all.prediction","predict.data.new",
            "predict.data.new","data.new.prediction",
            "end","end")
                        ))
#print(ggplot.petrinet(glb_analytics_pn))
print(ggplot.petrinet(glb_analytics_pn) + coord_flip())
glb_analytics_avl_objs <- NULL

glb_chunks_df <- myadd_chunk(NULL, "import.data")
```

## Step ``r mydsp_chunk(glb_chunks_df)``
#### chunk option: eval=<r condition>
```{r import.data, cache=FALSE, echo=FALSE} 
#glb_chunks_df <- myadd_chunk(NULL, "import.data")

glb2Sav <- function() {
    savObsAll <<- glbObsAll 
    savObsTrn <<- glbObsTrn
    if (any(grepl("^glbObsFit$", ls(envir=globalenv()))) &&
        !is.null(glbObsFit)) savObsFit <<- glbObsFit    
    if (any(grepl("^glbObsOOB$", ls(envir=globalenv()))) &&
        !is.null(glbObsOOB)) savObsOOB <<- glbObsOOB    
    if (any(grepl("^glbObsNew$", ls(envir=globalenv()))) &&
        !is.null(glbObsNew)) {
        #print("Attempting to save glbObsNew...")
        savObsNew <<- glbObsNew    
    }

    if (any(grepl("^glbFeatsTextCorpus$", ls(envir=globalenv()))) &&
        !is.null(glbFeatsTextCorpus)) savFeatsTextCorpus <<- glbFeatsTextCorpus

    if (any(grepl("^glbLvlCategory$", ls(envir=globalenv()))) &&
        !is.null(glbLvlCategory)) savLvlCategory <<- glbLvlCategory    

    if (!is.null(glb_models_lst )) sav_models_lst  <<- glb_models_lst
    if (!is.null(glb_models_df  )) sav_models_df   <<- glb_models_df

    if (any(grepl("^glb_feats_df$", ls(envir=globalenv()))) &&
        !is.null(glb_feats_df)) sav_feats_df <<- glb_feats_df    
    if (any(grepl("^glb_featsimp_df$", ls(envir=globalenv()))) &&
        !is.null(glb_featsimp_df)) sav_featsimp_df <<- glb_featsimp_df    
}

glbObsTrn <- myimport_data(specs = glbObsTrnFile, comment = "glbObsTrn", force_header = TRUE)
# glbObsTrn <- read.delim("data/hygiene.txt", header=TRUE, fill=TRUE, sep="\t",
#                             fileEncoding='iso-8859-1')
# glbObsTrn <- read.table("data/hygiene.dat.labels", col.names=c("dirty"),
#                             na.strings="[none]")
# glbObsTrn$review <- readLines("data/hygiene.dat", n =-1)
# comment(glbObsTrn) <- "glbObsTrn"                                

# glbObsTrn <- data.frame()
# for (symbol in c("Boeing", "CocaCola", "GE", "IBM", "ProcterGamble")) {
#     sym_trnobs_df <- 
#         myimport_data(url=gsub("IBM", symbol, glbObsTrnFile), comment="glbObsTrn", 
#                                     force_header=TRUE)
#     sym_trnobs_df$Symbol <- symbol
#     glbObsTrn <- myrbind_df(glbObsTrn, sym_trnobs_df)
# }
                                
# glbObsTrn <- 
#     glbObsTrn %>% dplyr::filter(Year >= 1999)
                                
if (glb_is_separate_newobs_dataset) {
    glbObsNew <- myimport_data(specs = glbObsNewFile, comment = "glbObsNew", force_header = TRUE)
    
    # To make plots / stats / checks easier in chunk:inspectORexplore.data
    glbObsAll <- myrbind_df(glbObsTrn, glbObsNew); 
    comment(glbObsAll) <- "glbObsAll"
} else {
    glbObsAll <- glbObsTrn; comment(glbObsAll) <- "glbObsAll"
    if (!glb_split_entity_newobs_datasets) {
        stop("!glb_split_entity_newobs_datasets: not implemented yet") 
        glbObsNew <- glbObsTrn[sample(1:nrow(glbObsTrn),
                                          max(2, nrow(glbObsTrn) / 1000)),]                    
    } else      if (is.null(glbObsNewFile$splitSpecs) || 
                    is.null(glbObsNewFile$splitSpecs$method)) {
            stop("glbObsNewFile$splitSpecs$method required when glb_is_separate_newobs_dataset is FALSE")
        } else  if (glbObsNewFile$splitSpecs$method == "condition") {
            glbObsNew <- do.call("subset", 
                list(glbObsTrn, parse(text=glb_split_newdata_condition)))
            glbObsTrn <- do.call("subset", 
                list(glbObsTrn, parse(text=paste0("!(", 
                                                      glb_split_newdata_condition,
                                                      ")"))))
        } else if (glbObsNewFile$splitSpecs$method == "sample") {
                require(caTools)
                
                set.seed(glbObsNewFile$splitSpecs$seed)
                split <- sample.split(glbObsTrn[, glb_rsp_var_raw], 
                                      SplitRatio = (1 - glbObsNewFile$splitSpecs$nRatio))
                glbObsNew <- glbObsTrn[!split, ] 
                glbObsTrn <- glbObsTrn[split ,]
        } else if (glbObsNewFile$splitSpecs$method == "copy") {  
            glbObsTrn <- glbObsAll
            comment(glbObsTrn) <- "glbObsTrn"
            glbObsNew <- glbObsAll
            comment(glbObsNew) <- "glbObsNew"
        } else stop("glbObsNewFile$splitSpecs$method should be %in% c('condition', 'sample', 'copy')")   

    comment(glbObsNew) <- "glbObsNew"
    myprint_df(glbObsNew)
    str(glbObsNew)

    if (glb_split_entity_newobs_datasets) {
        myprint_df(glbObsTrn)
        str(glbObsTrn)        
    }
}         

if ((num_nas <- sum(is.na(glbObsTrn[, glb_rsp_var_raw]))) > 0)
    stop("glbObsTrn$", glb_rsp_var_raw, " contains NAs for ", num_nas, " obs")

if (nrow(glbObsTrn) == nrow(glbObsAll))
    warning("glbObsTrn same as glbObsAll")
if (nrow(glbObsNew) == nrow(glbObsAll))
    warning("glbObsNew same as glbObsAll")

if (length(glb_drop_vars) > 0) {
    warning("dropping vars: ", paste0(glb_drop_vars, collapse=", "))
    glbObsAll <- glbObsAll[, setdiff(names(glbObsAll), glb_drop_vars)]
    glbObsTrn <- glbObsTrn[, setdiff(names(glbObsTrn), glb_drop_vars)]    
    glbObsNew <- glbObsNew[, setdiff(names(glbObsNew), glb_drop_vars)]    
}

#stop(here"); savObsAll <- glbObsAll # glbObsAll <- savObsAll
# Combine trnent & newobs into glbObsAll for easier manipulation
glbObsTrn$.src <- "Train"; glbObsNew$.src <- "Test"; 
glbFeatsExclude <- union(glbFeatsExclude, ".src")
glbObsAll <- myrbind_df(glbObsTrn, glbObsNew)
comment(glbObsAll) <- "glbObsAll"

### Derivations using mapping functions
###     glbFeatsId might be in glbFeatsDerive
set.seed(169); glbObsAll[, ".rnorm"] <- rnorm(n = nrow(glbObsAll))
for (new_feat in glb_derive_vars) {
    print(sprintf("Creating new feature: %s...", new_feat))
    args_lst <- NULL 
    for (arg in glbFeatsDerive[[new_feat]]$args) 
        args_lst[[arg]] <- glbObsAll[, arg]
    glbObsAll[, new_feat] <- do.call(glbFeatsDerive[[new_feat]]$mapfn, args_lst)
}

# Check for duplicates in glbFeatsId
if (length(glbFeatsId) == 0) {
    warning("using .rownames as identifiers for observations")
    glbObsAll$.rownames <- as.numeric(rownames(glbObsAll))
    glbObsTrn$.rownames <- as.numeric(rownames(subset(glbObsAll, .src == "Train")))
    glbObsNew$.rownames <- as.numeric(rownames(subset(glbObsAll, .src == "Test")))
    glbFeatsId <- ".rownames"
}
if ((glbObsNewFile$splitSpecs$method != "copy") && 
    ((nDups <- sum(duplicated(glbObsAll[, glbFeatsId, FALSE]))) > 0)) {
    print(sprintf("Found %d duplicates in glbObsAll", nDups))
    myprint_df(orderBy(as.formula(paste0("~", glbFeatsId)), 
                       glbObsAll[duplicated2(glbObsAll[, glbFeatsId, FALSE]),]))
    stop(glbFeatsId, " duplicated in glbObsAll")
}
glbFeatsExclude <- union(glbFeatsExclude, glbFeatsId)

glbObsAll <- orderBy(reformulate(glbFeatsId), glbObsAll)
glbObsTrn <- glbObsNew <- NULL

# For Tableau
if (!is.null(glbOutDataVizFname))
    write.csv(glbObsAll, glbOutDataVizFname, row.names=FALSE)

# - Merge glb_obs_stack_condition & glbObsDropCondition
# - Derive glb_obs_stack|drop_chk_vars from condition automatically
# - Implement glb_obs_stack_condition & glb_obs_stack_chk_vars options

dsp_partition_stats <- function(obs_df, vars=NULL) {
    
    lcl_vars <- NULL
    for (var in c(vars, glb_rsp_var_raw)) {
        if (!(var %in% names(obs_df)))
            next
        
        if ((length(unique(obs_df[, var])) > 5) && is.numeric(obs_df[, var])) {
            cut_var <- paste0(var, ".cut.fctr")
            obs_df[, cut_var] <- cut(obs_df[, var], 3)
            lcl_vars <- union(lcl_vars, cut_var)
        } else lcl_vars <- union(lcl_vars, var)   
    }

    print("Partition stats:")
    print(mycreate_sqlxtab_df(obs_df, union(lcl_vars, ".src")))
    for (var in lcl_vars) {
        myprint_df(freq_df <- mycreate_sqlxtab_df(obs_df, union(var, ".src")))
        if ((nrow(freq_df) > 10) && is.character(obs_df[, var])) {
            first <- 10
            while ((nrow(freq_df) > 10) && (first >= 2)) {
                newVar <- sprintf("%s.first.%d", var, first)
                obs_df[, newVar] <- substr(obs_df[, var], 1, first)
                myprint_df(freq_df <- mycreate_sqlxtab_df(obs_df, union(newVar, ".src")))
                if (nrow(freq_df) > 10) first <- first - 2
            }
            var <- newVar
        }
        
        print(myplot_hbar(freq_df, ".src", ".n", colorcol_name = var))
    }
    print(mycreate_sqlxtab_df(obs_df, ".src"))
        
}

myget_symbols <- function(txt) {
    if (is.null(txt)) return(NULL)
    #print(getParseData(parse(text=txt, keep.source=TRUE)))
    return(unique(subset(getParseData(parse(text=txt, keep.source=TRUE)), 
                         token == "SYMBOL")$text))
}
# tokens <- unlist(strsplit(gsub("[[:punct:]|[:space:]]", " ", glbObsDropCondition), " "))
# tokens <- tokens[tokens != ""]
# glb_obs_drop_chk_vars <- c("biddable") # or NULL

dsp_partition_stats(obs_df=glbObsAll, vars=myget_symbols(glbObsDropCondition))
if (!is.null(glbObsDropCondition)) {
    print(sprintf("Running glbObsDropCondition filter: %s", glbObsDropCondition))
    glbObsAll <- do.call("subset", 
                list(glbObsAll, parse(text=paste0("!(", glbObsDropCondition, ")"))))
    dsp_partition_stats(obs_df=glbObsAll, vars=myget_symbols(glbObsDropCondition))    
}

# Check for duplicates by all features
# Refactor to utilize glbSpecs
require(dplyr)
require(lazyeval)
require(gdata)

if (glb_is_separate_newobs_dataset || (glbObsNewFile$splitSpecs$method != "copy")) {
    dupObsAll <- glbObsAll[duplicated2(
        # subset(glbObsAll, select = -c(UniqueID, Popular, .src))
        glbObsAll[, setdiff(names(glbObsAll), c(glbFeatsId, glb_rsp_var_raw, ".src"))]
                                       ), ]
    dupObsAllIx <- glbObsAll %>% 
        dplyr::select_(.dots = setdiff(names(glbObsAll), 
                                       c(glbFeatsId, glb_rsp_var_raw, ".src"))) %>%    
        #dplyr::select(-c(UniqueID, Popular, .src)) %>%
        duplicated2()
    dupObsAll <- glbObsAll[dupObsAllIx, ] %>% 
        dplyr::arrange_(.dots = setdiff(names(glbObsAll), 
                                       c(glbFeatsId, glb_rsp_var_raw, ".src")))
    print(sprintf("Found %d duplicates by all features:", nrow(dupObsAll)))
    myprint_df(dupObsAll)
} else {
    print("Skipping duplicates check since glbObsNewFile$splitSpecs$method == 'copy'")
    dupObsAll <- data.frame()
}    

# print(dupObsAll[, c(glbFeatsId, glb_rsp_var_raw, 
#                          "description", "startprice", "biddable")])
# write.csv(dupObsAll[, c("UniqueID"), FALSE], "ebayipads_dups.csv", row.names=FALSE)

if (nrow(dupObsAll) > 0) {
    dupobs_df <- tidyr::unite(dupObsAll, "allfeats", -c(sold, UniqueID, .src), sep="*")
    # dupobs_df <- dplyr::group_by(dupobs_df, allfeats)
    # dupobs_df <- dupobs_df[, "UniqueID", FALSE]
    # dupobs_df <- ungroup(dupobs_df)
    # 
    # dupobs_df$.rownames <- row.names(dupobs_df)
    grpobs_df <- data.frame(allfeats=unique(dupobs_df[, "allfeats"]))
    grpobs_df$.grpid <- row.names(grpobs_df)
    dupobs_df <- merge(dupobs_df, grpobs_df)
    
    # dupobs_tbl <- table(dupobs_df$.grpid)
    # print(max(dupobs_tbl))
    # print(dupobs_tbl[which.max(dupobs_tbl)])
    # print(dupobs_df[dupobs_df$.grpid == names(dupobs_tbl[which.max(dupobs_tbl)]), ])
    # print(dupobs_df[dupobs_df$.grpid == 106, ])
    # for (grpid in c(9, 17, 31, 36, 53))
    #     print(dupobs_df[dupobs_df$.grpid == grpid, ])
    dupgrps_df <- as.data.frame(table(dupobs_df$.grpid, dupobs_df$sold, useNA="ifany"))
    names(dupgrps_df)[c(1,2)] <- c(".grpid", "sold")
    dupgrps_df$.grpid <- as.numeric(as.character(dupgrps_df$.grpid))
    dupgrps_df <- tidyr::spread(dupgrps_df, sold, Freq)
    names(dupgrps_df)[-1] <- paste("sold", names(dupgrps_df)[-1], sep=".")
    dupgrps_df$.freq <- sapply(1:nrow(dupgrps_df), function(row) sum(dupgrps_df[row, -1]))
    myprint_df(orderBy(~-.freq, dupgrps_df))
    
    print("sold Conflicts:")
    print(subset(dupgrps_df, (sold.0 > 0) & (sold.1 > 0)))
    #dupobs_df[dupobs_df$.grpid == 4, ]
    glbObsAll <- merge(glbObsAll, dupobs_df[, c(glbFeatsId, ".grpid")], 
                           by=glbFeatsId, all.x=TRUE)
    if (nrow(subset(dupgrps_df, (sold.0 > 0) & (sold.1 > 0) & (sold.0 != sold.1))) > 0)
        stop("Duplicate conflicts are resolvable")
    #subset(glbObsAll, .grpid %in% c(25))
    #mydspObs(list(productline.contains="iPad 1", storage.contains="16", color.contains="Black", carrier.contains="None", cellular.contains="0", condition.contains="Used", startprice=80), cols=c("productline", "storage", "color", "carrier", "cellular", "condition", "startprice", "sold"))

    print("Test & Train Groups:")
stop("aargh!!! here"); print("Fix this ...")    
    #print(subset(dupgrps_df, (<gsb_rsp_var_raw>.NA > 0)))

    glbFeatsExclude <- c(".grpid", glbFeatsExclude)
}

if (!is.null(glbInpMerge)) {
    print("Running glbInpMerge specs...")
    obsMrg <- data.frame()
    for (fName in glbInpMerge$fnames) {
        print(sprintf("    Appending rows from %s...", fName))
        obsMrg <- rbind(obsMrg, read.csv(fName))
    }
    glbObsAll <- merge(glbObsAll, obsMrg, all.x = TRUE)
}

if (!is.null(glb_obs_repartition_train_condition)) {
    print(sprintf("Running glb_obs_repartition_train_condition filter: %s",
                  glb_obs_repartition_train_condition))
    glbObsTrn <- do.call("subset", list(glbObsAll, 
                        parse(text=paste0(" (", glb_obs_repartition_train_condition, ")"))))
    glbObsTrn$.src <- "Train"
    glbObsNew <- do.call("subset", list(glbObsAll, 
                        parse(text=paste0("!(", glb_obs_repartition_train_condition, ")"))))
    glbObsNew$.src <- "Test"
    glbObsAll <- rbind(glbObsTrn, glbObsNew)

    dsp_partition_stats(obs_df = glbObsAll,
                        vars = myget_symbols(glb_obs_repartition_train_condition))    
}

glb_chunks_df <- myadd_chunk(glb_chunks_df, "inspect.data", major.inc=TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r inspect.data, cache=FALSE, echo=FALSE}
#print(str(glbObsAll))
#View(glbObsAll)

dsp_class_dstrb <- function(var) {
    xtab_df <- mycreate_xtab_df(glbObsAll, c(".src", var))
    rownames(xtab_df) <- xtab_df$.src
    xtab_df <- subset(xtab_df, select=-.src)
    print(xtab_df)
    print(xtab_df / rowSums(xtab_df, na.rm=TRUE))    
}    

# Performed repeatedly in other chunks
glb_chk_data <- function(featsExclude = glbFeatsExclude, 
                         fctrMaxUniqVals = glbFctrMaxUniqVals) {
    # Histogram of predictor in glbObsTrn & glbObsNew
    if (is.numeric(glbObsAll[, glb_rsp_var_raw]))
        print(myplot_histogram(glbObsAll, glb_rsp_var_raw) + facet_wrap(~ .src)) else
        print(ggplot(glbObsAll, aes_string(x = glb_rsp_var_raw)) + 
                  geom_bar(stat = "count") + facet_wrap(~ .src))
    
    if (glb_is_classification) 
        dsp_class_dstrb(var=ifelse(glb_rsp_var %in% names(glbObsAll), 
                                   glb_rsp_var, glb_rsp_var_raw))
    mycheck_problem_data(glbObsAll, featsExclude, fctrMaxUniqVals)
}
glb_chk_data()

# Create new features that help diagnostics
if (!is.null(glb_map_rsp_raw_to_var)) {
    glbObsAll[, glb_rsp_var] <- 
        glb_map_rsp_raw_to_var(glbObsAll[, glb_rsp_var_raw])
    mycheck_map_results(mapd_df=glbObsAll, 
                        from_col_name=glb_rsp_var_raw, to_col_name=glb_rsp_var)
        
    if (glb_is_classification) dsp_class_dstrb(glb_rsp_var)
}

# check distribution of all numeric data
dsp_numeric_feats_dstrb <- function(feats_vctr) {
    grid.newpage()
    pushViewport(viewport(layout = grid.layout(ceiling(length(feats_vctr) / 2.0), 2)))
    pltIx <- 1
    for (feat in feats_vctr) {
        #print(sprintf("feat: %s", feat))
        if (glb_is_regression)
            gp <- myplot_scatter(df=glbObsAll, ycol_name=glb_rsp_var, xcol_name=feat,
                                 smooth=TRUE)
        if (glb_is_classification)
            #gp <- myplot_box(df=glbObsAll, ycol_names=feat, xcol_name=glb_rsp_var)
            gp <- myplot_violin(glbObsAll, ycol_names = feat, xcol_name = glb_rsp_var)
        if (inherits(glbObsAll[, feat], "factor"))
            gp <- gp + facet_wrap(reformulate(feat))
        print(gp + labs(title = feat), 
              vp = viewport(layout.pos.row = ceiling(pltIx / 2.0), 
                            layout.pos.col = ((pltIx - 1) %% 2) + 1))  
        
        pltIx <- pltIx + 1        
    }
}

require(dplyr)

# Display distribution of glbFeatsId
pltObsAll <- glbObsAll
if (!is.numeric(glbObsAll[, glbFeatsId])) {
    featsId <- paste0(glbFeatsId,".first.1", sep = "")
    pltObsAll[, featsId] <- substr(glbObsAll[, glbFeatsId], 1, 1)
} else featsId <- glbFeatsId
    
if (is.numeric(glbObsAll[, glb_rsp_var_raw]) && 
    length(unique(glbObsAll[, glb_rsp_var_raw]) > 5)) {
    facetVar <- paste0(glb_rsp_var_raw, ".cut.fctr")
    pltObsAll[, facetVar] <- cut(pltObsAll[, glb_rsp_var_raw], 5)
} else facetVar <- glb_rsp_var_raw

if (is.numeric(glbObsAll[, glbFeatsId]))
    print(gp <- myplot_histogram(pltObsAll, featsId, fill_col_name = ".src") + 
                  facet_wrap(as.formula(paste0("~", facetVar)))) else {
    pltObsFrq <- mycreate_sqlxtab_df(pltObsAll, c(featsId, facetVar, ".src"))                      
    print(gp <- myplot_hbar(pltObsFrq, featsId, ".n", colorcol_name = ".src") + 
                  facet_wrap(as.formula(paste0("~", facetVar))))
}                      

# Check distributions of newly transformed / extracted vars
#   Enhancement: remove vars that were displayed ealier
dsp_numeric_feats_dstrb(feats_vctr=setdiff(names(glbObsAll), 
        c(myfind_chr_cols_df(glbObsAll), glb_rsp_var_raw, glb_rsp_var, 
          glbFeatsExclude)))

#   Convert factors to dummy variables
#   Build splines   require(splines); bsBasis <- bs(training$age, df=3)

#pairs(subset(glbObsTrn, select=-c(col_symbol)))
# Check for glbObsNew & glbObsTrn features range mismatches

# Other diagnostics:
# print(subset(glbObsTrn, <col1_name> == max(glbObsTrn$<col1_name>, na.rm=TRUE) & 
#                         <col2_name> <= mean(glbObsTrn$<col1_name>, na.rm=TRUE)))

# print(glbObsTrn[which.max(glbObsTrn$<col_name>),])

# print(<col_name>_freq_glbObsTrn <- mycreate_tbl_df(glbObsTrn, "<col_name>"))
# print(which.min(table(glbObsTrn$<col_name>)))
# print(which.max(table(glbObsTrn$<col_name>)))
# print(which.max(table(glbObsTrn$<col1_name>, glbObsTrn$<col2_name>)[, 2]))
# print(table(glbObsTrn$<col1_name>, glbObsTrn$<col2_name>))
# print(table(is.na(glbObsTrn$<col1_name>), glbObsTrn$<col2_name>))
# print(table(sign(glbObsTrn$<col1_name>), glbObsTrn$<col2_name>))
# print(mycreate_xtab_df(glbObsTrn, <col1_name>))
# print(mycreate_xtab_df(glbObsTrn, c(<col1_name>, <col2_name>)))
# print(<col1_name>_<col2_name>_xtab_glbObsTrn <- 
#   mycreate_xtab_df(glbObsTrn, c("<col1_name>", "<col2_name>")))
# <col1_name>_<col2_name>_xtab_glbObsTrn[is.na(<col1_name>_<col2_name>_xtab_glbObsTrn)] <- 0
# print(<col1_name>_<col2_name>_xtab_glbObsTrn <- 
#   mutate(<col1_name>_<col2_name>_xtab_glbObsTrn, 
#             <col3_name>=(<col1_name> * 1.0) / (<col1_name> + <col2_name>))) 
# print(mycreate_sqlxtab_df(glbObsAll, c("<col1_name>", "<col2_name>")))

# print(<col2_name>_min_entity_arr <- 
#    sort(tapply(glbObsTrn$<col1_name>, glbObsTrn$<col2_name>, min, na.rm=TRUE)))
# print(<col1_name>_na_by_<col2_name>_arr <- 
#    sort(tapply(glbObsTrn$<col1_name>.NA, glbObsTrn$<col2_name>, mean, na.rm=TRUE)))

# Other plots:
# print(myplot_box(df=glbObsTrn, ycol_names="<col1_name>"))
# print(myplot_box(df=glbObsTrn, ycol_names="<col1_name>", xcol_name="<col2_name>"))
# print(myplot_line(subset(glbObsTrn, Symbol %in% c("CocaCola", "ProcterGamble")), 
#                   "Date.POSIX", "StockPrice", facet_row_colnames="Symbol") + 
#     geom_vline(xintercept=as.numeric(as.POSIXlt("2003-03-01"))) +
#     geom_vline(xintercept=as.numeric(as.POSIXlt("1983-01-01")))        
#         )
# print(myplot_line(subset(glbObsTrn, Date.POSIX > as.POSIXct("2004-01-01")), 
#                   "Date.POSIX", "StockPrice") +
#     geom_line(aes(color=Symbol)) + 
#     coord_cartesian(xlim=c(as.POSIXct("1990-01-01"),
#                            as.POSIXct("2000-01-01"))) +     
#     coord_cartesian(ylim=c(0, 250)) +     
#     geom_vline(xintercept=as.numeric(as.POSIXlt("1997-09-01"))) +
#     geom_vline(xintercept=as.numeric(as.POSIXlt("1997-11-01")))        
#         )
# print(myplot_scatter(glbObsAll, "<col1_name>", "<col2_name>", smooth=TRUE))
# print(myplot_scatter(glbObsAll, "<col1_name>", "<col2_name>", colorcol_name="<Pred.fctr>") + 
#         geom_point(data=subset(glbObsAll, <condition>), 
#                     mapping=aes(x=<x_var>, y=<y_var>), color="red", shape=4, size=5) +
#         geom_vline(xintercept=84))

glb_chunks_df <- myadd_chunk(glb_chunks_df, "scrub.data", major.inc = FALSE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r scrub.data, cache=FALSE, echo=FALSE}
mycheck_problem_data(glbObsAll, featsExclude = glbFeatsExclude, 
                     fctrMaxUniqVals = glbFctrMaxUniqVals)

findOffendingCharacter <- function(x, maxStringLength=256){  
  print(x)
  for (c in 1:maxStringLength){
    offendingChar <- substr(x,c,c)
    #print(offendingChar) #uncomment if you want the indiv characters printed
    #the next character is the offending multibyte Character
  }    
}
# string_vector <- c("test", "Se\x96ora", "works fine")
# lapply(string_vector, findOffendingCharacter)
# lapply(glbObsAll$description[29], findOffendingCharacter)

dsp_hdlxtab <- function(str) 
    print(mycreate_sqlxtab_df(glbObsAll[sel_obs(Headline.contains=str), ],
                           c("Headline.pfx", "Headline", glb_rsp_var)))
#dsp_hdlxtab("(1914)|(1939)")

dsp_catxtab <- function(str) 
    print(mycreate_sqlxtab_df(glbObsAll[sel_obs(Headline.contains=str), ],
        c("Headline.pfx", "NewsDesk", "SectionName", "SubsectionName", glb_rsp_var)))
# dsp_catxtab("1914)|(1939)")
# dsp_catxtab("19(14|39|64):")
# dsp_catxtab("19..:")

# Merge some categories
# glbObsAll$myCategory <-
#     plyr::revalue(glbObsAll$myCategory, c(      
#         "#Business Day#Dealbook"            = "Business#Business Day#Dealbook",
#         "#Business Day#Small Business"      = "Business#Business Day#Small Business",
#         "dummy" = "dummy"
#     ))

# ctgry_xtab_df <- orderBy(reformulate(c("-", ".n")),
#                           mycreate_sqlxtab_df(glbObsAll,
#     c("myCategory", "NewsDesk", "SectionName", "SubsectionName", glb_rsp_var)))
# myprint_df(ctgry_xtab_df)
# write.table(ctgry_xtab_df, paste0(glb_out_pfx, "ctgry_xtab.csv"), 
#             row.names=FALSE)

# ctgry_cast_df <- orderBy(~ -Y -NA, dcast(ctgry_xtab_df, 
#                        myCategory + NewsDesk + SectionName + SubsectionName ~ 
#                            Pplr.fctr, sum, value.var=".n"))
# myprint_df(ctgry_cast_df)
# write.table(ctgry_cast_df, paste0(glb_out_pfx, "ctgry_cast.csv"), 
#             row.names=FALSE)

# print(ctgry_sum_tbl <- table(glbObsAll$myCategory, glbObsAll[, glb_rsp_var], 
#                              useNA="ifany"))

dsp_chisq.test <- function(...) {
    sel_df <- glbObsAll[sel_obs(...) & 
                            !is.na(glbObsAll$Popular), ]
    sel_df$.marker <- 1
    ref_df <- glbObsAll[!is.na(glbObsAll$Popular), ]
    mrg_df <- merge(ref_df[, c(glbFeatsId, "Popular")],
                    sel_df[, c(glbFeatsId, ".marker")], all.x=TRUE)
    mrg_df[is.na(mrg_df)] <- 0
    print(mrg_tbl <- table(mrg_df$.marker, mrg_df$Popular))
    print("Rows:Selected; Cols:Popular")
    #print(mrg_tbl)
    print(chisq.test(mrg_tbl))
}
# dsp_chisq.test(Headline.contains="[Ee]bola")
# dsp_chisq.test(Snippet.contains="[Ee]bola")
# dsp_chisq.test(Abstract.contains="[Ee]bola")

# print(mycreate_sqlxtab_df(glbObsAll[sel_obs(Headline.contains="[Ee]bola"), ], 
#                           c(glb_rsp_var, "NewsDesk", "SectionName", "SubsectionName")))

# print(table(glbObsAll$NewsDesk, glbObsAll$SectionName))
# print(table(glbObsAll$SectionName, glbObsAll$SubsectionName))
# print(table(glbObsAll$NewsDesk, glbObsAll$SectionName, glbObsAll$SubsectionName))

# glbObsAll$myCategory.fctr <- as.factor(glbObsAll$myCategory)
glb_chunks_df <- myadd_chunk(glb_chunks_df, "transform.data", major.inc = FALSE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r transform.data, cache=FALSE, echo=FALSE}

### Mapping dictionary
#savObsAll <- glbObsAll; glbObsAll <- savObsAll
if (!is.null(glb_map_vars)) {
    for (feat in glb_map_vars) {
        map_df <- myimport_data(url=glb_map_urls[[feat]], 
                                            comment="map_df", 
                                           print_diagn=TRUE)
        glbObsAll <- mymap_codes(glbObsAll, feat, names(map_df)[2], 
                                     map_df, map_join_col_name=names(map_df)[1], 
                                     map_tgt_col_name=names(map_df)[2])
    }
    glbFeatsExclude <- union(glbFeatsExclude, glb_map_vars)
}

### Forced Assignments
#stop(here"); savObsAll <- glbObsAll; glbObsAll <- savObsAll
for (feat in glb_assign_vars) {
    new_feat <- paste0(feat, ".my")
    print(sprintf("Forced Assignments for: %s -> %s...", feat, new_feat))
    glbObsAll[, new_feat] <- glbObsAll[, feat]
    
    pairs <- glb_assign_pairs_lst[[feat]]
    for (pair_ix in 1:length(pairs$from)) {
        if (is.na(pairs$from[pair_ix]))
            nobs <- nrow(filter(glbObsAll, 
                                is.na(eval(parse(text=feat),
                                            envir=glbObsAll)))) else
            nobs <- sum(glbObsAll[, feat] == pairs$from[pair_ix])
        #nobs <- nrow(filter(glbObsAll, is.na(Married.fctr)))    ; print(nobs)
        
        if ((is.na(pairs$from[pair_ix])) && (is.na(pairs$to[pair_ix])))
            stop("what are you trying to do ???")
        if (is.na(pairs$from[pair_ix]))
            glbObsAll[is.na(glbObsAll[, feat]), new_feat] <- 
                pairs$to[pair_ix] else
            glbObsAll[glbObsAll[, feat] == pairs$from[pair_ix], new_feat] <- 
                pairs$to[pair_ix]
                    
        print(sprintf("    %s -> %s for %s obs", 
                      pairs$from[pair_ix], pairs$to[pair_ix], format(nobs, big.mark=",")))
    }

    glbFeatsExclude <- union(glbFeatsExclude, glb_assign_vars)
}

#stop(here")
#hex_vctr <- c("\n", "\211", "\235", "\317", "\333")
hex_regex <- paste0(c("\n", "\211", "\235", "\317", "\333"), collapse="|")
for (obs_id in c(10029, 10948, 10136, 10178, 11514, 11904, 12157, 12210, 12659)) {
#     tmp_str <- unlist(strsplit(glbObsAll[row_pos, "descr.my"], ""))
#     glbObsAll[row_pos, "descr.my"] <- paste0(tmp_str[!tmp_str %in% hex_vctr],
#                                                          collapse="")
    row_pos <- which(glbObsAll$UniqueID == obs_id)
#     glbObsAll[row_pos, "descr.my"] <- 
#         gsub(hex_regex, " ", glbObsAll[row_pos, "descr.my"])
}
glb_chunks_df <- myadd_chunk(glb_chunks_df, "extract.features", major.inc = TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r extract.features, cache=FALSE, echo=FALSE}
glb_chunks_df <- myadd_chunk(glb_chunks_df, "extract.features.string", major.inc = FALSE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r extract.features.string, cache=FALSE, echo=FALSE}
extract.features.string.chunk.df <- myadd_chunk(NULL, "extract.features.string.bgn")

# Create dummy category if glbFeatsCategory is NULL
if (is.null(glbFeatsCategory)) {
    glbFeatsCategory <- ".category"
    glbObsAll[, glbFeatsCategory] <- as.factor(".dummy")
    glbFeatsExclude <- union(glbFeatsExclude, glbFeatsCategory)
}

#   Create factors of string variables excl. dates/times since they come in as chars
extract.features.string.chunk.df <- myadd_chunk(extract.features.string.chunk.df, 
            paste0("extract.features.string", "factorize.str.vars"), major.inc = TRUE)

#stop(here"); savObsAll <- glbObsAll; #glbObsAll <- savObsAll
print(str_vars <- myfind_chr_cols_df(glbObsAll))
if (length(str_vars <- setdiff(str_vars, 
                               c(glbFeatsExclude, glbFeatsDateTime, names(glbFeatsText)))) > 0) {
    for (var in str_vars) {
        warning("Creating factors of string variable: ", var, 
                ": # of unique values: ", length(unique(glbObsAll[, var])))
        glbObsAll[, paste0(var, ".fctr")] <- 
            relevel(factor(glbObsAll[, var]),
                    names(which.max(table(glbObsAll[, var], useNA = "ifany"))))
    }
    glbFeatsExclude <- union(glbFeatsExclude, str_vars)
}
glb_chunks_df <- myadd_chunk(glb_chunks_df, "extract.features.datetime", major.inc = FALSE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r extract.features.datetime, cache=FALSE, echo=FALSE}
extract.features.datetime.chunk.df <- myadd_chunk(NULL, "extract.features.datetime.bgn")

#   Extract date & time features
#       typically, dates/times come in as chars; 

if (length(glbFeatsDateTime) > 0) {
    extract.features.datetime.chunk.df <- myadd_chunk(extract.features.datetime.chunk.df, 
                            paste0("extract.features_", "xtract.DateTime.vars"), major.inc = TRUE)
    
    print(sprintf("Extracting features from DateTime(s): %s", names(glbFeatsDateTime)))
    glbObsAll <- cbind(glbObsAll, 
                       myextract_dates_df(df = glbObsAll, vars = names(glbFeatsDateTime), 
                                          id_vars = glbFeatsId, rsp_var = glb_rsp_var))
    for (sfx in c("", ".POSIX", ".day.minutes"))
        glbFeatsExclude <- 
        union(glbFeatsExclude, 
              paste(names(glbFeatsDateTime), sfx, sep = ""))
    
    for (feat in names(glbFeatsDateTime)) {
        pltObsAll <- 
            glbObsAll[, c(paste(feat, c(".month.fctr", ".wkend"), sep = ""), glb_rsp_var)]
        pltObsAll[, ".freq"] <- 1
        print(myplot_bar(pltObsAll, xcol_name = paste0(feat, ".month.fctr"), 
                         ycol_names = ".freq", colorcol_name = paste0(feat, ".wkend")) + 
                  facet_wrap(reformulate(glb_rsp_var)))
#         print(myplot_scatter(glbObsAll[glbObsAll[, paste0(feat, ".POSIX")] >=
#                                                strptime("2012-12-01", "%Y-%m-%d"), ], 
#                              xcol_name=paste0(feat, ".POSIX"),
#                              ycol_name=glb_rsp_var, colorcol_name=paste0(feat, ".wkend")))

        # Create Time Series Polynomial features (overall & by category)
#stop(here"); glb2Sav(); all.equal(savObsAll, glbObsAll); glbObsAll <- savObsAll
        glbObsAll <- cbind(glbObsAll, myextractTimePoly(glbObsAll, feat))
        for (poly in grep(paste0(feat, ".day.minutes.poly."), names(glbObsAll), fixed = TRUE,
                          value = TRUE)) {
            print(myplot_scatter(glbObsAll, paste0(feat, ".day.minutes"), poly, 
                                 colorcol_name = glb_rsp_var))    
        }
    
        if (as.logical(glbFeatsDateTime[[feat]]["poly.ctg"])) {
            tmpTimePoly <- data.frame()
            for (category in sort(unique(glbObsAll[, glbFeatsCategory]))) {
                ctgObsAll <- glbObsAll[(glbObsAll[, glbFeatsCategory] %in% category), ]
                ctgTimePoly <- myextractTimePoly(ctgObsAll, feat)
                names(ctgTimePoly) <- paste(names(ctgTimePoly), ".ctg", sep = "")
                
                tmpTimePoly <- myrbind_df(tmpTimePoly, ctgTimePoly)
            }
            glbObsAll <- cbind(glbObsAll, tmpTimePoly)

            for (ctgFeat in names(tmpTimePoly))
                glbFeatsInteractionOnly[[ctgFeat]] <- ifelse(grepl("\\.fctr", glbFeatsCategory),
                                                             glbFeatsCategory, 
                                                            paste0(glbFeatsCategory, ".fctr"))
        }

        # Create features that measure the gap between previous timestamp in the data
        glbObsAll <- cbind(glbObsAll, 
                            myextractTimeLags(glbObsAll, feat, glb_rsp_var, glb_rsp_var_raw,
                                              impute.na = glbFeatsDateTime[[feat]]["impute.na"]))
        for (last in grep(paste0(feat, ".last"), names(glbObsAll), fixed = TRUE,
                          value = TRUE)) {
            print(myplot_histogram(glbObsAll, last) +
                            facet_grid(as.formula(paste0(feat, ".year.fctr", "~", glb_rsp_var))))
        }
        glbFeatsExclude <- union(glbFeatsExclude, c(paste0(feat, ".zoo")))
        
        if (as.logical(glbFeatsDateTime[[feat]]["last.ctg"])) {        
        # Create features that measure the gap between previous timestamp in the data by category
            tmpTimeLags <- data.frame()
            for (category in sort(unique(glbObsAll[, glbFeatsCategory]))) {
                ctgObsAll <- glbObsAll[(glbObsAll[, glbFeatsCategory] %in% category), ]
                ctgTimeLags <- myextractTimeLags(ctgObsAll, feat, glb_rsp_var, glb_rsp_var_raw, 
                                                impute.na = glbFeatsDateTime[[feat]]["impute.na"])
                names(ctgTimeLags) <- paste(names(ctgTimeLags), ".ctg", sep = "")
                
                tmpTimeLags <- myrbind_df(tmpTimeLags, ctgTimeLags)
            }
            if (length(missingFeats <- myfind_numerics_missing(tmpTimeLags)) > 0) {
                if (!as.logical(glbFeatsDateTime[[feat]]["impute.na"])) {
                    for (mf in missingFeats)
                        tmpTimeLags[is.na(tmpTimeLags[, mf]), mf] <- 0
                } else {
                    stop("refactor missing data code here also; cannot let it pass-thru to missing data chunk since these features are added to glbFeatsInteractionOnly")
                }    
            }
            glbObsAll <- cbind(glbObsAll, tmpTimeLags)
            glbFeatsExclude <- union(glbFeatsExclude, c(paste0(feat, ".zoo.ctg")))
            for (ctgFeat in setdiff(names(tmpTimeLags), grep("\\.zoo\\.ctg", names(tmpTimeLags), 
                                                             value = TRUE)))
                glbFeatsInteractionOnly[[ctgFeat]] <- ifelse(grepl("\\.fctr", glbFeatsCategory),
                                                             glbFeatsCategory, 
                                                            paste0(glbFeatsCategory, ".fctr"))
        }
        
        # ## fill in NAs
        # # count averages
        # na.avg = all2 %>% group_by(weekend, hour) %>% dplyr::summarise(
        #     last1=mean(last1, na.rm=TRUE),
        #     last3=mean(last3, na.rm=TRUE),
        #     last5=mean(last5, na.rm=TRUE),
        #     last10=mean(last10, na.rm=TRUE),
        #     last20=mean(last20, na.rm=TRUE),
        #     last50=mean(last50, na.rm=TRUE)
        # )
        # 
        # # fill in averages
        # na.merge = merge(all2, na.avg, by=c("weekend","hour"))
        # na.merge = na.merge[order(na.merge$id),]
        # for(i in c("last1", "last3", "last5", "last10", "last20", "last50")) {
        #     y = paste0(i, ".y")
        #     idx = is.na(all2[[i]])
        #     all2[idx,][[i]] <- na.merge[idx,][[y]]
        # }
        # rm(na.avg, na.merge, b, i, idx, n, pd, sec, sh, y, z)
    }
}
#rm(last1, last10, last100)

glb_chunks_df <- myadd_chunk(glb_chunks_df, "extract.features.price", major.inc = FALSE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r extract.features.price, cache=FALSE, echo=FALSE}
extract.features.price.chunk.df <- myadd_chunk(NULL, "extract.features.price.bgn")

if (!is.null(glbFeatsPrice)) {
    extract.features.price.chunk.df <- 
        myadd_chunk(extract.features.price.chunk.df, paste0("extract.features", ".price"), 
                    major.inc = TRUE)

    for (var in glbFeatsPrice) {
        for (digit in 1:(log10(max(glbObsAll[, var], na.rm=TRUE)) + 1)) {
            glbObsAll[, paste0(var, ".dgt", digit, ".is9")] <- 
                as.numeric(as.integer((as.integer(glbObsAll[, var]) %% (10 ^ digit)) / 
                                          (10 ^ (digit - 1))) == 9)
#             glbObsAll[, paste0(var, ".dgt", digit, ".is9.fctr")] <- 
#                 as.factor(as.integer((as.integer(glbObsAll[, var]) %% (10 ^ digit)) / 
#                                           (10 ^ (digit - 1))) == 9)
        }
        for (decimal in 1:2) {
            glbObsAll[, paste0(var, ".dcm", decimal, ".is9")] <- 
                as.numeric(as.integer(glbObsAll[, var] * (10 ^ decimal)) %% 10 == 9)
#             glbObsAll[, paste0(var, ".dcm", decimal, ".is9.fctr")] <- 
#                 as.factor(as.integer(glbObsAll[, var] * (10 ^ decimal)) %% 10 == 9)
        }
    }
    #as.numeric((as.integer(startprice) %% 10) == 9)    
    rm(corpus_lst
       , glb_sprs_DTM_lst #, glb_full_DTM_lst
       , txt_corpus, txt_vctr)
}

glb_chunks_df <- myadd_chunk(glb_chunks_df, "extract.features.text", major.inc = FALSE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r extract.features.text, cache=FALSE, echo=FALSE}
extract.features.text.chunk.df <- myadd_chunk(NULL, "extract.features.text.bgn")

if (length(glbFeatsText) > 0) {
    require(foreach)
    require(gsubfn)
    require(stringr)
    require(tm)
    
    print(sprintf("Extracting features from Text(s): %s", names(glbFeatsText)))
    
    chk_pattern_freq <- function(rex_str, ignore.case = TRUE) {
        match_mtrx <- str_extract_all(txt_vctr, regex(rex_str, ignore_case = ignore.case), 
                                      simplify = TRUE)
        match_df <- as.data.frame(match_mtrx[match_mtrx != ""])
        if (nrow(match_df) <= 10)
            print(txt_vctr[match_mtrx != ""])
        names(match_df) <- "pattern"
        return(mycreate_sqlxtab_df(match_df, "pattern"))        
    }

#     match_lst <- gregexpr("\\bok(?!ay)", txt_vctr[746], ignore.case = FALSE, perl=TRUE); print(match_lst)
    dsp_pattern <- function(rex_str, ignore.case=TRUE, print.all=TRUE) {
        match_lst <- gregexpr(rex_str, txt_vctr, ignore.case = ignore.case, perl=TRUE)
        match_lst <- regmatches(txt_vctr, match_lst)
        match_df <- data.frame(matches=sapply(match_lst, 
                                              function (elems) paste(elems, collapse="*")))
        match_df <- subset(match_df, matches != "")
        if (print.all)
            print(match_df)
        return(match_df)
    }
    
    dsp_matches <- function(rex_str, ix) {
        print(match_pos <- gregexpr(rex_str, txt_vctr[ix], perl=TRUE))
        print(str_sub(txt_vctr[ix], (match_pos[[1]] / 100) *  99 +   0, 
                                    (match_pos[[1]] / 100) * 100 + 100))        
    }

    myapply_txtmap <- function(txt_vctr, bgn = 1, end = nrow(glbFeatsTextMap), ...) {
        #nrows <- nrow(glbFeatsTextMap)
        for (ptn_ix in bgn:end) {
            if ((ptn_ix %% 10) == 0)
                print(sprintf("running gsub for %02d (of %02d): #%s#...", ptn_ix, 
                                # nrows, glbFeatsTextMap[ptn_ix, "rex_str"]))
                                end, glbFeatsTextMap[ptn_ix, "rex_str"]))            
            txt_vctr <- gsub(glbFeatsTextMap[ptn_ix, "rex_str"], 
                             glbFeatsTextMap[ptn_ix, "rpl_str"], 
                               txt_vctr, ...)
        }
        return(txt_vctr)
        #print(txt_vctr <- glbObsAll[7322, txtFeat])
        #print(glbFeatsTextMap[grepl("upper", glbFeatsTextMap$rex_str), ])
        #strsplit(txt_vctr, "")[[1]][1]
        #ptn_ix <- 2; glbFeatsTextMap[ptn_ix, ]
        #gsub(glbFeatsTextMap[ptn_ix <- 109, "rex_str"], glbFeatsTextMap[ptn_ix, "rpl_str"], txt_vctr)
        #myapply_txtmap(txt_vctr, bgn = 1, end = 153)
        #print(match_lst <- gregexpr(glbFeatsTextMap[ptn_ix, "rex_str"], txt_vctr))
        #strsplit(glbFeatsTextMap[ptn_ix, "rex_str"], "")[[1]]
    }    

    chk.equal <- function(bgn, end) {
        print(all.equal(sav_txt_lst[["Headline"]][bgn:end], 
                        glb_txt_chr_lst[["Headline"]][bgn:end]))
    }    
    dsp.equal <- function(bgn, end) {
        print(sav_txt_lst[["Headline"]][bgn:end])
        print(glb_txt_chr_lst[["Headline"]][bgn:end])
    }    
#sav_txt_lst <- glb_txt_chr_lst; all.equal(sav_txt_lst, glb_txt_chr_lst)
#all.equal(sav_txt_lst[["Headline"]][1:4200], glb_txt_chr_lst[["Headline"]][1:4200])
#chk.equal( 1, 100)
#dsp.equal(86, 90)
    
    txt_map_filename <- paste0(glb_txt_munge_filenames_pfx, "map.csv")
    if (!file.exists(txt_map_filename))
        stop(txt_map_filename, " not found!")
    glbFeatsTextMap <- read.csv(txt_map_filename, comment.char = "#", strip.white = TRUE)
    # If nchar(rpl_str) == 0, strip all blanks from rex_str along with first word delimiter
    mask <- (nchar(glbFeatsTextMap$rpl_str) == 0)
    glbFeatsTextMap[mask, "rpl_str"] <- 
        gsub(" ", "", gsub("^\\\\b", "", glbFeatsTextMap[mask, "rex_str"]))

    glb_txt_chr_lst <- list(); 
    print(sprintf("Building glb_txt_chr_lst..."))
    glb_txt_chr_lst <- foreach(txtFeat = names(glbFeatsText)) %dopar% {   
#     for (txtFeat in glbFeatsText) {
        txt_vctr <- glbObsAll[, txtFeat]
        names(txt_vctr) <- glbObsAll[, glbFeatsId]
        
        # myapply_txtmap shd be created as a tm_map::content_transformer ?
        # 
        #txtFeat=glbFeatsText[3]; txt_vctr <- glb_txt_chr_lst[[txtFeat]]
        #print(rex_str <- glbFeatsTextMap[glbFeatsTextMap$rex_str == "&(.+)circ;", "rex_str"])
        #print(rex_str <- glbFeatsTextMap[grepl("du Pont", glbFeatsTextMap$rex_str), "rex_str"])        
        #print(rex_str <- glbFeatsTextMap[glbFeatsTextMap$rpl_str == "versus", "rex_str"])             
        #print(tmp_vctr <- grep(rex_str, "Anniversary Dinners for Relais & Ch&acirc;teaux", value=TRUE, ignore.case=FALSE))
        #ret_lst <- regexec(rex_str, txt_vctr, ignore.case=FALSE); ret_lst <- regmatches(txt_vctr, ret_lst); ret_vctr <- sapply(1:length(ret_lst), function(pos_ix) ifelse(length(ret_lst[[pos_ix]]) > 0, ret_lst[[pos_ix]], "")); print(ret_vctr <- ret_vctr[ret_vctr != ""])
        #gsub(rex_str, glbFeatsTextMap[glbFeatsTextMap$rex_str == rex_str, "rpl_str"], tmp_vctr, ignore.case=FALSE)
        #grep("Hong Hong", txt_vctr, value=TRUE)
    
        txt_vctr <- myapply_txtmap(txt_vctr, ignore.case = FALSE)    
    }
    names(glb_txt_chr_lst) <- names(glbFeatsText)

#stop(here")    
    for (txtFeat in names(glbFeatsText)) {
        print(sprintf("Remaining & in %s:", txtFeat))
        txt_vctr <- glb_txt_chr_lst[[txtFeat]]
        
        print(chk_pattern_freq(rex_str <- "\\w&\\w", ignore.case = FALSE))        
        print(chk_pattern_freq(rex_str <- "\\w&", ignore.case = FALSE))
        print(chk_pattern_freq(rex_str <- "&\\w", ignore.case = FALSE))        
#         match_df <- dsp_pattern(rex_str, ignore.case=FALSE, print.all=FALSE)
#         for (row in row.names(match_df))
#             dsp_matches(rex_str, ix=as.numeric(row))
    }    
    
    for (txtFeat in names(glbFeatsText)) {
        print(sprintf("Remaining OK in %s:", txtFeat))
        txt_vctr <- glb_txt_chr_lst[[txtFeat]]
        
        print(chk_pattern_freq(rex_str <- "\\bOK\\b", ignore.case = FALSE))
#         match_df <- dsp_pattern(rex_str, ignore.case=FALSE, print.all=FALSE)
#         for (row in row.names(match_df))
#             dsp_matches(rex_str, ix=as.numeric(row))
    }    
    # txt_vctr <- glb_txt_chr_lst[[glbFeatsText[1]]]
    # print(chk_pattern_freq(rex_str <- "(?<!( b| c| C| p|\\(b|bo|co|lo|Lo|Sp|to|To))ok(?!(ay|e |e\\)|e,|e\\.|ed|el|en|es|ey|ie|in|on|ra))", ignore.case=FALSE))
    # print(chk_pattern_freq(rex_str <- "ok(?!(ay|el|on|ra))", ignore.case=FALSE))
    # dsp_pattern(rex_str, ignore.case=FALSE, print.all=FALSE)
    # dsp_matches(rex_str, ix=8)
    # substr(txt_vctr[86], 5613, 5620)
    # substr(glbObsAll[301, "review"], 550, 650)

#stop(here"); sav_txt_lst <- glb_txt_chr_lst    
    for (txtFeat in names(glbFeatsText)) {
        print(sprintf("Remaining Acronyms in %s:", txtFeat))
        txt_vctr <- glb_txt_chr_lst[[txtFeat]]
        
        print(chk_pattern_freq(rex_str <- "([[:upper:]]\\.( *)){2,}", ignore.case = FALSE))
        
        # Check for names
        print(subset(chk_pattern_freq(rex_str <- "(([[:upper:]]+)\\.( *)){1}"),
                     !grepl("^[Jan|Feb|Apr|Jun|Jul|Sep|Oct|Nov|Dec|Mr|Mrs|Ms|Rev]", pattern)))
        # dsp_pattern(rex_str="(OK\\.( *)){1}", ignore.case=FALSE)
        # dsp_matches(rex_str="(OK\\.( *)){1}", ix=557)
        #dsp_matches(rex_str="\\bR\\.I\\.P(\\.*)(\\B)", ix=461)
        #dsp_matches(rex_str="\\bR\\.I\\.P(\\.*)", ix=461)        
        #print(str_sub(txt_vctr[676], 10100, 10200))
        #print(str_sub(txt_vctr[74], 1, -1))        
    }

    for (txtFeat in names(glbFeatsText)) {
        re_str <- "\\b(Fort|Ft\\.|Hong|Las|Los|New|Puerto|Saint|San|Santa|St\\.)( |-)(\\w)+"
        print(sprintf("Remaining #%s# terms in %s: ", re_str, txtFeat))
        txt_vctr <- glb_txt_chr_lst[[txtFeat]]        
        print(orderBy(~ -.n +pattern, subset(chk_pattern_freq(re_str, ignore.case = FALSE), 
                                             grepl("( |-)[[:upper:]]", pattern))))
        print("    consider cleaning if relevant to problem domain; geography name; .n > 1")
        #grep("New G", txt_vctr, value=TRUE, ignore.case=FALSE)
        #grep("St\\. Wins", txt_vctr, value=TRUE, ignore.case=FALSE)
    }        
        
#stop(here"); sav_txt_lst <- glb_txt_chr_lst    
    for (txtFeat in names(glbFeatsText)) {
        re_str <- "\\b(N|S|E|W|C)( |\\.)(\\w)+"
        print(sprintf("Remaining #%s# terms in %s: ", re_str, txtFeat))        
        txt_vctr <- glb_txt_chr_lst[[txtFeat]]                
        print(orderBy(~ -.n +pattern, subset(chk_pattern_freq(re_str, ignore.case=FALSE), 
                                             grepl(".", pattern))))
        #grep("N Weaver", txt_vctr, value=TRUE, ignore.case=FALSE)        
    }    

    for (txtFeat in names(glbFeatsText)) {
        re_str <- "\\b(North|South|East|West|Central)( |\\.)(\\w)+"
        print(sprintf("Remaining #%s# terms in %s: ", re_str, txtFeat))        
        txt_vctr <- glb_txt_chr_lst[[txtFeat]]                        
        if (nrow(filtered_df <- subset(chk_pattern_freq(re_str, ignore.case=FALSE), 
                                             grepl(".", pattern))) > 0)
            print(orderBy(~ -.n +pattern, filtered_df))
        #grep("Central (African|Bankers|Cast|Italy|Role|Spring)", txt_vctr, value=TRUE, ignore.case=FALSE)
        #grep("East (Africa|Berlin|London|Poland|Rivals|Spring)", txt_vctr, value=TRUE, ignore.case=FALSE)
        #grep("North (American|Korean|West)", txt_vctr, value=TRUE, ignore.case=FALSE)        
        #grep("South (Pacific|Street)", txt_vctr, value=TRUE, ignore.case=FALSE)
        #grep("St\\. Martins", txt_vctr, value=TRUE, ignore.case=FALSE)
    }    

    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
            paste0("extract.features_", "build.corpus"), major.inc = TRUE)
    
    get_txt_terms <- function(terms_TDM, 
                              compute.cor.y = FALSE, compute.nzv = FALSE, compute.chisq = FALSE, 
                              compute.classWeights = FALSE) {
        tmEnter <- as.numeric(proc.time()["elapsed"])
        print(sprintf("get_txt_terms: enter: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
    
        terms_mtrx <- as.matrix(as.TermDocumentMatrix(terms_TDM))
        docms_mtrx <- as.matrix(as.DocumentTermMatrix(terms_TDM))        
        terms_df <- data.frame(term = dimnames(terms_mtrx)$Terms,
                               weight = rowSums(terms_mtrx),
                               freq = rowSums(terms_mtrx > 0))
        terms_df$pos <- 1:nrow(terms_df)
        print(sprintf("get_txt_terms: terms_df setup: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        if (compute.cor.y) {
            terms_df$cor.y <- 
                cor(docms_mtrx[glbObsAll$.src == "Train",], 
                    as.numeric(glbObsAll[glbObsAll$.src == "Train", glb_rsp_var]),
                                  use = "pairwise.complete.obs")
            terms_df$cor.y.abs <- abs(terms_df$cor.y)
    #         .rnorm.cor.y.abs <- abs(cor(glbObsAll[glbObsAll$.src == "Train", ".rnorm"],
    #                         as.numeric(glbObsAll[glbObsAll$.src == "Train", glb_rsp_var]),
    #                                 use = "pairwise.complete.obs"))
            print(sprintf("get_txt_terms: terms_df cor.y: elapsed: %0.2f secs",
                          as.numeric(proc.time()["elapsed"]) - tmEnter))
        }
        
        # compute nzv before chisq.test since terms with nzv.freqRatio beyond a max threshold are not chisq.significant ???
        if (compute.nzv) {
            nzv_df <- 
#                 caret::nzv(docms_mtrx[glbObsAll$.src == "Train",], freqCut = glbFeatsNzvFreqMax,
#                             uniqueCut = glbFeatsNzvUniqMin, saveMetrics = TRUE)
                # caret::nearZeroVar(docms_mtrx[glbObsAll$.src == "Train",], 
                mycaret.nearZeroVar(docms_mtrx[glbObsAll$.src == "Train",],
                                   freqCut = glbFeatsNzvFreqMax, uniqueCut = glbFeatsNzvUniqMin,
                                   saveMetrics = TRUE, foreach = TRUE)
            terms_df$nzv.freqRatio <- nzv_df$freqRatio
    #         terms_df$nzv.freqRatio.cut.fctr <- cut(terms_df$nzv.freqRatio, 
    #                                                breaks = sort(c(min(terms_df$nzv.freqRatio), 
    #                                                                 glbFeatsNzvFreqMax,
    #                                                            max(terms_df$nzv.freqRatio))))
            terms_df$nzv.percentUnique <- nzv_df$percentUnique
    #         terms_df$nzv.percentUnique.cut.fctr <- cut(terms_df$nzv.percentUnique, 
    #               breaks = sort(c(min(terms_df$nzv.percentUnique) - .Machine$double.neg.eps, 
    #                                                             glbFeatsNzvUniqMin,
    #                                                       max(terms_df$nzv.percentUnique))))
    #         terms_df$nzv.quad.fctr <- as.factor(paste0("fRatio:",
    #          terms_df$nzv.freqRatio.cut.fctr,
    #                                             "\n%Unq:", terms_df$nzv.percentUnique.cut.fctr))
            terms_df$nzv <- nzv_df$nzv
            print(sprintf("get_txt_terms: terms_df nzv: elapsed: %0.2f secs",
                          as.numeric(proc.time()["elapsed"]) - tmEnter))
        }
        
        # to check is nzv.freqRatio max threshold is properly set
# summary(full_terms_df$nzv.freqRatio)
# full_terms_df$chisq.pval.significant <-
#     cut(full_terms_df$chisq.pval, breaks = c(0, 0.05, max(full_terms_df$chisq.pval, na.rm = TRUE)))
# ggplot(full_terms_df, mapping = aes(x = nzv.percentUnique, y = nzv.freqRatio)) +
#     geom_point(aes(color = chisq.pval.significant), position = "jitter")
# significant_terms_df <- subset(full_terms_df, chisq.pval < 0.05)
# summary(significant_terms_df$nzv.freqRatio)
        
        if (compute.chisq) {
            naDf <- data.frame(chisq.stat = NA, chisq.pval = NA)
            trnObsIx <- (glbObsAll$.src == "Train")
            chisqDf <- foreach(ix = 1:nrow(terms_df), .combine = rbind.data.frame) %dopar% 
            # chisqDf <- foreach(ix = 1:nrow(terms_df), .combine = rbind.data.frame) %do% 
            # chisqDf <- foreach(ix = 1:30, .combine = rbind) %do% 
#                 if ((length(unique(docms_mtrx[trnObsIx, ix])) > 1) && 
#                     (terms_df[ix, "nzv.freqRatio"] <= 3265)) # Empirical setting
                if (length(unique(docms_mtrx[trnObsIx, ix])) > 1)
                {
                    chisq <- chisq.test(docms_mtrx[trnObsIx, ix], 
                                        glbObsAll[trnObsIx, glb_rsp_var])
                    tmpDf <- data.frame(chisq.stat = chisq$statistic,
                                        chisq.pval = chisq$p.value)
                } else {
                    tmpDf <- naDf
                }
    #         terms_df[ix, "chisq.stat"] <- chisq$statistic
    #         terms_df[ix, "chisq.pval"] <- chisq$p.value 
            terms_df <- cbind(terms_df, chisqDf)       
            print(sprintf("get_txt_terms: terms_df chisq.test: elapsed: %0.2f secs",
                          as.numeric(proc.time()["elapsed"]) - tmEnter))
        }
        
        if (compute.classWeights) {
            for (cls in unique(glbObsAll[, glb_txt_cor_var])) {
                if (!is.na(cls)) {
                    obsMask <- as.numeric(!is.na(glbObsAll[, glb_txt_cor_var]) &
                                            (glbObsAll[, glb_txt_cor_var] == cls))
                } else {
                    obsMask <- as.numeric(is.na(glbObsAll[, glb_txt_cor_var]))
                }    
                terms_df[, paste0("weight.mean.", as.character(cls))] <- 
                    colSums(t(terms_mtrx) * obsMask) * 1.0 / sum(obsMask)
            }    
            print(sprintf("get_txt_terms: terms_df weight.glb_txt_cor_var: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        }    
        
        print(sprintf("get_txt_terms: exit: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        # Check all calls to get_terms_DTM_terms to change returned order assumption
        return(terms_df <- orderBy(~ -weight, terms_df))
    }
    #plt_full_df <- get_terms_DTM_terms(terms_DTM=glb_full_terms_DTM_lst[[txtFeat]])
    
    get_corpus_terms <- function(txt_corpus, ...) {
        set.seed(glbFeatsTextSeed)        
        terms_TDM <- TermDocumentMatrix(txt_corpus, control = glb_txt_terms_control)
        return(terms_df <- get_txt_terms(terms_TDM, ...))
    }
    
#stop(here"); glb2Sav()    
    glbFeatsTextCorpus <- list()
    print(sprintf("Building glbFeatsTextCorpus..."))
    glbFeatsTextCorpus <- foreach(txtFeat = names(glbFeatsText), .verbose = FALSE) %dopar% {
    #glbFeatsTextCorpus <- foreach(txtFeat = glbFeatsText, .verbose = TRUE) %do% {        
    #for (txtFeat in glbFeatsText) {
        txt_corpus <- Corpus(VectorSource(glb_txt_chr_lst[[txtFeat]]))
        txt_corpus <- tm_map(txt_corpus, PlainTextDocument, lazy = TRUE)
        txt_corpus <- tm_map(txt_corpus, content_transformer(tolower), lazy = TRUE) #nuppr
        # removePunctuation does not replace with whitespace. Use a custom transformer ???
        #txt_corpus <- tm_map(txt_corpus, removePunctuation, lazy = TRUE) #npnct<chr_ix>
        txt_corpus <- tm_map(txt_corpus, content_transformer(myreplacePunctuation)
                             , lazy = TRUE) #npnct<chr_ix>
#         txt-corpus <- tm_map(txt_corpus, content_transformer(function(x, pattern) gsub(pattern, "", x))   
        if (!is.null(glb_txt_stop_words[[txtFeat]]))
            txt_corpus <- tm_map(txt_corpus, removeWords, glb_txt_stop_words[[txtFeat]],
                                 lazy = FALSE)#, lazy=TRUE) #nstopwrds

        # foreach result is based on .Last.Eval
        txt_corpus <- txt_corpus
        # glbFeatsTextCorpus[[txtFeat]] <- txt_corpus
    }
    names(glbFeatsTextCorpus) <- names(glbFeatsText)
    
mycombineSynonyms <- content_transformer(function(x, syn=NULL) { 
    Reduce(function(a,b) {
        gsub(paste0("\\b(", paste(b$syns, collapse = "|"),")\\b"), b$word, a)}, syn, x)   
})    
    
mytokenize3grams <- content_transformer(function(x) { 
    tokens <- scan_tokenizer(x)
    return(paste(head(tokens, -2), 
                 head(tail(tokens, -1), -1), 
                 tail(tokens, -2), sep = " "))
})    

    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
            paste0("extract.features_", "load.validwords"), major.inc = TRUE)
    glbSCOWLWordsFilename <- paste0("scowl-2015.08.24-english-american-35.txt")
    if (!file.exists(glbSCOWLWordsFilename))
        stop(glbSCOWLWordsFilename, " not found!")
    glbSCOWLWords <- read.csv(glbSCOWLWordsFilename, header = FALSE, comment.char = "#", 
                           strip.white = TRUE, encoding = "UTF-8")
    glbSCOWLWords <- iconv(glbSCOWLWords[, 1], "latin1", "UTF-8")
    glbSCOWLWords <- myreplacePunctuation(glbSCOWLWords)
    glbSCOWLWords <- glbSCOWLWords[!duplicated(glbSCOWLWords)]
                        
    vldTerms <- union(glbSCOWLWords, myreplacePunctuation(str_to_lower(c(NULL
        # Month names
        ,"jan","feb","mar","April","may","jun","jul","aug","sep","oct","nov","dec"
        # Day names
        ,"sunday","monday","tuesday","wednesday","thursday","friday","saturday"
        
        # Place names
        ,"afghan","Afghans","Afghanistan","africa","african","africans","africanamerican"
            ,"alaska","Alaskan"
            ,"Amazon","america","americas","american","Americana","americans","Amsterdam"
            ,"Antwerp"
            ,"arab","arabia","Argentina","argentine","arizona","Arkansas"
            ,"asia","Asian"
            ,"atlanta"
            ,"Australia","australian","Australias"
        ,"Baja","beijing","Belgian","Belgians","belgium","benghazi","Berlin","boston"
            ,"Brazil","Brazilian","Brazos","britain","british","Britons"
                ,"broadway","bronx","brooklyn"
            ,"Burma"
        ,"Canada","canadian","canadians","caribbean"
            ,"centralpark"
            ,"chechnya","chicago","Chicago's","Chile","Chilean","chinese"
            ,"Cleveland"
            ,"Colombia","Colombian","Colombians","colorado","Connecticut"
            ,"cuba","cuba's","cuban","cubans"
            ,"Cyprus"
        ,"Dakota","dallas","Danish","Denmark","detroit","Deutsche","Dutch"
        ,"Egypt","England","english","euro","europe","european","Europeans","Europes"
        ,"ferguson","finland","Finnish","Finns","florida","France","frankfurt","Frankfurter"
        ,"georgia","Georgian","german","Germans","germany","glencore","greece","greek"
        ,"Haiti","Haitian","havana","Hawaii","Hawaiian","Helsinki","Hispanic"
            ,"hollywood","hongkong","HongKongs","Houston","Hungarian"
        ,"Ibiza","Iceland","illinois","india","indias","indian","indians","iowa"
            ,"Iran","Iranian","iraq","iraqi","Ireland","irish"
            ,"israel","Israeli"
            ,"Italia","Italian"
        ,"japan","japanese"
        ,"kansas","kentucky","Kenya","Kenyans","Kiev","korea"
        ,"lasvegas","Libya","Libyan","Lima","Lisbon","london","LosAngeles"
        ,"madrid","maine","malaysia","maryland","massachusetts","mediterranean","Mexican","mexico"
            ,"Miami","Michigan","milan","minnesota","missouri","moscow","myanmar"
        ,"NATO"
            ,"NewDelhi","NewHampshire","neworleans","newyork","newyorker","newyorkers"
                ,"newzealand"
            ,"Nigeria","nigerian"
            ,"NorthAmerican","northcarolina","NorthKorea","NorthKorean","norway","Norwegian"
        ,"ohio","omaha","ottawa"
        ,"pakistan","palestine","palestinian","paris","peking","Pennsylvania","peshawar"
            ,"Philadelphia","philippine"
            ,"poland","Ponytail","Portland","portugal","portuguese"
            ,"Prague","PuertoRico"
        ,"rome","Russia","russian","russians"
        ,"saigon","saintlouis","sanfrancisco","Saudi"
            ,"Scandinavian","Scot","scots","scotland","scottish"
            ,"Seattle","Selma"
            ,"Shanghai","Singapore"
            ,"SouthAfrica","SouthAfrican","southcarolina","southdakota","southkorea","southsudan"
            ,"spain","Spains","spanish"
            ,"stockton"
            ,"sweden","Swedish","swiss","Switzerland"
            ,"Sydney","Syria","Syrian","syrians"
        ,"Taiwan","Taiwanese","Tanzania","Taos","tennessee","texas","thai","Thailand"
            ,"Tiananmen","Tibetan"
            ,"Tokyo","toronto"
            ,"turks","turkish"
        ,"Uighur","Uighurs","ukraine","Ukraines","Ukrainian","Utah"
        ,"vermont","Vienna","Vietnam","Vietnamese","Vietnams","virginia","virginia's"
        ,"Wales","wallstreet","washington","WashingtonDC","wisconsin"
        ,"Xinjiang"
        ,"Yangtze","yemen","Ypres"
        ,"Zhejiang"
    ))))    
        
    # Add "rpl_str" from glb_txt_map (that don't contain \\#)    
    txtMapRplStr <- glbFeatsTextMap$rpl_str[!grepl("\\\\", glbFeatsTextMap$rpl_str)]
    txtMapRplStr <- txtMapRplStr[!grepl("[[:punct:]]|[[:digit:]]", txtMapRplStr)]
    txtMapRplStr <- str_trim(txtMapRplStr)
    txtMapRplStr <- txtMapRplStr[txtMapRplStr != ""]
    txtMapRplStr <- str_to_lower(txtMapRplStr[!grepl(" ", txtMapRplStr)])
    vldTerms <- union(vldTerms, txtMapRplStr)
                    
    # Set up vldTerms for each txtFeat
    for (feat in names(glbFeatsText)) {
        glbFeatsText[[feat]]$vldTerms <- vldTerms
    }    
    
    # Add "rpl_str" from glbFeatsDerive[[txtFeat]] (that don't contain \\#)
    for (feat in intersect(names(glbFeatsText), names(glbFeatsDerive))) {
        userDefTerms <- deparse(glbFeatsDerive[[feat]]$mapfn, width.cutoff = 100)
        userDefTerms <- userDefTerms[grepl("gsub", userDefTerms)]
        userDefTerms <- gsub("modRaw <- gsub\\(\"(.+)\", \"(.+)\", (.+)\\)", "\\2", userDefTerms)
        #            regexpr("modRaw <- gsub\\(\"(.+)\", \"(.+)\", ",         userDefTerms[111])
        userDefTerms <- str_trim(gsub("modRaw (.+)", "\\1", userDefTerms))
        userDefTerms <- userDefTerms[!grepl("\\\\", userDefTerms)]
        userDefTerms <- unique(str_to_lower(unlist(str_split(userDefTerms, " "))))
        glbFeatsText[[feat]]$vldTerms <- union(glbFeatsText[[feat]]$vldTerms, userDefTerms)
    }    
    
    # Add names & rareWords from glbFeatsText[[txtFeat]]
    for (feat in names(glbFeatsText)) {
        glbFeatsText[[feat]]$vldTerms <- union(glbFeatsText[[feat]]$vldTerms,
                                               glbFeatsText[[feat]]$names)
        glbFeatsText[[feat]]$vldTerms <- union(glbFeatsText[[feat]]$vldTerms,
                                               glbFeatsText[[feat]]$rareWords)        
    }    
                        
    # Add synonyms from glbFeatsTextSynonyms[[txtFeat]]
    for (feat in intersect(names(glbFeatsText), names(glbFeatsTextSynonyms))) {
        userDefTerms <- unlist(sapply(1:length(glbFeatsTextSynonyms[[feat]]), function(elemIx)
                                                    glbFeatsTextSynonyms[[feat]][[elemIx]]$syns))
        glbFeatsText[[feat]]$vldTerms <- union(glbFeatsText[[feat]]$vldTerms, userDefTerms)
    }    

#stop(here"); glb2Sav(); all.equal(savFeatsTextCorpus, glbFeatsTextCorpus); glbFeatsTextCorpus[[txtFeat]] <- savFeatsTextCorpus[[txtFeat]]
    glb_post_stop_words_terms_df_lst <- list(); 
    glb_post_stop_words_terms_mtrx_lst <- list();     
    glb_post_stem_words_terms_df_lst <- list(); 
    glb_post_stem_words_terms_mtrx_lst <- list();     
    glb_full_DTM_lst <- list();     
    for (txtFeat in names(glbFeatsText)) {
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "post.stop.weights.df.", txtFeat), major.inc = TRUE)

        print(sprintf("    Top_n post-stop term weights for %s:", txtFeat))
        # This impacts stemming probably due to lazy parameter
        print(myprint_df(full_terms_df <-
                get_corpus_terms(txt_corpus = glbFeatsTextCorpus[[txtFeat]]), 
                        glbFeatsTextTermsMax[[txtFeat]]))
        glb_post_stop_words_terms_df_lst[[txtFeat]] <- full_terms_df
        
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "post.stop.weights.mtrx.", txtFeat), major.inc = TRUE)
        
        set.seed(glbFeatsTextSeed)
        terms_stop_mtrx <- as.matrix(DocumentTermMatrix(glbFeatsTextCorpus[[txtFeat]], 
                                        control=glb_txt_terms_control))
        rownames(terms_stop_mtrx) <- rownames(glbObsAll) # print undreadable otherwise
        glb_post_stop_words_terms_mtrx_lst[[txtFeat]] <- terms_stop_mtrx
        
        tmp_allobs_df <- glbObsAll[, c(glbFeatsId, glb_rsp_var)]
        tmp_allobs_df$terms.post.stop.n <- rowSums(terms_stop_mtrx > 0)
        tmp_allobs_df$terms.post.stop.n.log <- log(1 + tmp_allobs_df$terms.post.stop.n)
        tmp_allobs_df$weight.post.stop.sum <- rowSums(terms_stop_mtrx)        
        
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "find.ngrams.", txtFeat), major.inc = TRUE)
        
        txt_corpus <- tm_map(glbFeatsTextCorpus[[txtFeat]], removeNumbers, lazy = TRUE)
        # myngramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
        TdmNgrams <- TermDocumentMatrix(txt_corpus, control = list(tokenize = mytokenize3grams))
        txtTdmNgrams <- get_txt_terms(TdmNgrams,
                        compute.nzv = FALSE, compute.chisq = FALSE, compute.classWeights = FALSE)
#         txtTdmNgrams$vldTerms <- sapply(1:nrow(txtTdmNgrams), function(rowIx) {
#             tokens <- unlist(str_split(txtTdmNgrams[rowIx, "term"], " ")); 
#             return(all(tokens %in% glbFeatsText[[txtFeat]]$vldTerms)) })
#         txtTdmNgrams$vldTerms <- unlist(mclapply(1:nrow(txtTdmNgrams), function(rowIx) {
#             tokens <- unlist(str_split(txtTdmNgrams[rowIx, "term"], " "));
#             return(all(tokens %in% glbFeatsText[[txtFeat]]$vldTerms)) },
#                                                     mc.cores = glbCores))
        inVldTerms <- setdiff(unique(unlist(strsplit(txtTdmNgrams$term, " "))),
                            glbFeatsText[[txtFeat]]$vldTerms)
        inVldTermsRegEx <- paste0("\\b(", paste(inVldTerms, collapse = "|"), ")\\b")
        txtTdmNgrams$vldTerms <- !grepl(inVldTermsRegEx, txtTdmNgrams$term)
        
        potentialNames <- subset(txtTdmNgrams, !vldTerms)
        print(sprintf("Remaining potential names: %d", nrow(potentialNames)))
        myprint_df(subset(potentialNames, select = -term))
        # to id invalid terms
        #unlist(strsplit("zzz z sleep", " ")) %in% glbFeatsText[[txtFeat]]$vldTerms
        #grep("\\bzzz", glb_txt_chr_lst[[txtFeat]], ignore.case = TRUE, value = TRUE)
        #grep("\\bzzzz", glbObsAll[, txtFeat], ignore.case = TRUE, value = TRUE)
        # to id docs that have a specific ngram using txtTdmNgrams[, pos]
        #glbObsAll[TdmNgrams$j[which(TdmNgrams$i == 40052)], glb_dsp_cols]
        #glbObsAll[unique(TdmNgrams$j[TdmNgrams$i %in% invldWords$pos]), "Hdln.my"]
    
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "stem.corpus.", txtFeat), major.inc = TRUE)
    
        glbFeatsTextCorpus[[txtFeat]] <- tm_map(glbFeatsTextCorpus[[txtFeat]], stemDocument,
                                            "english", lazy = FALSE)
        if (!is.null(glbFeatsTextSynonyms[[txtFeat]])) {
            # First entry in glbFeatsTextSynonyms[[txtFeat]] is NULL due to programming aesthetics
            #syn_lst <- myrmNullObj(glbFeatsTextSynonyms[[txtFeat]])
            syn_lst <- tail(glbFeatsTextSynonyms[[txtFeat]], -1) 
            glbFeatsTextCorpus[[txtFeat]] <- tm_map(glbFeatsTextCorpus[[txtFeat]],
                                                    mycombineSynonyms,
                                                    syn_lst, lazy = FALSE)
            glbFeatsText[[txtFeat]]$postStemSynonymsCorpus <- glbFeatsTextCorpus[[txtFeat]]
            
#             as.character(glbFeatsTextCorpus[[txtFeat]][c(5732, 5754)])
#             tmpFeatsTextCorpus <- tm_map(glbFeatsTextCorpus[[txtFeat]],
#                                                     mycombineSynonyms,
#                                                     syn_lst[1:133], # Crashes                 
#                                                     #syn_lst[1:2], 
#                                                     #syn_lst[1:4], 
#                                                     #syn_lst[1:8], 
#                                                     #syn_lst[1:16],
#                                                     #syn_lst[1:33], 
#                                                     #syn_lst[1:66], # Works                    
#                                                     #syn_lst[67:133], # Works
#                                                     lazy = FALSE)
#             as.character(tmpFeatsTextCorpus[c(5732, 5754)])
        }    
        
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "stem.weights.", txtFeat), major.inc = TRUE)
    
        print(sprintf("    Top_n stem term weights for %s:", txtFeat))        
        print(myprint_df(full_terms_df <- 
                             get_corpus_terms(txt_corpus = glbFeatsTextCorpus[[txtFeat]]), 
                   glbFeatsTextTermsMax[[txtFeat]]))
        glb_post_stem_words_terms_df_lst[[txtFeat]] <- full_terms_df    
        
        set.seed(glbFeatsTextSeed)
        glb_full_DTM_lst[[txtFeat]] <- 
            DocumentTermMatrix(glbFeatsTextCorpus[[txtFeat]], control = glb_txt_terms_control)
#         terms_stem_mtrx <- as.matrix(DocumentTermMatrix(glbFeatsTextCorpus[[txtFeat]], 
#                                         control=glb_txt_terms_control))
        terms_stem_mtrx <- as.matrix(glb_full_DTM_lst[[txtFeat]])
        rownames(terms_stem_mtrx) <- rownames(glbObsAll) # print undreadable otherwise
        glb_post_stem_words_terms_mtrx_lst[[txtFeat]] <- terms_stem_mtrx
        
        tmp_allobs_df$terms.post.stem.n <- rowSums(terms_stem_mtrx > 0)
        tmp_allobs_df$terms.post.stem.n.log <- log(1 + tmp_allobs_df$terms.post.stem.n)
        tmp_allobs_df$weight.post.stem.sum <- rowSums(terms_stem_mtrx)
        
        tmp_allobs_df$terms.n.stem.stop.Ratio <- 
            1.0 * tmp_allobs_df$terms.post.stem.n / tmp_allobs_df$terms.post.stop.n
        tmp_allobs_df[(is.nan(tmp_allobs_df$terms.n.stem.stop.Ratio) | 
                       is.infinite(tmp_allobs_df$terms.n.stem.stop.Ratio)), 
                      "terms.n.stem.stop.Ratio"] <- 1.0                
        if ((n.errors <- sum(tmp_allobs_df$terms.n.stem.stop.Ratio > 1)) > 0)
            stop(n.errors, " obs in tmp_allobs_df have terms.n.stem.stop.Ratio > 1", 
                 " happening due to terms filtered by glb_txt_terms_control$bounds$global[1] but stemmable to other terms")
        #print(head(subset(tmp_allobs_df, terms.n.stem.stop.Ratio > 1)))
        #glbObsAll[(row_ix <- which(glbObsAll$UniqueID == 4253)), glb_dsp_cols]
        #terms_stop_mtrx[row_ix, terms_stop_mtrx[row_ix, ] > 0]
        #terms_stem_mtrx[row_ix, terms_stem_mtrx[row_ix, ] > 0]
        #setdiff(names(terms_stem_mtrx[row_ix, terms_stem_mtrx[row_ix, ] > 0]), names(terms_stop_mtrx[row_ix, terms_stop_mtrx[row_ix, ] > 0]))
        #mydspObs(list(descr.my.contains="updat"))
        
        tmp_allobs_df$weight.sum.stem.stop.Ratio <- 
            1.0 * tmp_allobs_df$weight.post.stem.sum / tmp_allobs_df$weight.post.stop.sum
        tmp_allobs_df[is.nan(tmp_allobs_df$weight.sum.stem.stop.Ratio) | 
                      is.infinite(tmp_allobs_df$weight.sum.stem.stop.Ratio), 
                      "weight.sum.stem.stop.Ratio"] <- 1.0                
        
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "stem.weights.cor.", txtFeat), major.inc = TRUE)
    
        tmp_trnobs_df <- tmp_allobs_df[!is.na(tmp_allobs_df[, glb_rsp_var]), ]
        print(cor(as.matrix(tmp_trnobs_df[, -c(1, 2)]), 
                  as.numeric(tmp_trnobs_df[, glb_rsp_var])))

        txtFeat_pfx <- toupper(substr(txtFeat, 1, 1))
        tmp_allobs_df <- tmp_allobs_df[, -c(1, 2)]
        names(tmp_allobs_df) <- paste(paste0(txtFeat_pfx, "."), names(tmp_allobs_df), sep = "")
        glbObsAll <- cbind(glbObsAll, tmp_allobs_df)
        glbFeatsExclude <- c(glbFeatsExclude, 
                paste(paste0(txtFeat_pfx, ".terms.post."), c("stop.n", "stem.n"), sep = ""))
    }
    
    # Report terms that are present only in Trn or New (may be eliminated from analysis)    
    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
            paste0("extract.features_", "report.corpus.stem.partition"), major.inc = TRUE)
    for (txtFeat in names(glbFeatsText)) {
        m <- glb_post_stem_words_terms_mtrx_lst[[txtFeat]]
        mTrn <- m[glbObsAll$.src == "Train", ]
        termsTrn <- data.frame(term = dimnames(mTrn)$Terms, weight.Trn = colSums(mTrn))
        mNew <- m[glbObsAll$.src == "Test", ]
        termsNew <- data.frame(term = dimnames(mNew)$Terms, weight.New = colSums(mNew))
        termsAll <- merge(termsTrn, termsNew)
        termsAll$pos <- seq(1:nrow(termsAll))
        termsAll <- termsAll[(termsAll$weight.Trn == 0) | (termsAll$weight.New == 0), ]
        print(sprintf("    Partition post-stem term weights for %s: %d", txtFeat, nrow(termsAll)))
        myprint_df(termsAll)
        glbFeatsText[[txtFeat]]$filterTerms <- termsAll$term
    }
    
    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
            paste0("extract.features_", "report.corpus.wordcloud"), major.inc = TRUE)
    
    myplot_wordcloud <- function(DTMtrx) {
        require(wordcloud)

        # calculate the frequency of words
        v <- sort(colSums(DTMtrx), decreasing = TRUE)
        myNames <- names(v)
        minFrq <- glb_txt_terms_control$bounds$global[1]
        d <- subset(data.frame(word = myNames, freq = v), freq >= minFrq)
        while (nrow(d) > 200) {
            # wordcloud takes a long time
            minFrq <- min(d$freq)
            d <- subset(d, freq > minFrq)
        }
        minFrq <- min(d$freq)
        if (minFrq > glb_txt_terms_control$bounds$global[1])
            print(sprintf("        Wordcloud filtered for freq: %d vs. control: %d", 
                          floor(minFrq), glb_txt_terms_control$bounds$global[1]))
        print(wordcloud(d$word, d$freq, min.freq = minFrq))
    }
        
    for (txtFeat in names(glbFeatsText)) {
        print(sprintf("    Wordcloud (All) post-stem term weights for %s:", txtFeat))
        myplot_wordcloud(glb_post_stem_words_terms_mtrx_lst[[txtFeat]])
        print(sprintf("    Wordcloud (Trn only) post-stem term weights for %s:", txtFeat))
        myplot_wordcloud(glb_post_stem_words_terms_mtrx_lst[[txtFeat]][glbObsAll$.src == "Train", ])
        print(sprintf("    Wordcloud (New only) post-stem term weights for %s:", txtFeat))
        myplot_wordcloud(glb_post_stem_words_terms_mtrx_lst[[txtFeat]][glbObsAll$.src == "Test", ])
    }    

    for (txtFeat in names(glbFeatsText)) {
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "filter.corpus.", txtFeat), major.inc = TRUE)
    
        if (!is.null(glbFeatsText[[txtFeat]]$filterTerms)) {
#             glbFeatsTextCorpus[[txtFeat]] <- 
#                 tm_map(glbFeatsTextCorpus[[txtFeat]], removeWords,
#                         glbFeatsText[[txtFeat]]$filterTerms, lazy = FALSE) #nFltrTerms
            
            # if length(glbFeatsText[[txtFeat]]$filterTerms) > 3000 tm_map crashes
            stepVal <- 1000
            for (rng in seq(1, length(glbFeatsText[[txtFeat]]$filterTerms), stepVal)) {
                bgn <- rng
                end <- bgn + stepVal - 1
                if (end > length(glbFeatsText[[txtFeat]]$filterTerms))
                    end <- length(glbFeatsText[[txtFeat]]$filterTerms)
                print(sprintf("bgn:%d; end:%d", bgn, end))
                glbFeatsTextCorpus[[txtFeat]] <- 
                    tm_map(glbFeatsTextCorpus[[txtFeat]], removeWords,
                        glbFeatsText[[txtFeat]]$filterTerms[bgn:end], lazy = FALSE) #nFltrTerms
            }
        }            

        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "filter.weights.", txtFeat), major.inc = TRUE)
    
        print(sprintf("    Top_n filter term weights for %s:", txtFeat))        
        print(myprint_df(full_terms_df <- 
                    get_corpus_terms(txt_corpus = glbFeatsTextCorpus[[txtFeat]], 
                        compute.cor.y = TRUE, compute.nzv = TRUE, 
                        compute.chisq = TRUE, compute.classWeights = TRUE), 
                   glbFeatsTextTermsMax[[txtFeat]]))
        glbFeatsText[[txtFeat]]$postFilterTermsDf <- full_terms_df    
        
        set.seed(glbFeatsTextSeed)
        glb_full_DTM_lst[[txtFeat]] <- 
            DocumentTermMatrix(glbFeatsTextCorpus[[txtFeat]], control = glb_txt_terms_control)
        termsFilterMtrx <- as.matrix(glb_full_DTM_lst[[txtFeat]])
        rownames(termsFilterMtrx) <- rownames(glbObsAll) # print undreadable otherwise
        glbFeatsText[[txtFeat]]$postFilterTermsMtrx <- termsFilterMtrx
        
        txtFeat_pfx <- toupper(substr(txtFeat, 1, 1))
        tmpObsAll <- glbObsAll[, c(glbFeatsId, glb_rsp_var)] 
        tmpObsAll$terms.post.stem.n <- 
            glbObsAll[, paste0(txtFeat_pfx, ".terms.post.stem.n")]
        tmpObsAll$weight.post.stem.sum <- 
            glbObsAll[, paste0(txtFeat_pfx, ".weight.post.stem.sum")]
        tmpObsAll$terms.post.filter.n <- rowSums(termsFilterMtrx > 0)
        tmpObsAll$terms.post.filter.n.log <- log1p(tmpObsAll$terms.post.filter.n)
        tmpObsAll$weight.post.filter.sum <- rowSums(termsFilterMtrx)
        
        tmpObsAll$terms.n.filter.stem.Ratio <- 
            1.0 * tmpObsAll$terms.post.filter.n / tmpObsAll$terms.post.stem.n
        tmpObsAll[(is.nan(tmpObsAll$terms.n.filter.stem.Ratio) | 
                       is.infinite(tmpObsAll$terms.n.filter.stem.Ratio)), 
                      "terms.n.filter.stem.Ratio"] <- 1.0                
        if ((n.errors <- sum(tmpObsAll$terms.n.filter.stem.Ratio > 1)) > 0)
            stop(n.errors, " obs in tmpObsAll have terms.n.filter.stem.Ratio > 1", 
                 " happening due to terms ???")
        #print(head(subset(tmpObsAll, terms.n.stem.stop.Ratio > 1)))
        #glbObsAll[(row_ix <- which(glbObsAll$UniqueID == 4253)), glb_dsp_cols]
        #terms_stop_mtrx[row_ix, terms_stop_mtrx[row_ix, ] > 0]
        #terms_stem_mtrx[row_ix, terms_stem_mtrx[row_ix, ] > 0]
        #setdiff(names(terms_stem_mtrx[row_ix, terms_stem_mtrx[row_ix, ] > 0]), names(terms_stop_mtrx[row_ix, terms_stop_mtrx[row_ix, ] > 0]))
        #mydspObs(list(descr.my.contains="updat"))
        
        tmpObsAll$weight.sum.filter.stem.Ratio <- 
            1.0 * tmpObsAll$weight.post.filter.sum / tmpObsAll$weight.post.stem.sum
        tmpObsAll[is.nan(tmpObsAll$weight.sum.filter.stem.Ratio) | 
                      is.infinite(tmpObsAll$weight.sum.filter.stem.Ratio), 
                      "weight.sum.filter.stem.Ratio"] <- 1.0                
        
        extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                paste0("extract.features_", "filter.weights.cor.", txtFeat), major.inc = TRUE)
    
        tmpObsTrn <- tmpObsAll[!is.na(tmpObsAll[, glb_rsp_var]), ]
        print(cor(as.matrix(tmpObsTrn[, -c(1, 2)]), 
                  as.numeric(tmpObsTrn[, glb_rsp_var])))

        tmpObsAll <- tmpObsAll[, -c(1, 2)]
        names(tmpObsAll) <- paste(paste0(txtFeat_pfx, "."), names(tmpObsAll), sep = "")
        glbObsAll <- cbind(glbObsAll, tmpObsAll[, setdiff(names(tmpObsAll), names(glbObsAll))])
        glbFeatsExclude <- union(glbFeatsExclude, 
            paste(paste0(txtFeat_pfx, ".terms.post."), c("stop.n","stem.n","filter.n"), sep = ""))
    }    

    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
            paste0("extract.features_", "report.corpus.termggplot"), major.inc = TRUE)
    
    for (txtFeat in names(glbFeatsText)) {    
        .rnorm.cor.y.abs <- abs(cor(glbObsAll[glbObsAll$.src == "Train", ".rnorm"],
                    as.numeric(glbObsAll[glbObsAll$.src == "Train", glb_rsp_var]),
                                use = "pairwise.complete.obs"))
        # plt_df <- subset(glb_post_stem_words_terms_df_lst[[txtFeat]], !is.na(cor.y))
        plt_df <- subset(glbFeatsText[[txtFeat]]$postFilterTermsDf, !is.na(cor.y))        
        plt_df$nzv.freqRatio.cut.fctr <- cut(plt_df$nzv.freqRatio, 
                                            breaks = sort(c(min(plt_df$nzv.freqRatio), 
                                                                glbFeatsNzvFreqMax,
                                                            max(plt_df$nzv.freqRatio))))
        plt_df$nzv.percentUnique.cut.fctr <- cut(plt_df$nzv.percentUnique, 
                breaks = sort(c(min(plt_df$nzv.percentUnique) - .Machine$double.neg.eps, 
                                                            glbFeatsNzvUniqMin,
                                                        max(plt_df$nzv.percentUnique))))
        plt_df$nzv.quad.fctr <- as.factor(paste0("fRatio:", plt_df$nzv.freqRatio.cut.fctr,
                                            "\n%Unq:", plt_df$nzv.percentUnique.cut.fctr))
        labelCnd <- !plt_df$nzv | 
                    (!is.na(plt_df$chisq.pval) & (plt_df$chisq.pval < 0.05)) & 
                (!is.na(plt_df$cor.y.abs) & (plt_df$cor.y.abs > .rnorm.cor.y.abs))
        plt_df$label <- NA; plt_df[labelCnd, "label"] <- plt_df[labelCnd, "term"]
        print(ggplot(plt_df, aes(x = cor.y, y = chisq.stat)) + 
                  geom_point(aes(color = nzv.quad.fctr, size = weight)) +
                  geom_text(aes(label = label), color = "gray50") + 
                  # geom_vline(xintercept = 0) + 
            geom_vline(xintercept = c(-1, +1) * .rnorm.cor.y.abs, color = "gray", 
                       linetype = "dashed") + 
            ggtitle(txtFeat))
    }
        
    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
            paste0("extract.features_", "extract.DTM"), major.inc = TRUE)

    glb_sprs_DTM_lst <- list();
    for (txtFeat in names(glbFeatsText)) {
        print(sprintf("Extracting term weights for %s...", txtFeat))        
#         txt_corpus <- glbFeatsTextCorpus[[txtFeat]]
#         
#         full_DTM <- DocumentTermMatrix(txt_corpus, 
#                                           control=glb_txt_terms_control)
#         sprs_DTM <- removeSparseTerms(full_DTM, 
#                                             glb_sprs_thresholds[txtFeat])
        # glb_full_DTM_lst[[txtFeat]] <- full_DTM
        
        glb_sprs_DTM_lst[[txtFeat]] <- 
            removeSparseTerms(glb_full_DTM_lst[[txtFeat]], glb_sprs_thresholds[txtFeat])
    }

    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
            paste0("extract.features_", "report.DTM"), major.inc=TRUE)
    
    require(reshape2)
    for (txtFeat in names(glbFeatsText)) {
        print(sprintf("Reporting term weights for %s...", txtFeat))        
        full_DTM <- glb_full_DTM_lst[[txtFeat]]
        sprs_DTM <- glb_sprs_DTM_lst[[txtFeat]]        

        print("   Full TermMatrix:"); print(full_DTM)
        # full_terms_df <- get_txt_terms(full_DTM)
        # terms_full_df <- glb_post_stem_words_terms_df_lst[[txtFeat]]
        terms_full_df <- glbFeatsText[[txtFeat]]$postFilterTermsDf

        print("   Sparse TermMatrix:"); print(sprs_DTM)
        terms_sprs_df <- get_txt_terms(sprs_DTM, compute.classWeights = TRUE)
#         sprs_terms_df <- sprs_terms_df[, c(2, 1, 3, 4)]
#         col_names <- names(sprs_terms_df)
#         col_names[2:length(col_names)] <- 
#             paste(col_names[2:length(col_names)], ".sprs", sep="")
#         names(sprs_terms_df) <- col_names

        #intersect(names(full_terms_df), names(sprs_terms_df))
        terms_df <- merge(terms_full_df, terms_sprs_df, by = c("term", "weight", "freq",
                                        grep("weight\\.", names(full_terms_df), value = TRUE)),
                          all.x = TRUE, suffixes = c(".full", ".sprs"))
        terms_df$in.sprs <- !is.na(terms_df$pos.sprs)
        plt_terms_df <- subset(terms_df, 
                        weight >= min(terms_df$weight[!is.na(terms_df$pos.sprs)], na.rm=TRUE))
        plt_terms_df$label <- ""
        plt_terms_df[is.na(plt_terms_df$pos.sprs), "label"] <- 
            plt_terms_df[is.na(plt_terms_df$pos.sprs), "term"]
#         glb_important_terms[[txtFeat]] <- union(glb_important_terms[[txtFeat]],
#             plt_terms_df[is.na(plt_terms_df$TfIdf.sprs), "term"])
        print(myplot_scatter(plt_terms_df, "freq", "weight", 
                             colorcol_name="in.sprs") + 
                  geom_text(aes(label=label), color="Black", size=3.5))
        
        melt_terms_df <- orderBy(~ -value, 
                            melt(terms_df, id.vars="term", measure.vars = c("weight", "freq")))
        print(ggplot(melt_terms_df, aes(value, color=variable)) + stat_ecdf() + 
                  geom_hline(yintercept=glb_sprs_thresholds[txtFeat], 
                             linetype = "dotted"))
        
        melt_terms_df <- orderBy(~-value, 
                        melt(subset(terms_df, in.sprs), id.vars = "term",
                             measure.vars = grep("weight.", names(terms_df), value = TRUE)))
        print(myplot_hbar(melt_terms_df, "term", "value", colorcol_name = "variable"))
        
        melt_terms_df <- orderBy(~ -value, 
                        melt(subset(terms_df, !in.sprs), id.vars="term",
                             measure.vars=grep("weight.", names(terms_df), value=TRUE)))
        print(myplot_hbar(head(melt_terms_df, glbFeatsTextTermsMax[[txtFeat]]), "term", "value",
                          colorcol_name="variable"))
    }

#     sav_full_DTM_lst <- glb_full_DTM_lst
#     print(identical(sav_glbFeatsTextCorpus, glbFeatsTextCorpus))
#     print(all.equal(length(sav_glbFeatsTextCorpus), length(glbFeatsTextCorpus)))
#     print(all.equal(names(sav_glbFeatsTextCorpus), names(glbFeatsTextCorpus)))
#     print(all.equal(sav_glbFeatsTextCorpus[["Headline"]], glbFeatsTextCorpus[["Headline"]]))

#     print(identical(sav_full_DTM_lst, glb_full_DTM_lst))
        
    # Create txt features
    if ((length(names(glbFeatsText)) > 1) &&
        (length(unique(pfxs <- sapply(names(glbFeatsText), 
                    function(txt) toupper(substr(txt, 1, 1))))) < length(names(glbFeatsText))))
            stop("Prefixes for corpus freq terms not unique: ", pfxs)
    
    extract.features.text.chunk.df <- myadd_chunk(extract.features.text.chunk.df, 
                            paste0("extract.features_", "bind.DTM"), 
                                         major.inc = TRUE)
#stop(here"); glb2Sav(); all.equal(savObsAll, glbObsAll); glbObsAll <- savObsAll
    require(tidyr)
    for (txtFeat in names(glbFeatsText)) {
        print(sprintf("Binding DTM for %s...", txtFeat))
        txtFeat_pfx <- toupper(substr(txtFeat, 1, 1))
        
        txt_full_X_df <- as.data.frame(as.matrix(glb_full_DTM_lst[[txtFeat]]))
        #terms_full_df <- glb_post_stem_words_terms_df_lst[[txtFeat]]
        terms_full_df <- glbFeatsText[[txtFeat]]$postFilterTermsDf        
        # make.names adds a period to R keywords e.g. "in", "function"
        colnames(txt_full_X_df) <- paste(txtFeat_pfx, ".T.",
                                    make.names(colnames(txt_full_X_df)), sep = "")
        rownames(txt_full_X_df) <- rownames(glbObsAll) # warning otherwise
        
        if (glbFeatsTextFilter == "sparse") {
            txt_X_df <- as.data.frame(as.matrix(glb_sprs_DTM_lst[[txtFeat]]))
            select_terms <- make.names(colnames(txt_X_df))
        } else if (glbFeatsTextFilter == "top.val") {
            select_terms <- orderBy(~-weight,
                                    terms_full_df)$term[1:glbFeatsTextTermsMax[[txtFeat]]]
        } else if (glbFeatsTextFilter == "top.cor") {
            select_terms <- orderBy(~-cor.y.abs,
                                    terms_full_df)$term[1:glbFeatsTextTermsMax[[txtFeat]]]
        } else if (glbFeatsTextFilter == "top.chisq") {
            select_terms <- orderBy(~-chisq.stat,
                                    subset(terms_full_df, chisq.pval < 0.05)
                                    )$term[1:glbFeatsTextTermsMax[[txtFeat]]]
        } else if (glbFeatsTextFilter == "union.top.val.cor") {
            select_terms <- union(
                orderBy(~-weight   , terms_full_df)$term[1:glbFeatsTextTermsMax[[txtFeat]]],
                orderBy(~-cor.y.abs, terms_full_df)$term[1:glbFeatsTextTermsMax[[txtFeat]]])
        } else stop(
        "glbFeatsTextFilter should be one of c('sparse', 'top.val', 'top.cor', 'union.top.val.cor', 'top.chisq') vs. '",
                    glbFeatsTextFilter, "'")    
        
        assoc_terms_lst <- findAssocs(glb_full_DTM_lst[[txtFeat]], select_terms, 
                                      glbFeatsTextAssocCor[[txtFeat]])
        assoc_terms <- c(NULL)
        for (term in names(assoc_terms_lst))
            if (length(assoc_terms_lst[[term]]) > 0)
                assoc_terms <- union(assoc_terms, names(assoc_terms_lst[[term]]))

#stop(here"); glb2Sav()
        txt_X_df <- txt_full_X_df[, 
                        subset(terms_full_df, term %in% c(select_terms, assoc_terms))$pos,
                                    FALSE]
        glbObsAll <- cbind(glbObsAll, txt_X_df) # TfIdf is normalized
        #glbObsAll <- cbind(glbObsAll, log_X_df) # if using non-normalized metrics 
    }
    #identical(chk_entity_df, glbObsAll)
    #chk_entity_df <- glbObsAll

    extract.features.text.chunk.df <- 
        myadd_chunk(extract.features.text.chunk.df, paste0("extract.features_", "bind.DXM"), 
                    major.inc = TRUE)

#stop(here"); glb2Sav(); all.equal(savObsAll, glbObsAll); savObsAll <- glbObsAll; glbObsAll <- savObsAll
    glb_punct_vctr <- c("!", "\"", "#", "\\$", "%", "&", "'", 
                        "\\(|\\)",# "\\(", "\\)", 
                        "\\*", "\\+", ",", "-", "\\.", "/", ":", ";", 
                        "<|>", # "<", 
                        "=", 
                        # ">", 
                        "\\?", "@", "\\[", "\\\\", "\\]", "\\^", "_", "`", 
                        "\\{", "\\|", "\\}", "~")
    txt_X_df <- glbObsAll[, c(glbFeatsId, ".rnorm"), FALSE]
    #txt_X_df <- foreach(txtFeat=glbFeatsText, .combine=cbind) %dopar% {   
    txt_X_df <- foreach(txtFeat = names(glbFeatsText), .combine = cbind) %do% {           
    # for (txtFeat in glbFeatsText) {
        print(sprintf("Binding DXM for %s...", txtFeat))
        txtFeat_pfx <- toupper(substr(txtFeat, 1, 1))        
        
        tmEnter <- as.numeric(proc.time()["elapsed"])
        print(sprintf("Binding DXM Loop: enter: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        txt_full_DTM_mtrx <- as.matrix(glb_full_DTM_lst[[txtFeat]])
        rownames(txt_full_DTM_mtrx) <- rownames(glbObsAll) # print undreadable otherwise
        #print(txt_full_DTM_mtrx[txt_full_DTM_mtrx[, "ebola"] != 0, "ebola"])
        
        # Create <txtFeat>.T.<term> for glb_important_terms
        print(sprintf("Binding DXM Loop: glb_important_terms: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        for (term in glb_important_terms[[txtFeat]])
            txt_X_df[, paste0(txtFeat_pfx, ".T.", make.names(term))] <- 
                txt_full_DTM_mtrx[, term]
                
        # Create <txtFeat>.wrds.n.log & .wrds.unq.n.log
        print(sprintf("Binding DXM Loop: .wrds.n.log: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        txt_X_df[, paste0(txtFeat_pfx, ".wrds.n.log")] <- 
            log(1 + mykntpar_pattern_occ("\\w+", glbObsAll[, txtFeat]))
        txt_X_df[, paste0(txtFeat_pfx, ".wrds.unq.n.log")] <- 
            log(1 + rowSums(txt_full_DTM_mtrx != 0))
        txt_X_df[, paste0(txtFeat_pfx, ".weight.sum")] <- 
            rowSums(txt_full_DTM_mtrx) 
        txt_X_df[, paste0(txtFeat_pfx, ".ratio.weight.sum.wrds.n")] <- 
            txt_X_df[, paste0(txtFeat_pfx, ".weight.sum")] / 
            (exp(txt_X_df[, paste0(txtFeat_pfx, ".wrds.n.log")]) - 1)
        txt_X_df[is.nan(txt_X_df[, paste0(txtFeat_pfx, ".ratio.weight.sum.wrds.n")]),
                 paste0(txtFeat_pfx, ".ratio.weight.sum.wrds.n")] <- 0

        # Create <txtFeat>.chrs.n.log
        txt_X_df[, paste0(txtFeat_pfx, ".chrs.n.log")] <- 
            log(1 + mykntpar_pattern_occ(".", glbObsAll[, txtFeat]))
        txt_X_df[, paste0(txtFeat_pfx, ".chrs.uppr.n.log")] <- 
            log(1 + mykntpar_pattern_occ("[[:upper:]]", glbObsAll[, txtFeat]))
        txt_X_df[, paste0(txtFeat_pfx, ".dgts.n.log")] <- 
            log(1 + mykntpar_pattern_occ("[[:digit:]]", glbObsAll[, txtFeat]))

        # Create <txtFeat>.npnct?.log
        # would this be faster if it's iterated over each row instead of 
        #   each created column ???
        print(sprintf("Binding DXM Loop: glb_punct_vctr: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
# stop(here"); sav_X_df <- txt_X_df; all.equal(sav_X_df, txt_X_df); txt_X_df <- sav_X_df; head(txt_X_df)
#         for (punct_ix in 1:length(glb_punct_vctr)) { 
#             txt_X_df[, 
#                 paste0(txtFeat_pfx, ".chrs.pnct", sprintf("%02d", punct_ix), ".n.log")] <-
#                 log(1 + mykntpar_pattern_occ(glb_punct_vctr[punct_ix], 
#                                             glbObsAll[, txtFeat]))
#         }
# res_X_df <- txt_X_df; all.equal(res_X_df, txt_X_df)

        thsXMtrx <- foreach(punct_ix = 1:length(glb_punct_vctr), .combine = cbind) %dopar% {
        # for (punct_ix in 1:length(glb_punct_vctr)) { 
#             smp0 <- " "
#             smp1 <- "! \" # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~"
#             smp2 <- paste(smp1, smp1, sep=" ")
#             print(sprintf("Testing %s pattern:", glb_punct_vctr[punct_ix])) 
#             results <- mycount_pattern_occ(glb_punct_vctr[punct_ix], c(smp0, smp1, smp2))
#             names(results) <- NULL; print(results)
            thsXClmn <-
                log1p(mykntpar_pattern_occ(glb_punct_vctr[punct_ix], glbObsAll[, txtFeat]))
        }
        dimnames(thsXMtrx)[[2]] <- 
            paste0(txtFeat_pfx, ".chrs.pnct", sprintf("%02d", 1:length(glb_punct_vctr)), ".n.log")
        txt_X_df <- cbind(txt_X_df, thsXMtrx)
        
#         print(head(glbObsAll[glbObsAll[, "A.npnct23.log"] > 0, 
#                                     c("UniqueID", "Popular", "Abstract", "A.npnct23.log")]))    
        
        # Create <txtFeat>.niso8859.log
        txt_X_df[, paste0(txtFeat_pfx, ".chrs.iso8859.n.log")] <- 
            log1p(mykntpar_pattern_occ("&#[[:digit:]]{3};", glbObsAll[, txtFeat]))
                
        # Create <txtFeat>.wrds.stop.n.log & <txtFeat>ratio.wrds.stop.n.wrds.n
        if (!is.null(glb_txt_stop_words[[txtFeat]])) {
            print(sprintf("Binding DXM Loop: .wrds.stop.n.log: elapsed: %0.2f secs",
                          as.numeric(proc.time()["elapsed"]) - tmEnter))
            
#           stop_words_rex_str <- paste0("\\b(", paste0(glb_txt_stop_words[[txtFeat]][1501:2000], collapse = "|"),")\\b")
#             stopLen <- sapply(glb_txt_stop_words[[txtFeat]], function(x) length(unlist(strsplit(x, " "))));print(stopLen[stopLen > 1])
#           
            # if length(glb_txt_stop_words[[txtFeat]]) > 500 it crashes
            txt_X_df[, paste0(txtFeat_pfx, ".wrds.stop.n.log")] <- 0            
            stepVal <- 500
            for (rng in seq(1, length(glb_txt_stop_words[[txtFeat]]), stepVal)) {
                bgn <- rng
                end <- bgn + stepVal - 1
                if (end > length(glb_txt_stop_words[[txtFeat]]))
                    end <- length(glb_txt_stop_words[[txtFeat]])
                print(sprintf("bgn:%d; end:%d", bgn, end))
                stop_words_rex_str <- paste0("\\b(", 
                                paste0(glb_txt_stop_words[[txtFeat]][bgn:end], collapse = "|"),
                                             ")\\b")
                txt_X_df[, paste0(txtFeat_pfx, ".wrds.stop.n.log")] <-
                    txt_X_df[, paste0(txtFeat_pfx, ".wrds.stop.n.log")] + 
                    mykntpar_pattern_occ(stop_words_rex_str, glbObsAll[, txtFeat])
                    #log1p(mykntpar_pattern_occ(stop_words_rex_str, glb_txt_chr_lst[[txtFeat]][1]))   
            }
            txt_X_df[, paste0(txtFeat_pfx, ".wrds.stop.n.log")] <-
                    log1p(txt_X_df[, paste0(txtFeat_pfx, ".wrds.stop.n.log")])
         
            txt_X_df[, paste0(txtFeat_pfx, ".ratio.wrds.stop.n.wrds.n")] <-
                exp(txt_X_df[, paste0(txtFeat_pfx, ".wrds.stop.n.log")] - 
                    txt_X_df[, paste0(txtFeat_pfx, ".wrds.n.log")])
        }

        # Create <txtFeat>.wrds.SCOWL.n.log & <txtFeat>ratio.wrds.SCOWL.n.wrds.n
        if (TRUE) {
            print(sprintf("Binding DXM Loop: .wrds.SCOWL.n.log: elapsed: %0.2f secs",
                          as.numeric(proc.time()["elapsed"]) - tmEnter))
            
            mask <- dimnames(glb_post_stem_words_terms_mtrx_lst[[txtFeat]])$Terms %in%
                            glbSCOWLWords
            txt_X_df[, paste0(txtFeat_pfx, ".wrds.SCOWL.n.log")] <-
                log1p(rowSums(glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, mask] > 0))
                
            txt_X_df[, paste0(txtFeat_pfx, ".ratio.wrds.SCOWL.n.wrds.n")] <-
                exp(txt_X_df[, paste0(txtFeat_pfx, ".wrds.SCOWL.n.log")] - 
                    txt_X_df[, paste0(txtFeat_pfx, ".wrds.n.log")])
        }

        # Create <txtFeat>.wrds.rareWords.n.log & <txtFeat>ratio.wrds.rareWords.n.wrds.n
        print(sprintf("Binding DXM Loop: .wrds.rareWords.n.log: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        mask <- dimnames(glb_post_stem_words_terms_mtrx_lst[[txtFeat]])$Terms %in%
                glbFeatsText[[txtFeat]]$rareWords
        txt_X_df[, paste0(txtFeat_pfx, ".wrds.rareWords.n.log")] <-
            log1p(rowSums(glb_post_stem_words_terms_mtrx_lst[[txtFeat]][, mask] > 0))
        
        txt_X_df[, paste0(txtFeat_pfx, ".ratio.wrds.rareWords.n.wrds.n")] <-
            exp(txt_X_df[, paste0(txtFeat_pfx, ".wrds.rareWords.n.log")] - 
                    txt_X_df[, paste0(txtFeat_pfx, ".wrds.n.log")])
        
        # Create <txtFeat>.P.http
        print(sprintf("Binding DXM Loop: .P.<pattern>.log1p: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        txt_X_df[, paste(txtFeat_pfx, ".P.http.log1p", sep = "")] <- 
            log1p(mykntpar_pattern_occ("http", glbObsAll[, txtFeat]))
    
        # Create <txtFeat>.P.<user-spec-pattern>
        for (pattern in names(glbFeatsTextPatterns[[txtFeat]]))
            txt_X_df[, paste(txtFeat_pfx, ".P.", pattern, sep = "")] <- 
                log1p(mykntpar_pattern_occ(glbFeatsTextPatterns[[txtFeat]][pattern], 
                                            glbObsAll[, txtFeat]))
    
        print(sprintf("Binding DXM Loop: exit: elapsed: %0.2f secs",
                      as.numeric(proc.time()["elapsed"]) - tmEnter))
        
        txt_X_df <- subset(txt_X_df, select = -.rnorm)
        txt_X_df <- txt_X_df[, -grep(glbFeatsId, names(txt_X_df), fixed = TRUE), FALSE]
        #glbObsAll <- cbind(glbObsAll, txt_X_df)
    }
    glbObsAll <- cbind(glbObsAll, txt_X_df)
    #myplot_box(glbObsAll, "A.sum.TfIdf", glb_rsp_var)
    colCmp <- grep("\\.P\\.", names(glbObsAll), value = TRUE)
    if (sum(is.na(txt_X_df [, colCmp])) > 0)
        stop("colCmp is/are NA")
    if (sum(is.na(glbObsAll[, colCmp])) > 0)
        stop("colCmp is/are NA")        
    if (!(identical(txt_X_df[, colCmp], glbObsAll[, colCmp])))
        stop("colCmp is/are NA")        
    #all.equal(txt_X_df[, colCmp], glbObsAll[, colCmp])
    #head(glbObsAll[, union(colCmp, glb_dsp_cols)])

    # Generate summaries
#     print(summary(glbObsAll))
#     print(sapply(names(glbObsAll), function(col) sum(is.na(glbObsAll[, col]))))
#     print(summary(glbObsTrn))
#     print(sapply(names(glbObsTrn), function(col) sum(is.na(glbObsTrn[, col]))))
#     print(summary(glbObsNew))
#     print(sapply(names(glbObsNew), function(col) sum(is.na(glbObsNew[, col]))))

    glbFeatsExclude <- union(glbFeatsExclude, names(glbFeatsText))
    #rm(log_X_df, txt_X_df)
    
    extract.features.text.chunk.df <- 
        myadd_chunk(extract.features.text.chunk.df, paste0("extract.features.text", ".bind.DXM"), 
                    major.inc = TRUE)
}

glb_chunks_df <- myadd_chunk(glb_chunks_df, "extract.features.end", major.inc = FALSE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r extract.features.end, cache=FALSE, echo=FALSE}

# print(sapply(names(glbObsTrn), function(col) sum(is.na(glbObsTrn[, col]))))
# print(sapply(names(glbObsNew), function(col) sum(is.na(glbObsNew[, col]))))

# print(myplot_scatter(glbObsTrn, "<col1_name>", "<col2_name>", smooth=TRUE))

#stop(here"); glb2Sav(); glbObsAll <- savObsAll

# if (glb_save_envir)
#     save(glb_feats_df, 
#          glbObsAll, #glbObsTrn, glbObsFit, glbObsOOB, glbObsNew,
#          file=paste0(glb_out_pfx, "extract_features_dsk.RData"))
# load(paste0(glb_out_pfx, "extract_features_dsk.RData"))

replay.petrisim(pn=glb_analytics_pn, 
    replay.trans=(glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
        "data.training.all","data.new")), flip_coord=TRUE)
glb_chunks_df <- myadd_chunk(glb_chunks_df, "manage.missing.data", major.inc = TRUE)
```

### Step ``r mydsp_chunk(glb_chunks_df)``
```{r manage.missing.data, cache=FALSE, echo=FALSE}

# If mice crashes with error: Error in get(as.character(FUN), mode = "function", envir = envir) : object 'State' of mode 'function' was not found
#   consider excluding 'State' as a feature

# print(sapply(names(glbObsTrn), function(col) sum(is.na(glbObsTrn[, col]))))
# print(sapply(names(glbObsNew), function(col) sum(is.na(glbObsNew[, col]))))
# glbObsTrn <- na.omit(glbObsTrn)
# glbObsNew <- na.omit(glbObsNew)
# df[is.na(df)] <- 0

mycheck_problem_data(glbObsAll, featsExclude = glbFeatsExclude, 
                     fctrMaxUniqVals = glbFctrMaxUniqVals)
# glbObsAll <- na.omit(glbObsAll)

featsExclude <- union(glbFeatsExclude, c(glb_rsp_var_raw, glb_rsp_var))
if (glb_impute_na_data && 
    (length(myfind_numerics_missing(glbObsAll, featsExclude = featsExclude)) > 0)) {
    
    # Factors are useful in imputation 
    #   (only delete chars, glb_rsp_var_raw, glb_rsp_var & numerics in FeatsExclude)
    feedImp <- setdiff(names(glbObsAll), myfind_chr_cols_df(glbObsAll))
    feedImp <- setdiff(feedImp, union(glb_rsp_var, glb_rsp_var_raw))
    #feedImp <- setdiff(feedImp, grep("\\.[POSIX|zoo]", feedImp, value = TRUE))
    feedImp <- setdiff(feedImp, 
                       intersect(glbFeatsExclude, 
                                 setdiff(names(glbObsAll), myfind_chr_cols_df(glbObsAll))))
    feedImp <- setdiff(feedImp, ".rnorm")
    nonna_df <- myimputeMissingData(glbObsAll[, feedImp], miceSeed = glb_mice_complete.seed)
#colIx=6
#cbind(nonna_df[is.na(glbObsAll[, names(nonna_df[colIx]), FALSE]), names(nonna_df[colIx])], glbObsAll[is.na(glbObsAll[, names(nonna_df[colIx])]), c(glb_dsp_cols, names(nonna_df[colIx]))]); 
#summary(glbObsAll[, names(nonna_df[colIx])]); summary(nonna_df[, names(nonna_df[colIx])]) 

#stop(here"); glb2Sav(); all.equal(glbObsAll, savObsAll); glbObsAll <- savObsAll    
    for (col in names(nonna_df)) {
        glbObsAll[, paste0(col, ".nonNA")] <- nonna_df[, col]
        glbFeatsExclude <- c(glbFeatsExclude, col)        
    }
}    
    
mycheck_problem_data(glbObsAll, featsExclude = glbFeatsExclude, 
                     fctrMaxUniqVals = glbFctrMaxUniqVals, terminate = TRUE)

glb_chunks_df <- myadd_chunk(glb_chunks_df, "cluster.data", major.inc = TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r cluster.data, cache=FALSE, echo=FALSE}
mycompute_entropy_df <- function(obs_df, entropy_var, by_var=NULL) {   
    require(lazyeval)
    require(dplyr)
    require(tidyr)

    if (is.null(by_var)) {
        by_var <- ".default"
        obs_df$.default <- as.factor(".default") 
    }
    
    if (!any(grepl(".clusterid", names(obs_df), fixed=TRUE)))
        obs_df$.clusterid <- 1
        
    cluster_df <- obs_df %>%
            count_(c(by_var, ".clusterid", entropy_var)) %>%
            dplyr::filter(n > 0) %>%
            dplyr::filter_(interp(~(!is.na(var)), var=as.name(entropy_var))) %>%
            unite_(paste0(by_var, ".clusterid"),
                   c(interp(by_var), ".clusterid")) %>%
            spread_(interp(entropy_var), "n", fill=0) 

#     head(cluster_df)
#     sum(cluster_df$n)
    tmp.entropy <- sapply(1:nrow(cluster_df),
            function(row) entropy(as.numeric(cluster_df[row, -1]), method = "ML"))
    tmp.knt <- sapply(1:nrow(cluster_df),
                    function(row) sum(as.numeric(cluster_df[row, -1])))
    cluster_df$.entropy <- tmp.entropy; cluster_df$.knt <- tmp.knt
    #print(cluster_df)
    return(cluster_df)
}
    
if (glb_cluster) {
    require(proxy)
    #require(hash)
    require(dynamicTreeCut)
    require(entropy)
    require(tidyr)
    require(ggdendro)

    mywgtdcosine_dist <- function(x, y=NULL, weights=NULL) {
        if (!inherits(x, "matrix"))
            x <- as.matrix(x)
    
        if (is.null(weights))
            weights <- rep(1, ncol(x))
    
        wgtsx <- matrix(rep(weights / sum(weights), nrow(x)), nrow = nrow(x),
                        byrow = TRUE)
        wgtdx <- x * wgtsx
    
        wgtdxsqsum <- as.matrix(rowSums((x ^ 2) * wgtsx), byrow=FALSE)
        denom <- sqrt(wgtdxsqsum %*% t(wgtdxsqsum))
    
        ret_mtrx <- 1 - ((sum(weights) ^ 1) * (wgtdx %*% t(wgtdx)) / denom)
        ret_mtrx[is.nan(ret_mtrx)] <- 1
        diag(ret_mtrx) <- 0
        return(ret_mtrx)
    }
    #pr_DB$delete_entry("mywgtdcosine"); 
    # Need to do this only once across runs ?
    if (!pr_DB$entry_exists("mywgtdcosine")) {
        pr_DB$set_entry(FUN = mywgtdcosine_dist, names = c("mywgtdcosine"))
        pr_DB$modify_entry(names="mywgtdcosine", type="metric", loop=FALSE)
    }
    #pr_DB$get_entry("mywgtdcosine")

#     glb_hash <- hash(key=unique(glbObsAll$myCategory), 
#                      values=1:length(unique(glbObsAll$myCategory)))
#     glb_hash_lst <- hash(key=unique(glbObsAll$myCategory), 
#                      values=1:length(unique(glbObsAll$myCategory)))
#stop(here"); glb2Sav(); glbObsAll <- savObsAll
    cluster_vars <- grep(paste0("[", 
                        toupper(paste0(substr(glbFeatsText, 1, 1), collapse = "")),
                                      "]\\.[PT]\\."), 
                               names(glbObsAll), value = TRUE)
    # Assign correlations with rsp_var as weights for cosine distance
    print("Clustering features: ")
    cluster_vars_df <- data.frame(abs.cor.y = abs(cor(
                        glbObsAll[glbObsAll$.src == "Train", cluster_vars],
            as.numeric(glbObsAll[glbObsAll$.src == "Train", glb_rsp_var]),
                                    use = "pairwise.complete.obs")))
    print(tail(cluster_vars_df <- orderBy(~ abs.cor.y, 
                                    subset(cluster_vars_df, !is.na(abs.cor.y))), 5))
    print(sprintf("    .rnorm cor: %0.4f",
        cor(glbObsAll[glbObsAll$.src == "Train", ".rnorm"], 
            as.numeric(glbObsAll[glbObsAll$.src == "Train", glb_rsp_var]),
            use = "pairwise.complete.obs")))
    
    print(sprintf("glbObsAll Entropy: %0.4f", 
        allobs_ent <- entropy(table(glbObsAll[, glb_cluster_entropy_var]),
                              method="ML")))
    
    print(category_df <- mycompute_entropy_df(obs_df=glbObsAll,
                                             entropy_var=glb_cluster_entropy_var,
                                             by_var=glbFeatsCategory))
    print(sprintf("glbObsAll$%s Entropy: %0.4f (%0.4f pct)",
                    glbFeatsCategory,
            category_ent <- weighted.mean(category_df$.entropy, category_df$.knt),
                    100 * category_ent / allobs_ent))

    glbObsAll$.clusterid <- 1    
    #print(max(table(glbObsAll$myCategory.fctr) / 20))
        
#stop(here"); glb2Sav()    
    grp_ids <- sort(unique(glbObsAll[, glbFeatsCategory]))
    glb_cluster_size_df_lst <- list()
    png(paste0(glb_out_pfx, "FeatsTxtClusters.png"), 
        width = 480 * 2, height = 480 * length(grp_ids))
    grid.newpage()
    pushViewport(viewport(layout = grid.layout(nrow = length(grp_ids), ncol = 2)))
    pltIx <- 1
    for (grp in grp_ids) {
# if (grep(grp, levels(grp_ids)) <= 6) next                
# if (grep(grp, levels(grp_ids)) > 9) next        
# if (grep(grp, levels(grp_ids)) != 10) next        
        print(sprintf("Category: %s", grp))
        ctgry_allobs_df <- glbObsAll[glbObsAll[, glbFeatsCategory] == grp, ]
        if (!inherits(ctgry_allobs_df[, glb_cluster_entropy_var], "factor"))
            ctgry_allobs_df[, glb_cluster_entropy_var] <- 
                as.factor(ctgry_allobs_df[, glb_cluster_entropy_var])
        
        #dstns_dist <- proxy::dist(ctgry_allobs_df[, cluster_vars], method = "cosine")
        dstns_dist <- proxy::dist(ctgry_allobs_df[, row.names(cluster_vars_df)], 
                                  method = "mywgtdcosine",
                                  weights = cluster_vars_df$abs.cor.y)
        # Custom distance functions return a crossdist object
        #dstns_mtrx <- as.matrix(dstns_dist)
        dstns_mtrx <- matrix(as.vector(dstns_dist), nrow=attr(dstns_dist, "dim")[1],
                             dimnames=attr(dstns_dist, "dimnames"))
        dstns_dist <- as.dist(dstns_mtrx)

        print(sprintf("max distance(%0.4f) pair:", max(dstns_mtrx)))
#         print(dim(dstns_mtrx))        
#         print(sprintf("which.max: %d", which.max(dstns_mtrx)))
        row_ix <- ceiling(which.max(dstns_mtrx) / ncol(dstns_mtrx))
        col_ix <- which.max(dstns_mtrx[row_ix, ])
#         print(sprintf("row_ix: %d", row_ix)); print(sprintf("col_ix: %d", col_ix));
#         print(dim(ctgry_allobs_df))
        print(ctgry_allobs_df[c(row_ix, col_ix), 
            c(glbFeatsId, glb_cluster_entropy_var, glbFeatsCategory, glbFeatsText, cluster_vars)])
    
        min_dstns_mtrx <- dstns_mtrx
        diag(min_dstns_mtrx) <- 1
        # Float representations issue -2.22e-16 vs. 0.0000
        print(sprintf("min distance(%0.4f) pair:", min(min_dstns_mtrx)))
        row_ix <- ceiling(which.min(min_dstns_mtrx) / ncol(min_dstns_mtrx))
        col_ix <- which.min(min_dstns_mtrx[row_ix, ])
        print(ctgry_allobs_df[c(row_ix, col_ix), 
            c(glbFeatsId, glb_cluster_entropy_var, glbFeatsCategory, glbFeatsText,
              cluster_vars)])
    
        set.seed(glb_cluster.seed)
        clusters <- hclust(dstns_dist, method = "ward.D2")
        # Workaround to avoid "Error in cutree(dendro, h = heightcutoff) : the 'height' component of 'tree' is not sorted (increasingly)"
        if (with(clusters,all.equal(height,sort(height))))
            clusters$height <- round(clusters$height,6)
        
        clusters$labels <- ctgry_allobs_df[, glbFeatsId]
        clustersDD <- dendro_data(clusters)
        clustersDD$labels[, glb_rsp_var] <- sapply(clustersDD$labels$label, function(id)
            ctgry_allobs_df[id == ctgry_allobs_df[, glbFeatsId], glb_rsp_var])
        print(ggdendrogram(clustersDD, rotate = TRUE, size = 2) + 
                geom_point(data = clustersDD$labels, 
            aes_string(x = "x", color = glb_rsp_var), y = min(clustersDD$segments$y)) + 
                coord_flip(ylim = c(min(clustersDD$segments$y),
                                         max(clustersDD$segments$y))) + 
                ggtitle(grp),
            vp = viewport(layout.pos.row = pltIx, layout.pos.col = 1))  
        
#         clusters$labels <- ctgry_allobs_df[, glbFeatsId]
#         clustersDD <- dendro_data(clusters)
#         clustersDD$labels$color <- sapply(clustersDD$labels$label, function(id)
#             ctgry_allobs_df[id == ctgry_allobs_df[, glbFeatsId], glb_rsp_var])
#         print(ggdendrogram(clustersDD, rotate = TRUE, size = 2) + 
#                 geom_point(data = clustersDD$labels, 
#                 aes_string(x = "x", color = "color"), y = min(clustersDD$segments$y)) + 
#                 coord_flip(ylim = c(min(clustersDD$segments$y),
#                                          max(clustersDD$segments$y))))
#         print(ggdendrogram(clustersDD, rotate = TRUE, size = 2) + 
#                 geom_point(data = clustersDD$labels, 
#                           aes_string(x = "x", y = "y", color = "color")))
#         myplclust(clusters, lab=ctgry_allobs_df[, glbFeatsId], 
#                   lab.col=unclass(ctgry_allobs_df[, glb_cluster_entropy_var]))

        opt_minclustersize_df <- data.frame(minclustersize = nrow(ctgry_allobs_df), 
            entropy = entropy(table(ctgry_allobs_df[, glb_cluster_entropy_var]),
                              method = "ML"))
        for (minclustersize in 
             as.integer(seq(nrow(ctgry_allobs_df) / 2, nrow(ctgry_allobs_df) / 10, 
                            length = 5))) {
            clusterGroups <- cutreeDynamic(clusters, minClusterSize = minclustersize,
                                           method = "tree", deepSplit = 0)
            # Unassigned groups are labeled 0; the largest group has label 1
            clusterGroups[clusterGroups == 0] <- 1
            ctgry_allobs_df$.clusterid <- clusterGroups
            ctgry_clstrs_df <- mycompute_entropy_df(ctgry_allobs_df,
                                                    glb_cluster_entropy_var)
            opt_minclustersize_df <- rbind(opt_minclustersize_df, 
                                           data.frame(minclustersize = minclustersize,
                entropy = weighted.mean(ctgry_clstrs_df$.entropy, ctgry_clstrs_df$.knt)))
        }
        opt_minclustersize <-
            opt_minclustersize_df$minclustersize[which.min(opt_minclustersize_df$entropy)]
        opt_minclustersize_df$.color <- 
            ifelse(opt_minclustersize_df$minclustersize == opt_minclustersize,
                   "red", "blue")
        print(ggplot(data = opt_minclustersize_df, 
                     mapping = aes(x = minclustersize, y = entropy)) + 
                geom_point(aes(color = .color)) + scale_color_identity() + 
                guides(color = "none") + geom_line(),
            vp = viewport(layout.pos.row = pltIx, layout.pos.col = 2))
        glb_cluster_size_df_lst[[grp]] <- opt_minclustersize_df
        
        # select minclustersize that minimizes entropy
        clusterGroups <- cutreeDynamic(clusters, minClusterSize = opt_minclustersize,
                                       method = "tree",
                                       deepSplit = 0)
        # Unassigned groups are labeled 0; the largest group has label 1
        table(clusterGroups, ctgry_allobs_df[, glb_cluster_entropy_var], 
              useNA = "ifany")   
        clusterGroups[clusterGroups == 0] <- 1
        table(clusterGroups, ctgry_allobs_df[, glb_cluster_entropy_var], useNA = "ifany") 
        glbObsAll[glbObsAll[, glbFeatsCategory] == grp,]$.clusterid <-
            clusterGroups
        
        pltIx <- pltIx + 1
    }
    dev.off()
    #all.equal(savObsAll_clusterid, glbObsAll$.clusterid)
    
    print(cluster_df <- mycompute_entropy_df(obs_df=glbObsAll,
                                             entropy_var=glb_cluster_entropy_var,
                                             by_var=glbFeatsCategory))
    print(sprintf("glbObsAll$%s$.clusterid Entropy: %0.4f (%0.4f pct)",
                    glbFeatsCategory,
                cluster_ent <- weighted.mean(cluster_df$.entropy, cluster_df$.knt),
                    100 * cluster_ent / category_ent))
    
    glbObsAll$.clusterid.fctr <- as.factor(glbObsAll$.clusterid)
    # .clusterid.fctr is created automatically (probably ?) later
    glbFeatsExclude <- c(glbFeatsExclude, ".clusterid")
    if (!is.null(glbFeatsCategory))
#         glbFeatsInteractionOnly[ifelse(grepl("\\.fctr", glbFeatsCategory),
#                                             glbFeatsCategory, 
#                                             paste0(glbFeatsCategory, ".fctr"))] <-
#             c(".clusterid.fctr")
        glbFeatsInteractionOnly[[".clusterid.fctr"]] <-
            ifelse(grepl("\\.fctr", glbFeatsCategory), glbFeatsCategory, 
                                                        paste0(glbFeatsCategory, ".fctr"))
            
    if (glbFeatsTextClusterVarsExclude)
        glbFeatsExclude <- c(glbFeatsExclude, cluster_vars)
}

# Last call for data modifications 
#stop(here") # savObsAll <- glbObsAll
# glbObsAll[(glbObsAll$PropR == 0.75) & (glbObsAll$State == "Hawaii"), "PropR.fctr"] <- "N"

# Re-partition
glbObsTrn <- subset(glbObsAll, .src == "Train")
glbObsNew <- subset(glbObsAll, .src == "Test")

glb_chunks_df <- myadd_chunk(glb_chunks_df, "partition.data.training", major.inc = TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r partition.data.training, cache=FALSE, echo=FALSE}
if (all(is.na(glbObsNew[, glb_rsp_var]))) {
    set.seed(glbObsTrnPartitionSeed)
    OOB_size <- nrow(glbObsNew) * 1.1
    
    if (is.null(glbFeatsCategory)) {
        require(caTools)
        split <- sample.split(glbObsTrn[, glb_rsp_var_raw], 
                              SplitRatio=OOB_size / nrow(glbObsTrn))
        glbObsOOB <- glbObsTrn[split ,]            
        glbObsFit <- glbObsTrn[!split, ] 
    } else {
        sample_vars <- c(glbFeatsCategory, glb_rsp_var_raw)
        rspvar_freq_df <- orderBy(reformulate(glb_rsp_var_raw), 
                                 mycreate_sqlxtab_df(glbObsTrn, glb_rsp_var_raw))
        OOB_rspvar_size <- 
            1.0 * OOB_size * rspvar_freq_df$.n / sum(rspvar_freq_df$.n) 
        names(OOB_rspvar_size) <- as.character(rspvar_freq_df[, glb_rsp_var_raw])
        newobs_freq_df <- orderBy(reformulate(glbFeatsCategory),
                                mycreate_sqlxtab_df(glbObsNew, glbFeatsCategory))
        trnobs_freq_df <- orderBy(reformulate(glbFeatsCategory),
                                mycreate_sqlxtab_df(glbObsTrn, glbFeatsCategory))
        ctgry_freq_df <- merge(newobs_freq_df, trnobs_freq_df, 
                                by = glbFeatsCategory,
                                all = TRUE, sort = TRUE, 
                                suffixes = c(".tst", ".trn"))
        ctgry_freq_df[is.na(ctgry_freq_df)] <- 0
        
        obs_freq_df <- mycreate_xtab_df(glbObsTrn, 
                                        c(glbFeatsCategory, glb_rsp_var_raw))
        newobs_freq_df <- orderBy(reformulate(glbFeatsCategory),
                                mycreate_sqlxtab_df(glbObsNew, glbFeatsCategory))
        names(newobs_freq_df) <- gsub(".n", ".n.tst", names(newobs_freq_df), 
                                      fixed = TRUE)
        obs_freq_df <- merge(obs_freq_df, newobs_freq_df, all = TRUE)
        strata_mtrx <- ceiling(
            matrix(obs_freq_df$.n.tst * 1.0 / sum(obs_freq_df$.n.tst)) %*%
                      matrix(OOB_rspvar_size, nrow = 1))
        dimnames(strata_mtrx)[[1]] <- obs_freq_df[, glbFeatsCategory]
        dimnames(strata_mtrx)[[2]] <- 
            as.character(rspvar_freq_df[, glb_rsp_var_raw])
        for (val in rspvar_freq_df[, glb_rsp_var_raw]) {
            trn <- paste0(glb_rsp_var_raw, ".", as.character(val))
            strata <- paste0(".strata.", as.character(val))            
            obs_freq_df[, strata] <- strata_mtrx[, as.character(val)]
            
            if (length(ix <- which(is.na(obs_freq_df[, trn]))) > 0) {
                # NA obs in a particular category 
                print("Prediction Hints by Catgeory:")
                print(obs_freq_df[ix, ])
                obs_freq_df[ix, strata] <- NA
            }

            if (length((ix <- which(obs_freq_df[, trn] < 
                                    2.0 * obs_freq_df[, strata]))) > 0)
                # More obs in OOB compared to fit currently
                obs_freq_df[ix, strata] <- floor(obs_freq_df[ix, trn] / 2.0)
        
            if (length((ix <- which(obs_freq_df[, strata] == 0))) > 0)
                obs_freq_df[ix, strata] <- 1
        }    
        #print(colSums(obs_freq_df[, -1]))
        #glbObsFrq <- obs_freq_df
    
        
        OOB_strata_size <- as.vector(as.matrix(obs_freq_df[, 
                                    grepl("^\\.strata\\.", names(obs_freq_df))]))
        tmp_trnobs_df <- orderBy(reformulate(c(glb_rsp_var_raw, glbFeatsCategory)),
                                glbObsTrn)
        require(sampling)
        split_strata <- sampling::strata(tmp_trnobs_df, 
                               stratanames = c(glb_rsp_var_raw, glbFeatsCategory),
                               size = OOB_strata_size[!is.na(OOB_strata_size)],
                               method = "srswor")
        glbObsOOB <- getdata(tmp_trnobs_df, split_strata)[, names(glbObsTrn)]
        glbObsFit <- glbObsTrn[!glbObsTrn[, glbFeatsId] %in% 
                                        glbObsOOB[, glbFeatsId], ]
    }
} else {
    print(sprintf("Newdata contains non-NA data for %s; setting OOB to Newdata", 
                  glb_rsp_var))
    glbObsFit <- glbObsTrn; glbObsOOB <- glbObsNew
}

if (!is.null(glb_max_fitobs) && (nrow(glbObsFit) > glb_max_fitobs)) {
    warning("glbObsFit restricted to glb_max_fitobs: ", 
            format(glb_max_fitobs, big.mark = ","))
    org_fitobs_df <- glbObsFit
    glbObsFit <- 
        org_fitobs_df[split <- sample.split(org_fitobs_df[, glb_rsp_var_raw], 
                                            SplitRatio = glb_max_fitobs), ]
    org_fitobs_df <- NULL
}

glbObsAll$.lcn <- ""; glbObsTrn$.lcn <- "";
glbObsAll[glbObsAll[, glbFeatsId] %in% 
              glbObsFit[, glbFeatsId], ".lcn"] <- "Fit"
glbObsTrn[glbObsTrn[, glbFeatsId] %in% 
              glbObsFit[, glbFeatsId], ".lcn"] <- "Fit"
glbObsAll[glbObsAll[, glbFeatsId] %in% 
              glbObsOOB[, glbFeatsId], ".lcn"] <- "OOB"
glbObsTrn[glbObsTrn[, glbFeatsId] %in% 
              glbObsOOB[, glbFeatsId], ".lcn"] <- "OOB"

dsp_class_dstrb <- function(obs_df, location_var, partition_var) {
    xtab_df <- mycreate_xtab_df(obs_df, c(location_var, partition_var))
    rownames(xtab_df) <- xtab_df[, location_var]
    xtab_df <- xtab_df[, -grepl(location_var, names(xtab_df))]
    print(xtab_df)
    print(xtab_df / rowSums(xtab_df, na.rm=TRUE))    
}    

# Ensure proper splits by glb_rsp_var_raw & user-specified feature for OOB vs. new
if (!is.null(glbFeatsCategory)) {
    if (glb_is_classification)
        dsp_class_dstrb(glbObsAll, ".lcn", glb_rsp_var_raw)
    newobs_ctgry_df <- mycreate_sqlxtab_df(subset(glbObsAll, .src == "Test"), 
                                           glbFeatsCategory)
    names(newobs_ctgry_df) <- 
        gsub(".n", ".n.Tst", names(newobs_ctgry_df), fixed = TRUE)
    OOBobs_ctgry_df <- mycreate_sqlxtab_df(subset(glbObsAll, .lcn == "OOB"), 
                                           glbFeatsCategory)
    names(OOBobs_ctgry_df) <- 
        gsub(".n", ".n.OOB", names(OOBobs_ctgry_df), fixed = TRUE)
    fitobs_ctgry_df <- mycreate_sqlxtab_df(subset(glbObsAll, .lcn == "Fit"), 
                                           glbFeatsCategory)
    names(fitobs_ctgry_df) <- 
        gsub(".n", ".n.Fit", names(fitobs_ctgry_df), fixed = TRUE)
    
    if (nrow(subset(glbObsAll, .lcn == "Fit")) == 0) {
        # Fit same as OOB
        OOBobs_ctgry_df$.n.OOB <- OOBobs_ctgry_df$.n.OOB / 2;
        fitobs_ctgry_df <- OOBobs_ctgry_df
        names(fitobs_ctgry_df) <- 
            gsub(".n.OOB", ".n.Fit", names(fitobs_ctgry_df), fixed = TRUE)
    } else {
        fitobs_ctgry_df <- mycreate_sqlxtab_df(subset(glbObsAll, .lcn == "Fit"), 
                                               glbFeatsCategory)
        names(fitobs_ctgry_df) <- 
            gsub(".n", ".n.Fit", names(fitobs_ctgry_df), fixed = TRUE)
    }
    
    glbLvlCategory <- merge(fitobs_ctgry_df, OOBobs_ctgry_df, by = glbFeatsCategory
                          , all = TRUE)
    glbLvlCategory <- merge(glbLvlCategory, newobs_ctgry_df, by = glbFeatsCategory
                          , all = TRUE)
    glbLvlCategory$.freqRatio.Fit <- 
        glbLvlCategory$.n.Fit / sum(glbLvlCategory$.n.Fit, na.rm = TRUE)
    glbLvlCategory$.freqRatio.OOB <- 
        glbLvlCategory$.n.OOB / sum(glbLvlCategory$.n.OOB, na.rm = TRUE)
    glbLvlCategory$.freqRatio.Tst <- 
        glbLvlCategory$.n.Tst / sum(glbLvlCategory$.n.Tst, na.rm = TRUE)
    print(orderBy(~-.freqRatio.Tst-.freqRatio.OOB-.freqRatio.Fit, glbLvlCategory))
}

print("glbObsAll: "); print(dim(glbObsAll))
print("glbObsTrn: "); print(dim(glbObsTrn))
print("glbObsFit: "); print(dim(glbObsFit))
print("glbObsOOB: "); print(dim(glbObsOOB))
print("glbObsNew: "); print(dim(glbObsNew))
# # Does not handle NULL or length(glbFeatsId) > 1

if (glb_save_envir)
    save(glbObsAll, #glbObsTrn, glbObsFit, glbObsOOB, glbObsNew,
         file=paste0(glb_out_pfx, "blddfs_dsk.RData"))
# load(paste0(glb_out_pfx, "blddfs_dsk.RData"))

rm(split)

glb_chunks_df <- myadd_chunk(glb_chunks_df, "select.features", major.inc = TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r select.features, cache=FALSE, echo=FALSE}
#stop(here"); glb2Sav(); glbObsAll <- savObsAll
glb_feats_df <- myselect_features(entity_df = glbObsTrn, 
                                  exclude_vars_as_features = glbFeatsExclude, 
                                  rsp_var = glb_rsp_var)

glb_feats_df <- orderBy(~-cor.y, 
          myfind_cor_features(feats_df=glb_feats_df, obs_df=glbObsTrn, rsp_var=glb_rsp_var,
                              nzv.freqCut = glbFeatsNzvFreqMax, 
                              nzv.uniqueCut = glbFeatsNzvUniqMin))
print(subset(glb_feats_df, select = -id))
                              
plt_feats_df <- glb_feats_df
print(myplot_scatter(plt_feats_df, "percentUnique", "freqRatio", 
                     colorcol_name="nzv", jitter=TRUE) + 
          #geom_point(aes(shape=nzv)) +           
          geom_point() + 
          xlim(-5, 25) + 
          geom_hline(yintercept=glbFeatsNzvFreqMax) +
          geom_vline(xintercept=glbFeatsNzvUniqMin))
print(subset(glb_feats_df, nzv, select = -id))

tmp_allobs_df <- 
    glbObsAll[, union(setdiff(names(glbObsAll), subset(glb_feats_df, nzv)$id),
                      c(glbFeatsCategory, glb_cluster_entropy_var))]
glbObsTrn <- subset(tmp_allobs_df, .src == "Train")
glbObsNew <- subset(tmp_allobs_df, .src == "Test")

glb_feats_df$interaction.feat <- NA
for (feat in names(glbFeatsInteractionOnly))
    glb_feats_df[glb_feats_df$id %in% feat, "interaction.feat"] <-
        glbFeatsInteractionOnly[[feat]]
        
#stop(here"); glb2Sav(); glbObsAll <- savObsAll
indep_vars <- subset(glb_feats_df, !nzv & (exclude.as.feat != 1))[, "id"]
numeric_indep_vars <- indep_vars[!grepl(".fctr", indep_vars, fixed = TRUE)]
glb_feats_df$shapiro.test.p.value <- NA
glb_feats_df[glb_feats_df$id %in% numeric_indep_vars, "shapiro.test.p.value"] <- 
    sapply(numeric_indep_vars, function(feat) {
        # shapiro.test(glbObsTrn[, var][1:min(5000, nrow(glbObsTrn))])$p.value)
        if (nrow(glbObsTrn) <= 5000) smplVals <- glbObsTrn[, feat] else
            smplVals <- glbObsTrn[, feat][sample.int(nrow(glbObsTrn), 5000)]
        return(shapiro.test(smplVals)$p.value) })

not_nrml_feats_df <- glb_feats_df %>%
                        subset(!is.na(shapiro.test.p.value)) %>%
                    subset((shapiro.test.p.value < 0.05) || (id == ".rnorm")) %>%
                        arrange(shapiro.test.p.value)
row.names(not_nrml_feats_df) <- not_nrml_feats_df$id

#pltTrnObs <- glbObsTrn[, c("D.npnct05.log", ".rnorm")]
pltTrnObs <- glbObsTrn[, c(union(".rnorm",
                        not_nrml_feats_df$id[1:min(5, nrow(not_nrml_feats_df))]),
                            glb_cluster_entropy_var), FALSE]
if (is.null(glb_cluster_entropy_var))
    gp <- myplot_violin(pltTrnObs, 
                    setdiff(names(pltTrnObs), glb_cluster_entropy_var)) else
    gp <- myplot_violin(pltTrnObs, 
                    setdiff(names(pltTrnObs), glb_cluster_entropy_var), 
                    xcol_name = glb_cluster_entropy_var)
if ("variable" %in% names(pltTrnObs))
    gp <- gp + facet_wrap(~variable, scales = "free")
print(gp)
#myplot_histogram(pltTrnObs, "D.npnct11.log", fill_col_name="sold", show_stats = TRUE)

myadjust_interaction_feats <- function(vars_vctr) {
    for (feat in subset(glb_feats_df, !is.na(interaction.feat))$id)
        if (feat %in% vars_vctr)
            vars_vctr <- union(setdiff(vars_vctr, feat), 
                paste0(glb_feats_df[glb_feats_df$id == feat, "interaction.feat"], ":",
                       feat))
    return(vars_vctr)
}

# Need to create an option for allowParallel ???
myrun_rfe <- function(obs_df, indep_vars, sizes = NULL) {
    rfe_obs_df <- myget_vectorized_obs_df(obs_df, glb_rsp_var, indep_vars)
    predictors_vctr <- setdiff(names(rfe_obs_df), glb_rsp_var)
    
    if (glb_is_regression)  rfeFuncs <- lmFuncs else {    
        rfeFuncs <- ldaFuncs
        
        # Delete non-variant columns
        predictors_unqLen <- sapply(predictors_vctr, function(col)
                                                length(unique(rfe_obs_df[, col])))
        predictors_vctr <- predictors_vctr[predictors_unqLen > 1]
        # Delete freqRatio >= 291
        #   plagiarized from caret:::nzv
        predictors_freqRatio <- 
            apply(rfe_obs_df[, predictors_vctr, FALSE], 2, function(data) {
                t <- table(data[!is.na(data)])
                if (length(t) <= 1) {
                    return(0)
                }
                w <- which.max(t)
                return(max(t, na.rm = TRUE)/max(t[-w], na.rm = TRUE))
            })
        predictors_vctr <- predictors_vctr[predictors_freqRatio < 172]
    }                        
    
    if (is.null(sizes))
        sizes <- tail(2 ^ (1:as.integer(log2(length(predictors_vctr)))), 5)
    
    rfe_control <- rfeControl(functions = rfeFuncs, method = "repeatedcv",
                             number = glb_rcv_n_folds, repeats = glb_rcv_n_repeats,
                              verbose = TRUE, returnResamp = "all",
        seeds = mygen_seeds(seeds_lst_len = 
                                (glb_rcv_n_folds * glb_rcv_n_repeats) + 1,
                            seeds_elmnt_lst_len = (length(sizes) + 1))
, allowParallel = FALSE
                            )
    set.seed(113)
    rfe_results <- rfe(rfe_obs_df[, predictors_vctr, FALSE], 
                       rfe_obs_df[, glb_rsp_var],
                       sizes = sizes, 
                       # metric = unlist(strsplit(glbMdlMetricsEval, "[.]"))[2],
#         maximize = ifelse(unlist(strsplit(glbMdlMetricsEval, "[.]"))[1] == "max",
#                                        TRUE, FALSE),
                       rfeControl = rfe_control)
    print(rfe_results)
    print(predictors(rfe_results))
    # print(plot(rfe_results, type=c("g", "o")))
    # print(plot(rfe_results))
    print(ggplot(rfe_results))

    return(rfe_results)
}

#stop(here"); glb2Sav()
# shd .clusterid.fctr be excluded from this ? or include encoding of glbFeatsCategory:.clusterid.fctr ?
if (grepl("RFE\\.X", names(glbMdlFamilies))) {
    indep_vars <- myadjust_interaction_feats(subset(glb_feats_df, 
                                                    !nzv & (exclude.as.feat != 1))$id)
    rfe_fit_results <- myrun_rfe(obs_df = glbObsFit, indep_vars = indep_vars, 
                                 sizes = glbRFESizes[["RFE.X"]])
}

# print(all.equal(rfe_results[-which(names(rfe_results) == "times")], 
#                 sav_rfe_results[-which(names(sav_rfe_results) == "times")]))

# require(mRMRe)
# indep_vars_vctr <- subset(glb_feats_df, !nzv &
#                                         (exclude.as.feat != 1))[, "id"]
# indep_vars_vctr <- setdiff(indep_vars_vctr, 
#                     myfind_fctr_cols_df(glbObsTrn[, c(glb_rsp_var, indep_vars_vctr)]))
# tmp_trnobs_df <- glbObsTrn[, c(glb_rsp_var, indep_vars_vctr)]
# tmp_trnobs_df$biddable <- as.numeric(tmp_trnobs_df$biddable)
# dd <- mRMR.data(data = tmp_trnobs_df)
# mRMRe.fltr <- mRMR.classic(data = dd, target_indices = c(1), feature_count = 10)
# print(solutions(mRMRe.fltr)[[1]])
# print(apply(solutions(mRMRe.fltr)[[1]], 2, function(x, y) { return(y[x]) },
#             y=featureNames(dd)))
# print(featureNames(dd)[solutions(mRMRe.fltr)[[1]]])
# print(mRMRe.fltr@filters); print(mRMRe.fltr@scores)

mycheck_problem_data(glbObsAll, featsExclude = glbFeatsExclude, 
                     fctrMaxUniqVals = glbFctrMaxUniqVals, terminate = TRUE)
# glbObsAll %>% filter(is.na(Married.fctr)) %>% tbl_df()
# glbObsAll %>% count(Married.fctr)
# levels(glbObsAll$Married.fctr)

print("glb_feats_df:");   print(dim(glb_feats_df))
sav_feats_df <- glb_feats_df
glb_feats_df <- sav_feats_df

glb_feats_df[, "rsp_var_raw"] <- FALSE
glb_feats_df[glb_feats_df$id == glb_rsp_var_raw, "rsp_var_raw"] <- TRUE 
glb_feats_df$exclude.as.feat <- (glb_feats_df$exclude.as.feat == 1)
if (!is.null(glbFeatsId) && glbFeatsId != ".rownames")
    glb_feats_df[glb_feats_df$id %in% glbFeatsId, "id_var"] <- TRUE 
add_feats_df <- data.frame(id=glb_rsp_var, exclude.as.feat=TRUE, rsp_var=TRUE)
row.names(add_feats_df) <- add_feats_df$id; print(add_feats_df)
glb_feats_df <- myrbind_df(glb_feats_df, add_feats_df)
if (glbFeatsId != ".rownames")
    print(subset(glb_feats_df, rsp_var_raw | rsp_var | id_var)) else
    print(subset(glb_feats_df, rsp_var_raw | rsp_var))    

print("glb_feats_df vs. glbObsAll: "); 
print(setdiff(glb_feats_df$id, names(glbObsAll)))
print("glbObsAll vs. glb_feats_df: "); 
# Ensure these are only chr vars
print(setdiff(setdiff(names(glbObsAll), glb_feats_df$id), 
                myfind_chr_cols_df(glbObsAll)))

if (glb_save_envir)
    save(glb_feats_df, 
         glbObsAll, #glbObsTrn, glbObsFit, glbObsOOB, glbObsNew,
         file=paste0(glb_out_pfx, "selfts_dsk.RData"))
# load(paste0(glb_out_pfx, "blddfs_dsk.RData"))

# if (!all.equal(tmp_feats_df, glb_feats_df))
#     stop("glb_feats_df r/w not working")
# if (!all.equal(tmp_entity_df, glbObsAll))
#     stop("glbObsAll r/w not working")

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc=TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r fit.models_0, cache=FALSE}
fit.models_0_chunk_df <- myadd_chunk(NULL, "fit.models_0_bgn", label.minor = "setup")
# load(paste0(glb_out_pfx, "dsk.RData"))

get_model_sel_frmla <- function() {
    model_evl_terms <- c(NULL)
    # min.aic.fit might not be avl
    lclMdlEvlCriteria <- 
        glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)]
    for (metric in lclMdlEvlCriteria)
        model_evl_terms <- c(model_evl_terms, 
                             ifelse(length(grep("max", metric)) > 0, "-", "+"), metric)
    if (glb_is_classification && glb_is_binomial)
        model_evl_terms <- c(model_evl_terms, "-", "opt.prob.threshold.OOB")
    model_sel_frmla <- as.formula(paste(c("~ ", model_evl_terms), collapse = " "))
    return(model_sel_frmla)
}

get_dsp_models_df <- function() {
    dsp_models_cols <- c("id", 
                    glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                    grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
    dsp_models_df <- 
        #orderBy(get_model_sel_frmla(), glb_models_df)[, c("id", glbMdlMetricsEval)]
        orderBy(get_model_sel_frmla(), glb_models_df)[, dsp_models_cols]    
    nCvMdl <- sapply(glb_models_lst, function(mdl) nrow(mdl$results))
    nParams <- sapply(glb_models_lst, function(mdl) ifelse(mdl$method == "custom", 0, 
        nrow(subset(modelLookup(mdl$method), parameter != "parameter"))))
    
#     nCvMdl <- nCvMdl[names(nCvMdl) != "avNNet"]
#     nParams <- nParams[names(nParams) != "avNNet"]    
    
    if (length(cvMdlProblems <- nCvMdl[nCvMdl <= nParams]) > 0) {
        print("Cross Validation issues:")
        warning("Cross Validation issues:")        
        print(cvMdlProblems)
    }
    
    pltMdls <- setdiff(names(nCvMdl), names(cvMdlProblems))
    pltMdls <- setdiff(pltMdls, names(nParams[nParams == 0]))
    
    # length(pltMdls) == 21
    png(paste0(glb_out_pfx, "bestTune.png"), width = 480 * 2, height = 480 * 4)
    grid.newpage()
    pushViewport(viewport(layout = grid.layout(ceiling(length(pltMdls) / 2.0), 2)))
    pltIx <- 1
    for (mdlId in pltMdls) {
        print(ggplot(glb_models_lst[[mdlId]], highBestTune = TRUE) + labs(title = mdlId),   
              vp = viewport(layout.pos.row = ceiling(pltIx / 2.0), 
                            layout.pos.col = ((pltIx - 1) %% 2) + 1))  
        pltIx <- pltIx + 1
    }
    dev.off()

    if (all(row.names(dsp_models_df) != dsp_models_df$id))
        row.names(dsp_models_df) <- dsp_models_df$id
    return(dsp_models_df)
}
#get_dsp_models_df()

if (glb_is_classification && glb_is_binomial && 
        (length(unique(glbObsFit[, glb_rsp_var])) < 2))
    stop("glbObsFit$", glb_rsp_var, ": contains less than 2 unique values: ",
         paste0(unique(glbObsFit[, glb_rsp_var]), collapse=", "))

max_cor_y_x_vars <- orderBy(~ -cor.y.abs, 
        subset(glb_feats_df, (exclude.as.feat == 0) & !nzv & !is.cor.y.abs.low & 
                                is.na(cor.high.X)))[1:2, "id"]
max_cor_y_x_vars <- max_cor_y_x_vars[!is.na(max_cor_y_x_vars)]

if (!is.null(glb_Baseline_mdl_var)) {
    if ((max_cor_y_x_vars[1] != glb_Baseline_mdl_var) & 
        (glb_feats_df[glb_feats_df$id == max_cor_y_x_vars[1], "cor.y.abs"] > 
         glb_feats_df[glb_feats_df$id == glb_Baseline_mdl_var, "cor.y.abs"]))
        stop(max_cor_y_x_vars[1], " has a higher correlation with ", glb_rsp_var, 
             " than the Baseline var: ", glb_Baseline_mdl_var)
}

glb_model_type <- ifelse(glb_is_regression, "regression", "classification")
    
# Model specs
c("id.prefix", "method", "type",
  # trainControl params
  "preProc.method", "cv.n.folds", "cv.n.repeats", "summary.fn",
  # train params
  "metric", "metric.maximize", "tune.df")

# Baseline
if (!is.null(glb_Baseline_mdl_var)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                            paste0("fit.models_0_", "Baseline"), major.inc = FALSE,
                                    label.minor = "mybaseln_classfr")
    ret_lst <- myfit_mdl(mdl_id="Baseline", 
                         model_method="mybaseln_classfr",
                        indep_vars_vctr=glb_Baseline_mdl_var,
                        rsp_var=glb_rsp_var,
                        fit_df=glbObsFit, OOB_df=glbObsOOB)
}    

# Most Frequent Outcome "MFO" model: mean(y) for regression
#   Not using caret's nullModel since model stats not avl
#   Cannot use rpart for multinomial classification since it predicts non-MFO
if (glb_is_classification) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                                paste0("fit.models_0_", "MFO"), major.inc = FALSE,
                                        label.minor = "myMFO_classfr")
    
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "MFO", type = glb_model_type, trainControl.method = "none",
        train.method = ifelse(glb_is_regression, "lm", "myMFO_classfr"))),
                            indep_vars = ".rnorm", rsp_var = glb_rsp_var,
                            fit_df = glbObsFit, OOB_df = glbObsOOB)
    
        # "random" model - only for classification; 
        #   none needed for regression since it is same as MFO
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                                paste0("fit.models_0_", "Random"), major.inc = FALSE,
                                        label.minor = "myrandom_classfr")

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)    
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Random", type = glb_model_type, trainControl.method = "none",
        train.method = "myrandom_classfr")),
                        indep_vars = ".rnorm", rsp_var = glb_rsp_var,
                        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

# Max.cor.Y
#   Check impact of cv
#       rpart is not a good candidate since caret does not optimize cp (only tuning parameter of rpart) well
fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                        paste0("fit.models_0_", "Max.cor.Y.rcv.*X*"), major.inc = FALSE,
                                    label.minor = "glmnet")

ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
    id.prefix="Max.cor.Y.rcv.1X1", type=glb_model_type, trainControl.method="none",
    train.method="glmnet")),
                    indep_vars=max_cor_y_x_vars, rsp_var=glb_rsp_var, 
                    fit_df=glbObsFit, OOB_df=glbObsOOB)

if (glbMdlCheckRcv) {
    # rcv_n_folds == 1 & rcv_n_repeats > 1 crashes
    for (rcv_n_folds in seq(3, glb_rcv_n_folds + 2, 2))
        for (rcv_n_repeats in seq(1, glb_rcv_n_repeats + 2, 2)) {
            
            # Experiment specific code to avoid caret crash
    #         lcl_tune_models_df <- rbind(data.frame()
    #                             ,data.frame(method = "glmnet", parameter = "alpha", 
    #                                         vals = "0.100 0.325 0.550 0.775 1.000")
    #                             ,data.frame(method = "glmnet", parameter = "lambda",
    #                                         vals = "9.342e-02")    
    #                                     )
            
            ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst =
                list(
                id.prefix = paste0("Max.cor.Y.rcv.", rcv_n_folds, "X", rcv_n_repeats), 
                type = glb_model_type, 
    # tune.df = lcl_tune_models_df,            
                trainControl.method = "repeatedcv",
                trainControl.number = rcv_n_folds, 
                trainControl.repeats = rcv_n_repeats,
                trainControl.classProbs = glb_is_classification,
                trainControl.summaryFunction = glbMdlMetricSummaryFn,
                train.method = "glmnet", train.metric = glbMdlMetricSummary, 
                train.maximize = glbMdlMetricMaximize)),
                                indep_vars = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                                fit_df = glbObsFit, OOB_df = glbObsOOB)
        }
    # Add parallel coordinates graph of glb_models_df[, glbMdlMetricsEval] to evaluate cv parameters
    tmp_models_cols <- c("id", "max.nTuningRuns",
                        glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                        grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
    print(myplot_parcoord(obs_df = subset(glb_models_df, 
                                          grepl("Max.cor.Y.rcv.", id, fixed = TRUE), 
                                            select = -feats)[, tmp_models_cols],
                          id_var = "id"))
}
        
# Useful for stacking decisions
# fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
#                     paste0("fit.models_0_", "Max.cor.Y[rcv.1X1.cp.0|]"), major.inc = FALSE,
#                                     label.minor = "rpart")
# 
# ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
#     id.prefix = "Max.cor.Y.rcv.1X1.cp.0", type = glb_model_type, trainControl.method = "none",
#     train.method = "rpart",
#     tune.df=data.frame(method="rpart", parameter="cp", min=0.0, max=0.0, by=0.1))),
#                     indep_vars=max_cor_y_x_vars, rsp_var=glb_rsp_var, 
#                     fit_df=glbObsFit, OOB_df=glbObsOOB)

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)
# if (glb_is_regression || glb_is_binomial) # For multinomials this model will be run next by default
ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                        id.prefix = "Max.cor.Y", 
                        type = glb_model_type, trainControl.method = "repeatedcv",
                        trainControl.number = glb_rcv_n_folds, 
                        trainControl.repeats = glb_rcv_n_repeats,
                        trainControl.classProbs = glb_is_classification,
                        trainControl.summaryFunction = glbMdlMetricSummaryFn,
                        trainControl.allowParallel = glbMdlAllowParallel,                        
                        train.metric = glbMdlMetricSummary, 
                        train.maximize = glbMdlMetricMaximize,    
                        train.method = "rpart")),
                    indep_vars = max_cor_y_x_vars, rsp_var = glb_rsp_var, 
                    fit_df = glbObsFit, OOB_df = glbObsOOB)

if ((length(glbFeatsDateTime) > 0) && 
    (sum(grepl(paste(names(glbFeatsDateTime), "\\.day\\.minutes\\.poly\\.", sep = ""),
               names(glbObsAll))) > 0)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Max.cor.Y.Time.Poly"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars, 
            grep(paste(names(glbFeatsDateTime), "\\.day\\.minutes\\.poly\\.", sep = ""),
                        names(glbObsAll), value = TRUE))
    indepVars <- myadjust_interaction_feats(indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = "Max.cor.Y.Time.Poly", 
            type = glb_model_type, trainControl.method = "repeatedcv",
            trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = "glmnet")),
        indep_vars = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

if ((length(glbFeatsDateTime) > 0) && 
    (sum(grepl(paste(names(glbFeatsDateTime), "\\.last[[:digit:]]", sep = ""),
               names(glbObsAll))) > 0)) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Max.cor.Y.Time.Lag"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars, 
            grep(paste(names(glbFeatsDateTime), "\\.last[[:digit:]]", sep = ""),
                        names(glbObsAll), value = TRUE))
    indepVars <- myadjust_interaction_feats(indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Time.Lag", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indep_vars = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

if (length(glbFeatsText) > 0) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Txt.*"), major.inc = FALSE,
                                    label.minor = "glmnet")

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.(?!([T|P]\\.))", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjust_interaction_feats(indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.nonTP", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,                                
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indep_vars = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.T\\.", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjust_interaction_feats(indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.onlyT", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indep_vars = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

    indepVars <- c(max_cor_y_x_vars)
    for (txtFeat in names(glbFeatsText))
        indepVars <- union(indepVars, 
            grep(paste(str_to_upper(substr(txtFeat, 1, 1)), "\\.P\\.", sep = ""),
                        names(glbObsAll), perl = TRUE, value = TRUE))
    indepVars <- myadjust_interaction_feats(indepVars)
    ret_lst <- myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
        id.prefix = "Max.cor.Y.Text.onlyP", 
        type = glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method = "repeatedcv",
        trainControl.number = glb_rcv_n_folds, trainControl.repeats = glb_rcv_n_repeats,
        trainControl.classProbs = glb_is_classification,
        trainControl.summaryFunction = glbMdlMetricSummaryFn,
        trainControl.allowParallel = glbMdlAllowParallel,        
        train.metric = glbMdlMetricSummary, 
        train.maximize = glbMdlMetricMaximize,    
        train.method = "glmnet")),
        indep_vars = indepVars,
        rsp_var = glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)
}

# Interactions.High.cor.Y
if (length(int_feats <- setdiff(setdiff(unique(glb_feats_df$cor.high.X), NA), 
                                subset(glb_feats_df, nzv)$id)) > 0) {
    fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                    paste0("fit.models_0_", "Interact.High.cor.Y"), major.inc = FALSE,
                                    label.minor = "glmnet")

    ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
        id.prefix="Interact.High.cor.Y", 
        type=glb_model_type, trainControl.method="repeatedcv",
        trainControl.number=glb_rcv_n_folds, trainControl.repeats=glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
        train.method="glmnet")),
        indep_vars=c(max_cor_y_x_vars, paste(max_cor_y_x_vars[1], int_feats, sep=":")),
        rsp_var=glb_rsp_var, 
        fit_df=glbObsFit, OOB_df=glbObsOOB)
}    

# Low.cor.X
fit.models_0_chunk_df <- myadd_chunk(fit.models_0_chunk_df, 
                        paste0("fit.models_0_", "Low.cor.X"), major.inc = FALSE,
                                     label.minor = "glmnet")
indep_vars <- subset(glb_feats_df, is.na(cor.high.X) & !nzv & 
                              (exclude.as.feat != 1))[, "id"]  
indep_vars <- myadjust_interaction_feats(indep_vars)
ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
        id.prefix="Low.cor.X", 
        type=glb_model_type, 
        tune.df = glbMdlTuneParams,        
        trainControl.method="repeatedcv",
        trainControl.number=glb_rcv_n_folds, trainControl.repeats=glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
        train.method="glmnet")),
        indep_vars=indep_vars, rsp_var=glb_rsp_var, 
        fit_df = glbObsFit, OOB_df = glbObsOOB)

fit.models_0_chunk_df <- 
    myadd_chunk(fit.models_0_chunk_df, "fit.models_0_end", major.inc = FALSE,
                label.minor = "teardown")

rm(ret_lst)

glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc = FALSE)
```

```{r fit.models_1, cache=FALSE, fig.height=10, fig.width=15}
fit.models_1_chunk_df <- myadd_chunk(NULL, "fit.models_1_bgn", label.minor="setup")

#stop(here"); glb2Sav(); all.equal(glb_models_df, sav_models_df)
topindep_var <- NULL; interact_vars <- NULL;
for (mdl_id_pfx in names(glbMdlFamilies)) {
    fit.models_1_chunk_df <- 
        myadd_chunk(fit.models_1_chunk_df, paste0("fit.models_1_", mdl_id_pfx),
                    major.inc = FALSE, label.minor = "setup")

    indep_vars <- NULL;

    if (grepl("\\.Interact", mdl_id_pfx)) {
        if (is.null(topindep_var) && is.null(interact_vars)) {
        #   select best glmnet model upto now
            dsp_models_df <- orderBy(model_sel_frmla <- get_model_sel_frmla(),
                                     glb_models_df)
            dsp_models_df <- subset(dsp_models_df, 
                                    grepl(".glmnet", id, fixed = TRUE))
            bst_mdl_id <- dsp_models_df$id[1]
            mdl_id_pfx <- 
                paste(c(head(unlist(strsplit(bst_mdl_id, "[.]")), -1), "Interact"),
                      collapse=".")
        #   select important features
            if (is.null(bst_featsimp_df <- 
                        myget_feats_importance(glb_models_lst[[bst_mdl_id]]))) {
                warning("Base model for RFE.Interact: ", bst_mdl_id, 
                        " has no important features")
                next
            }    
            
            topindep_ix <- 1
            while (is.null(topindep_var) && (topindep_ix <= nrow(bst_featsimp_df))) {
                topindep_var <- row.names(bst_featsimp_df)[topindep_ix]
                if (grepl(".fctr", topindep_var, fixed=TRUE))
                    topindep_var <- 
                        paste0(unlist(strsplit(topindep_var, ".fctr"))[1], ".fctr")
                if (topindep_var %in% names(glbFeatsInteractionOnly)) {
                    topindep_var <- NULL; topindep_ix <- topindep_ix + 1
                } else break
            }
            
        #   select features with importance > max(10, importance of .rnorm) & is not highest
        #       combine factor dummy features to just the factor feature
            if (length(pos_rnorm <- 
                       grep(".rnorm", row.names(bst_featsimp_df), fixed=TRUE)) > 0)
                imp_rnorm <- bst_featsimp_df[pos_rnorm, 1] else
                imp_rnorm <- NA    
            imp_cutoff <- max(10, imp_rnorm, na.rm=TRUE)
            interact_vars <- 
                tail(row.names(subset(bst_featsimp_df, 
                                      imp > imp_cutoff)), -1)
            if (length(interact_vars) > 0) {
                interact_vars <-
                    myadjust_interaction_feats(myextract_actual_feats(interact_vars))
                interact_vars <- 
                    interact_vars[!grepl(topindep_var, interact_vars, fixed=TRUE)]
            }
            ### bid0_sp only
#             interact_vars <- c(
#     "biddable", "D.ratio.sum.TfIdf.wrds.n", "D.TfIdf.sum.stem.stop.Ratio", "D.sum.TfIdf",
#     "D.TfIdf.sum.post.stop", "D.TfIdf.sum.post.stem", "D.ratio.wrds.stop.n.wrds.n", "D.chrs.uppr.n.log",
#     "D.chrs.n.log", "color.fctr"
#     # , "condition.fctr", "prdl.my.descr.fctr"
#                                 )
#            interact_vars <- setdiff(interact_vars, c("startprice.dgt2.is9", "color.fctr"))
            ###
            indep_vars <- myextract_actual_feats(row.names(bst_featsimp_df))
            indep_vars <- setdiff(indep_vars, topindep_var)
            if (length(interact_vars) > 0) {
                indep_vars <- 
                    setdiff(indep_vars, myextract_actual_feats(interact_vars))
                indep_vars <- c(indep_vars, 
                    paste(topindep_var, setdiff(interact_vars, topindep_var), 
                          sep = "*"))
            } else indep_vars <- union(indep_vars, topindep_var)
        }
    }
    
    if (is.null(indep_vars))
        indep_vars <- glb_mdl_feats_lst[[mdl_id_pfx]]

    if (is.null(indep_vars) && grepl("RFE\\.", mdl_id_pfx))
        indep_vars <- myextract_actual_feats(predictors(rfe_fit_results))
    
    if (is.null(indep_vars))
        indep_vars <- subset(glb_feats_df, !nzv & (exclude.as.feat != 1))[, "id"]
    
    if ((length(indep_vars) == 1) && (grepl("^%<d-%", indep_vars))) {    
        indep_vars <- 
            eval(parse(text = str_trim(unlist(strsplit(indep_vars, "%<d-%"))[2])))
    }    

    indep_vars <- myadjust_interaction_feats(indep_vars)
    
    if (grepl("\\.Interact", mdl_id_pfx)) { 
        # if (method != tail(unlist(strsplit(bst_mdl_id, "[.]")), 1)) next
        if (is.null(glbMdlFamilies[[mdl_id_pfx]])) {
            if (!is.null(glbMdlFamilies[["Best.Interact"]]))
                glbMdlFamilies[[mdl_id_pfx]] <-
                    glbMdlFamilies[["Best.Interact"]]
        }
    }
    
    if (!is.null(glbObsFitOutliers[[mdl_id_pfx]])) {
        fitobs_df <- glbObsFit[!(glbObsFit[, glbFeatsId] %in%
                                         glbObsFitOutliers[[mdl_id_pfx]]), ]
    } else fitobs_df <- glbObsFit

    if (is.null(glbMdlFamilies[[mdl_id_pfx]]))
        mdl_methods <- glbMdlMethods else
        mdl_methods <- glbMdlFamilies[[mdl_id_pfx]]    

    for (method in mdl_methods) {
        if (method %in% c("rpart", "rf")) {
            # rpart:    fubar's the tree
            # rf:       skip the scenario w/ .rnorm for speed
            indep_vars <- setdiff(indep_vars, c(".rnorm"))
            #mdl_id <- paste0(mdl_id_pfx, ".no.rnorm")
        } 

        fit.models_1_chunk_df <- myadd_chunk(fit.models_1_chunk_df, 
                            paste0("fit.models_1_", mdl_id_pfx), major.inc = FALSE,
                                    label.minor = method)

        ret_lst <- 
            myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
            id.prefix = mdl_id_pfx, 
            type = glb_model_type, 
            tune.df = glbMdlTuneParams,
            trainControl.method = "repeatedcv", # or "none" if nominalWorkflow is crashing
            trainControl.number = glb_rcv_n_folds,
            trainControl.repeats = glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            trainControl.allowParallel = glbMdlAllowParallel,            
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method = method)),
            indep_vars = indep_vars, rsp_var = glb_rsp_var, 
            fit_df = fitobs_df, OOB_df = glbObsOOB)
        
#         ntv_mdl <- glmnet(x = as.matrix(
#                               fitobs_df[, indep_vars]), 
#                           y = as.factor(as.character(
#                               fitobs_df[, glb_rsp_var])),
#                           family = "multinomial")
#         bgn = 1; end = 100;
#         ntv_mdl <- glmnet(x = as.matrix(
#                               subset(fitobs_df, pop.fctr != "crypto")[bgn:end, indep_vars]), 
#                           y = as.factor(as.character(
#                               subset(fitobs_df, pop.fctr != "crypto")[bgn:end, glb_rsp_var])),
#                           family = "multinomial")
    }
}

# Check if other preProcess methods improve model performance
fit.models_1_chunk_df <- 
    myadd_chunk(fit.models_1_chunk_df, "fit.models_1_preProc", major.inc = FALSE,
                label.minor = "preProc")

mdl_id <- orderBy(get_model_sel_frmla(), glb_models_df)[1, "id"]
indep_vars_vctr <- trim(unlist(strsplit(glb_models_df[glb_models_df$id == mdl_id,
                                                      "feats"], "[,]")))
method <- tail(unlist(strsplit(mdl_id, "[.]")), 1)
mdl_id_pfx <- paste0(head(unlist(strsplit(mdl_id, "[.]")), -1), collapse = ".")
if (!is.null(glbObsFitOutliers[[mdl_id_pfx]])) {
    fitobs_df <- glbObsFit[!(glbObsFit[, glbFeatsId] %in%
                                     glbObsFitOutliers[[mdl_id_pfx]]), ]
} else fitobs_df <- glbObsFit

for (prePr in glb_preproc_methods) {   
    # The operations are applied in this order: 
    #   Box-Cox/Yeo-Johnson transformation, centering, scaling, range, imputation, PCA, ICA then spatial sign.
    
    ret_lst <- myfit_mdl(mdl_specs_lst=myinit_mdl_specs_lst(mdl_specs_lst=list(
            id.prefix=mdl_id_pfx, 
            type=glb_model_type, tune.df=glbMdlTuneParams,
            trainControl.method="repeatedcv",
            trainControl.number=glb_rcv_n_folds,
            trainControl.repeats=glb_rcv_n_repeats,
            trainControl.classProbs = glb_is_classification,
            trainControl.summaryFunction = glbMdlMetricSummaryFn,
            train.metric = glbMdlMetricSummary, 
            train.maximize = glbMdlMetricMaximize,    
            train.method=method, train.preProcess=prePr)),
            indep_vars=indep_vars_vctr, rsp_var=glb_rsp_var, 
            fit_df=fitobs_df, OOB_df=glbObsOOB)
}            
    
    # If (All|RFE).X.glm is less accurate than Low.Cor.X.glm
    #   check NA coefficients & filter appropriate terms in indep_vars_vctr
#     if (method == "glm") {
#         orig_glm <- glb_models_lst[[paste0(mdl_id, ".", model_method)]]$finalModel
#         orig_glm <- glb_models_lst[["All.X.glm"]]$finalModel; print(summary(orig_glm))
#         orig_glm <- glb_models_lst[["RFE.X.glm"]]$finalModel; print(summary(orig_glm))
#           require(car)
#           vif_orig_glm <- vif(orig_glm); print(vif_orig_glm)
#           # if vif errors out with "there are aliased coefficients in the model"
#               alias_orig_glm <- alias(orig_glm); alias_complete_orig_glm <- (alias_orig_glm$Complete > 0); alias_complete_orig_glm <- alias_complete_orig_glm[rowSums(alias_complete_orig_glm) > 0, colSums(alias_complete_orig_glm) > 0]; print(alias_complete_orig_glm)
#           print(vif_orig_glm[!is.na(vif_orig_glm) & (vif_orig_glm == Inf)])
#           print(which.max(vif_orig_glm))
#           print(sort(vif_orig_glm[vif_orig_glm >= 1.0e+03], decreasing=TRUE))
#           glbObsFit[c(1143, 3637, 3953, 4105), c("UniqueID", "Popular", "H.P.quandary", "Headline")]
#           glb_feats_df[glb_feats_df$id %in% grep("[HSA]\\.chrs.n.log", glb_feats_df$id, value=TRUE) | glb_feats_df$cor.high.X %in%    grep("[HSA]\\.chrs.n.log", glb_feats_df$id, value=TRUE), ]
#           all.equal(glbObsAll$S.chrs.uppr.n.log, glbObsAll$A.chrs.uppr.n.log)
#           cor(glbObsAll$S.T.herald, glbObsAll$S.T.tribun)
#           mydspObs(Abstract.contains="[Dd]iar", cols=("Abstract"), all=TRUE)
#           subset(glb_feats_df, cor.y.abs <= glb_feats_df[glb_feats_df$id == ".rnorm", "cor.y.abs"])
#         corxx_mtrx <- cor(data.matrix(glbObsAll[, setdiff(names(glbObsAll), myfind_chr_cols_df(glbObsAll))]), use="pairwise.complete.obs"); abs_corxx_mtrx <- abs(corxx_mtrx); diag(abs_corxx_mtrx) <- 0
#           which.max(abs_corxx_mtrx["S.T.tribun", ])
#           abs_corxx_mtrx["A.npnct08.log", "S.npnct08.log"]
#         step_glm <- step(orig_glm)
#     }
    # Since caret does not optimize rpart well
#     if (method == "rpart")
#         ret_lst <- myfit_mdl(mdl_id=paste0(mdl_id_pfx, ".cp.0"), model_method=method,
#                                 indep_vars_vctr=indep_vars_vctr,
#                                 model_type=glb_model_type,
#                                 rsp_var=glb_rsp_var,
#                                 fit_df=glbObsFit, OOB_df=glbObsOOB,        
#             n_cv_folds=0, tune_models_df=data.frame(parameter="cp", min=0.0, max=0.0, by=0.1))

# User specified
#   Ensure at least 2 vars in each regression; else varImp crashes
# sav_models_lst <- glb_models_lst; sav_models_df <- glb_models_df; sav_featsimp_df <- glb_featsimp_df; all.equal(sav_featsimp_df, glb_featsimp_df)
# glb_models_lst <- sav_models_lst; glb_models_df <- sav_models_df; glm_featsimp_df <- sav_featsimp_df

    # easier to exclude features
# require(gdata) # needed for trim
# mdl_id <- "";
# indep_vars_vctr <- head(subset(glb_models_df, grepl("All\\.X\\.", mdl_id), select=feats)
#                         , 1)[, "feats"]
# indep_vars_vctr <- trim(unlist(strsplit(indep_vars_vctr, "[,]")))
# indep_vars_vctr <- setdiff(indep_vars_vctr, ".rnorm")

    # easier to include features
#stop(here"); sav_models_df <- glb_models_df; glb_models_df <- sav_models_df
# !_sp
# mdl_id <- "csm"; indep_vars_vctr <- c(NULL
#     ,"prdline.my.fctr", "prdline.my.fctr:.clusterid.fctr"
#     ,"prdline.my.fctr*biddable"
#     #,"prdline.my.fctr*startprice.log"
#     #,"prdline.my.fctr*startprice.diff"    
#     ,"prdline.my.fctr*condition.fctr"
#     ,"prdline.my.fctr*D.terms.post.stop.n"
#     #,"prdline.my.fctr*D.terms.post.stem.n"
#     ,"prdline.my.fctr*cellular.fctr"    
# #    ,"<feat1>:<feat2>"
#                                            )
# for (method in glbMdlMethods) {
#     ret_lst <- myfit_mdl(mdl_id=mdl_id, model_method=method,
#                                 indep_vars_vctr=indep_vars_vctr,
#                                 model_type=glb_model_type,
#                                 rsp_var=glb_rsp_var,
#                                 fit_df=glbObsFit, OOB_df=glbObsOOB,
#                     n_cv_folds=glb_rcv_n_folds, tune_models_df=glbMdlTuneParams)
#     csm_mdl_id <- paste0(mdl_id, ".", method)
#     csm_featsimp_df <- myget_feats_importance(glb_models_lst[[paste0(mdl_id, ".",
#                                                                      method)]]);               print(head(csm_featsimp_df))
# }
###

# Ntv.1.lm <- lm(reformulate(indep_vars_vctr, glb_rsp_var), glbObsTrn); print(summary(Ntv.1.lm))

#glb_models_df[, "max.Accuracy.OOB", FALSE]
#varImp(glb_models_lst[["Low.cor.X.glm"]])
#orderBy(~ -Overall, varImp(glb_models_lst[["All.X.2.glm"]])$imp)
#orderBy(~ -Overall, varImp(glb_models_lst[["All.X.3.glm"]])$imp)
#glb_feats_df[grepl("npnct28", glb_feats_df$id), ]

    # User specified bivariate models
#     indep_vars_vctr_lst <- list()
#     for (feat in setdiff(names(glbObsFit), 
#                          union(glb_rsp_var, glbFeatsExclude)))
#         indep_vars_vctr_lst[["feat"]] <- feat

    # User specified combinatorial models
#     indep_vars_vctr_lst <- list()
#     combn_mtrx <- combn(c("<feat1_name>", "<feat2_name>", "<featn_name>"), 
#                           <num_feats_to_choose>)
#     for (combn_ix in 1:ncol(combn_mtrx))
#         #print(combn_mtrx[, combn_ix])
#         indep_vars_vctr_lst[[combn_ix]] <- combn_mtrx[, combn_ix]
    
    # template for myfit_mdl
    #   rf is hard-coded in caret to recognize only Accuracy / Kappa evaluation metrics
    #       only for OOB in trainControl ?
    
#     ret_lst <- myfit_mdl_fn(mdl_id=paste0(mdl_id_pfx, ""), model_method=method,
#                             indep_vars_vctr=indep_vars_vctr,
#                             rsp_var=glb_rsp_var,
#                             fit_df=glbObsFit, OOB_df=glbObsOOB,
#                             n_cv_folds=glb_rcv_n_folds, tune_models_df=glbMdlTuneParams,
#                             model_loss_mtrx=glbMdlMetric_terms,
#                             model_summaryFunction=glbMdlMetricSummaryFn,
#                             model_metric=glbMdlMetricSummary,
#                             model_metric_maximize=glbMdlMetricMaximize)

# Simplify a model
# fit_df <- glbObsFit; glb_mdl <- step(<complex>_mdl)

# Non-caret models
#     rpart_area_mdl <- rpart(reformulate("Area", response=glb_rsp_var), 
#                                data=glbObsFit, #method="class", 
#                                control=rpart.control(cp=0.12),
#                            parms=list(loss=glbMdlMetric_terms))
#     print("rpart_sel_wlm_mdl"); prp(rpart_sel_wlm_mdl)
# 

print(glb_models_df)

rm(ret_lst)
fit.models_1_chunk_df <- 
    myadd_chunk(fit.models_1_chunk_df, "fit.models_1_end", major.inc = FALSE,
                label.minor = "teardown")
glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc = FALSE)
```

```{r fit.models_2, cache=FALSE, fig.height=10, fig.width=15}
fit.models_2_chunk_df <- 
    myadd_chunk(NULL, "fit.models_2_bgn", label.minor = "setup")

plt_models_df <- glb_models_df[, -grep("SD|Upper|Lower", names(glb_models_df))]
for (var in grep("^min.", names(plt_models_df), value=TRUE)) {
    plt_models_df[, sub("min.", "inv.", var)] <- 
        #ifelse(all(is.na(tmp <- plt_models_df[, var])), NA, 1.0 / tmp)
        1.0 / plt_models_df[, var]
    plt_models_df <- plt_models_df[ , -grep(var, names(plt_models_df))]
}
print(plt_models_df)
# print(myplot_radar(radar_inp_df=plt_models_df))
# print(myplot_radar(radar_inp_df=subset(plt_models_df, 
#         !(mdl_id %in% grep("random|MFO", plt_models_df$id, value=TRUE)))))

# Compute CI for <metric>SD
glb_models_df <- mutate(glb_models_df, 
                max.df = ifelse(max.nTuningRuns > 1, max.nTuningRuns - 1, NA),
                min.sd2ci.scaler = ifelse(is.na(max.df), NA, qt(0.975, max.df)))
for (var in grep("SD", names(glb_models_df), value=TRUE)) {
    # Does CI alredy exist ?
    var_components <- unlist(strsplit(var, "SD"))
    varActul <- paste0(var_components[1],          var_components[2])
    varUpper <- paste0(var_components[1], "Upper", var_components[2])
    varLower <- paste0(var_components[1], "Lower", var_components[2])
    if (varUpper %in% names(glb_models_df)) {
        warning(varUpper, " already exists in glb_models_df")
        # Assuming Lower also exists
        next
    }    
    print(sprintf("var:%s", var))
    # CI is dependent on sample size in t distribution; df=n-1
    glb_models_df[, varUpper] <- glb_models_df[, varActul] + 
        glb_models_df[, "min.sd2ci.scaler"] * glb_models_df[, var]
    glb_models_df[, varLower] <- glb_models_df[, varActul] - 
        glb_models_df[, "min.sd2ci.scaler"] * glb_models_df[, var]
}

# Plot metrics with CI
plt_models_df <- glb_models_df[, "id", FALSE]
pltCI_models_df <- glb_models_df[, "id", FALSE]
for (var in grep("Upper", names(glb_models_df), value=TRUE)) {
    var_components <- unlist(strsplit(var, "Upper"))
    col_name <- unlist(paste(var_components, collapse=""))
    plt_models_df[, col_name] <- glb_models_df[, col_name]
    for (name in paste0(var_components[1], c("Upper", "Lower"), var_components[2]))
        pltCI_models_df[, name] <- glb_models_df[, name]
}

build_statsCI_data <- function(plt_models_df) {
    mltd_models_df <- melt(plt_models_df, id.vars="id")
    mltd_models_df$data <- sapply(1:nrow(mltd_models_df), 
        function(row_ix) tail(unlist(strsplit(as.character(
            mltd_models_df[row_ix, "variable"]), "[.]")), 1))
    mltd_models_df$label <- sapply(1:nrow(mltd_models_df), 
        function(row_ix) head(unlist(strsplit(as.character(
            mltd_models_df[row_ix, "variable"]), 
            paste0(".", mltd_models_df[row_ix, "data"]))), 1))
    #print(mltd_models_df)
    
    return(mltd_models_df)
}
mltd_models_df <- build_statsCI_data(plt_models_df)

mltdCI_models_df <- melt(pltCI_models_df, id.vars="id")
for (row_ix in 1:nrow(mltdCI_models_df)) {
    for (type in c("Upper", "Lower")) {
        if (length(var_components <- unlist(strsplit(
                as.character(mltdCI_models_df[row_ix, "variable"]), type))) > 1) {
            #print(sprintf("row_ix:%d; type:%s; ", row_ix, type))
            mltdCI_models_df[row_ix, "label"] <- var_components[1]
            mltdCI_models_df[row_ix, "data"] <- 
                unlist(strsplit(var_components[2], "[.]"))[2]
            mltdCI_models_df[row_ix, "type"] <- type
            break
        }
    }    
}
wideCI_models_df <- reshape(subset(mltdCI_models_df, select=-variable), 
                            timevar="type", 
        idvar=setdiff(names(mltdCI_models_df), c("type", "value", "variable")), 
                            direction="wide")
#print(wideCI_models_df)
mrgdCI_models_df <- merge(wideCI_models_df, mltd_models_df, all.x=TRUE)
#print(mrgdCI_models_df)

# Merge stats back in if CIs don't exist
goback_vars <- c()
for (var in unique(mltd_models_df$label)) {
    for (type in unique(mltd_models_df$data)) {
        var_type <- paste0(var, ".", type)
        # if this data is already present, next
        if (var_type %in% unique(paste(mltd_models_df$label, mltd_models_df$data,
                                       sep=".")))
            next
        #print(sprintf("var_type:%s", var_type))
        goback_vars <- c(goback_vars, var_type)
    }
}

if (length(goback_vars) > 0) {
    mltd_goback_df <- build_statsCI_data(glb_models_df[, c("id", goback_vars)])
    mltd_models_df <- rbind(mltd_models_df, mltd_goback_df)
}

# mltd_models_df <- merge(mltd_models_df, glb_models_df[, c("id", "model_method")], 
#                         all.x=TRUE)

png(paste0(glb_out_pfx, "models_bar.png"), width=480*3, height=480*2)
#print(gp <- myplot_bar(mltd_models_df, "id", "value", colorcol_name="model_method") + 
print(gp <- myplot_bar(df=mltd_models_df, xcol_name="id", ycol_names="value") + 
        geom_errorbar(data=mrgdCI_models_df, 
            mapping=aes(x=mdl_id, ymax=value.Upper, ymin=value.Lower), width=0.5) + 
          facet_grid(label ~ data, scales="free") + 
          theme(axis.text.x = element_text(angle = 90,vjust = 0.5)))
dev.off()
print(gp)

dsp_models_cols <- c("id", 
                    glbMdlMetricsEval[glbMdlMetricsEval %in% names(glb_models_df)],
                    grep("opt.", names(glb_models_df), fixed = TRUE, value = TRUE)) 
# if (glb_is_classification && glb_is_binomial) 
#     dsp_models_cols <- c(dsp_models_cols, "opt.prob.threshold.OOB")
print(dsp_models_df <- orderBy(get_model_sel_frmla(), glb_models_df)[, dsp_models_cols])
# print(myplot_radar(radar_inp_df = dsp_models_df))
print("Metrics used for model selection:"); print(get_model_sel_frmla())
print(sprintf("Best model id: %s", dsp_models_df[1, "id"]))

glb_get_predictions <- function(df, mdl_id, rsp_var, prob_threshold_def=NULL, verbose=FALSE) {
    mdl <- glb_models_lst[[mdl_id]]
    
    clmnNames <- mygetPredictIds(rsp_var, mdl_id)
    predct_var_name <- clmnNames$value        
    predct_prob_var_name <- clmnNames$prob
    predct_accurate_var_name <- clmnNames$is.acc
    predct_error_var_name <- clmnNames$err
    predct_erabs_var_name <- clmnNames$err.abs

    if (glb_is_regression) {
        df[, predct_var_name] <- predict(mdl, newdata=df, type="raw")
        if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_var_name) + 
                  facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="glm"))

        df[, predct_error_var_name] <- df[, predct_var_name] - df[, glb_rsp_var]
        if (verbose) print(myplot_scatter(df, predct_var_name, predct_error_var_name) + 
                  #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="auto"))
        if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_error_var_name) + 
                  #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
                  stat_smooth(method="glm"))
        
        df[, predct_erabs_var_name] <- abs(df[, predct_error_var_name])
        if (verbose) print(head(orderBy(reformulate(c("-", predct_erabs_var_name)), df)))
        
        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])
    }

    if (glb_is_classification && glb_is_binomial) {
        prob_threshold <- glb_models_df[glb_models_df$id == mdl_id, 
                                        "opt.prob.threshold.OOB"]
        if (is.null(prob_threshold) || is.na(prob_threshold)) {
            warning("Using default probability threshold: ", prob_threshold_def)
            if (is.null(prob_threshold <- prob_threshold_def))
                stop("Default probability threshold is NULL")
        }
        
        df[, predct_prob_var_name] <- predict(mdl, newdata = df, type = "prob")[, 2]
        df[, predct_var_name] <- 
        		factor(levels(df[, glb_rsp_var])[
    				(df[, predct_prob_var_name] >=
    					prob_threshold) * 1 + 1], levels(df[, glb_rsp_var]))
    
#         if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_var_name) + 
#                   facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="glm"))

        df[, predct_error_var_name] <- df[, predct_var_name] != df[, glb_rsp_var]
#         if (verbose) print(myplot_scatter(df, predct_var_name, predct_error_var_name) + 
#                   #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="auto"))
#         if (verbose) print(myplot_scatter(df, glb_rsp_var, predct_error_var_name) + 
#                   #facet_wrap(reformulate(glbFeatsCategory), scales = "free") + 
#                   stat_smooth(method="glm"))
        
        # if prediction is a TP (true +ve), measure distance from 1.0
        tp <- which((df[, predct_var_name] == df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[2]))
        df[tp, predct_erabs_var_name] <- abs(1 - df[tp, predct_prob_var_name])
        #rowIx <- which.max(df[tp, predct_erabs_var_name]); df[tp, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a TN (true -ve), measure distance from 0.0
        tn <- which((df[, predct_var_name] == df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[1]))
        df[tn, predct_erabs_var_name] <- abs(0 - df[tn, predct_prob_var_name])
        #rowIx <- which.max(df[tn, predct_erabs_var_name]); df[tn, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a FP (flse +ve), measure distance from 0.0
        fp <- which((df[, predct_var_name] != df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[2]))
        df[fp, predct_erabs_var_name] <- abs(0 - df[fp, predct_prob_var_name])
        #rowIx <- which.max(df[fp, predct_erabs_var_name]); df[fp, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]
        
        # if prediction is a FN (flse -ve), measure distance from 1.0
        fn <- which((df[, predct_var_name] != df[, glb_rsp_var]) &
                    (df[, predct_var_name] == levels(df[, glb_rsp_var])[1]))
        df[fn, predct_erabs_var_name] <- abs(1 - df[fn, predct_prob_var_name])
        #rowIx <- which.max(df[fn, predct_erabs_var_name]); df[fn, c(glbFeatsId, glb_rsp_var, predct_var_name, predct_prob_var_name, predct_erabs_var_name)][rowIx, ]

        
        if (verbose) print(head(orderBy(reformulate(c("-", predct_erabs_var_name)), df)))
        
        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])
    }    
    
    if (glb_is_classification && !glb_is_binomial) {
        df[, predct_var_name] <- predict(mdl, newdata = df, type = "raw")
        probCls <- predict(mdl, newdata = df, type = "prob")        
        df[, predct_prob_var_name] <- NA
        for (cls in names(probCls)) {
            mask <- (df[, predct_var_name] == cls)
            df[mask, predct_prob_var_name] <- probCls[mask, cls]
        }    
        if (verbose) print(myplot_histogram(df, predct_prob_var_name, 
                                            fill_col_name = predct_var_name))
        if (verbose) print(myplot_histogram(df, predct_prob_var_name, 
                                            facet_frmla = paste0("~", glb_rsp_var)))
        
        df[, predct_error_var_name] <- df[, predct_var_name] != df[, glb_rsp_var]
        
        # if prediction is erroneous, measure predicted class prob from actual class prob
        df[, predct_erabs_var_name] <- 0
        for (cls in names(probCls)) {
            mask <- (df[, glb_rsp_var] == cls) & (df[, predct_error_var_name])
            df[mask, predct_erabs_var_name] <- probCls[mask, cls]
        }    

        df[, predct_accurate_var_name] <- (df[, glb_rsp_var] == df[, predct_var_name])        
    }

    return(df)
}    

#stop(here"); glb2Sav(); glbObsAll <- savObsAll; glbObsTrn <- savObsTrn; glbObsFit <- savObsFit; glbObsOOB <- savObsOOB; sav_models_df <- glb_models_df; glb_models_df <- sav_models_df; glb_featsimp_df <- sav_featsimp_df    

myget_category_stats <- function(obs_df, mdl_id, label) {
    require(dplyr)
    require(lazyeval)
    
    predct_var_name <- mygetPredictIds(glb_rsp_var, mdl_id)$value        
    predct_error_var_name <- mygetPredictIds(glb_rsp_var, mdl_id)$err.abs
    
    if (!predct_var_name %in% names(obs_df))
        obs_df <- glb_get_predictions(obs_df, mdl_id, glb_rsp_var)
    
    tmp_obs_df <- obs_df[, c(glbFeatsCategory, glb_rsp_var, 
                             predct_var_name, predct_error_var_name)]
#     tmp_obs_df <- obs_df %>%
#         dplyr::select_(glbFeatsCategory, glb_rsp_var, predct_var_name, predct_error_var_name) 
    #dplyr::rename(startprice.log10.predict.RFE.X.glmnet.err=error_abs_OOB)
    names(tmp_obs_df)[length(names(tmp_obs_df))] <- paste0("err.abs.", label)
    
    ret_ctgry_df <- tmp_obs_df %>%
        dplyr::group_by_(glbFeatsCategory) %>%
        dplyr::summarise_(#interp(~sum(abs(var)), var=as.name(glb_rsp_var)), 
            interp(~sum(var), var=as.name(paste0("err.abs.", label))), 
            interp(~mean(var), var=as.name(paste0("err.abs.", label))),
            interp(~n()))
    names(ret_ctgry_df) <- c(glbFeatsCategory, 
                             #paste0(glb_rsp_var, ".abs.", label, ".sum"),
                             paste0("err.abs.", label, ".sum"),                             
                             paste0("err.abs.", label, ".mean"), 
                             paste0(".n.", label))
    ret_ctgry_df <- dplyr::ungroup(ret_ctgry_df)
    #colSums(ret_ctgry_df[, -grep(glbFeatsCategory, names(ret_ctgry_df))])
    
    return(ret_ctgry_df)    
}
#print(colSums((ctgry_df <- myget_category_stats(obs_df=glbObsFit, mdl_id="", label="fit"))[, -grep(glbFeatsCategory, names(ctgry_df))]))

if (!is.null(glb_mdl_ensemble)) {
    fit.models_2_chunk_df <- myadd_chunk(fit.models_2_chunk_df, 
                            paste0("fit.models_2_", mdl_id_pfx), major.inc = TRUE, 
                                                label.minor = "ensemble")
    
    mdl_id_pfx <- "Ensemble"

    if (#(glb_is_regression) | 
        ((glb_is_classification) & (!glb_is_binomial)))
        stop("Ensemble models not implemented yet for multinomial classification")
    
    mygetEnsembleAutoMdlIds <- function() {
        tmp_models_df <- orderBy(get_model_sel_frmla(), glb_models_df)
        row.names(tmp_models_df) <- tmp_models_df$id
        mdl_threshold_pos <- 
            min(which(grepl("MFO|Random|Baseline", tmp_models_df$id))) - 1
        mdlIds <- tmp_models_df$id[1:mdl_threshold_pos]
        return(mdlIds[!grepl("Ensemble", mdlIds)])
    }
    
    if (glb_mdl_ensemble == "auto") {
        glb_mdl_ensemble <- mygetEnsembleAutoMdlIds()
        mdl_id_pfx <- paste0(mdl_id_pfx, ".auto")        
    } else if (grepl("^%<d-%", glb_mdl_ensemble)) {
        glb_mdl_ensemble <- eval(parse(text =
                        str_trim(unlist(strsplit(glb_mdl_ensemble, "%<d-%"))[2])))
    }
    
    for (mdl_id in glb_mdl_ensemble) {
        if (!(mdl_id %in% names(glb_models_lst))) {
            warning("Model ", mdl_id, " in glb_model_ensemble not found !")
            next
        }
        glbObsFit <- glb_get_predictions(df = glbObsFit, mdl_id, glb_rsp_var)
        glbObsOOB <- glb_get_predictions(df = glbObsOOB, mdl_id, glb_rsp_var)
    }
    
#mdl_id_pfx <- "Ensemble.RFE"; mdlId <- paste0(mdl_id_pfx, ".glmnet")
#glb_mdl_ensemble <- gsub(mygetPredictIds$value, "", grep("RFE\\.X\\.(?!Interact)", row.names(glb_featsimp_df), perl = TRUE, value = TRUE), fixed = TRUE)
#varImp(glb_models_lst[[mdlId]])
    
#cor_df <- data.frame(cor=cor(glbObsFit[, glb_rsp_var], glbObsFit[, paste(mygetPredictIds$value, glb_mdl_ensemble)], use="pairwise.complete.obs"))
#glbObsFit <- glb_get_predictions(df=glbObsFit, "Ensemble.glmnet", glb_rsp_var);print(colSums((ctgry_df <- myget_category_stats(obs_df=glbObsFit, mdl_id="Ensemble.glmnet", label="fit"))[, -grep(glbFeatsCategory, names(ctgry_df))]))
    
    ### bid0_sp
    #  Better than MFO; models.n=28; min.RMSE.fit=0.0521233; err.abs.fit.sum=7.3631895
    #  old: Top x from auto; models.n= 5; min.RMSE.fit=0.06311047; err.abs.fit.sum=9.5937080
    #  RFE only ;       models.n=16; min.RMSE.fit=0.05148588; err.abs.fit.sum=7.2875091
    #  RFE subset only ;models.n= 5; min.RMSE.fit=0.06040702; err.abs.fit.sum=9.059088
    #  RFE subset only ;models.n= 9; min.RMSE.fit=0.05933167; err.abs.fit.sum=8.7421288
    #  RFE subset only ;models.n=15; min.RMSE.fit=0.0584607; err.abs.fit.sum=8.5902066
    #  RFE subset only ;models.n=17; min.RMSE.fit=0.05496899; err.abs.fit.sum=8.0170431
    #  RFE subset only ;models.n=18; min.RMSE.fit=0.05441577; err.abs.fit.sum=7.837223
    #  RFE subset only ;models.n=16; min.RMSE.fit=0.05441577; err.abs.fit.sum=7.837223
    ### bid0_sp
    ### bid1_sp
    # "auto"; err.abs.fit.sum=76.699774; min.RMSE.fit=0.2186429
    # "RFE.X.*"; err.abs.fit.sum=; min.RMSE.fit=0.221114
    ### bid1_sp

    indep_vars <- paste(mygetPredictIds(glb_rsp_var)$value, glb_mdl_ensemble, sep = "")
    if (glb_is_classification)
        indep_vars <- paste(indep_vars, ".prob", sep = "")
    # Some models in glb_mdl_ensemble might not be fitted e.g. RFE.X.Interact
    indep_vars <- intersect(indep_vars, names(glbObsFit))
    
#     indep_vars <- grep(mygetPredictIds(glb_rsp_var)$value, names(glbObsFit), fixed=TRUE, value=TRUE)
#     if (glb_is_regression)
#         indep_vars <- indep_vars[!grepl("(err\\.abs|accurate)$", indep_vars)]
#     if (glb_is_classification && glb_is_binomial)
#         indep_vars <- grep("prob$", indep_vars, value=TRUE) else
#         indep_vars <- indep_vars[!grepl("err$", indep_vars)]

    #rfe_fit_ens_results <- myrun_rfe(glbObsFit, indep_vars)
    
    for (method in c("glm", "glmnet")) {
        for (trainControlMethod in 
             c("boot", "boot632", "cv", "repeatedcv"
               #, "LOOCV" # tuneLength * nrow(fitDF)
               , "LGOCV", "adaptive_cv"
               #, "adaptive_boot"  #error: adaptive$min should be less than 3 
               #, "adaptive_LGOCV" #error: adaptive$min should be less than 3 
               )) {
            #sav_models_df <- glb_models_df; all.equal(sav_models_df, glb_models_df)
            #glb_models_df <- sav_models_df; print(glb_models_df$id)
                
            if ((method == "glm") && (trainControlMethod != "repeatedcv"))
                # glm used only to identify outliers
                next
            
            ret_lst <- myfit_mdl(
                mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                    id.prefix = paste0(mdl_id_pfx, ".", trainControlMethod), 
                    type = glb_model_type, tune.df = NULL,
                    trainControl.method = trainControlMethod,
                    trainControl.number = glb_rcv_n_folds,
                    trainControl.repeats = glb_rcv_n_repeats,
                    trainControl.classProbs = glb_is_classification,
                    trainControl.summaryFunction = glbMdlMetricSummaryFn,
                    train.metric = glbMdlMetricSummary, 
                    train.maximize = glbMdlMetricMaximize,    
                    train.method = method)),
                indep_vars = indep_vars, rsp_var = glb_rsp_var, 
                fit_df = glbObsFit, OOB_df = glbObsOOB)
        }
    }
    dsp_models_df <- get_dsp_models_df()
}

if (is.null(glb_sel_mdl_id)) 
    glb_sel_mdl_id <- dsp_models_df[1, "id"] else 
    print(sprintf("User specified selection: %s", glb_sel_mdl_id))   
    
myprint_mdl(glb_sel_mdl <- glb_models_lst[[glb_sel_mdl_id]])
 
# From here to save(), this should all be in one function
#   these are executed in the same seq twice more:
#       fit.data.training & predict.data.new chunks
print(sprintf("%s fit prediction diagnostics:", glb_sel_mdl_id))
glbObsFit <- glb_get_predictions(df = glbObsFit, mdl_id = glb_sel_mdl_id, 
                                 rsp_var = glb_rsp_var)
print(sprintf("%s OOB prediction diagnostics:", glb_sel_mdl_id))
glbObsOOB <- glb_get_predictions(df = glbObsOOB, mdl_id = glb_sel_mdl_id, 
                                     rsp_var = glb_rsp_var)

print(glb_featsimp_df <- myget_feats_importance(mdl = glb_sel_mdl, featsimp_df = NULL))
#mdl_id <-"RFE.X.glmnet"; glb_featsimp_df <- myget_feats_importance(glb_models_lst[[mdl_id]], glb_featsimp_df); glb_featsimp_df[, paste0(mdl_id, ".imp")] <- glb_featsimp_df$imp; print(glb_featsimp_df)
#print(head(sbst_featsimp_df <- subset(glb_featsimp_df, is.na(RFE.X.glmnet.imp) | (abs(RFE.X.YeoJohnson.glmnet.imp - RFE.X.glmnet.imp) > 0.0001), select=-imp)))
#print(orderBy(~ -cor.y.abs, subset(glb_feats_df, id %in% c(row.names(sbst_featsimp_df), "startprice.dcm1.is9", "D.weight.post.stop.sum"))))

# Used again in fit.data.training & predict.data.new chunks
glb_analytics_diag_plots <- function(obs_df, mdl_id, prob_threshold=NULL) {
    if (!is.null(featsimp_df <- glb_featsimp_df)) {
        featsimp_df$feat <- gsub("`(.*?)`", "\\1", row.names(featsimp_df))    
        featsimp_df$feat.interact <- gsub("(.*?):(.*)", "\\2", featsimp_df$feat)
        featsimp_df$feat <- gsub("(.*?):(.*)", "\\1", featsimp_df$feat)    
        featsimp_df$feat.interact <- 
            ifelse(featsimp_df$feat.interact == featsimp_df$feat, 
                                            NA, featsimp_df$feat.interact)
        featsimp_df$feat <- 
            gsub("(.*?)\\.fctr(.*)", "\\1\\.fctr", featsimp_df$feat)
        featsimp_df$feat.interact <- 
            gsub("(.*?)\\.fctr(.*)", "\\1\\.fctr", featsimp_df$feat.interact) 
        featsimp_df <- orderBy(~ -imp.max, 
            summaryBy(imp ~ feat + feat.interact, data=featsimp_df,
                      FUN=max))    
        #rex_str=":(.*)"; txt_vctr=tail(featsimp_df$feat); ret_lst <- regexec(rex_str, txt_vctr); ret_lst <- regmatches(txt_vctr, ret_lst); ret_vctr <- sapply(1:length(ret_lst), function(pos_ix) ifelse(length(ret_lst[[pos_ix]]) > 0, ret_lst[[pos_ix]], "")); print(ret_vctr <- ret_vctr[ret_vctr != ""])    
        
        featsimp_df <- subset(featsimp_df, !is.na(imp.max))
        if (nrow(featsimp_df) > 5) {
            warning("Limiting important feature scatter plots to 5 out of ",
                    nrow(featsimp_df))
            featsimp_df <- head(featsimp_df, 5)
        }
        
    #     if (!all(is.na(featsimp_df$feat.interact)))
    #         stop("not implemented yet")
        rsp_var_out <- mygetPredictIds(glb_rsp_var, mdl_id)$value
        for (var in featsimp_df$feat) {
            plot_df <- melt(obs_df, id.vars = var, 
                            measure.vars = c(glb_rsp_var, rsp_var_out))
    
            print(myplot_scatter(plot_df, var, "value", colorcol_name = "variable",
                                facet_colcol_name = "variable", jitter = TRUE) + 
                          guides(color = FALSE))
        }
    }
    
    if (glb_is_regression) {
        if (is.null(featsimp_df) || (nrow(featsimp_df) == 0))
            warning("No important features in glb_fin_mdl") else
            print(myplot_prediction_regression(df=obs_df, 
                        feat_x=ifelse(nrow(featsimp_df) > 1, featsimp_df$feat[2],
                                      ".rownames"), 
                                               feat_y=featsimp_df$feat[1],
                        rsp_var=glb_rsp_var, rsp_var_out=rsp_var_out,
                        id_vars=glbFeatsId)
    #               + facet_wrap(reformulate(featsimp_df$feat[2])) # if [1 or 2] is a factor
    #               + geom_point(aes_string(color="<col_name>.fctr")) #  to color the plot
                  )
    }    
    
    if (glb_is_classification) {
        if (is.null(featsimp_df) || (nrow(featsimp_df) == 0))
            warning("No features in selected model are statistically important")
        else print(myplot_prediction_classification(df = obs_df, 
                                feat_x = ifelse(nrow(featsimp_df) > 1, 
                                                featsimp_df$feat[2], ".rownames"),
                                               feat_y = featsimp_df$feat[1],
                                                rsp_var = glb_rsp_var, 
                                                rsp_var_out = rsp_var_out, 
                                                id_vars = glbFeatsId,
                                                prob_threshold = prob_threshold))
    }    
}

if (glb_is_classification && glb_is_binomial)
    glb_analytics_diag_plots(obs_df = glbObsOOB, mdl_id = glb_sel_mdl_id, 
            prob_threshold = glb_models_df[glb_models_df$id == glb_sel_mdl_id, 
                                           "opt.prob.threshold.OOB"]) else
    glb_analytics_diag_plots(obs_df = glbObsOOB, mdl_id = glb_sel_mdl_id)                  

if (!is.null(glbFeatsCategory)) {
    glbLvlCategory <- merge(glbLvlCategory, 
            myget_category_stats(obs_df = glbObsFit, mdl_id = glb_sel_mdl_id, 
                                 label = "fit"), 
                            by = glbFeatsCategory, all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    glbLvlCategory <- merge(glbLvlCategory, 
            myget_category_stats(obs_df = glbObsOOB, mdl_id = glb_sel_mdl_id,
                                 label="OOB"),
                          #by=glbFeatsCategory, all=TRUE) glb_ctgry-df already contains .n.OOB ?
                          all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    if (any(grepl("OOB", glbMdlMetricsEval)))
        print(orderBy(~-err.abs.OOB.mean, glbLvlCategory)) else
            print(orderBy(~-err.abs.fit.mean, glbLvlCategory))
    print(colSums(glbLvlCategory[, -grep(glbFeatsCategory, names(glbLvlCategory))]))
}

write.csv(glbObsOOB[, c(glbFeatsId, 
                grep(glb_rsp_var, names(glbObsOOB), fixed=TRUE, value=TRUE))], 
    paste0(gsub(".", "_", paste0(glb_out_pfx, glb_sel_mdl_id), fixed=TRUE), 
           "_OOBobs.csv"), row.names=FALSE)

fit.models_2_chunk_df <- 
    myadd_chunk(NULL, "fit.models_2_bgn", label.minor = "teardown")
glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.models", major.inc=FALSE)
```

```{r fit.models_3, cache=FALSE, fig.height=10, fig.width=15}
# if (sum(is.na(glbObsAll$D.P.http)) > 0)
#         stop("fit.models_3: Why is this happening ?")

#stop(here"); glb2Sav()
sync_glb_obs_df <- function() {
    # Merge or cbind ?
    for (col in setdiff(names(glbObsFit), names(glbObsTrn)))
        glbObsTrn[glbObsTrn$.lcn == "Fit", col] <<- glbObsFit[, col]
    for (col in setdiff(names(glbObsFit), names(glbObsAll)))
        glbObsAll[glbObsAll$.lcn == "Fit", col] <<- glbObsFit[, col]
    if (all(is.na(glbObsNew[, glb_rsp_var])))
        for (col in setdiff(names(glbObsOOB), names(glbObsTrn)))
            glbObsTrn[glbObsTrn$.lcn == "OOB", col] <<- glbObsOOB[, col]
    for (col in setdiff(names(glbObsOOB), names(glbObsAll)))
        glbObsAll[glbObsAll$.lcn == "OOB", col] <<- glbObsOOB[, col]
}
sync_glb_obs_df()
    
print(setdiff(names(glbObsNew), names(glbObsAll)))

if (glb_save_envir)
    save(glb_feats_df, 
         glbObsAll, #glbObsTrn, glbObsFit, glbObsOOB, glbObsNew,
         glb_models_df, dsp_models_df, glb_models_lst, glb_sel_mdl, glb_sel_mdl_id,
         glb_model_type,
        file=paste0(glb_out_pfx, "selmdl_dsk.RData"))
#load(paste0(glb_out_pfx, "selmdl_dsk.RData"))

rm(ret_lst)
replay.petrisim(pn=glb_analytics_pn, 
    replay.trans=(glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
        "model.selected")), flip_coord=TRUE)
glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.data.training", major.inc=TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r fit.data.training_0, cache=FALSE}
#load(paste0(glb_inp_pfx, "dsk.RData"))

if (!is.null(glb_fin_mdl_id) && (glb_fin_mdl_id %in% names(glb_models_lst))) {
    warning("Final model same as user selected model")
    glb_fin_mdl <- glb_models_lst[[glb_fin_mdl_id]]
} else 
# if (nrow(glbObsFit) + length(glbObsFitOutliers) == nrow(glbObsTrn))
if (!all(is.na(glbObsNew[, glb_rsp_var])))
{    
    warning("Final model same as glb_sel_mdl_id")
    glb_fin_mdl_id <- paste0("Final.", glb_sel_mdl_id)
    glb_fin_mdl <- glb_sel_mdl
    glb_models_lst[[glb_fin_mdl_id]] <- glb_fin_mdl
} else {    
            if (grepl("RFE\\.X", names(glbMdlFamilies))) {
                indep_vars <- myadjust_interaction_feats(subset(glb_feats_df, 
                                                    !nzv & (exclude.as.feat != 1))[, "id"])
                rfe_trn_results <- 
                    myrun_rfe(glbObsTrn, indep_vars, glbRFESizes[["Final"]])
                if (!isTRUE(all.equal(sort(predictors(rfe_trn_results)),
                                      sort(predictors(rfe_fit_results))))) {
                    print("Diffs predictors(rfe_trn_results) vs. predictors(rfe_fit_results):")
                    print(setdiff(predictors(rfe_trn_results), predictors(rfe_fit_results)))
                    print("Diffs predictors(rfe_fit_results) vs. predictors(rfe_trn_results):")
                    print(setdiff(predictors(rfe_fit_results), predictors(rfe_trn_results)))
            }
        }
    # }    

    if (grepl("Ensemble", glb_sel_mdl_id)) {
        # Find which models are relevant
        mdlimp_df <- subset(myget_feats_importance(glb_sel_mdl), imp > 5)
        # Fit selected models on glbObsTrn
        for (mdl_id in gsub(".prob", "", 
gsub(mygetPredictIds(glb_rsp_var)$value, "", row.names(mdlimp_df), fixed = TRUE),
                            fixed = TRUE)) {
            mdl_id_components <- unlist(strsplit(mdl_id, "[.]"))
            mdlIdPfx <- paste0(c(head(mdl_id_components, -1), "Train"), 
                               collapse = ".")
            if (grepl("RFE\\.X\\.", mdlIdPfx)) 
                mdlIndepVars <- myadjust_interaction_feats(myextract_actual_feats(
                    predictors(rfe_trn_results))) else
                mdlIndepVars <- trim(unlist(
            strsplit(glb_models_df[glb_models_df$id == mdl_id, "feats"], "[,]")))
            ret_lst <- 
                myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                        id.prefix = mdlIdPfx, 
                        type = glb_model_type, tune.df = glbMdlTuneParams,
                        trainControl.method = "repeatedcv",
                        trainControl.number = glb_rcv_n_folds,
                        trainControl.repeats = glb_rcv_n_repeats,
                        trainControl.classProbs = glb_is_classification,
                        trainControl.summaryFunction = glbMdlMetricSummaryFn,
                        train.metric = glbMdlMetricSummary, 
                        train.maximize = glbMdlMetricMaximize,    
                        train.method = tail(mdl_id_components, 1))),
                    indep_vars = mdlIndepVars,
                    rsp_var = glb_rsp_var, 
                    fit_df = glbObsTrn, OOB_df = NULL)
            
            glbObsTrn <- glb_get_predictions(df = glbObsTrn,
                                                mdl_id = tail(glb_models_df$id, 1), 
                                                rsp_var = glb_rsp_var,
                                                prob_threshold_def = 
                    subset(glb_models_df, id == mdl_id)$opt.prob.threshold.OOB)
            glbObsNew <- glb_get_predictions(df = glbObsNew,
                                                mdl_id = tail(glb_models_df$id, 1), 
                                                rsp_var = glb_rsp_var,
                                                prob_threshold_def = 
                    subset(glb_models_df, id == mdl_id)$opt.prob.threshold.OOB)
        }    
    }
    
    # "Final" model
    if ((model_method <- glb_sel_mdl$method) == "custom")
        # get actual method from the mdl_id
        model_method <- tail(unlist(strsplit(glb_sel_mdl_id, "[.]")), 1)
        
    if (grepl("Ensemble", glb_sel_mdl_id)) {
        # Find which models are relevant
        mdlimp_df <- subset(myget_feats_importance(glb_sel_mdl), imp > 5)
        if (glb_is_classification && glb_is_binomial)
            indep_vars_vctr <- gsub("(.*)\\.(.*)\\.prob", "\\1\\.Train\\.\\2\\.prob",
                                    row.names(mdlimp_df)) else
            indep_vars_vctr <- gsub("(.*)\\.(.*)", "\\1\\.Train\\.\\2",
                                    row.names(mdlimp_df))
    } else 
    if (grepl("RFE.X", glb_sel_mdl_id, fixed = TRUE)) {
        indep_vars_vctr <- myextract_actual_feats(predictors(rfe_trn_results))
    } else indep_vars_vctr <- 
                trim(unlist(strsplit(glb_models_df[glb_models_df$id ==
                                                   glb_sel_mdl_id
                                                   , "feats"], "[,]")))
        
    if (!is.null(glb_preproc_methods) &&
        ((match_pos <- regexpr(gsub(".", "\\.", 
                                    paste(glb_preproc_methods, collapse = "|"),
                                   fixed = TRUE), glb_sel_mdl_id)) != -1))
        ths_preProcess <- str_sub(glb_sel_mdl_id, match_pos, 
                                match_pos + attr(match_pos, "match.length") - 1) else
        ths_preProcess <- NULL                                      

    mdl_id_pfx <- ifelse(grepl("Ensemble", glb_sel_mdl_id),
                                   "Final.Ensemble", "Final")
    trnobs_df <- if (is.null(glbObsTrnOutliers[[mdl_id_pfx]])) glbObsTrn else 
        glbObsTrn[!(glbObsTrn[, glbFeatsId] %in%
                            glbObsTrnOutliers[[mdl_id_pfx]]), ]
        
    # Force fitting of Final.glm to identify outliers
    method_vctr <- unique(c(myparseMdlId(glb_sel_mdl_id)$alg, glbMdlFamilies[["Final"]]))
    for (method in method_vctr) {
        #source("caret_nominalTrainWorkflow.R")
        
        # glmnet requires at least 2 indep vars
        if ((length(indep_vars_vctr) == 1) && (method %in% "glmnet"))
            next
        
        ret_lst <- 
            myfit_mdl(mdl_specs_lst = myinit_mdl_specs_lst(mdl_specs_lst = list(
                    id.prefix = mdl_id_pfx, 
                    type = glb_model_type, trainControl.method = "repeatedcv",
                    trainControl.number = glb_rcv_n_folds, 
                    trainControl.repeats = glb_rcv_n_repeats,
                    trainControl.classProbs = glb_is_classification,
                    trainControl.summaryFunction = glbMdlMetricSummaryFn,
                    trainControl.allowParallel = glbMdlAllowParallel,
                    train.metric = glbMdlMetricSummary, 
                    train.maximize = glbMdlMetricMaximize,    
                    train.method = method,
                    train.preProcess = ths_preProcess)),
                indep_vars = indep_vars_vctr, rsp_var = glb_rsp_var, 
                fit_df = trnobs_df, OOB_df = NULL)
    }
        
    if ((length(method_vctr) == 1) || (method != "glm")) {
        glb_fin_mdl <- glb_models_lst[[length(glb_models_lst)]] 
        glb_fin_mdl_id <- glb_models_df[length(glb_models_lst), "id"]
    }
}

rm(ret_lst)
glb_chunks_df <- myadd_chunk(glb_chunks_df, "fit.data.training", major.inc=FALSE)
```

```{r fit.data.training_1, cache=FALSE}
#stop(here"); glb2Sav()
if (glb_is_classification && glb_is_binomial) 
    prob_threshold <- glb_models_df[glb_models_df$id == glb_sel_mdl_id,
                                        "opt.prob.threshold.OOB"] else 
    prob_threshold <- NULL

if (grepl("Ensemble", glb_fin_mdl_id)) {
    # Get predictions for each model in ensemble; Outliers that have been moved to OOB might not have been predicted yet
    mdlEnsembleComps <- unlist(str_split(subset(glb_models_df, 
                                                id == glb_fin_mdl_id)$feats, ","))
    if (glb_is_classification && glb_is_binomial)
        mdlEnsembleComps <- gsub("\\.prob$", "", mdlEnsembleComps)
    mdlEnsembleComps <- gsub(paste0("^", 
                        gsub(".", "\\.", mygetPredictIds(glb_rsp_var)$value, fixed = TRUE)),
                             "", mdlEnsembleComps)
    for (mdl_id in mdlEnsembleComps) {
        glbObsTrn <- glb_get_predictions(df = glbObsTrn, mdl_id = mdl_id, 
                                            rsp_var = glb_rsp_var,
                                            prob_threshold_def = prob_threshold)
        glbObsNew <- glb_get_predictions(df = glbObsNew, mdl_id = mdl_id, 
                                            rsp_var = glb_rsp_var,
                                            prob_threshold_def = prob_threshold)
    }    
}
glbObsTrn <- glb_get_predictions(df = glbObsTrn, mdl_id = glb_fin_mdl_id, 
                                     rsp_var = glb_rsp_var,
                                    prob_threshold_def = prob_threshold)

glb_featsimp_df <- myget_feats_importance(mdl=glb_fin_mdl,
                                          featsimp_df=glb_featsimp_df)
glb_featsimp_df[, paste0(glb_fin_mdl_id, ".imp")] <- glb_featsimp_df$imp
print(glb_featsimp_df)
if (glb_is_classification && glb_is_binomial)
    glb_analytics_diag_plots(obs_df=glbObsTrn, mdl_id=glb_fin_mdl_id, 
            prob_threshold=glb_models_df[glb_models_df$id == glb_sel_mdl_id, 
                                         "opt.prob.threshold.OOB"]) else
    glb_analytics_diag_plots(obs_df=glbObsTrn, mdl_id=glb_fin_mdl_id)                  

dsp_feats_vctr <- c(NULL)
for(var in grep(".imp", names(glb_feats_df), fixed=TRUE, value=TRUE))
    dsp_feats_vctr <- union(dsp_feats_vctr, 
                            glb_feats_df[!is.na(glb_feats_df[, var]), "id"])

# print(glbObsTrn[glbObsTrn$UniqueID %in% FN_OOB_ids, 
#                     grep(glb_rsp_var, names(glbObsTrn), value=TRUE)])

print(setdiff(names(glbObsTrn), names(glbObsAll)))
for (col in setdiff(names(glbObsTrn), names(glbObsAll)))
    # Merge or cbind ?
    glbObsAll[glbObsAll$.src == "Train", col] <- glbObsTrn[, col]

print(setdiff(names(glbObsFit), names(glbObsAll)))
print(setdiff(names(glbObsOOB), names(glbObsAll)))
for (col in setdiff(names(glbObsOOB), names(glbObsAll)))
    # Merge or cbind ?
    glbObsAll[glbObsAll$.lcn == "OOB", col] <- glbObsOOB[, col]
    
print(setdiff(names(glbObsNew), names(glbObsAll)))

if (glb_save_envir)
    save(glb_feats_df, glbObsAll, 
         #glbObsTrn, glbObsFit, glbObsOOB, glbObsNew,
         glb_models_df, dsp_models_df, glb_models_lst, glb_model_type,
         glb_sel_mdl, glb_sel_mdl_id,
         glb_fin_mdl, glb_fin_mdl_id,
        file = paste0(glb_out_pfx, "dsk.RData"))

#glb2Sav(); all.equal(savObsAll, glbObsAll); all.equal(sav_models_lst, glb_models_lst)
#load(file = paste0(glb_out_pfx, "dsk_knitr.RData"))
#cmpCols <- names(glbObsAll)[!grepl("\\.Final\\.", names(glbObsAll))]; all.equal(savObsAll[, cmpCols], glbObsAll[, cmpCols]); all.equal(savObsAll[, "H.P.http"], glbObsAll[, "H.P.http"]); 

replay.petrisim(pn = glb_analytics_pn, 
    replay.trans = (glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
        "data.training.all.prediction","model.final")), flip_coord = TRUE)
glb_chunks_df <- myadd_chunk(glb_chunks_df, "predict.data.new", major.inc = TRUE)
```

## Step ``r mydsp_chunk(glb_chunks_df)``
```{r predict.data.new, cache=FALSE, echo=FALSE}
# Compute final model predictions

#glb2Sav(); all.equal(savObsAll, glbObsAll); all.equal(savObsTrn, glbObsTrn); all.equal(savObsNew, glbObsNew)  
if (glb_is_classification && glb_is_binomial)
    prob_threshold_def <- 
        glb_models_df[glb_models_df$id == glb_sel_mdl_id, "opt.prob.threshold.OOB"] else
    prob_threshold_def <- NULL
for (obsSet in c("trn", "new")) {
    obs_df <- switch(obsSet, all = glbObsAll, trn = glbObsTrn, new = glbObsNew)
    obs_df <- glb_get_predictions(obs_df, mdl_id = glb_fin_mdl_id, 
                                    rsp_var = glb_rsp_var, 
                                    prob_threshold_def = prob_threshold_def)
    if (obsSet == "all") glbObsAll <- obs_df else
    if (obsSet == "trn") glbObsTrn <- obs_df else
    if (obsSet == "new") glbObsNew <- obs_df
}
rm(obs_df)
glbObsAll <- orderBy(reformulate(glbFeatsId), myrbind_df(glbObsTrn, glbObsNew))

glb_analytics_diag_plots(obs_df = glbObsNew, mdl_id = glb_fin_mdl_id, 
                         prob_threshold = prob_threshold_def)

if (is.null(glb_out_obs)) obs_df <- glbObsNew else
    obs_df <- switch(glb_out_obs, 
                     all = glbObsAll, trn = glbObsTrn, new = glbObsNew)

require(stringr)
obsout_df <- obs_df[, glbFeatsId, FALSE]
for (clmn in names(glb_out_vars_lst))
    if (!grepl("^%<d-%", glb_out_vars_lst[[clmn]]))
        obsout_df[, clmn] <- obs_df[, glb_out_vars_lst[[clmn]]] else {
            feat <- str_trim(unlist(strsplit(glb_out_vars_lst[[clmn]], "%<d-%"))[2])
            obsout_df[, clmn] <- obs_df[, eval(parse(text = feat))]
    }        

if (glb_is_classification) {
    rsp_var_out <- paste0(mygetPredictIds(glb_rsp_var)$value, glb_fin_mdl_id)
    if (".grpid" %in% names(glbObsNew)) {
        # Dups were found in glbObsAll
        tmp_newobs_df <- subset(glbObsNew[, c(glbFeatsId, ".grpid", rsp_var_out)],
                                !is.na(.grpid))
        tmp_newobs_df <- 
            merge(tmp_newobs_df, dupgrps_df, by = ".grpid", all.x = TRUE)
        tmp_newobs_df <- 
            merge(tmp_newobs_df, obsout_df, by = glbFeatsId, all.x = TRUE)
        tmp_newobs_df$.err <- 
            ((tmp_newobs_df$Probability1 > 0.5) & (tmp_newobs_df$sold.0 > 0) |
             (tmp_newobs_df$Probability1 < 0.5) & (tmp_newobs_df$sold.1 > 0))
        tmp_newobs_df <- orderBy(~UniqueID, subset(tmp_newobs_df, .err == TRUE))
        print(sprintf("ObsNew Prediction errors in duplicates: %d",
                      nrow(tmp_newobs_df)))
        print(tmp_newobs_df)
    }
    
    # Check for prediction errors based on category X glb_rsp_var distribution
    TrnLvlCategory <- 
        mycreate_sqlxtab_df(glbObsTrn, c(glbFeatsCategory, glb_rsp_var)) %>%
            tidyr::spread_(glb_rsp_var, ".n")
    names(TrnLvlCategory)[2:ncol(TrnLvlCategory)] <- 
        paste(".n.Trn.", names(TrnLvlCategory)[2:ncol(TrnLvlCategory)], sep = "")
    glbLvlCategory <- 
        merge(glbLvlCategory, TrnLvlCategory, by = glbFeatsCategory, all.x = TRUE)
    
    predctId <- mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$value
    NewLvlCategory <- 
        mycreate_sqlxtab_df(glbObsNew, c(glbFeatsCategory, predctId)) %>%
            tidyr::spread_(predctId, ".n")
    names(NewLvlCategory)[2:ncol(NewLvlCategory)] <- 
        paste(".n.New.", names(NewLvlCategory)[2:ncol(NewLvlCategory)], sep = "")
    glbLvlCategory <- 
        merge(glbLvlCategory, NewLvlCategory, by = glbFeatsCategory, all.x = TRUE)
    
    tmpLvlCategory <- glbLvlCategory[, c(glbFeatsCategory, 
                grep("\\.n\\.(.+)\\.", names(glbLvlCategory), value = TRUE))]
    tmpLvlCategory <- tmpLvlCategory[rowSums(is.na(tmpLvlCategory)) > 0, ]
    errLvlIx <- c(NULL)
    for (clss in unique(glbObsTrn[, glb_rsp_var])) {
        newClmn <- paste0(".n.New.", as.character(clss))
        trnClmn <- paste0(".n.Trn.", as.character(clss))        
        errLvlIx <- union(errLvlIx, 
                            which(!is.na(tmpLvlCategory[, newClmn]) &
                                (tmpLvlCategory[, newClmn] > 0) & 
                                is.na(tmpLvlCategory[, trnClmn])))
    }
    if (length(errLvlIx) > 0) {
        print("ObsNew Prediction errors in categories:")
        print(tmpLvlCategory[errLvlIx, ])
        print(colSums(tmpLvlCategory[errLvlIx, -1], na.rm = TRUE))
        
        # By definition .err columns will be NA !!!
#predctId <- mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$value; errObsNew <- glbObsNew[(glbObsNew[, glbFeatsCategory] %in% tmpLvlCategory[, 1]), ];
#myprint_df(errObsNew[(errObsNew[, glbFeatsCategory] %in% tmpLvlCategory[errLvlIx, glbFeatsCategory]) & (errObsNew[, predctId] == "Y"), union(c(glbFeatsId, glbFeatsCategory, predctId), myextract_actual_feats(row.names(glb_featsimp_df[glb_featsimp_df[, paste0(glb_fin_mdl_id, ".imp")] > 10, ])))])
#myprint_df(errObsNew[(errObsNew[, glbFeatsCategory] %in% "Foreign#World#") & (errObsNew[, predctId] == "Y"), union(c(glbFeatsId, glbFeatsCategory, predctId), myextract_actual_feats(row.names(glb_featsimp_df[glb_featsimp_df[, paste0(glb_fin_mdl_id, ".imp")] > 10, ])))])
    }
    
    # Check predictions that are outside of data ranges
#stop(here")    
    require(stringr)
    tmp_feats_df <- subset(glb_feats_df, 
                           !nzv & 
                            (exclude.as.feat != 1) & 
                            !grepl(".fctr", id, fixed=TRUE))[, "id", FALSE]
    ranges_all_df <- glbObsAll[, tmp_feats_df$id] %>% 
                        dplyr::summarise_each(funs(min(., na.rm=TRUE), 
                                                   max(., na.rm=TRUE))) %>%
                        tidyr::gather() %>%
                        dplyr::mutate(id=str_sub(key, 1, -5), 
                                      stat=str_sub(key, -3)) %>% 
                        dplyr::select(-key) %>%
                        tidyr::spread(stat, value)
    
#     sav_ranges_trn_df <- ranges_trn_df; all.equal(sav_ranges_trn_df, ranges_trn_df)
#     sav_ranges_new_df <- ranges_new_df; all.equal(sav_ranges_new_df, ranges_new_df)    
    get_ranges_df <- function(obs_df, feats, class_var) {
        require(tidyr)
        
        tmpClass <- gsub("\\*", "_", class_var)
        tmpObs <- data.frame(.tmpClass = obs_df[, class_var])
        names(tmpObs) <- tmpClass
        tmpObs <- cbind(tmpObs, obs_df[, feats, FALSE])
        ranges_df <- tmpObs %>% 
            dplyr::group_by_(tmpClass) %>%
            dplyr::summarise_each(funs(min(., na.rm=TRUE), 
                                       max(., na.rm=TRUE))) %>%
            tidyr::gather(key, value, -1) %>%
            mutate(id=str_sub(key, 1, -5), 
                   stat.vname=paste0(str_sub(key, -3), ".", tmpClass)) %>%
            unite_("stat.class", c("stat.vname", tmpClass), sep=".") %>% 
            dplyr::select(-key) %>%
            spread(stat.class, value)
        return(ranges_df)
    }
    rsp_var_out_OOB <- mygetPredictIds(glb_rsp_var, glb_sel_mdl_id)$value
    rsp_var_out_new <- mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$value
    ranges_trn_df <- get_ranges_df(obs_df=glbObsTrn, feats=tmp_feats_df$id, 
                                   class_var=glb_rsp_var)
    ranges_fit_df <- get_ranges_df(obs_df=glbObsFit, feats=tmp_feats_df$id, 
                                   class_var=glb_rsp_var)
    ranges_OOB_df <- get_ranges_df(obs_df = glbObsOOB, feats = tmp_feats_df$id, 
                                   class_var = rsp_var_out_OOB)
    ranges_new_df <- get_ranges_df(obs_df=glbObsNew, feats=tmp_feats_df$id, 
                                   class_var=rsp_var_out_new)

    for (obsset in c("OOB", "new")) {
        if (obsset == "OOB") { 
            ranges_ref_df <- ranges_fit_df; obs_df <- glbObsOOB; 
            rsp_var_out_obs <- rsp_var_out_OOB; sprintf_pfx <- "OOBobs";
        } else { 
            ranges_ref_df <- ranges_trn_df; obs_df <- glbObsNew; 
            rsp_var_out_obs <- rsp_var_out_new; sprintf_pfx <- "newobs"; 
        }
        plt_feats_df <- glb_feats_df %>% 
                            merge(ranges_all_df, all=TRUE) %>%
                            merge(ranges_ref_df, all=TRUE) %>%
                            merge(ranges_OOB_df, all=TRUE) %>%        
                            merge(ranges_new_df, all=TRUE) %>%
                            subset(!is.na(min) & (id != ".rnorm"))
        row.names(plt_feats_df) <- plt_feats_df$id
        range_outlier_ids <- c(NULL)
        for (clss in unique(obs_df[, rsp_var_out_obs])) {
            tmp_rsp_var_out_obs <- gsub("\\*", "_", rsp_var_out_obs)
            for (stat in c("min", "max")) {
                if (stat == "min") {
                    dsp_feats <- plt_feats_df[
        which(plt_feats_df[, paste("min", tmp_rsp_var_out_obs, clss, sep = ".")] < 
                plt_feats_df[, paste("min", glb_rsp_var, clss, sep = ".")]),
                                            "id"]
                } else {
                    dsp_feats <- plt_feats_df[
        which(plt_feats_df[, paste("max", tmp_rsp_var_out_obs, clss, sep = ".")] > 
                plt_feats_df[, paste("max", glb_rsp_var, clss, sep = ".")]),
                                            "id"]
                }
                if (length(dsp_feats) > 0) {
                    ths_ids <- c(NULL)
                    for (feat in dsp_feats) {
                        if (stat == "min") {
                            ths_ids <- union(ths_ids, 
                                    obs_df[(obs_df[, rsp_var_out_obs] == clss) &
                                    (obs_df[, feat] < 
                                        plt_feats_df[plt_feats_df$id == feat, 
                                    paste("min", glb_rsp_var, clss, sep = ".")]), 
                                                    glbFeatsId])
                        } else {
                            ths_ids <- union(ths_ids, 
                                    obs_df[(obs_df[, rsp_var_out_obs] == clss) &
                                    (obs_df[, feat] > 
                                        plt_feats_df[plt_feats_df$id == feat,
                                    paste("max", glb_rsp_var, clss, sep = ".")]), 
                                                    glbFeatsId])
                        }
                    }
                    tmp_obs_df <- obs_df[obs_df[, glbFeatsId] %in% ths_ids, 
                                        c(glbFeatsId, rsp_var_out_obs, dsp_feats)]
                    if (stat == "min") {
                        print(sprintf("%s %s %s: min < min of Train range: %d", 
                            sprintf_pfx, rsp_var_out_obs, clss, nrow(tmp_obs_df)))
                    } else {
                        print(sprintf("%s %s %s: max > max of Train range: %d", 
                            sprintf_pfx, rsp_var_out_obs, clss, nrow(tmp_obs_df)))
                    }
                    myprint_df(tmp_obs_df)
                    print(subset(plt_feats_df, id %in% dsp_feats))
                    
                    range_outlier_ids <- union(range_outlier_ids, ths_ids)
                }
            }
        }
        print(sprintf("%s total range outliers: %d", sprintf_pfx,
                      length(range_outlier_ids)))
    }
}

#stop(here"); glb2Sav(); sav_obsout_df <- obsout_df; all.equal(sav_obsout_df, obsout_df); obsout_df <- sav_obsout_df

# This does not work for classification since AUC distribution might be different for different models
#   -> Run glm on .prob from this glb_fin_mdl_id & .prob from stacked file & stack condition as a feature
if (!is.null(glbOutStackFnames)) {
    for (fname in glbOutStackFnames) {
        print(sprintf("Stacking file %s to prediction output...", fname))
        #obsout_df <- dplyr::arrange_(rbind(obsout_df, read.csv(fname)), "UniqueID")
        obsout_df <- dplyr::arrange_(rbind(obsout_df, 
                #read.csv(fname) %>% filter(!(UniqueID %in% obsout_df$UniqueID))),
            #read.csv(fname) %>% filter(!(UniqueID %in% obsout_df[, glbFeatsId]))),
            read.csv(fname) %>% 
                dplyr::filter_(interp(~!(var %in% obsout_df$var), 
                                      var = as.name(glbFeatsId)))),
                                    glbFeatsId)
        
        if (nrow(obsout_df) != length(unique(obsout_df[, glbFeatsId])))
            stop("Potential dups in stacked prediction output")
    }
}

out_fname <- paste0(glb_out_pfx, "out.csv")
write.csv(obsout_df, out_fname, quote = FALSE, row.names = FALSE)
#cat(" ", "\n", file=submit_fn, append=TRUE)

# print(orderBy(~ -max.auc.OOB, glb_models_df[, c("id", 
#             "max.auc.OOB", "max.Accuracy.OOB")]))
for (txtFeat in names(glbFeatsText)) {
    print(sprintf("    All post-filter-words term weights for %s:", txtFeat))
#     myprint_df(glb_post_stem_words_terms_df_lst[[txtFeat]])
    myprint_df(glbFeatsText[[txtFeat]]$postFilterTermsDf)
#     terms_mtrx <- glb_post_stem_words_terms_mtrx_lst[[txtFeat]]
#     print(glbObsAll[
#         which(terms_mtrx[, tail(glb_post_stem_words_terms_df_lst[[txtFeat]], 1)$pos] > 0), 
#                         c(glbFeatsId, names(glbFeatsText))])
#     print(nrow(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], freq == 1)))

    print(sprintf("    All post-filter-words term freq distribution for %s:", txtFeat))
    # print(table(glb_post_stem_words_terms_df_lst[[txtFeat]]$freq))
    print(table(glbFeatsText[[txtFeat]]$postFilterTermsDf$freq))    

    print(sprintf("    All post-filter-words term length distribution for %s:", txtFeat))
#     print(table(nchar(glb_post_stem_words_terms_df_lst[[txtFeat]]$term)))
#     myprint_df(subset(glb_post_stem_words_terms_df_lst[[txtFeat]], nchar(term) >= 10))
    print(table(nchar(glbFeatsText[[txtFeat]]$postFilterTermsDf$term)))
    myprint_df(subset(glbFeatsText[[txtFeat]]$postFilterTermsDf, nchar(term) >= 10))

    print(sprintf("    Analyzed term weights for %s:", txtFeat))
    #tmp_df <- glb_post_stem_words_terms_df_lst[[txtFeat]]
    tmp_df <- glbFeatsText[[txtFeat]]$postFilterTermsDf
    anl_terms_vctr <- union(select_terms, assoc_terms)
    print(subset(tmp_df, term %in% anl_terms_vctr, select = -term))
}

if (glb_is_classification && glb_is_binomial)
    print(glb_models_df[glb_models_df$id == glb_sel_mdl_id, 
                        "opt.prob.threshold.OOB"])
print(sprintf("glb_sel_mdl_id: %s", glb_sel_mdl_id))
print(sprintf("glb_fin_mdl_id: %s", glb_fin_mdl_id))
print(subset(get_dsp_models_df(), select = -id))

if (glb_is_regression) {
    print(sprintf("%s OOB RMSE: %0.4f", glb_sel_mdl_id,
            glb_models_df[glb_models_df$id == glb_sel_mdl_id, "min.RMSE.OOB"]))
}    

if (glb_is_classification) {
    print(sprintf("%s OOB confusion matrix & accuracy: ", glb_sel_mdl_id))
    print(t(confusionMatrix(glbObsOOB[, mygetPredictIds(glb_rsp_var, glb_sel_mdl_id)$value],
                            glbObsOOB[, glb_rsp_var])$table))
}            

if (!is.null(glbFeatsCategory)) {
    glbLvlCategory <- merge(glbLvlCategory, 
                            myget_category_stats(obs_df = glbObsTrn, mdl_id = glb_fin_mdl_id, 
                                                 label = "trn"),
                            by = glbFeatsCategory, all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    
    glbLvlCategory <- merge(glbLvlCategory, 
                            myget_category_stats(obs_df = glbObsNew, mdl_id = glb_fin_mdl_id,
                                                 label = "new"),
                            by = glbFeatsCategory, all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    
    clmnOrder <- sort(setdiff(names(glbLvlCategory), glbFeatsCategory))
    names(clmnOrder) <- gsub("^err\\.abs\\.(.{3})\\.(.+)", "err\\.abs\\.\\2\\.\\1", clmnOrder)
    clmnOrder <- clmnOrder[sort(names(clmnOrder))]
    glbLvlCategory <- glbLvlCategory[, c(glbFeatsCategory, clmnOrder)]
    if (any(grepl("OOB", glbMdlMetricsEval)))
        print(orderBy(~-err.abs.OOB.mean, glbLvlCategory[, -1])) else
            print(orderBy(~-err.abs.fit.mean, glbLvlCategory[, -1]))
    print(colSums(glbLvlCategory[, -grep(glbFeatsCategory,
                                         names(glbLvlCategory))]))
    
    glbLvlCategory <- merge(glbLvlCategory, 
    myget_category_stats(obs_df = glbObsNew, mdl_id = glb_fin_mdl_id,
                             label = "new"),
                          by = glbFeatsCategory, all = TRUE)
    row.names(glbLvlCategory) <- glbLvlCategory[, glbFeatsCategory]
    
    clmnOrder <- sort(setdiff(names(glbLvlCategory), glbFeatsCategory))
    names(clmnOrder) <- gsub("^err\\.abs\\.(.{3})\\.(.+)", "err\\.abs\\.\\2\\.\\1", clmnOrder)
    clmnOrder <- clmnOrder[sort(names(clmnOrder))]
    glbLvlCategory <- glbLvlCategory[, c(glbFeatsCategory, clmnOrder)]
    if (any(grepl("OOB", glbMdlMetricsEval)))
        print(orderBy(~-err.abs.OOB.mean, glbLvlCategory[, -1])) else
        print(orderBy(~-err.abs.fit.mean, glbLvlCategory[, -1]))
    print(colSums(glbLvlCategory[, -grep(glbFeatsCategory,
                                       names(glbLvlCategory))]))
}

if ((glb_rsp_var %in% names(glbObsNew)) &&
    !(any(is.na(glbObsNew[, glb_rsp_var])))) {
    if (glb_is_regression) {    
        pred_stats_df <- 
            mypredict_mdl(mdl = glb_models_lst[[glb_fin_mdl_id]], 
                          df = glbObsNew, 
                          rsp_var = glb_rsp_var, 
                          label = "new",
                model_summaryFunction = glb_sel_mdl$control$summaryFunction,
                          model_metric = glb_sel_mdl$metric,
                          model_metric_maximize = glb_sel_mdl$maximize,
                          ret_type = "stats")        
        print(sprintf("%s prediction stats for glbObsNew:", glb_fin_mdl_id))
        print(pred_stats_df)
    } else {   
        print(sprintf("%s new confusion matrix & accuracy: ", glb_fin_mdl_id))
        print(t(confusionMatrix(glbObsNew[, mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$value],
                                glbObsNew[, glb_rsp_var])$table))
    }    
}    

tmpFeatsImp <- subset(glb_featsimp_df, select = -imp)
names(tmpFeatsImp) <- gsub("#", "_", names(tmpFeatsImp))
tmpFeatsImp[, "imp.max"] <- sapply(1:nrow(tmpFeatsImp), function(rowIx)
                                max(as.numeric(tmpFeatsImp[rowIx, ]), na.rm = TRUE))
print(orderBy(as.formula(paste0("~ -", gsub("#", "_", glb_sel_mdl_id), ".imp")), 
              subset(tmpFeatsImp, imp.max > 10, select = -imp.max)))

dsp_myCategory_conf_mtrx <- function(myCategory) {
    print(sprintf("%s OOB::myCategory=%s confusion matrix & accuracy: ", 
                  glb_sel_mdl_id, myCategory))
    print(t(confusionMatrix(
        glbObsOOB[glbObsOOB$myCategory == myCategory, 
                      paste0(mygetPredictIds(glb_rsp_var)$value, glb_sel_mdl_id)], 
        glbObsOOB[glbObsOOB$myCategory == myCategory, glb_rsp_var])$table))
    print(sum(glbObsOOB[glbObsOOB$myCategory == myCategory, 
                            predct_accurate_var_name]) / 
         nrow(glbObsOOB[glbObsOOB$myCategory == myCategory, ]))
    err_ids <- glbObsOOB[(glbObsOOB$myCategory == myCategory) & 
                             (!glbObsOOB[, predct_accurate_var_name]), glbFeatsId]

    OOB_FNerr_df <- glbObsOOB[(glbObsOOB$UniqueID %in% err_ids) & 
                               (glbObsOOB$Popular == 1), 
                        c(
                            ".clusterid", 
                            "Popular", "Headline", "Snippet", "Abstract")]
    print(sprintf("%s OOB::myCategory=%s FN errors: %d", glb_sel_mdl_id, myCategory,
                  nrow(OOB_FNerr_df)))
    print(OOB_FNerr_df)

    OOB_FPerr_df <- glbObsOOB[(glbObsOOB$UniqueID %in% err_ids) & 
                               (glbObsOOB$Popular == 0), 
                        c(
                            ".clusterid", 
                            "Popular", "Headline", "Snippet", "Abstract")]
    print(sprintf("%s OOB::myCategory=%s FP errors: %d", glb_sel_mdl_id, myCategory,
                  nrow(OOB_FPerr_df)))
    print(OOB_FPerr_df)
}
#dsp_myCategory_conf_mtrx(myCategory="OpEd#Opinion#")
#dsp_myCategory_conf_mtrx(myCategory="Business#Business Day#Dealbook")
#dsp_myCategory_conf_mtrx(myCategory="##")

# if (glb_is_classification) {
#     print("FN_OOB_ids:")
#     print(glbObsOOB[glbObsOOB$UniqueID %in% FN_OOB_ids, 
#                         grep(glb_rsp_var, names(glbObsOOB), value=TRUE)])
#     print(glbObsOOB[glbObsOOB$UniqueID %in% FN_OOB_ids, 
#                         glbFeatsText])
#     print(dsp_vctr <- colSums(glbObsOOB[glbObsOOB$UniqueID %in% FN_OOB_ids, 
#                         setdiff(grep("[HSA].", names(glbObsOOB), value=TRUE),
#                                 union(myfind_chr_cols_df(glbObsOOB),
#                     grep(".fctr", names(glbObsOOB), fixed=TRUE, value=TRUE)))]))
# }

print("glbObsNew prediction stats:")
if (glb_is_regression)
    print(myplot_histogram(glbObsNew, mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$value))
if (glb_is_classification)
    print(table(glbObsNew[, mygetPredictIds(glb_rsp_var, glb_fin_mdl_id)$value]))

# Use this to see how prediction changes by changing one or more values
# players_df <- data.frame(id=c("Chavez", "Giambi", "Menechino", "Myers", "Pena"),
#                          OBP=c(0.338, 0.391, 0.369, 0.313, 0.361),
#                          SLG=c(0.540, 0.450, 0.374, 0.447, 0.500),
#                         cost=c(1400000, 1065000, 295000, 800000, 300000))
# players_df$RS.predict <- predict(glb_models_lst[[csm_mdl_id]], players_df)
# print(orderBy(~ -RS.predict, players_df))
# dsp_chisq.test(Headline.contains="[Vi]deo")

if ((length(diff <- setdiff(names(glbObsTrn), names(glbObsAll))) > 0) ||
    (length(diff <- setdiff(names(glbObsFit), names(glbObsAll))) > 0) ||
    (length(diff <- setdiff(names(glbObsOOB), names(glbObsAll))) > 0) ||
    (length(diff <- setdiff(names(glbObsNew), names(glbObsAll))) > 0)) {
    print(diff)
    stop("glbObs* not in sync")
}

if (glb_save_envir)
    save(glb_feats_df, glbObsAll, 
         #glbObsTrn, glbObsFit, glbObsOOB, glbObsNew,
         glb_models_df, dsp_models_df, glb_models_lst, glb_model_type,
         glb_sel_mdl, glb_sel_mdl_id,
         glb_fin_mdl, glb_fin_mdl_id,
        file=paste0(glb_out_pfx, "prdnew_dsk.RData"))

# tmp_replay_lst <- replay.petrisim(pn=glb_analytics_pn, 
#     replay.trans=(glb_analytics_avl_objs <- c(glb_analytics_avl_objs, 
#         "data.new.prediction")), flip_coord=TRUE)
# print(ggplot.petrinet(tmp_replay_lst[["pn"]]) + coord_flip())

glb_chunks_df <- myadd_chunk(glb_chunks_df, "display.session.info", major.inc = TRUE)
```

Null Hypothesis ($\sf{H_{0}}$): mpg is not impacted by am_fctr.  
The variance by am_fctr appears to be independent. 
#```{r q1, cache=FALSE}
# print(t.test(subset(cars_df, am_fctr == "automatic")$mpg, 
#              subset(cars_df, am_fctr == "manual")$mpg, 
#              var.equal=FALSE)$conf)
#```
We reject the null hypothesis i.e. we have evidence to conclude that am_fctr impacts mpg (95% confidence). Manual transmission is better for miles per gallon versus automatic transmission.

```{r display.session.info, echo=FALSE}
myplt_chunk(glb_chunks_df)
chunkDFIds <- grep("chunk(_|\\.)df", ls(), value = TRUE)
chunkMaxElapsed <- orderBy(~-elapsed, glb_chunks_df)[1, ]
chunkMaxElapsedDFIds <- c(
    paste0(chunkMaxElapsed$label, "_", chunkMaxElapsed$step_minor, "_chunk_df"), 
    paste0(chunkMaxElapsed$label, ".", chunkMaxElapsed$step_minor, ".chunk.df"),     
    paste0(chunkMaxElapsed$label, "_chunk_df"),
    paste0(chunkMaxElapsed$label, ".chunk.df")    
                        )
for (dfId in intersect(chunkMaxElapsedDFIds, chunkDFIds)) {
    myplt_chunk(eval(parse(text = dfId)))
    break
}
#sessionInfo()
```