800_textMiningSE.Rmd

---
output:
  html_document: default
  pdf_document: default
---

# Text Mining Software Engineering Data 

In software engineering, there is a lot of information in plain text such as requirements, bug reports, mails, reviews from applicatons,  etc.  Typically that information can be extracted from Software Configuration Management Systems (SCM), Bug Tracking Systems (BTS) such as Bugzilla or application stores such as Google Play or Apple's AppStore, etc. can be mined to extract relevant information. Here we briefly explain the text mining process and how this can be done with R.

A well-known package for _text mining_ is `tm` [@FeinererH15,@FeinererHM08]. Another popular package is `wordcloud`. 

```{r setup, message=FALSE, eval=FALSE, echo=FALSE, warning=FALSE}
# - Install packages

#pckgs_needed <- c("tm", "wordcloud")
#install.packages(pckgs_needed, dependencies = TRUE)
```

## Terminology 

  The workflow that we follow for analyzing a set of text documents are: 

  1. Importing data. A _Corpus_ is a collection of text documents, implemented as VCorpus (corpora are R object held in memory). The `tm` provides several corpus constructors: `DirSource`, `VectorSource`, or `DataframeSource` (`getSources()`). 
  
  There are several parameters that control the creation of a _Corpus_.
  ((The parameter readerControl of the corpus constructor has to be a list with the named components reader and language))  

  2. Preprocessing: in this step we may remove common words, punctuation and we may perform other operations. We may do this operations after creating the DocumentTermMatrix.
  
  3. Inspecting and exploring data: Individual documents can be accessed via [[

  4. Transformations: Transformations are done via the `tm_map()` function. 
    + `tm_map(_____, stripWhitespace)`  
    + `tm_map(_____, content_transformer(tolower))`
    + `tm_map(_____, removeWords, stopwords("english"))`
    + `tm_map(_____, stemDocument)`
  5. Creating `Term-Document` Matrices: TermDocumentMatrix and DocumentTermMatrix
    + A document term matrix is a matrix with documents as the rows and terms as the columns. Each cell of the matrix contains the count of the frequency of words. We use DocumentTermMatrix()
to create the matrix.
    + `inspect(DocumentTermMatrix( newsreuters, list(dictionary = c("term1", "term2", "term3"))))`.   It displays detailed information on a corpus or a term-document matrix.

  6. Relationships between terms.
    + `findFreqTerms(_____, anumber)`
    + `findAssocs(Mydtm, "aterm", anumbercorrelation)`
    + A dictionary is a (multi-)set of strings. It is often used to denote relevant terms in text mining. 

  7. Clustering and Classification
  
    


## Example of classifying bugs from Bugzilla 


Bugzilla is Issue Tracking System that allow us to follow the evolution of a project.

The following example shows how to work with entries from Bugzilla. It is assumed that the data has been extracted and we have the records in a flat file (this can be done using Web crawlers or directly using the SQL database).


```{r readFileCompendiumARFF, message=FALSE, warning=FALSE}
library(foreign)
# path_name <- file.path("C:", "datasets", "textMining")
# path_name
# dir(path_name)

#Import data
options(stringsAsFactors = FALSE)
d <- read.arff("./datasets/textMining/reviewsBugs.arff" )
str(d) #print out information about d

head(d,2) # the first two rows of d. 
# fifth entry
d$revContent[5]
d$revBug[5]
```

Creating a Document-Term Matrix (DTM)

```{r Create DTM, message=FALSE, warning=FALSE, include=FALSE}
library(tm)

# creation of a data frame
ds <- DataframeSource(data.frame(doc_id=row.names(d),
                                 text=d$revContent))

# Corpus is the command that creates the corpus
dsc <- Corpus(ds)
str(dsc)

# weighting=TfIdf weighting is Tf*Idf. See documentation 'tm'.Term frequency*inverse term frequency
# minWordLength=WL the minimum word length is WL
# minDocFreq=ND each word must appear at least in ND docs

# Other options of DTM
# These are not really needed, if preprocessing has been carried out:
# stemming=TRUE stemming is applied
# stopwords=TRUE stopwords are eliminated
# removeNumbers=TRUE numbers are eliminated

dtm <- DocumentTermMatrix(dsc, control = list(weighting = weightTfIdf, minDocFreq=3, stopwords = TRUE, removeNumbers = TRUE))

dim(dtm)
inspect(dtm [1:5,1:10]) # view 5 first docs, 10 terms

# dtm.70=removeSparseTerms(dtm,sparse=0.7)
# dtm.70 # or dim(dtm.70)
# note that the term-document matrix needs to be transformed (casted)
# to a matrix form in the following barplot command
```

Now, we can explore things such as "which words are associated with "feature"?"

```{r}
# which words are associated with "bug"?
findAssocs(dtm, 'bug', .3) # minimum correlation of 0.3. Change accordingly. 
```

And find frequent terms.

```{r frequentTerms, message=FALSE}
findFreqTerms(dtm, 15) #terms that appear 15 or more times, in this case
```




Remove some terms 
```{r }
sparseparam <- 0.90 # will make the matrix 90% empty space, maximum. Change this, as you like.
dtm_sprs <- removeSparseTerms(dtm,sparse=sparseparam)
inspect(dtm_sprs)
maintitle <-paste0("Most frequent terms (sparseness=" ,sparseparam , "  )")
barplot(as.matrix(dtm_sprs),xlab="terms",ylab="number of occurrences", main=maintitle)

# organize terms by their frequency 

freq_dtm_sprs <- colSums(as.matrix(dtm_sprs))
length(freq_dtm_sprs)
sorted_freq_dtm_sprs <- sort(freq_dtm_sprs, decreasing = TRUE)
sorted_freq_dtm_sprs
```

Create a data frame that will be the input to the classifier.
Last column will be the label. 

As data frame:

```{r message=FALSE,results='hide'}
#dtmdf <- as.data.frame(dtm.90)
#dtmdf <- as.data.frame(inspect(dtm_sprs))
dtmdf <- as.data.frame(as.matrix(dtm_sprs))
# rownames(dtm)<- 1:nrow(dtm)

class <- d$revBug
dtmdf <- cbind(dtmdf,class)
head(dtmdf, 3)
```

Use any classifier now:
  - split the dataframe into training and testing
  - Build the classification model using the training subset
  - apply the model to the testing subset and obtain the Confusion Matrix
  - Analise the results

```{r tmCaret, message=FALSE}
library(caret)
library(randomForest)


inTraining <- createDataPartition(dtmdf$class, p = .75, list = FALSE)
training <- dtmdf[ inTraining,]
testing  <- dtmdf[-inTraining,]

fitControl <- trainControl(## 5-fold CV
                           method = "repeatedcv",
                           number = 5,
                           ## repeated ten times
                           repeats = 5)


gbmFit1 <- train(class ~ ., data = training,
                 method = "gbm",
                 trControl = fitControl,
                 ## This last option is actually one
                 ## for gbm() that passes through
                 verbose = FALSE)

gbmFit1

# trellis.par.set(caretTheme())
# plot(gbmFit1)
# 
# trellis.par.set(caretTheme())
# plot(gbmFit1, metric = "Kappa")

head(predict(gbmFit1, testing, type = "prob"))

conf_mat <- confusionMatrix(testing$class, predict(gbmFit1, testing))
conf_mat
```

We may compute manually all derived variables from the Confusion Matrix. See Section -- with the description of the Confusion Matrix

```{r confmat, message=FALSE}
# str(conf_mat)
TruePositive <- conf_mat$table[1,1]
TruePositive
FalsePositive <- conf_mat$table[1,2]
FalsePositive
FalseNegative <- conf_mat$table[2,1]
FalseNegative
TrueNegative <- conf_mat$table[2,2]
TrueNegative

# Sum columns in the confusion matrix
ConditionPositive <- TruePositive + FalseNegative
ConditionNegative <- FalsePositive + TrueNegative
TotalPopulation <- ConditionPositive + ConditionNegative
TotalPopulation

#Sum rows in the confusion matrix
PredictedPositive <- TruePositive + FalsePositive
PredictedNegative <- FalseNegative + TrueNegative
# Total Predicted must be equal to the total population
PredictedPositive+PredictedNegative

SensitivityRecall_TPR <- TruePositive / ConditionPositive
SensitivityRecall_TPR

Specificity_TNR_SPC <- TrueNegative / ConditionNegative
Specificity_TNR_SPC

Precision_PPV <- TruePositive / PredictedPositive
Precision_PPV 

NegativePredictedValue_NPV <- TrueNegative / PredictedNegative
NegativePredictedValue_NPV

Prevalence <- ConditionPositive / TotalPopulation
Prevalence

Accuracy_ACC <- (TruePositive + TrueNegative) / TotalPopulation
Accuracy_ACC

FalseDiscoveryRate_FDR <- FalsePositive / PredictedPositive
FalseDiscoveryRate_FDR

FalseOmisionRate_FOR <- FalseNegative / PredictedNegative 
FalseOmisionRate_FOR

FallOut_FPR <- FalsePositive / ConditionNegative
FallOut_FPR

MissRate_FNR <- FalseNegative / ConditionPositive
MissRate_FNR 

```

And finally, a word cloud as an example that appears everywhere these days.

```{r WordCloud, message=FALSE}
library(wordcloud)

# calculate the frequency of words and sort in descending order.
wordFreqs=sort(colSums(as.matrix(dtm_sprs)),decreasing=TRUE)

wordcloud(words=names(wordFreqs),freq=wordFreqs)
```


## Extracting data from Twitter

The hardest bit is to link with Twitter. Using the TwitteR package is explained following this [example](./twitter.Rmd).