Recoding Variables

Currently mpg variable of mtcars is a continuous numeric variable, but it may be more useful if mpg was a categorical variable that immedietly told you if the car had low or high miles per gallon.

# This is useful so that if you make a mistake your original dataset stays in tact!
mtcars2 <- mtcars
# Assign the label "high" to mpgcategory where mpg is greater than or equal to 20
mtcars2$mpgcategory[mtcars$mpg >= 20] <- 'high'
# Assign the label "low" to mpgcategory where mpg is less than 20
mtcars2$mpgcategory[mtcars$mpg < 20] <- 'low'
# Assign mpgcategory as factor to mpgfactor
mtcars2$mpgcategory <- as.factor(mtcars2$mpgcategory)
head(mtcars2$mpgcategory)

## [1] high high high high low  low 
## Levels: high low

Table

Frequency tables show you how often a given value occurs.
table(): The top row of the output is the value, and the bottom row is the frequency of the value.

table(mtcars2$mpgcategory)

## 
## high  low 
##   14   18

Bar Plot: Categorical Variable

barplot(): Base function for bar plot.

• ylab = '*': Add a label to the y axis.
  
• names.arg = '*': X axis labels to the bars.

barplot(table(mtcars2$mpgcategory), ylab = 'Number of Cars', names.arg = c('High','Low'))

Histograms: Continuous Variable

hist(): Base function for histogram.

• main = '*': Title.
  
• ylim = c(*, *): Set the range of y axis.
  
• xlab = '*': Title for x axis.

hist(mtcars$carb, main = "Carburetors", ylim = c(0,20), xlab = 'Number of Carburetors', col = 'lightblue')

Sample Space

You have a bag containing three A and two E scrabble letters.     
Consider the samplespace of drawing three letters from the bag without replacing them(each trial is dependent).

samplespace <- c('AAA', 'AAE', 'AEA', 'EAA', 'EEA', 'EAE', 'AEE')

The sample space contains each combination of A and E,      
except for three E's because this is not possible with only two E letters.      

Dependent

In your scrabble letter bag you have three A's and two E's.           
You take one letter out at a time, and return each letter to the bag before continuing with the following draw.       

# What is the probability of AAE?
aae <- 3/5 * 3/5 * 2/5
# What is the probability of EAE?
eae <- 2/5 * 2/5 * 3/5
# What is the probability of AAA or EEA?
aaaeea <- 3/5 * 3/5 * 3/5 + 2/5 * 2/5 * 3/5

Independent

This time, you take one letter out of the bag at a time,          
but you do not return the leters to the bag before continuing with the following draw.        

# What is the probability of AAE?
aae <- 3/5 * 2/4 * 2/3
# What is the probability of EAE?
eae <- 2/5 * 3/4 * 1/3
# What is the probability of AAA or EEA?
aaaeea <- 3/5 * 2/4 * 1/3 + 2/5 * 1/4 * 3/3

Probability Terms

• Complement: 1 - x
  
• Independent intersecting events: Two events that do not influence each other and can occur similtaneously.
  
• Disjoint exhaustive events: Mutually exclusive, so only one of the events can happen at a time.

Union

P(A or B) = P(A) + P(B) - P(A and B)

The probability of picking a flower that is at least partly blue is 0.4. 
The probability of picking a flower that is at least partly pink is 0.2.
Q: alculate the probability of picking a flower that is blue, pink or both?

0.2 + 0.4 - (0.2 * 0.4)

## [1] 0.52

Conditional Probability

P(A|B) = P(A + B)/P(B)

'|' means 'given', under `B` condition, the probability that `A` will happen.    

Independence

P(A|B) = P(A)

The probability of one event is independent of another 
if the probability of the first event occuring is unaffected by whether or not the other event occurs.

Bayesian Probability

P(A|B) = (P(B|A) * P(A))/P(B)

Information about A is can help tell us whether B will happen or not.
Because any information you have will reduces the sample space of possible events that can occur. 

Example: Simulated Normal Distribution

• dnorm(): Gives the density.
  
• pnorm(): Gives the distribution function.
  
• qnorm(): Gives the quantile function.
  
• rnorm(): Generates random deviates.

To get a probability,

• You will need to consider an interval under the curve of the probability density function.
  
• Probabilities here are thus considered surface areas.

# Simulate some random normally distributed data
set.seed(11225)
data <- rnorm(10000)

# Calculate the density of data and store it in the variable density
density <- dnorm(data)

# Make a plot with as x variable data and as y variable density
plot(data, density)

Cumulative Probability

Q: Probability that x is 0, smaller/equal to 1, smaller/equal to 2, and smaller/equal to 3.   

# Using cumsum() for cumulative sum.
cumsum(data$probs[1:4])

## [1] 0.1 0.3 0.6 0.8

Expected Value - The Mean

• The mean of a probability distribution is calculated by taking the weighted average of all possible values that a random variable can take.
  
• In the case of a discrete variable, you calculate the sum of each possible value times its probability.

Q: Calculate the expected probability value.

expected_score <- sum(data$outcome * data$probs)
expected_score

## [1] 2.3

Variance & Standard Deviation

• Variance
  • The variance is often taken as a measure of spread of a distribution.
    
  • It is the sum of the squared difference between the individual observation and multiplied by their probabilities.
    
• Standard Deviation
  • Take variance’s square root.

Q: Calculate the variance and store it in a variable called variance.     

variance <- sum((data$outcome -expected_score)^2 * data$probs)
variance

## [1] 2.01

Q: Calculate the standard deviation and store it in a variable called std.

std <- sqrt(variance)
std

## [1] 1.417745

pnorm() & Cumulative Probability

Hair length is considered to be normally distributed with a mean of 25 cm and a standard deviation of 5. 
Q: The probability that a woman's hair length is less than 30.

pnorm(30, mean = 25, sd = 5)

## [1] 0.8413447

Q: The probability of a woman having a hair length larger or equal to 30 centimers.

pnorm(30, mean = 25, sd = 5, lower.tail = FALSE)

## [1] 0.1586553

Let’s visualize this.
Note that the first example is visualized on the left, while the second example is visualized on the right.

qnorm() & Quantile

Sometimes we have a probability that we want to associate with a value.     
This is basically the opposite situation as the situation described in the previous question.     
Q: The value of a woman's hair length that corresponds with the 0.2 quantile.

qnorm(0.2, mean = 25, sd = 5)

## [1] 20.79189

Standard Normal Distribution & Z-scores

• A special form of the normal probability distribution is the standard normal distribution, also known as the z-distribution.
  
• A z distribution has a mean of 0 and a standard deviation of 1.
  
• Transform to z-scores.
  • The values of a variable subtract the mean.
    
  • Divide this by the standard deviation.

A woman with a hair length of 38 centimers and the average hair length was 25 centimers and the standard deviation was 5 centimers.    
Q: Change to z-scores.    

round(((38 - 25) / 5), 1)

## [1] 2.6

dbinom(), pbinom() & qbinom()

• dbinom(): Calculates an exact probability.
• pbinom(): Calculate an interval of probabilities.
  
• qbinom(): Calculate the value that is associated with certain quantile.

Q: Probability of answering 5 questions correctly.

dbinom(x = 5, size = 25, prob = 0.2)

## [1] 0.1960151

Q: Probability of answering at least 5 questions correctly.

pbinom(q = 4, size = 25, prob = 0.2, lower.tail = F)

## [1] 0.5793257

Q: Calculate the 60th percentile.   

qbinom(p = 0.6, size = 25, prob = 0.2)

## [1] 5

Sampling from Population

In this section you can see that the more you’ve sampled, the closer value to the actual mean you’ll get.

library(ggplot2)
# Use sample(), randomly sample 200 diamonds' prices.    
sample_price <- sample(diamonds$price, size = 200)
mean(sample_price)

## [1] 3509.43

# Use for loop do many samples and compare their means.
# Empty vector sample means
sample_means <- NULL

# Take 200 samples from diamonds' prices.
for (i in 1:500){
  samp <- sample(diamonds$price, 200)
  sample_means[i] <- mean(samp)
}

# Calculate the prices mean, that is, the mean of diamonds' prices and print it.
mean(diamonds$price)

## [1] 3932.8

# Calculate the mean of the sample means, that is, sample_means.
mean(sample_means)

## [1] 3926.979

Variance & Standard Deviation of Population

The denominator is n when computing statistics of population.

Variance & Standard Deviation of Sampling

However, the denominator is n - 1 when computing statistics of sampling.
The functions in R base: var() & sd() are calculating statistics of sampling.

Standard Error of the Mean: Standard Deviation of Sampling

• The standard deviation measures the amount of variability or dispersion for a subject set of data from the mean.
  
• The standard error of the mean measures how far the sample mean of the data is likely to be from the true population mean.
  
• The SEM is always smaller than the SD.

Sample Size: Safe Approach

This example work on proportion:
For safe approach p = 0.5 which is the value above which the output p*(1-p) cannot get any larger.

Now you're conducting a study on what proportion of students drink alcohol and want to know what sample size to use for a confidence interval of `95%`, with a margin of error of `0.05`.

# Assign the value of p(1-p) to object "p"
p <- 0.5*0.5
# Assign the value of the Z-score to new object "z"
z <- 1.96
# Assign the value of the margin of error squared to the new object "ms"
ms <-0.0025
# Calculate the neccessary sample size
p * z^2 / ms

## [1] 384.16

One-sided

The red area is considered the rejection region when we are doing a one-sided hypothesis where the alternative hypothesis checks whether the population mean of the beard length of scandinavian hipsters is larger than 25 millimeters.

# Calculate the value of cut_off
qnorm(0.95, mean = 25, sd = 0.55)

## [1] 25.90467

25.90 is the starting value of the rejection region.
Consider our example mentioned above with a mean beard length of 25 and a standard error of 0.55.

Imagine that we found a sample mean of 25.95 with a sample size of 40. 
Whether this result is significant depends on the test we do. 
If we would do a one-sided hypothesis test against a 5% significance level, 
we would only have to test for the effect in one direction.

# Calculate the z score and assign it to a variable called z_value
z_value <- round((25.95 - 25) / round(3.5 / sqrt(40), 2), 2)
z_value

## [1] 1.73

# Calculate the corresponding p value and store it in the variable called p_value
# We set the lower tail equal to FALSE 
# because pnorm() calculates the cumulative probability until the value of 1.81 
# and we want to know the probability of finding a value of 1.81 or larger.
pnorm(z_value, lower.tail = F)

## [1] 0.04181514

See that the p value is smaller than 0.05?
When testing against a significance level of 0.05, we have actually found a significant result.

Two-tailed

The red area is considered the rejection region when we are doing a two-tailed hypothesis test.   
This corresponds to the alternative hypothesis which checks whether the population mean of the beard length of scandinavian hipsters is not equal to 25 millimeters. As you can see the 0.05 probability is divided into two chunks of 0.025.   

# Calculate the value of the variable lower_cut_off
round(qnorm(0.025, mean = 25, sd = 0.55), 2)

## [1] 23.92

# Calculate the value of the variable upper_cut_off
round(qnorm(0.975, mean = 25, sd = 0.55), 2)

## [1] 26.08

23.92 & 26.08 indicate the start of the rejection region.
Consider our example mentioned above with a mean beard length of 25 and a standard error of 0.55.

If we would however do a two-sided hypothesis test, we should not only look for P(>z_value). 
In this case we should test for both P(>z_value) & (<-z_value).
Since the Z distribution we are working with is symmetric, we could multiply the outcome by 2.

# Calculate the z score and assign it to a variable called z_value
z_value <- round((25.95 - 25) / round(3.5 / sqrt(40), 2), 2)
z_value

## [1] 1.73

# Calculate the corresponding p value and store it in the variable called p_value
pnorm(z_value, lower.tail = F) * 2

## [1] 0.08363028

This time, p value is larger than 0.05.
In this case would work reject the null hypothesis if we would do a one-sided hypothesis test.
However, we would fail to reject it if we would do a two-sided hypothesis test.

Conclusion of Example Question

By checking whether the confidence interval contains our population mean,
we are essentially doing the same as testing a hypothesis.
For instance, we started out with assuming the population mean of the height of Dutch males is 185 centimers.
This could be our null hypothesis.
Our alternative hypothesis could be the fact that the Dutch male height is different from 185 centimeters.
After testing, we would reject the null hypothesis because the confidence interval does not contain the population mean.