---
title: "Statistical Literacy Project Template"
author: "Ms-RAZ"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

#Statistical Literacy Data Analysis Project 

In this RMarkdown (that's what this kind of file is called), we will walk through a dataset of Spotify's Top 50 tracks for the year 2019. Most of this code should not need to be modified. Just run the line of code to follow along.  

####A Few Tips to Get Started

* To run a line of code, you can highlight it or just put the cursor on the line you want to run and hit:  
"Command + Return" for Macs or "Ctrl + Enter" for PCs. 

* R is case sensitive, so "This" is not the same as "this" in R's eyes. 

* When you get an error, copy and paste that error directly into Google. Website likes Stackoverflow.com will typically pop up. More times than not, other people have encountered the same error and someone has already answered it.

* The data for this example from https://www.kaggle.com/leonardopena/top50spotify2019. You can find your own Spotify data with this cool tool. https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/ This link gives you more info about what each variable means. You can also get the info on any songs you want! That could let you learn about what music you like.  

###Pre-Steps: Installing Packages

There are countless "packages" in the R community that help you do all sorts of nifty thing. They can save you a huge amount of time because someone else already coded all of the complicated functions. For this class we will install two packages-- but you are more than welcome to install any packages you want! When you install packages, you only have to run this code one time (unless you update your R version, but don't worry about that). 
```{r}
# Install packages using the "install.packages()" function
# You only need to do this one time
install.packages("tidyverse")
install.packages("psych")

# Once they are installed, you can then load packages so you can use them
# You have to do this every time you reopen R
library(tidyverse)
library(psych)
```

###Step 1: Loading the Data

When coding, the first thing you need to know is where you are in your computer. When you tell R to look for something, it is going to look in the place you are in. This might not be the place the file is in. It's like looking for a file in the wrong filing cabinet. So, first let's see where you are in your computer. 
```{r}
# This line of code means "GET the Working Directory", other words, tell me where I am in my computer -- 
# and this is called a "comment", I can make a comment by using the pound symbol #
getwd() 

# Ideally, you've created your working Project/Directory already so your files of code and datasets will all be in the same folder and you will not have to mess with your working directory or specify file paths. 
```

When I run the line above, I get this: "/Users/RAZ/Desktop/My Stats Lit Class/My R Stuff". That is telling me I am in the folder where I saved my R Project/Directory. Hopefully, you've already downloaded the datasets and code from Canvas and saved them in this folder, "My R Stuff". Now we can load the data into R using the function, "read.csv" which reads in .csv files. We will read it in and assign it a name. 

```{r}
# The "<-" symbol is used to assign a name to something
spotify_data <- read.csv("spotify_top50.csv")

# If for some reason you have data elsewhere, you can tell R exactly where it is, but if you work in the correct Project/Directory you don't have to worry about this
spotify_data <- read.csv("/Users/RAZ/Desktop/My Stats Lit Class/My R Stuff/spotify_top50.csv")
```

Now I have a object in my environment called "spotify_data" which has 50 observations (rows, in this case different songs) and 13 variables (columns). Click on it in the "Global Environment" in the top right to look at the data. This will open up a viewer that looks like Excel. You can scroll through and look around at what we have in the dataset. Two other useful functions for looking at the data quickly to see what kind of variables we are dealing with are "head()" and "names()". Functions tell R to do something on an object. The object in this case is the dataset. Functions have a name and a set of parenthesis where you give it an "argument". The argument is the specifics, like the dataset you want to look at. For example, names(spotify_data) is telling R to give us the names of the spotify dataset. If ever you are unsure what a function does you can type a question mark at the beginning of the empty command, and R will pull up a help menu. 

```{r}
# Shows first few observations (rows) and columns
head(spotify_data) 

# Shows you the name of the columns
names(spotify_data) 

# Brings up help menu for the function in the lower right in RStudio
?names() 
```

###Step 2: Investigating the Data  

We can ask R to show us everything in one column using the "$" sign. It tells R you want a particular column. Let's look at all the AristNames in the Top 50. Notice how R will even bring up a list of all the possible columns. This can be very helpful and help prevent typos. 

```{r}
spotify_data$ArtistName
```

Let's look at the data in the Global Environment window (in the top right), click the little blue drop down icon next to the "spotify_data". This will show you all the names of the variables (columns) and you will be able to see what kind of data each variable is. We can see that the TrackName variable is a "chr" short for "character" which means this is variables is just words/names. ArtistName and Genre are also classified as character variables. The rest of the variables are all "int" short for "integer", which are whole numbers (discrete data). All the integer variables are correct, however, we need the ArtistName and Genre to be treated like categories (R calls them "factors) so we can work with that data later. We can change the "class" of a variable a few different ways in R. 

```{r}
# We can CHECK the class (i.e. data type) of any variable using the "class()" function
class(spotify_data$ArtistName)
class(spotify_data$Genre)

# Sometimes R does not read in the data the way we want
# In this case, I want R to treat the ArtistName and Genre as categories
# Categories in R are called "factors" 

# To CHANGE the class to a factor, we can use the function "as.factor()"
spotify_data$ArtistName <- as.factor(spotify_data$ArtistName)
spotify_data$Genre <- as.factor(spotify_data$Genre)

# This is a fancy way to change more than one column at a time, but the simplier code above is fine for our purposes
spotify_data[,2:3] <- lapply(spotify_data[,2:3], factor)
```

Now that we have our data in R with the correct classes, we can start playing with it.  
Let's look at just one column in the dataset more deeply, Popularity. I'll introduce a few different R functions. 

```{r}
# We can ask R to just give us the entire Popularity column from our "spotify_data" by using the "$" symbol. 
spotify_data$Popularity

# What is the RANGE of song popularity? 
range(spotify_data$Popularity)
# The Popularity scores for the Top 50 songs ranges from 70 to 95

#We can also ask R to give us the MAXimum value of Popularity 
max(spotify_data$Popularity)
# The highest level of popularity for the Top 50 songs was 95
# Unfortunetly, when we use this function, it only gives us the value,
# but we can't see which particular song each value belongs to for that we need to use a different function 
```

#####Which is the most popular song?  
As with all coding, there are a number of ways to do this, but here's one way. 
```{r}
#This code looks abut intimidating, so let's walk through some of it.
spotify_data[which.max(spotify_data$Popularity),]

# First we specify our dataset "spotify_data" and add a set of square brackets used for what is called "Indexing"
# Brackets are used to "index", basically telling R to grab certain rows and/or columns 
# The first number on the LEFT is for ROWS and the second letter on the RIGHT is the COLUMNS and a comma seperates them
# [rows, columns]

# For example, if you type the following line of code, you will get the first row of data and all the columns
# When you leave the right side empty, R assumes you want all of the columns 
spotify_data[1,]

# If you instead type the line of code below, R will give you the first column ("TrackName") and all the rows
# When you leave the left side empty, R assumes you want all of the rows
spotify_data[,1]

# So, in this code below, we are asking R for a certain row and all the columns because we have something written on the left of the comma (asking for a certain row) and nothing to the right (asking for all columns)
# But what are we asking for?
spotify_data[which.max(spotify_data$Popularity),]

# We want to know which song had the highest (maximum) value of Popularity
# So we are asking R to give us the row with the maximum value of Popularity. 
# The command "which.max" looks for the situation (here a row) that answers that question
# If you were to type only the line below, R will tell you which row number that is which is row 10
which.max(spotify_data$Popularity)

# But we don't want to have to go look up in the data which song row 10 is... We want R to give us all the info in the entire row. 
# For this, we use the indexing trick and the "which.max" command to get the full answer
# Rather than asking for a certain row by number, we are asking R to give us the row that is has the max Popularity 
spotify_data[which.max(spotify_data$Popularity),]

# This same trick can be used to find the least popular songs, too, by changing "which.max" to "which.min"
spotify_data[which.min(spotify_data$Popularity),]

# which() is a nifty function we can use in lots of ways to figure out "which" observation (row) answers a particular question

# Another way to do something similiar is with a function called top_n() from a library called "tidyverse", however this function does not sort them into order of Popularity
# To get the Top 3 songs by Popularity 
top_n(spotify_data, 3, wt = spotify_data$Popularity)

# We can use the same function to get the bottom 3, too, by adding a minus sign (-3)
top_n(spotify_data, -3, wt = spotify_data$Popularity)
```

**On Spotify for 2019 the most popular song was Billie Eilish's "bad guy" which a popularity rating of 95.**  

#####How many different musical genres are represented in the Top 50 Spotify songs?  

You could try to count how many different musical genres there are, but that would take a while, would be prone to errors, and would be impossible with larger data sets. Instead we can try a few different functions.  
```{r}
# You could use the function "levels()" to see all of the distinct factor "levels" there are, i.e. categories 
levels(spotify_data$Genre)

# But levels() only shows you all the levels. 
# What if I just want to know how many factor levels are there? How many categories? 
# We can use another nifty function, nlevels()
nlevels(spotify_data$Genre)
```

**There are 21 different music genres represented in the 2019 Top 50 Spotify songs.**

#####Which music genre is most common in the Top 50 songs for 2019?

You could try to count how many times you see "electropop" and other genres, but, again, that would take a while, would be prone to errors, and would be impossible with larger datasets. Instead we can create a table of the different levels (categories) which shows how many times each level was observered in the dataset. 

```{r}
# Using the "table()" function, R will create a table for the categorical variable. 
# Basically, it shows you how many distinct categories (in this case Genres) there are and tallies how many times each one comes up
table(spotify_data$Genre)

# You can sort a table by putting the "sort()" function around the table function 
# It sorts the categories by frequency
sort(table(spotify_data$Genre))
```

**The most common music Genre for the 2019 Top 50 songs on Spotify was "dance pop".**

#####Did any Artists have more than one Top 50 hit? If so, how many? Who had the most? 
```{r}
# Like before we can use the nifty table function, but again, it is hard to look through it all of the tallies 
table(spotify_data$ArtistName)

# Instead, we can save this table , but this time we are going to make it into a "dataframe". There are different types of objects in R, and I encourage you to explore them in more depth, but for now I won't explain more than just saying, dataframe play nicely and are easier to use. The function "as.data.frame()", takes anything you create and turns it into a dataframe. We will make a table like before using "table()", sort it so make it easier to look at, but now we will put those functions into the argument for as.data.frame and save it as a new object called "ArtistTable"
ArtistTable <- as.data.frame(sort(table(spotify_data$ArtistName)))

#Take a quick look at what you created by cicking the new object in the Global Environment in the top right

# Unfortunetly, as.data.frame sometimes names things in funny ways. So now, instead of "ArtistName", R renamed it "Var1". The frequency (i.e. number of Top 50 hits an artist had) column is called "Freq". There are ways to rename things, but for now, it is ok to just use the new names if we need them. Remeber we can check the names using the function names()
names(ArtistTable)

# We now have a new object that tells us the Artist and the number of Top 50 hits they had in 2019. First let's answer the questions 1) Did any Artists have more than one Top 50 hit? and 2) If so, how many?
# We want to know if the Freq for any person is higher than one, i.e. their name showed up more than once so they had more than one Top 50 hit. Here, I am asking R to sum up the number of Artists who have a Frequency greater than 1 (i.e. they had more than 1 hit)
sum(ArtistTable$Freq > 1)

#Looks like there are 10 artists who had more than 1 Top 50 hit in 2019. But which artists were they?
#Again using our "which()" function, we can ask R to tell us which artists had more than 1 Top 50 hit.
ArtistTable[which(ArtistTable$Freq > 1),]
```

**Yes, there are some artists with more than 1 Top 50 hit in 2019. There were 10 artists total. Ed Sheeran had 4 hits and Ariana Grande, Billie Eilish, J Balvin, Lil Nas X, Marshmello, Post Malone, Sech, Shawn Mendes, and The Chainsmokers each had 2 hits.**

#####What does the distribution in the Duration look like? 

```{r}
# R has some amazing plotting tools (like a package called "ggplot2" that are far beyond the scope of this class-- you should check them out! For now, we will use some simple base R plotting tools. 

# Let's make a histogram
hist(spotify_data$Duration)

# Let's make our histogram a little prettier by adding labels
hist(spotify_data$Duration,
     main = "Distribution of Song Duration for Top 50 Songs", #adds a title
     xlab = "Song Duration in Seconds", #names the x axis
     ylab = "Frequency") #names the y axis
```

#####Which artist had the longest song in 2019 and which artist has the shortest song in 2019?
```{r}
# We can use the which.max and which.min function again

spotify_data[which.max(spotify_data$Duration),]
spotify_data[which.min(spotify_data$Duration),]
```

**No me Conoce - Remix by Jhay Cortez was the longest song (309 seconds) and Panini by Lil Nas X was the shortest song (115 seconds).**

#####What is the average Duration (variable is in seconds) of the Top 50 songs? 
```{r}
# To calculate a mean, you could sum all the song Durations and divide by the number of songs 

Length_Sum <- sum(spotify_data$Duration) # Sums up all the song durations
Number_of_Song <- length(spotify_data$Duration) # Gets the "length" of the column which is the number of rows (50)
average_song_duration <- Length_Sum/Number_of_Song # Divides the Length_Sum by the Number_of_Songs to get the average

# But there is also a much easier way to do this in R. R has a vast variety of functions built in. 
# The built in function for R to calculate the mean is just called mean()
# This gives the same value as the longer code above
mean(spotify_data$Duration)
```

**The average song duration for the Top 50 Spotify songs of 2019 is 200.96 seconds (or about 3 minutes).**

#####How much does the song Duration vary? What is the standard deviation? 
```{r}
# Again, we could calculate this the long way or use a function. However, R assumes that you are working with a sample and not the population, so it divides the sum of squares by N-1 rather than by N. Technically, we are using population data here bc we have the Top 50 songs. We did not take a sample of the Top 50 songs. But for the sake of your project (and most real world applications you might apply these skills to), we can use the built in functions that treat the data as though it were from a sample. 

# To get the variance for a sample we use var()
# But remember, variance is in squared terms... It is not directly interpretable. 
var(spotify_data$Duration)

# To get the variation in a scale we can understand, we need it in the same scale as the variable Duration, i.e. seconds, which is what the standard deviation is
# We can use sd() to get the standard deviation of our "sample"
sd(spotify_data$Duration)
```

**The average about of variation (standard deviation) in song Duration is about 39 seconds. This means that on average the songs vary from one another by about 39 seconds.**

#####What is the correlation between Loudness_dB (the actually decibal level) and Energy (a measure of intensity, ex. death metal would have high energy)? 
```{r}
# You can get the correlation for any two quantiative ("num" or "int" in R) variables by using the function cor()
# You put the two variables of interest seperated by a comma into the function
cor(spotify_data$Loudness_dB, spotify_data$Energy)

# We can also plot this as a scatterplot, remember scatterplot require two quantitative variables
plot(spotify_data$Loudness_dB, spotify_data$Energy,
     main = "Scatterplot of Loudness and Energy",
     xlab = "Loudness", 
     ylab = "Energy")
```

**Loudness and Energy are correlated moderate to strongly at r = .67.** 

#####What are the correlations between the different variables? 
```{r}
# When you first start exploring data, you may not know which variables correlate. You can create an entire "matrix" of correlations using the same function, but now we tell it the data and use that indexing trick to tell it to only run a correlation on the numeric (quantiative) variables which are the 4th through the 13th columns
cor(spotify_data[,4:13])

# What happens if you try to run cor on all of the data, which includes catergorical data?
# Try running the code below, but first delete the "#" sign in from of it
cor(spotify_data)

# You will get an error that says "x" must be numeric. Here TrackName, ArtistName, and Genre are not numeric data, they are categories (in R called factors)
```

##Step 3: Statistical Testing

#####Are the Genre's "dance" and "dance pop" statistically significantly different from each other in terms of Popularity?

Now we will be getting into some statistical tests. R has many different statistical tests, but for now we will focus on an independent samples t-test to compare two categories (dance vs dance pop). 
```{r}
# This data set is quite small (only 50 observations). Data sets in the "real world" can be incredibly large with millions (even billions--or more??-- for "Big Data") of rows. Our two groups for our t-test here have very small sample sizes. We can check the sample size various ways. You can look back at the Genre table you made above or... 

# You can ask R to sum up the number of times a certain Genre appears in the data set
# "sum()" is the same as before but now we are specifying to R to only count those observations in the Genre column that are equal to a certain type

sum(spotify_data$Genre == "pop") # Sum up all of the "pop" songs
sum(spotify_data$Genre == "dance pop") # Sum up all of the "dance pop" songs

# We only have 7 and 8 observations respectively, but for now, it's ok for our example.
```

####Step 3a: Create two groups from the data.  

For this question we are going to create two objects one for "pop" and one for "dance pop". Though base R can do this, a package called "tidyverse" makes this easier. Let's walk through the steps of installing a package and loading the library that comes from that package. 
```{r}
# We already loaded our "tidyverse" library at the beginning of this file, but it doesn't hurt to load it again

library(tidyverse)

# Here we are telling R to "filter()" our spotify_data, specifically we are looking for observations where the Genre is equal to "pop"
pop <- filter(spotify_data, spotify_data$Genre == "pop")
dance_pop <- filter(spotify_data, spotify_data$Genre == "dance pop")

# If you look in your Global Environment you will see two new objects, "pop" and "dance pop". We can also make sure it correctly filtered bc we know that we calculated above that "pop" had n = 7 observations and "dance pop" had n = 8 observations, which is the same number of rows in the new objects. 
```

####Step 3b: Looks at the descriptive statistics for each group. 
```{r}
# Now we are going to use our other package we installed, "psych"
library(psych)

# The describe() function gets us all of the interesting descriptive statistics 
# Because the first 3 columns are not quantititaive data (they are characters and factors), I am only asking R to get the descriptive statistics for the numeric data, columns 4 through 13
describe(pop[,4:13])
describe(dance_pop[,4:13])

# You can also use describeBy() to get descriptive statisitics for different categories. We can use it the same way as describe where we first tell it the data, then we specify which variable is the grouping variable. In this case we are using the music Genre as the grouping variable. (This data set has very few observations for each group, but with larger datasets, this is a handy tool.)
describeBy(spotify_data, group = spotify_data$Genre)
```

Both groups have very similiar descriptive statistics. Look at Popularity, for example. The "pop" Genre has an average popularity of 85.86 while "dance pop" has an average of 85.75. We are testing if the difference in populatiry is statistically signficant between the two groups: 85.86 - 85.75 = 0.11 Do you think out t-test which is testing for a difference between "pop" and "dance pop" will be significant?

####Step 3c: Run the t-test.
```{r}
# R has its own t-test function ready to go, t.test()
# We need to give it the data from each of our groups and tell it which variable (column) we want to test, ex. pop$Popularity
# We can also specify the level of confidence (default is .95), if it is a one or two tailed test (default is two tailed) among other things. By default, R assumes the variance between the groups is NOT equal. In our example, both groups have exactly equal variance, which is usually rare. We can tell R to do a regular "Two Sample t-test" by telling it that "var.equal = TRUE", the default is "var.equal = FALSE". We will run both to see the differences. 

# Two-Sample t-test assuming equal variance between the groups 
t.test(pop$Popularity, dance_pop$Popularity, var.equal = TRUE)

# Welch Two-Sample t-test assuming unequal variance between the groups 
t.test(pop$Popularity, dance_pop$Popularity, var.equal = FALSE) 
```

Let's first look at the Two-Sample t-test assuming equal variance. R spits out a few different things to pay attention to. First, we see a t value, "t = 0.058944". Remember that the null hypothesis t value is t = 0. Notice how close our t-value is to 0. Next we see our degrees of freedom (df) which is equal to 13 (7 + 8 - 2) = 13. Next we can see the p-value,  p-value = 0.9539. It's huge. This means that there is about a 95% chance of seeing this tiny difference even if there was no *real* statistically significant different between the groups. We would expect to see this little difference under the null 95% of the time. This is VERY insignificant. We can also get to the same conclusion using the 95 percent confidence interval: -3.819776, 4.034062. What this is saying is that the true difference between "pop" and "dance pop" is somewhere between -3.8 points and +4.0 points. Meaning "pop" could have 3.8 points less than "dance pop" or 4.0 points more than dance pop. Definetly not significant. 

Now if we look at the Welch t-test, we see that the number are slightly different, but we still come to the same conclusion. Again, the difference here is that Welch does NOT assume equal variance. (This is R's default.) This is also why we have degrees of freedom that are not whole numbers. In practice, it is important to select the appropriate test based on your data, but for now, you can assume the variances are equal and stick with t.test(group1, group2, var.equal = FALSE) when running t-tests. 

**Based on the Independent Samples t-test (assuming equal variance), there is no statistically signficant difference between "pop" and "dance pop" in Popularity. t(13) = 0.06, p = 0.95. We fail to reject the null hypothesis, 95% CI [-3.82  4.03].**