From 134d5e743a7e0909c82ee1af87a6812fd4a2b2f0 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Wed, 29 May 2024 12:54:01 +0900 Subject: [PATCH 01/13] probably a typo --- EDA.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/EDA.qmd b/EDA.qmd index d41f46863..e18c83a0d 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -73,7 +73,7 @@ You can see variation easily in real life; if you measure any continuous variabl This is true even if you measure quantities that are constant, like the speed of light. Each of your measurements will include a small amount of error that varies from measurement to measurement. Variables can also vary if you measure across different subjects (e.g., the eye colors of different people) or at different times (e.g., the energy levels of an electron at different moments). -Every variable has its own pattern of variation, which can reveal interesting information about how that it varies between measurements on the same observation as well as across observations. +Every variable has its own pattern of variation, which can reveal interesting information about how it varies between measurements on the same observation as well as across observations. The best way to understand that pattern is to visualize the distribution of the variable's values, which you've learned about in @sec-data-visualization. We'll start our exploration by visualizing the distribution of weights (`carat`) of \~54,000 diamonds from the `diamonds` dataset. From 319ac91aad74157a99cf0ca894ff79424c734935 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Wed, 29 May 2024 17:04:20 +0900 Subject: [PATCH 02/13] `y` denotes width, and xlim is an argument. --- EDA.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index e18c83a0d..60f019bfd 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -161,7 +161,7 @@ The only evidence of outliers is the unusually wide limits on the x-axis. ```{r} #| fig-alt: | -#| A histogram of lengths of diamonds. The x-axis ranges from 0 to 60 and +#| A histogram of widths of diamonds. The x-axis ranges from 0 to 60 and #| the y-axis ranges from 0 to 12000. There is a peak around 5, and the #| data appear to be completely clustered around the peak. @@ -174,7 +174,7 @@ To make it easy to see the unusual values, we need to zoom to small values of th ```{r} #| fig-alt: | -#| A histogram of lengths of diamonds. The x-axis ranges from 0 to 60 and the +#| A histogram of widths of diamonds. The x-axis ranges from 0 to 60 and the #| y-axis ranges from 0 to 50. There is a peak around 5, and the data #| appear to be completely clustered around the peak. Other than those data, #| there is one bin at 0 with a height of about 8, one a little over 30 with @@ -185,7 +185,7 @@ ggplot(diamonds, aes(x = y)) + coord_cartesian(ylim = c(0, 50)) ``` -`coord_cartesian()` also has an `xlim()` argument for when you need to zoom into the x-axis. +`coord_cartesian()` also has an `xlim` argument for when you need to zoom into the x-axis. ggplot2 also has `xlim()` and `ylim()` functions that work slightly differently: they throw away the data outside the limits. This allows us to see that there are three unusual values: 0, \~30, and \~60. From 61ecccc7fe19a2a9f746cdac1492629f9324f4c3 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Wed, 29 May 2024 17:20:08 +0900 Subject: [PATCH 03/13] make it clearer to be missing, and `y` is width. --- EDA.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index 60f019bfd..71162acbb 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -212,10 +212,10 @@ options(old) ``` The `y` variable measures one of the three dimensions of these diamonds, in mm. -We know that diamonds can't have a width of 0mm, so these values must be incorrect. +We know that diamonds can't have a width of 0mm, so these must be missing values. By doing EDA, we have discovered missing data that was coded as 0, which we never would have found by simply searching for `NA`s. Going forward we might choose to re-code these values as `NA`s in order to prevent misleading calculations. -We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch long, but don't cost hundreds of thousands of dollars! +We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch wide, but don't cost hundreds of thousands of dollars! It's good practice to repeat your analysis with and without the outliers. If they have minimal effect on the results, and you can't figure out why they're there, it's reasonable to omit them, and move on. From 5418951eed485d18bd552b81e998e0961e57a2fa Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Wed, 29 May 2024 20:35:43 +0900 Subject: [PATCH 04/13] probably careless mistakes --- EDA.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index 71162acbb..cb87cc68c 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -299,7 +299,7 @@ You can do this by making a new variable, using `is.na()` to check if `dep_time` #| fig-alt: | #| A frequency polygon of scheduled departure times of flights. Two lines #| represent flights that are cancelled and not cancelled. The x-axis ranges -#| from 0 to 25 minutes and the y-axis ranges from 0 to 10000. The number of +#| from 0 to 25 hours and the y-axis ranges from 0 to 10000. The number of #| flights not cancelled are much higher than those cancelled. nycflights13::flights |> @@ -326,7 +326,7 @@ In the next section we'll explore some techniques for improving this comparison. 3. Recreate the frequency plot of `scheduled_dep_time` colored by whether the flight was cancelled or not. Also facet by the `cancelled` variable. - Experiment with different values of the `scales` variable in the faceting function to mitigate the effect of more non-cancelled flights than cancelled flights. + Experiment with different values of the `scales` argument in the faceting function to mitigate the effect of more non-cancelled flights than cancelled flights. ## Covariation From 9bbf7f9820b3a68a0a6aa92594ae81433a699524 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Wed, 29 May 2024 20:45:46 +0900 Subject: [PATCH 05/13] combine two sections of unsual values into one --- EDA.qmd | 44 ++++++++++++++++++++------------------------ 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index cb87cc68c..2e7261ca7 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -151,7 +151,7 @@ Some of these questions can be answered with the data while some will require do Many of them will prompt you to explore a relationship *between* variables, for example, to see if the values of one variable can explain the behavior of another variable. We'll get to that shortly. -### Unusual values +### Unusual values {#sec-unusual-values-eda} Outliers are observations that are unusual; data points that don't seem to fit the pattern. Sometimes outliers are data entry errors, sometimes they are simply values at the extremes that happened to be observed in this data collection, and other times they suggest important new discoveries. @@ -222,26 +222,6 @@ If they have minimal effect on the results, and you can't figure out why they're However, if they have a substantial effect on your results, you shouldn't drop them without justification. You'll need to figure out what caused them (e.g., a data entry error) and disclose that you removed them in your write-up. -### Exercises - -1. Explore the distribution of each of the `x`, `y`, and `z` variables in `diamonds`. - What do you learn? - Think about a diamond and how you might decide which dimension is the length, width, and depth. - -2. Explore the distribution of `price`. - Do you discover anything unusual or surprising? - (Hint: Carefully think about the `binwidth` and make sure you try a wide range of values.) - -3. How many diamonds are 0.99 carat? - How many are 1 carat? - What do you think is the cause of the difference? - -4. Compare and contrast `coord_cartesian()` vs. `xlim()` or `ylim()` when zooming in on a histogram. - What happens if you leave `binwidth` unset? - What happens if you try and zoom so only half a bar shows? - -## Unusual values {#sec-unusual-values-eda} - If you've encountered unusual values in your dataset, and simply want to move on to the rest of your analysis, you have two options. 1. Drop the entire row with the strange values: @@ -318,13 +298,29 @@ In the next section we'll explore some techniques for improving this comparison. ### Exercises -1. What happens to missing values in a histogram? +1. Explore the distribution of each of the `x`, `y`, and `z` variables in `diamonds`. + What do you learn? + Think about a diamond and how you might decide which dimension is the length, width, and depth. + +2. Explore the distribution of `price`. + Do you discover anything unusual or surprising? + (Hint: Carefully think about the `binwidth` and make sure you try a wide range of values.) + +3. How many diamonds are 0.99 carat? + How many are 1 carat? + What do you think is the cause of the difference? + +4. Compare and contrast `coord_cartesian()` vs. `xlim()` or `ylim()` when zooming in on a histogram. + What happens if you leave `binwidth` unset? + What happens if you try and zoom so only half a bar shows? + +5. What happens to missing values in a histogram? What happens to missing values in a bar chart? Why is there a difference in how missing values are handled in histograms and bar charts? -2. What does `na.rm = TRUE` do in `mean()` and `sum()`? +6. What does `na.rm = TRUE` do in `mean()` and `sum()`? -3. Recreate the frequency plot of `scheduled_dep_time` colored by whether the flight was cancelled or not. +7. Recreate the frequency plot of `scheduled_dep_time` colored by whether the flight was cancelled or not. Also facet by the `cancelled` variable. Experiment with different values of the `scales` argument in the faceting function to mitigate the effect of more non-cancelled flights than cancelled flights. From 7711b0b51457b57d9a12c2275b9ec7a0ebd3f7e5 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Wed, 29 May 2024 20:55:17 +0900 Subject: [PATCH 06/13] divide exercises --- EDA.qmd | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index 2e7261ca7..b9f7646f3 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -151,6 +151,20 @@ Some of these questions can be answered with the data while some will require do Many of them will prompt you to explore a relationship *between* variables, for example, to see if the values of one variable can explain the behavior of another variable. We'll get to that shortly. +#### Exercises + +1. Explore the distribution of each of the `x`, `y`, and `z` variables in `diamonds`. + What do you learn? + Think about a diamond and how you might decide which dimension is the length, width, and depth. + +2. Explore the distribution of `price`. + Do you discover anything unusual or surprising? + (Hint: Carefully think about the `binwidth` and make sure you try a wide range of values.) + +3. How many diamonds are 0.99 carat? + How many are 1 carat? + What do you think is the cause of the difference? + ### Unusual values {#sec-unusual-values-eda} Outliers are observations that are unusual; data points that don't seem to fit the pattern. @@ -296,31 +310,19 @@ nycflights13::flights |> However this plot isn't great because there are many more non-cancelled flights than cancelled flights. In the next section we'll explore some techniques for improving this comparison. -### Exercises - -1. Explore the distribution of each of the `x`, `y`, and `z` variables in `diamonds`. - What do you learn? - Think about a diamond and how you might decide which dimension is the length, width, and depth. - -2. Explore the distribution of `price`. - Do you discover anything unusual or surprising? - (Hint: Carefully think about the `binwidth` and make sure you try a wide range of values.) - -3. How many diamonds are 0.99 carat? - How many are 1 carat? - What do you think is the cause of the difference? +#### Exercises -4. Compare and contrast `coord_cartesian()` vs. `xlim()` or `ylim()` when zooming in on a histogram. +1. Compare and contrast `coord_cartesian()` vs. `xlim()` or `ylim()` when zooming in on a histogram. What happens if you leave `binwidth` unset? What happens if you try and zoom so only half a bar shows? -5. What happens to missing values in a histogram? +2. What happens to missing values in a histogram? What happens to missing values in a bar chart? Why is there a difference in how missing values are handled in histograms and bar charts? -6. What does `na.rm = TRUE` do in `mean()` and `sum()`? +3. What does `na.rm = TRUE` do in `mean()` and `sum()`? -7. Recreate the frequency plot of `scheduled_dep_time` colored by whether the flight was cancelled or not. +4. Recreate the frequency plot of `scheduled_dep_time` colored by whether the flight was cancelled or not. Also facet by the `cancelled` variable. Experiment with different values of the `scales` argument in the faceting function to mitigate the effect of more non-cancelled flights than cancelled flights. From 5396bc146942e2b9d91ee7f299131d7c24bc886d Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Thu, 30 May 2024 09:02:19 +0900 Subject: [PATCH 07/13] probably careless mistakes --- EDA.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index b9f7646f3..3a5aa1656 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -338,9 +338,9 @@ For example, let's explore how the price of a diamond varies with its quality (m ```{r} #| fig-alt: | -#| A frequency polygon of prices of diamonds where each cut of carat (Fair, +#| A frequency polygon of prices of diamonds where each cut of diamond (Fair, #| Good, Very Good, Premium, and Ideal) is represented with a different color -#| line. The x-axis ranges from 0 to 30000 and the y-axis ranges from 0 to +#| line. The x-axis ranges from 0 to 20000 and the y-axis ranges from 0 to #| 5000. The lines overlap a great deal, suggesting similar frequency #| distributions of prices of diamonds. One notable feature is that #| Ideal diamonds have the highest peak around 1500. From 86ac42297a28a34bc0e57d6fa648dabac6c734a2 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Thu, 30 May 2024 09:51:14 +0900 Subject: [PATCH 08/13] probably a careless mistake --- EDA.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/EDA.qmd b/EDA.qmd index 3a5aa1656..4c5536500 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -360,7 +360,7 @@ Instead of displaying count, we'll display the **density**, which is the count s ```{r} #| fig-alt: | #| A frequency polygon of densities of prices of diamonds where each cut of -#| carat (Fair, Good, Very Good, Premium, and Ideal) is represented with a +#| diamond (Fair, Good, Very Good, Premium, and Ideal) is represented with a #| different color line. The x-axis ranges from 0 to 20000. The lines overlap #| a great deal, suggesting similar density distributions of prices of #| diamonds. One notable feature is that all but Fair diamonds have high peaks From 8d5ae7cd075b460f4d5ae05f3fa534db10d52030 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Thu, 30 May 2024 15:57:04 +0900 Subject: [PATCH 09/13] a typo, and some points are smaller than 1000. --- EDA.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index 4c5536500..e834ed7bb 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -468,9 +468,9 @@ One way to do that is to rely on the built-in `geom_count()`: #| fig-alt: | #| A scatterplot of color vs. cut of diamonds. There is one point for each #| combination of levels of cut (Fair, Good, Very Good, Premium, and Ideal) -#| and color (D, E, F, G, G, I, and J). The sizes of the points represent +#| and color (D, E, F, G, H, I, and J). The sizes of the points represent #| the number of observations for that combination. The legend indicates -#| that these sizes range between 1000 and 4000. +#| that these sizes mainly range between 1000 and 4000. ggplot(diamonds, aes(x = cut, y = color)) + geom_count() From 564f4b42080f51db81af3b8e31b689a01b502583 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Thu, 30 May 2024 16:19:28 +0900 Subject: [PATCH 10/13] a careless mistake --- EDA.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/EDA.qmd b/EDA.qmd index e834ed7bb..666a831cf 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -494,7 +494,7 @@ Then visualize with `geom_tile()` and the fill aesthetic: #| cut/color combination and tiles are colored according to the number of #| observations in each tile. There are more Ideal diamonds than other cuts, #| with the highest number being Ideal diamonds with color G. Fair diamonds -#| and diamonds with color I are the lowest in frequency. +#| and diamonds with color J are the lowest in frequency. diamonds |> count(color, cut) |> From df07ab881c21918fc3acabbc081f3a761f29815b Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Thu, 30 May 2024 18:29:14 +0900 Subject: [PATCH 11/13] same notation of color and cut --- EDA.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/EDA.qmd b/EDA.qmd index 666a831cf..b7b5a2b81 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -509,7 +509,7 @@ For larger plots, you might want to try the heatmaply package, which creates int 1. How could you rescale the count dataset above to more clearly show the distribution of cut within color, or color within cut? -2. What different data insights do you get with a segmented bar chart if color is mapped to the `x` aesthetic and `cut` is mapped to the `fill` aesthetic? +2. What different data insights do you get with a segmented bar chart if `color` is mapped to the `x` aesthetic and `cut` is mapped to the `fill` aesthetic? Calculate the counts that fall into each of the segments. 3. Use `geom_tile()` together with dplyr to explore how average flight departure delays vary by destination and month of year. From 351590f013e52063410487035d73022678d6d196 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Thu, 30 May 2024 19:48:19 +0900 Subject: [PATCH 12/13] a typo --- EDA.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/EDA.qmd b/EDA.qmd index b7b5a2b81..01a63a67f 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -543,7 +543,7 @@ You've already seen one way to fix the problem: using the `alpha` aesthetic to a #| fig-alt: | #| A scatterplot of price vs. carat. The relationship is positive, somewhat #| strong, and exponential. The points are transparent, showing clusters where -#| the number of points is higher than other areas, The most obvious clusters +#| the number of points is higher than other areas. The most obvious clusters #| are for diamonds with 1, 1.5, and 2 carats. ggplot(smaller, aes(x = carat, y = price)) + From 1b4562dbc2d830bacdf1165ba2ed390345ca86e7 Mon Sep 17 00:00:00 2001 From: Mitsuo Shiota Date: Thu, 30 May 2024 20:59:05 +0900 Subject: [PATCH 13/13] a careless mistake, and "quite similar" contradicts with the sentence above. --- EDA.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/EDA.qmd b/EDA.qmd index 01a63a67f..14b198273 100644 --- a/EDA.qmd +++ b/EDA.qmd @@ -694,8 +694,8 @@ Once you've removed the strong relationship between carat and price, you can see ```{r} #| fig-alt: | #| Side-by-side box plots of residuals by cut. The x-axis displays the various -#| cuts (Fair to Ideal), the y-axis ranges from 0 to almost 5. The medians are -#| quite similar, between roughly 0.75 to 1.25. Each of the distributions of +#| cuts (Fair to Ideal), the y-axis ranges from 0 to almost 4. The medians are +#| between roughly 0.75 to 1.25, and increase in better cuts. Each of the distributions of #| residuals is right skewed, with many outliers on the higher end. ggplot(diamonds_aug, aes(x = cut, y = .resid)) +