Visualization II

Relationships Between Quantitative Variables¶

Up until now, we’ve discussed how to visualize single-variable distributions. Going beyond this, we want to understand the relationship between variables (i.e., bivariate or multivariate distributions)

The following diagram is a high-level overview of univariate, bivariate, and multivariate distributions.

Illustration of what plots are appropriate based on the variables being used.

Scatter Plots¶

Scatter plots are one of the most useful tools in representing the relationship between pairs of quantitative variables. They are particularly important in gauging the strength, or correlation, of the relationship between variables. Knowledge of these relationships can then motivate decisions in our modeling process.

In matplotlib, we use the function plt.scatter to generate a scatter plot. Notice that, in the example below (using data from the World Bank dataset wb) unlike our examples of plotting single-variable distributions, now we specify sequences of values to be plotted along the x-axis and the y-axis.

import polars as pl
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# The file's first column is an unnamed row label, which we drop
wb = pl.read_csv("data/world_bank.csv").drop("")
wb = wb.rename({'Antiretroviral therapy coverage: % of people living with HIV: 2015':"HIV rate",
                'Gross national income per capita, Atlas method: $: 2016':'gni'})
wb.head()

plt.scatter(wb["per capita: % growth: 2016"], \
            wb['Adult literacy rate: Female: % ages 15 and older: 2005-14'])

plt.xlabel("% growth per capita")
plt.ylabel("Female adult literacy rate")
plt.title("Female adult literacy against % growth");

In seaborn, we call the function sns.scatterplot. We use the x and y parameters to indicate the values to be plotted along the x and y axes, respectively. By using the hue parameter, we can specify a third variable to be used for coloring each scatter point.

sns.scatterplot(data = wb, x = "per capita: % growth: 2016", \
               y = "Adult literacy rate: Female: % ages 15 and older: 2005-14", 
               hue = "Continent")

plt.title("Female adult literacy against % growth");

Overplotting¶

Although the plots above communicate the general relationship between the two plotted variables, they both suffer a major limitation – overplotting. Overplotting occurs when scatter points with similar values are stacked on top of one another, making it difficult to see the number of scatter points actually plotted in the visualization. Notice how in the upper righthand region of the plots, we cannot easily tell just how many points have been plotted. This makes our visualizations difficult to interpret.

We have a few methods to help reduce overplotting:

Decreasing the size of the scatter point markers can improve readability. We do this by setting a new value to the size parameter, s, of plt.scatter or sns.scatterplot.
Jittering is the process of adding a small amount of random noise to all x and y values to slightly shift the position of each datapoint. By randomly shifting all the data by some small distance, we can discern individual points more clearly without modifying the major trends of the original dataset.

In the cell below, we first jitter the data using np.random.uniform, then re-plot it with smaller markers. The resulting plot is much easier to interpret.

# Setting a seed ensures that we produce the same plot each time
# This means that the course notes will not change each time you access them
np.random.seed(150)

# This call to np.random.uniform generates random numbers between -1 and 1
# We add these random numbers to the original x data to jitter it slightly
x_noise = np.random.uniform(-1, 1, len(wb))
jittered_x = wb["per capita: % growth: 2016"] + x_noise

# Repeat for y data
y_noise = np.random.uniform(-5, 5, len(wb))
jittered_y = wb["Adult literacy rate: Female: % ages 15 and older: 2005-14"] + y_noise

# Setting the size parameter `s` changes the size of each point
plt.scatter(jittered_x, jittered_y, s=15)

plt.xlabel("% growth per capita (jittered)")
plt.ylabel("Female adult literacy rate (jittered)")
plt.title("Female adult literacy against % growth");

`lmplot` and `jointplot`¶

seaborn also includes several built-in functions for creating more sophisticated scatter plots. Two of the most commonly used examples are sns.lmplot and sns.jointplot.

sns.lmplot plots both a scatter plot and a linear regression line, all in one function call. We’ll discuss linear regression in a few lectures.

sns.lmplot(data = wb, x = "per capita: % growth: 2016", \
           y = "Adult literacy rate: Female: % ages 15 and older: 2005-14")

plt.title("Female adult literacy against % growth");

sns.jointplot creates a visualization with three components: a scatter plot, a histogram of the distribution of x values, and a histogram of the distribution of y values.

sns.jointplot(data = wb, x = "per capita: % growth: 2016", \
           y = "Adult literacy rate: Female: % ages 15 and older: 2005-14")

# plt.suptitle allows us to shift the title up so it does not overlap with the histogram
plt.suptitle("Female adult literacy against % growth")
plt.subplots_adjust(top=0.9);

Hex plots¶

For datasets with a very large number of datapoints, jittering is unlikely to fully resolve the issue of overplotting. In these cases, we can attempt to visualize our data by its density, rather than displaying each individual datapoint.

Hex plots can be thought of as two-dimensional histograms that show the joint distribution between two variables. This is particularly useful when working with very dense data. In a hex plot, the x-y plane is binned into hexagons. Hexagons that are darker in color indicate a greater density of data – that is, there are more data points that lie in the region enclosed by the hexagon.

We can generate a hex plot using sns.jointplot modified with the kind parameter.

sns.jointplot(data = wb, x = "per capita: % growth: 2016", \
              y = "Adult literacy rate: Female: % ages 15 and older: 2005-14", \
              kind = "hex")

# plt.suptitle allows us to shift the title up so it does not overlap with the histogram
plt.suptitle("Female adult literacy against % growth")
plt.subplots_adjust(top=0.9);

Contour Plots¶

Contour plots are an alternative way of plotting the joint distribution of two variables. A contour plot can be interpreted in a similar way to a topographic map. Each contour line represents an area that has the same density of datapoints throughout the region. Contours marked with darker colors contain more datapoints (a higher density) in that region.

sns.kdeplot will generate a contour plot if we specify both x and y data.

sns.kdeplot(data = wb, x = "per capita: % growth: 2016", \
            y = "Adult literacy rate: Female: % ages 15 and older: 2005-14", \
            fill = True)

plt.title("Female adult literacy against % growth");

Transformations¶

We have now covered visualizations in great depth, looking into various forms of visualizations, plotting libraries, and high-level theory.

Much of this was done to uncover insights in data, which will prove necessary when we begin building models of data later in the course. A strong graphical correlation between two variables hints at an underlying relationship that we may want to study in greater detail. However, relying on visual relationships alone is limiting - not all plots show association. The presence of outliers and other statistical anomalies makes it hard to interpret data.

Transformations are the process of manipulating data to find significant relationships between variables. These are often found by applying mathematical functions to variables that “transform” their range of possible values and highlight some previously hidden associations between data.

To see why we may want to transform data, consider the following plot of population density against gross national income per capita.

# Some data cleaning to help with the next example
df = wb.select(
    pl.col("Population density: people per sq. km: 2016").alias("density"),
    pl.col("gni"),
).drop_nulls().filter(
    (pl.col("density") > 0) & (pl.col("gni") > 0)
)

plt.scatter(df["density"], df["gni"])

plt.xlabel("Population density: people per sq. km")
plt.ylabel("Gross national income per capita")
plt.title("Gross national income per capita against population density");

This plot is difficult to interpret for two reasons:

The data shown in the visualization appears almost “smushed” – it is heavily concentrated in the lower lefthand region of the plot. A few very large values stretch both the horizontal and vertical axes.
It is hard to generalize a clear relationship between the two plotted variables. We are not able to describe the specifics of this trend in much detail.

A transformation would allow us to visualize this data more clearly, which, in turn, would enable us to describe the underlying relationship between our variables of interest.

We will most commonly apply a transformation to linearize a relationship between variables. If we find a transformation to make a scatter plot of two variables linear, we can “backtrack” to find the exact relationship between the variables. This helps us in two major ways. Firstly, linear relationships are particularly simple to interpret – we have an intuitive sense of what the slope and intercept of a linear trend represent, and how they can help us understand the relationship between two variables. Secondly, linear relationships are the backbone of linear models. We will begin exploring linear modeling in great detail next week. As we’ll soon see, linear models become much more effective when we are working with linearized data.

Linearization and Applying Transformations¶

To linearize a relationship, begin by asking yourself: what makes the data non-linear? It is helpful to repeat this question for each variable in your visualization.

Let’s start by considering the gross national income per capita variable in our plot above. It sits on the vertical axis, and looking along it we can see that most countries are crushed against the bottom while a handful of very large incomes stretch the axis upwards. The scale of the vertical axis is also being distorted by the few large outlying y values.

If we decreased the size of these outliers relative to the bulk of the data, we could reduce the distortion of the vertical axis. How can we do this? We need a transformation that will:

Decrease the magnitude of large y values by a significant amount.
Not drastically change the magnitude of small y values.

One function that produces this result is the log transformation. When we take the logarithm of a large number, the original number will decrease in magnitude dramatically. Conversely, when we take the logarithm of a small number, the original number does not change its value by as significant of an amount (to illustrate this, consider the difference between $\log{(100)} = 4.61$ and $\log{(10)} = 2.3$ ).

In Data 100 (and most upper-division STEM classes), $\log$ is used to refer to the natural logarithm with base $e$ .

# np.log takes the logarithm of an array or Series
plt.scatter(df["density"], np.log(df["gni"]))

plt.xlabel("Population density: people per sq. km")
plt.ylabel("Log(gross national income per capita)")
plt.title("Log(GNI per capita) against population density");

After taking the logarithm of our y values, our plot appears much more balanced in its vertical scale. We no longer have many datapoints clumped on one end and a few outliers out at extreme values.

Let’s repeat this reasoning for the x values. Considering only the horizontal axis of the plot, notice how there are many datapoints concentrated at small x values. Only a few datapoints lie at much larger values of x.

If we were to “squish” these large values of x more, we would no longer see the dense concentration in one region of the x-axis. We need a transformation that will:

Decrease the magnitude of large values of x so these datapoints are distributed more broadly on the horizontal scale,
Not substantially alter the scaling of small values of x.

We can again apply a log transformation. Large values will decrease in magnitude proportionally much more than small values.

# Apply a log transformation to both the x and y values
plt.scatter(np.log(df["density"]), np.log(df["gni"]))

plt.xlabel("Log(population density: people per sq. km)")
plt.ylabel("Log(gross national income per capita)")
plt.suptitle("Log(GNI per capita) against Log(population density)")
plt.subplots_adjust(top=0.9);

Our scatter plot is looking a lot better! Now, we are plotting the log of our original x values on the horizontal axis, and the log of our original y values on the vertical axis. The transformations have reduced the influence of the large values that previously distorted our axes.

Let’s now consider another example where a transformation reveals an approximate linear relationship. Consider the relationship between the percentage of 15-year-olds at the lowest level of math proficiency and gross national income per capita.

What can we take away from this? After taking the log of gross national income per capita, the two variables are roughly linearly related. If we denote the original math proficiency values as $x$ and the original, untransformed gross national income per capita values as $y$ , we can use the standard form of a linear fit to express this relationship:

\log(y) = mx + b

Where $m$ represents the slope of the linear fit, while $b$ represents the intercept.

The cell below computes $m$ and $b$ for our transformed data. We’ll discuss how this code was generated in a future lecture.

# The code below fits a linear regression model. We'll discuss it at length in a future lecture
from sklearn.linear_model import LinearRegression

model_df = wb.select(
    pl.col(
        "Students at lowest proficiency on PISA: Mathematics: % of 15 year-olds: 2015"
    ).alias("math"),
    pl.col("gni"),
).drop_nulls().filter(pl.col("gni") > 0)

model = LinearRegression()
model.fit(model_df[["math"]], np.log(model_df["gni"]))
m, b = model.coef_[0], model.intercept_

print(f"The slope, m, of the transformed data is: {m}")
print(f"The intercept, b, of the transformed data is: {b}")

model_df = model_df.sort("math")

plt.scatter(
    model_df["math"],
    np.log(model_df["gni"]),
    label="Transformed data"
)
plt.plot(
    model_df["math"],
    m * model_df["math"] + b,
    c="red",
    label="Linear regression"
)
plt.xlabel("% of 15-year-olds at lowest math proficiency")
plt.ylabel("Log(gross national income per capita)")
plt.legend();

The slope, m, of the transformed data is: -0.04080981365001037
The intercept, b, of the transformed data is: 10.416008267765228

What if we want to understand the underlying relationship between our original variables, before they were transformed? We can simply rearrange our linear expression above!

Recall our linear relationship between the original x values and the transformed y values.

\log(y) = mx + b

By rearranging the equation, we find a relationship between the untransformed variables $x$ and $y$ .

y = e^{mx+b} = e^{mx}e^b

When we plug in the values for $m$ and $b$ computed above, something interesting happens.

# Now, plug the values for m and b into the relationship between the untransformed x and y
plt.scatter(
    model_df["math"],
    model_df["gni"],
    label="Untransformed data"
)
plt.plot(
    model_df["math"],
    np.exp(m * model_df["math"] + b),
    c="red",
    label="Modeled relationship"
)
plt.xlabel("% of 15-year-olds at lowest math proficiency")
plt.ylabel("Gross national income per capita")
plt.legend();

# Now, plug the values for m and b into the relationship between the untransformed x and y
plt.scatter(
    model_df["math"],
    model_df["gni"],
    label="Untransformed data"
)
plt.plot(
    model_df["math"],
    np.exp(m * model_df["math"] + b),
    c="red",
    label="Modeled relationship"
)
plt.xlabel("% of 15-year-olds at lowest math proficiency")
plt.ylabel("Gross national income per capita")
plt.legend();

We have found a relationship between our original variables – math proficiency and gross national income per capita!

Transformations are powerful tools for understanding our data in greater detail. To summarize what we just achieved:

We identified appropriate transformations to linearize the original data.
We used our knowledge of linear curves to compute the slope and intercept of the transformed data.
We used this slope and intercept information to derive a relationship in the untransformed data.

Linearization will be an important tool as we begin our work on linear modeling next week.

Additional Remarks¶

Visualization requires a lot of thought!

There are many tools for visualizing distributions.
- Distribution of a single variable:
  1. Rugplot
  2. Histogram
  3. Density plot
  4. Box plot
  5. Violin plot
- Joint distribution of two quantitative variables:
  1. Scatter plot
  2. Hex plot
  3. Contour plot

This class primarily uses seaborn and matplotlib. Many other visualization libraries exist, and plotly is one of them.

plotly makes it very easily creates interactive plots.
plotly will occasionally appear in lecture code, labs, and assignments!

Next, we’ll go deeper into the theory behind visualization.

Visualization Theory¶

This section marks a pivot to the second major topic of this lecture - visualization theory. We’ll discuss the abstract nature of visualizations and analyze how they convey information.

Remember, we had two goals for visualizing data. This section is particularly important in:

Helping us understand the data and results,
Communicating our results and conclusions with others.

Information Channels¶

Visualizations are able to convey information through various encodings. In the remainder of this lecture, we’ll look at the use of color, scale, and depth, to name a few.

Encodings in Rugplots¶

One detail that we may have overlooked in our earlier discussion of rugplots is the importance of encodings. Rugplots are effective visuals because they utilize line thickness to encode frequency. Consider the following diagram:

A dataframe is shown with an arrow pointing to lines, one per row labeled Mark (represents a datum). From there, another arrow points to values like 10x, 15px, and NONE labeled Encoding (Maps datum to visual position). Finally, a last arrow points to a rug plot.

Multi-Dimensional Encodings¶

Encodings are also useful for representing multi-dimensional data. Notice how the following visual highlights four distinct “dimensions” of data:

X-axis
Y-axis
Area
Color

Example plot using four dimensions of the data.

The human visual perception system is only capable of visualizing data in a three-dimensional plane, but as you’ve seen, we can encode many more channels of information.

Harnessing X|Y¶

Consider the Scale of the Data¶

However, we should be careful to not misrepresent relationships in our data by manipulating the scale or axes. The visualization below improperly portrays two seemingly independent relationships on the same plot. The authors have clearly changed the scale of the y-axis to mislead their audience.

Two lines representing two differnt things, cancer screenings and abortions are shown overlapping with each other.

Notice how the downwards-facing line segment contains values in the millions, while the upwards-trending segment only contains values near three hundred thousand. These lines should not be intersecting.

When there is a large difference in the magnitude of the data, it’s advised to analyze percentages instead of counts. The following diagrams correctly display the trends in cancer screening and abortion rates.

Now you can see that the cancer screen line is way above the line for abortions

Another way to visualize the percent change in the number of procedures, this time with a bar chart.

Reveal the Data¶

Great visualizations not only consider the scale of the data but also utilize the axes in a way that best conveys information. For example, data scientists commonly set certain axes limits to highlight parts of the visualization they are most interested in.

Line plot showing COVID-19 cases from March 2020 to April 2021.

The same data is shown, this time zoomed in on March 2020 to May 2020.

The visualization on the right captures the trend in coronavirus cases during March of 2020. From only looking at the visualization on the left, a viewer may incorrectly believe that coronavirus began to skyrocket on March 4^th^, 2020. However, the second illustration tells a different story - cases rose closer to March 21^th^, 2020.

Harnessing Color¶

Color is another important feature in visualizations that does more than what meets the eye.

We already explored using color to encode a categorical variable in our scatter plot. Let’s now discuss the uses of color in novel visualizations like colormaps and heatmaps.

5-8% of the world is red-green color blind, so we have to be very particular about our color scheme. We want to make these as accessible as possible. Choosing a set of colors that work together is evidently a challenging task!

Colormaps¶

Colormaps are mappings from pixel data to color values, and they’re often used to highlight distinct parts of an image. Let’s investigate a few properties of colormaps.

Jet Colormap:

Viridis Colormap

The jet colormap is infamous for being misleading. While it seems more vibrant than viridis, the aggressive colors poorly encode numerical data. To understand why, let’s analyze the following images.

Four different color palettes are used to show the same four sets of data.

The first diagram compares how a variety of colormaps represent pixel data that transitions from a high to low intensity. These include the jet colormap (row a) and grayscale (row b). Notice how the grayscale images do the best job in smoothly transitioning between pixel data. The jet colormap is the worst at this - the four images in row (a) look like a conglomeration of individual colors.

The difference is also evident in the images labeled (a) and (b) in the first diagram. The grayscale image is better at preserving finer detail in the vertical line strokes. Additionally, grayscale is preferred in X-ray scans for being more neutral. The intensity of the dark red color in the jet colormap is frightening and indicates something is wrong.

Why is the jet colormap so much worse? The answer lies in how its color composition is perceived to the human eye.

Jet Colormap Perception

Jet colormap is perceived as a piecewise linear function.

Viridis Colormap Perception

Viridis colormap is perceived as a straight line.

The jet colormap is largely misleading because it is not perceptually uniform. Perceptually uniform colormaps have the property that if the pixel data goes from 0.1 to 0.2, the perceptual change is the same as when the data goes from 0.8 to 0.9.

Notice how the said uniformity is present within the linear trend displayed in the viridis colormap. On the other hand, the jet colormap is largely non-linear - this is precisely why it’s considered a worse colormap.

Sequential and Diverging Colormaps¶

There are different types of colormaps that we can use depending on what we want to communicate with our quantitative data.

A sequential colormap is useful when our data progresses from low to high. The color scheme should similarly progress from light to dark or dark to light. For example, a heatmap of quantitative values across months and years can use increasingly intense colors to represent increasingly large values.

A diverging colormap is useful when both low and high values deserve equal emphasis. These colormaps typically use lighter colors for middle values and increasingly distinct colors towards either extreme. For example, a map displaying change around a meaningful middle value can use contrasting colors for changes in opposite directions, with lighter colors for values near the middle.

As always, the colormap we choose should reflect the information we want our visualization to communicate.

Harnessing Markings¶

In our earlier discussion of multi-dimensional encodings, we analyzed a scatter plot with four pseudo-dimensions: the two axes, area, and color. Were these appropriate to use? The following diagram analyzes how well the human eye can distinguish between these “markings”.

There are a few key takeaways from this diagram

Lengths are easy to discern. Don’t use plots with jiggled baselines - keep everything axis-aligned.
Avoid pie charts! Angle judgments are inaccurate.
Areas and volumes are hard to distinguish (area charts, word clouds, etc.).

Harnessing Conditioning¶

Conditioning is the process of comparing data that belong to separate groups. We’ve seen this before in overlaid distributions, side-by-side box plots, and scatter plots with categorical encodings. Here, we’ll introduce terminology that formalizes these examples.

Consider an example where we want to analyze income earnings for males and females with varying levels of education. There are multiple ways to compare this data.

Bar plot of earnings separated by gender.

Scatterplot of earnings separated by gender.

The barplot is an example of juxtaposition: placing multiple plots side by side, with the same scale. The scatter plot is an example of superposition: placing multiple density curves and scatter plots on top of each other.

Which is better depends on the problem at hand. Here, superposition makes the precise wage difference very clear from a quick glance. However, many sophisticated plots convey information that favors the use of juxtaposition. Below is one example.

Plot example with the geographic representation of the United States repeated multiple times.

Harnessing Context¶

The last component of a great visualization is perhaps the most critical - the use of context. Adding informative titles, axis labels, and descriptive captions are all best practices that we’ve heard repeatedly in Data 8.

A publication-ready plot (and every Data 100 plot) needs:

Informative title (takeaway, not description),
Axis labels,
Reference lines, markers, etc,
Legends, if appropriate,
Captions that describe data,

Captions should:

Be comprehensive and self-contained,
Describe what has been graphed,
Draw attention to important features,
Describe conclusions drawn from graphs.

Examples of Bad Visualization¶

After reviewing many good visualization practices, it is also helpful to examine examples of how improper use of visualization elements can negatively affect interpretation.

Bad visualization showing candidates with party support.

In this example, color is used poorly, making it difficult to distinguish and compare the “Yes” category with the other “No” category. In addition, the category ordering is not clearly aligned, which makes it harder for viewers to quickly understand the results.

Bad visualization number of month since the start of 2017 and the highway vehicle miles traveled for all systems.

In this example, “Months since the start of 2017” makes it difficult for readers to interpret the timeline. In addition, the scientific notation used on the y-axis is hard to read and may be confusing for a general audience.

Bad visualization showing density of Mother's single years of age.

In this example, the data appear to be missing because the histogram bins are set inappropriately, creating the impression that values are missing when this is unlikely to be the case.

Relationships Between Quantitative Variables¶

Scatter Plots¶

Overplotting¶

lmplot and jointplot¶

Hex plots¶

Contour Plots¶

Transformations¶

Linearization and Applying Transformations¶

Additional Remarks¶

Visualization Theory¶

Information Channels¶

Encodings in Rugplots¶

Multi-Dimensional Encodings¶

Harnessing X|Y¶

Consider the Scale of the Data¶

Reveal the Data¶

Harnessing Color¶

Colormaps¶

Sequential and Diverging Colormaps¶

Harnessing Markings¶

Harnessing Conditioning¶

Harnessing Context¶

Examples of Bad Visualization¶

`lmplot` and `jointplot`¶