We often come across the need to compare samples of two data sets to derive some conclusions about their similarity. Some common examples include assessing similarity of groups of users based on their purchase history, assessing similarity of servers based on resource utilization, or assessing similarity of applications based on their usage behavior, among others.
But how does one measure similarity? Let’s find out!
To present the concepts, we will compare three pairs of data sets. These pairs are created such that one pair is very similar, another pair is partially similar, and the third pair is very dissimilar. Through these three pairs, you will see how to measure the similarity of such sets.
Data summary stats
One of the most common ways to measure similarity oftwosets is to compare their data summary via mean and median. Figure 1 shows two graphsthatcompare the means and medians of thethreepairs ofdata sets respectively. As we know, thethreepairs have different levels of similarity but that is not exactly reflected in their mean or median values. Here, the data spread around their mean and median seems very similar.Using these graphs, you cannot separatethe mean and medianofthe very similar pairs from the dissimilar pairs. By just looking at the graphs, you wouldobservethat all thethreepairs are very similar, but you know that is not the case! Thus, only measuring the similarity based on mean and/or median can be deceptive.
Visualization tools
Another commonly used method to measure data similarity is using various visualization tools such as box plots, histograms, andcumulative distribution functions(CDFs).
Statistical significance tests
While visualization tools are a very expressive way to understand the data, they seldom can be scaled to large data sets. Given hundreds of data sets, comparing them by visualizing charts is an impractical approach.Hence, there is an imperative need to have some mathematical formulations to measure the similarity ofdata sets.Also, note that in many scenariosthedataof the entire population is either unknown or difficult to obtain. In such cases, one needs to work with a sampleofthepopulation.This is where statistical significance testscanhelp.
Student’s t-test
One of the popular statistical significance testsis theStudent’s t-test.An independent two-sample t-test is used to compare the means of two independent groups.The t-value is computed as follows:
This equation looks complicated, but it has a very simple explanation. It captures the signal to noise ratio. The numerator represents the signal, i.e., the difference in mean of two sets. The denominator represents the noise, i.e., total variance of the two groups. In other words, the denominator represents how scattered the data population is to trust the difference in the means. Heavily scattered data population indicates high noise. A high t-value indicates high signal and low noise. Similarly, a t-value closer to 0 indicates that the data sets are very similar without having a significant difference.
The t-value is often computed on a sample of the actual population. When you compute the t-value on another random sample of the same population, the t-value is likely to be a bit different. If you compute the t-value on thousands of such samples, you will end up with thousands of such t-values. Now, the question arises about which t-value to use and how large should the t-value of any random sample be to trust that the two populations are different. This is where p-value comes to help.
For a given pair of data-samples, Student’s t-test computes the t-value and a p-value. The t-value assesses the difference in the mean of two samples. The p-value provides the probability of getting a t-value at extreme ends in any random sample. In other words, t-value measures how different two samples are by measuring signal to noise ratio. The p-value measures how often we could observe such a signal to noise ratio.
Figure 5 compares the t-value and p-value of the three pairs of the data sets.
Different types of t-test
There are different variations of the Student’s t-test. For simplicity, we limited the discussion above to the independent two-sample t-test. Here are three different variations of this test:
The example presented in Figure 5 refers to the independent two-sample t-test. This test works under the assumption that there is no relationship between the two groups, the data in each group is approximately normally distributed and the two groups are obtained via random sampling from the population.
Hypothesis testing
Statistical significance tests belong to the wider class of tests called hypothesis testing. Hypothesis testing is a statistical method used to determine if a certain hypothesis is true or not based on the available sample data.
Other commonly used statistical significance tests
Conclusion
In this blog we explored how to define similarity between two groups. We often face such scenarios even in our day-to-day lives. For example, in clinical trials you can use these tests to determine similarities between different medicines. These tests can also be used in businesses to determine whether a new marketing campaign increased sales or not.
Statistical significance tests or hypothesis testing are some of the most important topics in statistics that are widely used in real-world scenarios and across many sectors such as finance, industries, healthcare, research, and economics. They are even used to determine the strength of the predictions of machine learning algorithms. Thus, statistical significance tests have a very significant usage in real life.
Written by Uday Chandra
