Exploratory Data Analysis (EDA)

Introduction to Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is a critical initial step in the data analysis process that involves the exploration, visualization, and summary of data to uncover patterns, trends, anomalies, and insights.

Key Objectives of EDA:

• Understand the Data: EDA helps you get a comprehensive understanding of your dataset, including its structure, size, and the variables it contains.

• Detect Patterns and Relationships: EDA aims to uncover relationships and patterns within the data, such as correlations between variables, trends over time, and clusters of similar data points.

• Identify Anomalies and Outliers: EDA helps you spot data points that deviate significantly from the norm, which could be errors or noteworthy observations.

• Prepare Data for Modeling: EDA assists in data preprocessing by revealing data quality issues, missing values, and helping with feature engineering.

Common Techniques in EDA:

• Descriptive Statistics: Calculating summary statistics like mean, median, variance, and quartiles to understand the central tendency and variability of your data.

• Data Visualization: Creating charts, graphs, and plots to visualize data distributions, relationships between variables, and trends. Common visualization tools include histograms, scatter plots, bar charts, and box plots.

• Correlation Analysis: Examining correlations between variables to understand how they are related.

• Outlier Detection: Identifying outliers using various methods, such as the Z-score, the IQR (Interquartile Range), or visualization techniques.

Descriptive Statistics

In data analysis, statistics are important for summarizing and understanding the data. Depending on whether the data attributes are discrete or continuous, different statistics are calculated:

Measures of Central Tendency:

In data analysis, measures of central tendency help us understand the central or typical value within a dataset. They provide insights into where the data tends to cluster. There are three primary measures of central tendency: the mean, the median, and the mode.

Mean:

The mean is often referred to as the average. It's calculated by adding up all the values in a dataset and then dividing by the number of data points. Here's the advantage and limitation of using the mean:

Advantage of the Mean:

• The mean can be used for both continuous and discrete numeric data.

Limitations of the Mean:

• The mean cannot be calculated for categorical data because the values cannot be summed.
• The mean is not robust against outliers, meaning that a single large value (an outlier) can significantly skew the average.

import pandas as pd

# Create a DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie'],
        'Age': [25, 30, 35],
        'Salary': [70000, 80000, 90000]}
df = pd.DataFrame(data)

# Mean of the Age and Salary columns
print(df[['Age', 'Salary']].mean())

# Output:
# Age          30.0
# Salary    80000.0
# dtype: float64

Median:

The median is the middle value in a dataset when it's ordered from smallest to largest. The median is less affected by outliers and skewed data, making it a preferred measure of central tendency when the data distribution is not symmetrical.

2 2 5 6 7 8 9

# Median of the Age and Salary columns
print(df[['Age', 'Salary']].median())

# Output:
# Age          30.0
# Salary    80000.0
# dtype: float64

Mode:

The mode is the value that occurs most frequently in a dataset.

2 2 5 6 7 8 9

Advantage of the Mode:

• Unlike the mean and median, which are mainly used for numeric data, the mode can be found for both numerical and categorical (non-numeric) data.

Limitation of the Mode:

• In some distributions, the mode may not reflect the center of the distribution very well, especially if the data is multimodal (has multiple modes) or if all values occur with similar frequencies.

import pandas as pd

# Create a DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie', 'Julia'],
        'Age': [25, 30, 35, 25],
        'Salary': [70000, 80000, 90000, 40000]}
df = pd.DataFrame(data)

# Mode of the Age column
print(df['Age'].mode())

# Output:
# 0    25
# Name: Age, dtype: int64

Measures of Dispersion:

Measures of dispersion provide insights into how data values are spread out or vary within a dataset. Common measures of dispersion include the range, variance, standard deviation.

Range:

The range is the simplest measure of dispersion. It's calculated by subtracting the minimum value from the maximum value in the dataset. While it's easy to compute, it's sensitive to extreme values (outliers) and may not provide a complete picture of data variability.

import pandas as pd

# Create a DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie', 'Julia'],
        'Age': [25, 30, 35, 25],
        'Salary': [70000, 80000, 90000, 40000]}
df = pd.DataFrame(data)

# Range of the Age column
print(df['Age'].max() - df['Age'].min())

# Output:
# 10

Variance and Standard Deviation:

Variance is a measure that quantifies how far each data point is from the mean.

Standard deviation is the square root of the variance and provides a measure of dispersion in the same units as the data.

These measures give us a more detailed understanding of how data points deviate from the mean. Higher variance and standard deviation values indicate greater data spread.

import pandas as pd

# Create a DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie', 'Julia'],
        'Age': [25, 30, 35, 25],
        'Salary': [70000, 80000, 90000, 40000]}
df = pd.DataFrame(data)

# Variance of the Age column
print(df['Age'].var())

# Output:
# 22.916666666666668

# Standard deviation of the Age column
print(df['Age'].std())

# Output:
# 4.7871355387816905

Data Distribution:

The shape of a data distribution can significantly affect the choice of measures of central tendency:

Normal or Symmetrical Distribution:

A normal distribution is a specific type of distribution where data tends to cluster around a central value with no bias to the left or right. It is often represented as a bell curve.

In a normal or symmetrical distribution, the mean, median, and mode are all centered at the same point. They are approximately equal, making any of these measures a suitable representation of central tendency.

Skewness:

Skewness measures the degree of asymmetry in a distribution. When a distribution is skewed, it deviates from the symmetrical bell curve of a normal distribution. In skewed distributions:

• Positive Skewness (Right Skewed): The tail on the right side is longer, and the mean is pulled toward the right tail. In such cases, the median is often preferred as a measure of central tendency because it's less affected by extreme values.

• Negative Skewness (Left Skewed): The tail on the left side is longer, and the mean is pulled toward the left tail. Again, the median is often a better choice in this situation.

Standard Normal Distribution:

The Standard Normal Distribution, also known as the Z-Distribution, is a specific type of probability distribution. It is a continuous probability distribution that is symmetrically shaped like a bell curve. This distribution has a mean (average) of 0 and a standard deviation of 1.

Z-Score:

A Z-Score, also known as a standard score, measures how many standard deviations a particular data point is away from the mean of a distribution. It's a way to standardize data and compare it to the standard normal distribution.

Z = x * μ / σ

• x = data point
• μ= mean
• σ = standard deviation

The Z-Score tells you how many standard deviations a data point is above or below the mean. A positive Z-Score indicates that the data point is above the mean, while a negative Z-Score indicates that it's below the mean.

The Z-Score is useful for comparing data points from different distributions, identifying outliers, and making statistical inferences. In particular, it helps determine how extreme or unusual a data point is in the context of its distribution.

Data Visualization

Distributions and Frequency plotting (Histograms):

Distributions and frequency plotting, particularly through histograms, are essential components of Exploratory Data Analysis (EDA). These tools help you understand how data is distributed, identify patterns, and visualize the central tendencies and variabilities in your dataset. Here's an explanation of distributions and how to create histograms:

Distributions:

• A distribution is a representation of how data is spread or arranged in a dataset.
• It describes the frequency of different values or ranges of values.
• Understanding the distribution of data is fundamental in EDA because it helps you identify the characteristics and properties of your dataset.

Frequency Plotting (Histograms):

• A histogram is a graphical representation of the distribution of a dataset.
• It divides the data into discrete intervals or "bins" and shows the number of data points that fall into each bin.

Here's how you can create a histogram in Python using the Seaborn library:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Sample dataset
books = pd.DataFrame({
    'title': [
        'To Kill a Mockingbird', '1984', 'The Great Gatsby', 'The Catcher in the Rye', 
        'The Hobbit', 'Fahrenheit 451', 'The Lord of the Rings', 'Pride and Prejudice', 
        'Jane Eyre', 'Animal Farm', 'The Book Thief', 'Wuthering Heights',
        'Brave New World', 'Harry Potter and the Sorcerer\'s Stone', 'Moby-Dick'
    ],
    'author': [
        'Harper Lee', 'George Orwell', 'F. Scott Fitzgerald', 'J.D. Salinger', 
        'J.R.R. Tolkien', 'Ray Bradbury', 'J.R.R. Tolkien', 'Jane Austen', 
        'Charlotte Bronte', 'George Orwell', 'Markus Zusak', 'Emily Bronte',
        'Aldous Huxley', 'J.K. Rowling', 'Herman Melville'
    ],
    'rating': [
        4.28, 4.17, 3.91, 3.80, 4.27, 3.99, 4.36, 4.26, 4.12, 3.92,
        4.37, 3.85, 3.99, 4.47, 3.50
    ]
})

# Create a histogram of book ratings
sns.histplot(data=books, x="rating", binwidth=0.1)
plt.title('Histogram of Books Rating')
plt.xlabel('Rating')
plt.ylabel('Frequency')

# Show the plot
plt.show()

Common observations you can make from a histogram include:

• Shape: Is the distribution symmetric or skewed (left or right)?
• Central Tendency: Where is the peak of the distribution, often indicated by the mode, median, or mean?
• Spread: How spread out are the data points?
• Kurtosis: Is a statistical measure that quantifies the shape of the probability distribution of a dataset, specifically how "heavy-tailed" or "light-tailed" the distribution is compared to a normal distribution. It provides information about the presence of outliers and the degree of peakedness in the distribution. There are typically three common types of kurtosis:

1. Mesokurtic (Kurtosis = 3): The distribution has kurtosis equal to 3, which is the kurtosis of a normal distribution. It indicates that the distribution is neither heavily tailed nor too peaked.
2. Leptokurtic (Kurtosis > 3): A leptokurtic distribution has positive kurtosis, indicating heavy tails and a higher peak than a normal distribution. It implies that the dataset has more outliers and is more "pointy."
3. Platykurtic (Kurtosis < 3): A platykurtic distribution has negative kurtosis, meaning it has lighter tails and is flatter than a normal distribution. It implies that the dataset has fewer outliers and is less peaked.

Data Spread, Range, and Outlier Analysis (Box-and-Whisker plots):

• A box-and-whisker plot is a graphical representation of the spread and distribution of a dataset. It displays the central tendency, data spread, and identifies potential outliers.
• The key components of a box-and-whisker plot include:
• A rectangular box, which represents the interquartile range (IQR), with the lower boundary being the first quartile (Q1) and the upper boundary being the third quartile (Q3).
• A horizontal line inside the box, which represents the median (Q2), also known as the second quartile.
• Whiskers extending from the box, which can help identify potential outliers.
• Individual data points beyond the whiskers, which are considered potential outliers.

Here's an example of creating a box-and-whisker plot in Python using Matplotlib:

import matplotlib.pyplot as plt

data = [10, 15, 20, 25, 30, 35, 40, 100]

# Create a box-and-whisker plot
plt.boxplot(data)

plt.title('Box-and-Whisker Plot')
plt.ylabel('Value')
plt.show()

Correlation Analysis (Scatter plots):

Correlation analysis is a fundamental technique in exploratory data analysis (EDA) that helps you understand the relationships between variables in your dataset. Scatter charts, also known as scatter plots or scatter diagrams, are commonly used to visualize these relationships.

Scatter Charts:

• A scatter chart is a graphical representation of data points in a Cartesian coordinate system. Each data point is represented as a dot on the chart.
• Scatter charts are used to visualize the relationship between two continuous variables, making them suitable for assessing correlation.
• In correlation analysis, you create scatter plots to visually inspect how data points are distributed and whether there's a discernible pattern in their arrangement.

Here's an example of creating a scatter plot in Python using Seaborn:

import seaborn as sns
import matplotlib.pyplot as plt

# Sample data for two variables
x = [2, 3, 5, 7, 8, 10, 12, 15]
y = [12, 13, 15, 18, 20, 22, 26, 30]

# Create a scatter plot
sns.scatterplot(x, y, marker='o', color='red')

plt.title('Scatter Plot')
plt.xlabel('Variable X')
plt.ylabel('Variable Y')
plt.show()

Here are the common types of correlation that can be assessed through scatter plots:

1. Positive Correlation:

In a positively correlated relationship, as one variable increases, the other variable tends to increase as well. When plotted on a scatter plot, data points tend to form an upward-sloping pattern from the bottom left to the top right.

2. Negative Correlation:

In a negatively correlated relationship, as one variable increases, the other variable tends to decrease. On a scatter plot, data points tend to form a downward-sloping pattern from the top left to the bottom right.

3. No Correlation (Zero Correlation):

When there is no correlation, the two variables do not show any consistent pattern or trend on the scatter plot. Data points are scattered randomly, without forming any noticeable direction.

4. Strong and Weak Correlations:

You can also assess the strength of the relationship. A strong correlation indicates a tight and consistent pattern in the scatter plot, while a weak correlation suggests a less consistent or scattered pattern.

Correlation Analysis

Correlation is a statistical measure that helps us understand the relationship or association between two or more variables in a dataset. It tells us how these variables change in relation to each other. Correlation is often used to determine whether there's a connection between variables and, if so, the strength and direction of that connection.

Here are some key points about correlation:

1. Positive Correlation: When two variables have a positive correlation, it means that as one variable increases, the other tends to increase as well.
2. Negative Correlation: Conversely, a negative correlation indicates that as one variable increases, the other tends to decrease. They move in opposite directions.
3. No Correlation: If there's no apparent pattern or relationship between two variables, they are said to have no correlation. Changes in one variable do not have a consistent effect on the other.

Correlation Coefficient:

To quantify the strength and direction of the correlation between two variables, we use a number called the correlation coefficient. This number ranges between -1 and 1.

• A correlation coefficient of 1 indicates a perfect positive correlation.
• A correlation coefficient of -1 indicates a perfect negative correlation.
• A correlation coefficient close to 0 suggests little to no correlation.

The .corr() method in pandas calculates the Pearson correlation coefficient by default, which measures the linear relationship between two variables.

Here's an example:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Sample dataset
books = pd.DataFrame({
    'title': [
        'To Kill a Mockingbird', '1984', 'The Great Gatsby', 'The Catcher in the Rye',
        'The Hobbit', 'Fahrenheit 451', 'The Lord of the Rings', 'Pride and Prejudice',
        'Jane Eyre', 'Animal Farm', 'The Book Thief', 'Wuthering Heights',
        'Brave New World', 'Harry Potter', 'Moby-Dick'
    ],
    'author': [
        'Harper Lee', 'George Orwell', 'F. Scott Fitzgerald', 'J.D. Salinger',
        'J.R.R. Tolkien', 'Ray Bradbury', 'J.R.R. Tolkien', 'Jane Austen',
        'Charlotte Bronte', 'George Orwell', 'Markus Zusak', 'Emily Bronte',
        'Aldous Huxley', 'J.K. Rowling', 'Herman Melville'
    ],
    'rating': [
        4.28, 4.17, 3.91, 3.80, 4.27, 3.99, 4.36, 4.26, 4.12, 3.92,
        4.37, 3.85, 3.99, 4.47, 3.50
    ],
    'pages': [
        324, 328, 180, 214, 310, 194, 1178, 279, 500, 112,
        552, 416, 268, 309, 635
    ]
})

# Calculate the correlation
correlation = books['rating'].corr(books["pages"])
print("Correlation:\n", correlation)

# Correlation:
# 0.20381989732822434

# Calculate the correlation matrix
correlation_matrix = books[['rating', 'pages']].corr()
print("\nCorrelation Matrix:\n", correlation_matrix)

# Correlation Matrix:
#          rating  pages
# rating   1.000  0.204
# pages    0.204  1.000

# Create a heatmap to visualize the correlation matrix
sns.heatmap(correlation_matrix, annot=True)
plt.title('Correlation Matrix Heatmap')
plt.show()

Data Quality

Data quality is a measure of how good or reliable data is. It considers factors like accuracy, completeness, consistency, reliability, and up-to-date. High-quality data is essential for digital businesses.

Dimensions of Data Quality:

1. Accuracy (Is the information correct?)

• Data should be error-free and reflect real-world scenarios.
• Errors can lead to significant problems, such as unauthorized access to bank accounts.

2. Completeness (How comprehensive is the information?)

• Completeness measures how exhaustive a dataset is.
• It ensures that all required values are available, making the information usable.

3. Consistency/Reliability (Does the information contradict other trusted resources?)

• Consistency refers to data uniformity across networks and applications.
• Data in different locations should not conflict with each other.

4. Relevance/Timeliness (Is the information needed, and is it up-to-date?)

• Relevance checks if data fulfills its intended purpose.
• Timeliness ensures data is available when required, preventing wrong decisions.

5. Interpretability (How easy is it to understand the data?)

• Interpretability reflects how easily data can be understood.

Measuring these data quality dimensions helps organizations identify and resolve data errors, ensuring that their data is fit for its intended purpose. High-quality data is the cornerstone of effective digital businesses.

Class Balance

Class Balance or Data Balance refers to the distribution of data points among different categories or classes within a dataset.

Imagine you have a dataset of customer reviews for a product, and you want to classify these reviews as either "positive" or "negative". If you have 90% positive reviews and only 10% negative reviews, you have an imbalance because one class (positive) dominates the dataset, while the other class (negative) is underrepresented.

Here's a simple explanation:

• Balanced Data: When you have roughly an equal number of data points for each class or category, it's called balanced data. For example, if you have 50 positive reviews and 50 negative reviews in your dataset, it's balanced.
• Imbalanced Data: When one class has significantly more data points than the other class, it's called imbalanced data. For example, if you have 90 positive reviews and only 10 negative reviews, it's imbalanced.

Why is Class Balance Important?

Class balance is essential because it can impact the performance of machine learning models. In an imbalanced dataset, models may become biased towards the majority class (the one with more data points) because they see more examples of it. This can lead to poor predictions for the minority class.