Streamlining Repetitive Tasks During Exploratory Data Analysis

This article discusses automation in data science, particularly in the area of exploratory data analysis (EDA). The author emphasizes the importance of automating repetitive EDA tasks and demonstrates the creation of a utility to automate these tasks. The utility includes features such as summary statistics, statistical tests, correlation heatmap, category averages, and data distribution visualizations. By automating these tasks, data scientists can save time and focus on higher-value areas of analysis.

Automation in Data Science

An invitation to identify your repetitive EDA tasks and create an automated workflow, illustrated through an example utility.

Programming Principle: Automate the Mundane

Skilled programmers automate repetitive tasks to save time and effort. By creating tools and using smart software, they avoid redundancy and make their work easier to maintain and refactor.

The Repetitive Nature of EDA

Exploratory data analysis (EDA) involves repetitive tasks such as statistical analysis and visualization. Automation can greatly benefit EDA by saving time and effort.

Limits of Full Automation

Complete automation of EDA is hindered by the unique challenges of each dataset. Standardization is difficult due to factors like encoding strategies and data types.

A Modular Approach

To address this limitation, a utility has been created that assumes minimal data processing and requires the definition of numerical, categorical, and target columns.

What does it contain?

The utility provides high-level statistics, statistical tests, a correlation heatmap, category averages, and data distribution visualizations. Optional parameters allow for flexibility in enabling or disabling specific functionalities.

The Dataset

The utility was applied to a dataset examining factors predictive of stroke diagnosis.

Light Pre-processing and Feature Engineering

The dataset underwent pre-processing steps such as extracting cholesterol values, generating binary indicator columns for symptoms, and converting categorical columns and the target column into numerical codes.

Summary()

The summary() function generates a summary of data exploration tasks, including categorical and numerical summaries, statistical tests, a correlation heatmap, category averages, and data distribution visualizations.

Categorical and Numerical Summaries

The categorical summary provides insight into each category, including unique values, most frequent value, percentage of missing values, and entropy. The numerical summary calculates descriptive stats and identifies outliers.

Statistical Tests

The statistical test summary evaluates the relationship between each feature and the target variable using chi-squared tests for categorical variables and t-tests for numerical variables.

Correlation Heatmap

The correlation heatmap visualizes the Spearman correlation between numerical variables, ordinal variables, and the target variable.

Plots

The summary() function generates barplots for categorical variables and histograms and boxplots for numerical variables to visualize data distributions.

Concluding Remarks

Creating customized EDA utilities allows for rapid exploration of new datasets and provides insights for targeted analysis. Automating repetitive tasks frees up cognitive resources for higher-value areas like domain knowledge and modeling.