Normality Test for Data using R

Hello Data Experts,

Let me continue from my last blog http://outstandingoutlier.blogspot.in/2017/08/exploratory-data-analysis-using-r.html “ Exploratory Data Analysis using R where I had covered four moments of statistics. I will help recap all those 4 moments

There are 4 moments of statistics.

*  First step covers Mean, Median and Mode, it is a measure of central tendency.

*  Second step covers Variance Standard Deviation, Range, it is a measure of dispersion.

*  Third step covers Skewness, it is a measure of asymmetry.

*  Fourth step covers Kurtosis, it is a measure of peakness.

To complete Exploratory Data Analysis, 4 moments cover very basic aspects however there are few other techniques which are necessary/mandatory for an individual to understand.  Now we will focus on Graphical Visualization Techniques and Standard Normal Distribution.

Let me continue with my example from last blog with CarsMileage dataset.

CarsMileage <- c(12, 14, 12.5, 13.5, 15, 10, 11, 12, 12, 14, 12, 11.5, 12.5, 13.5, 15, 10.5, 15, 12, 14, 14)

Bar Graph is the basic graph that is drawn on the datasets to understand comparative analysis of each data point. Bar Graph can be placed Horizontal or Vertical.

barplot(CarsMileage)

Histogram, represent different heights of bars where each bar and height has its interpretation. Height represent count or frequency whereas each bin reflects event by logically grouping an event in a specific range.  Distribution of these bins/bars reflect a distribution analysis. Taking an example of retail, low frequency can be an area of opportunity for a business to focus on to grab niche market, where as higher frequency is already encashed.

hist(CarsMileage)

Boxplot, reflect data distribution in quartiles. This graph play a vital role in understanding the data especially outliers. It has six key measures

·         Outliers

·         Lower extreme
·         1^st Quartile
·         Median
·         3^rd Quartile
·         Upper Extreme
 
boxplot(CarsMileage)

Transposing boxplot horizontally help us understand data distribution.  

Just like data is distributed and there are graphs described about help understand Skewness etc. We will touch upon important aspect of Probability and Probability distribution.

Probability in simple terms defined as count of happening of the expecting event/ Total number of events.

Probability distribution: If Probability values drawn on Y axis and Random variable values are drawn on X Axis, that represents Probability distribution.  It has certain characteristics:

·         Random variable value ranges from -Infinite to +Infinite.
·         Probability associated with any single value is always Zero because if we go by the logic of probability where chances of happening expected event is 1 whereas total event that could happen are infinite thus 1/Infinite is always ZERO.
·         Area under probability distribution curve will be 1.

These concepts are important as they lay down the foundation for statistical analysis.

Normal Distribution is characterized as when Mean = Median = Mode for probability distribution and forms a perfect bell curve. Normal distribution curve is symmetrical across Mean (Central Tendency)  that is 50% area under curve on both sides of mean. Standard Deviation under bell curve represents sigma level starting 1 Sigma to 6 Sigma where Six Sigma represent 3.4 DPMO (Defects per Million Opportunities). Let us understand how 1 Sigma is different from 6 Sigma. If we take an example of Photoshoot where photographer takes photos for models. If photographer is professional and proficient number of retakes will be low. so  what does retake shows, if photographer is skilled at 6 Sigma he will take 3.4 retakes out of million photoshoots whereas if Photographer is  1 Sigma skilled he will take 32 retakes in every 100 photoshoots. In photography it is fine to have retakes however industries like Airline and Healthcare which are related to human life and death, it is a norm to have 6 Sigma adherence.  Let me share Sigma values:
1 Sigma reflect 68.26% adherence.
2 Sigma reflects 95.44% adherence
3 Sigma reflects 99.77% adherence
4 Sigma reflects 99.9937% adherence
5 Sigma reflects 99.99997% adherence
6 Sigma reflects 99.9999998& adherence.

Let us take hypothetical example from BPO where # of incidents could be a result of

·         # of new rollout to production.
·         # of test scenarios
·         # of line of code  

In this example, there are various parameter which can influence number of defects based on various factors of different unit. To avoid any influence, we should change each unit value to a unit less value. Approach to convert different unit value to unit less value is called standardization.  Plotting the standardized data distribution is called Standardized Normal distribution. Key characteristic of standardized data is Mean will be Zero and Standard Deviation will be always be 1. So if we have an output dependent of multiple inputs of varied units and size, it is recommended to standardized it make it unit less. Once  data is unit less , output will not get influenced by any specific parameter. 

Statistically standardized value can be derived using a formula ((X-Mean)/Standard Deviation) or using excel we can use "norm.dist" inbuilt statistical formula to achieve the same. R makes life simple and keep all these complexity as black box for data scientist to focus on the core. "pnorm" can be used in R to get the same outcome. It is a straight forward simple formula which can help us derive standard data in a single shot by executing following command using R.

scale(CarsMileage) # this will result in standardized data.

Output of this will be as below

[1,] -2.5006312
[2,]  0.5304369
[3,] -0.2273301
[4,]  0.2778479
[5,]  1.0356149
[6,]  2.0459710
[7,] -0.9850971
[8,] -0.4799191
[9,] -0.4799191
[10,]  0.5304369
[11,] -0.4799191
[12,] -0.7325081
[13,] -0.2273301
[14,]  0.2778479
[15,]  1.0356149
[16,] -1.2376862
[17,]  1.0356149
[18,] -0.4799191
[19,]  0.5304369
[20,]  0.5304369
 
Once data is standardized, plotting of these points will follow standardized normal distribution or Z distribution. Probability of any value will be Zero but we can calculate the probability of a value grater than or less than using standard normal distribution technique. To derive the probability we can use Z table from statistics where values are in there in X axis and Y axis that matching value will be the probability. This is very time consuming if done manually. But by using R it is very simple and probability value can be derived using is single command.

# pnorm(Probability of particular value, Mean, Standard Deviation)

pnorm(14,mean(CarsMileage),sd(CarsMileage))

Probability of mileage to be 14 will be 78%

What is probability of value to be in a range i.e., greater than X and lesser than y value? Using R it can be derived as shown below?
pnorm(y, mean, SD) - pnorm(x, mean, SD)

All this is relevant if data follows normal distribution, so it is important for us to understand if data follows normal distribution of not using R. If the outcome from below commands have straight line that reflects data is Normalized and good for statistical analysis.

qqnorm(CarsMileage)

qqline(CarsMileage)

Now that we had talked about key concept, it is very important that data follows normal distribution to have meaningful statistical analysis. If data in its original form does not follow Normal distribution, we should try transforming that dataset and measure Normality. Data can be transformed by executing Square Root, Cube Root or Exponential equations. There are multiple other ways to transform data and check its normality.

Once normality of data is derived, right statistical analysis can be executed and help infer right outcome to make right decisions.

I hope this blog helped you gain insight into Normal Probability distribution and normality check. In next blog, I will cover what statistical analysis techniques can be executed on normal and non-normal data. This completes the Exploratory Data Analysis, we covered four moments of statistical decision and normality check of data. This is the key foundation for rest of the statistical word since 60% -70% of statistician’s time goes in executing EDA to make sure data captured is complete and unbiased which will help one take right decision and right outcome.  Getting right data be it a population or sample it is very important for one to make a right choice. Now that basics for EDA we will cover “Advance statistical concepts like confidence level using R Studio”.

Thank you for going through this blog, I hope it helped you built sound foundation of statistics using R. Kindly share your valuable and kind opinion. Please do not forget to suggest what you would like to understand and hear from me in my future blogs. 

Thank you...

Outstanding Outliers:: "AG".