Hi.
In this video we will learn how to do univer analysis of numeric
variable. First, uh,
let's examine the duration variable a bit. Uh,
this is the duration in seconds of each connection.
Uh, the first,
what we will always do is to calculate the descriptive statistics of some
variable, which we did in the describe method.
And now we can plug the histogram.
Histogram is basically only a bar chart where the values of the variable are
split into bin with equal value range.
And in each bin we see the counts of observations falling into that bin.
For example, we see here that most of the values are very close to zero.
Uh, we know from the describe method, uh,
that at least 75% are zero. We see that, uh,
maximum is, uh, close to, uh, 60 K as well.
Uh, to have a better view, we can plot the counts on the log axis.
We can do this with this parameter, log y, uh, equals through. We can
do also to calculate the duration value, uh,
for each of the percentile from, uh,
0.01 to 0.99 and plus it, uh,
via a line chart. This is, uh, what I did Here.
We see that, uh, maybe around, uh,
97% of the duration variable is uh,
zero.
We can share directly in the data frame.
We can also now upload the histogram to better see the distribution and
apply the filters. Uh, like removing the zero values and, uh,
values about 200, for example, we
see that this distribution is, uh, highly right, skewed,
uh, basically it's very similar to an exponential distribution.
Now, let's check the source bytes variable. Uh, we see here the same pattern,
uh, that most of the values are zero, but now, uh,
we can check the log by parameter. We see that it doesn't help much as well.
We see here, uh, only one very big outlier.
We can apply now some filters to have a better view of the distribution. Uh,
let's say apply a filter up to 10 K.
We see it now a little bit better, but let's zoom in.
Now up to two K,
we see that most of the observations have source by value around 1000.
Uh, then a lot of, uh, zeros,
then around 250 and 500.
Other interesting chart where we can see the shape of the distribution is the
density plot. Uh, we can plot it, uh,
with a kind, uh, k d e.
This is the shortcut for kernel density estimation.
Uh, sometimes, uh, when we use this chart,
we can see the distribution shape more clearly, uh,
because we do not need to specify the number of bins.
Now let's examine the count variable. Uh,
it is the number of connections to the same host as the current connection. In,
uh, past two seconds. We see here few bumps,
uh, but this one seems as there are multiple, uh, different, uh,
populations present. Uh, you see that when you examine, uh,
some variable. Usually when you have only one population present,
the distribution has a clear shape.
This one is clearly different around zero. Uh,
we have some sort of exponential distribution,
around 100 and 200, uh, vc,
some sort of uniform distributions,
but hard to say we'll investigate, uh, more
we see here also the density product as well.
Now, let's go a little bit into restrictive statistics and to the most common
distributions.
Here I put a few charts to explain you the few basic concepts.
Uh, this is the normal distribution and it is the most common type here,
the median mode, and the average is the same.
Value mode is the most common value. Median plays the data in half.
That means 50% of the data is below and 50% of the data
is above this value. Around, uh,
68% of the data are within one standard deviation from the mean.
95% of the data are within two standard deviations from
the mean. And, uh,
99 a percent of the data are within three standard deviations from the
mean.
I'm sure that you have also the experience that when people do some analysis,
they use mostly the average. This is the synonym for mean,
but this is not always the proper value to use. Now, I will explain to you why
in symmetrical distributions like the normal distribution and uniform
distribution, this is this one,
the mode median and the average is, uh, approximately the same value.
But when the distribution is cute and the average is pulled by the tail,
we can see it here. The red mark is the mean,
and the blue line is the median,
longer the tail and more and bigger outliers present.
The difference between the median and average will be bigger.
Uh, we can see this uh, in this, uh,
left skew distribution and also in this right skew distribution.
Another form of the skew distribution is the exponential distribution.
In these cases, the proper statistics, uh,
to use to describe the variable is the median, not the average.
Now to transformations, uh, you see that, uh,
when a variable has some distribution type, uh, the distribution is not fixed.
Uh, you can change it via applying some function.
The most common transformations are square, where root or lock.
Let's see how the distributions will change.
Uh, here in the first example,
I transformed the left queue distribution to approximately normal,
the original distribution. It looks like this,
but then we apply square.
We see slowly that the shape of the distribution changes
and when it's to the power of eight, we see that it's approximately normal,
it's symmetrical.
When there is a right skew distribution, we can apply the root or the logarithm.
We can see this applying, uh, dysfunction on the exponential distribution. And,
uh, we see that it's changing the shape.
This one looks the best. I think.
Here is an example of the algorithm transformation. Uh, we see that I,
uh, try to transform, uh, the exponential distribution to normal, but, uh,
we see that the log transformation is already too much. Uh, we completely,
uh, changed the distribution, uh, from right skews to left skews.
Uh, two more things. Uh, first regarding these, uh, box plot,
what do you see here? Let's go to the first one here.
This is a very common way to show you the distribution. Uh,
we can see here the first quartile, uh,
which is basically the 25th percentile. Then we see the second quartile.
This is the median. Then we see the third quartile, uh,
which is the 75th percentile.
The difference between the third and first quartile is called the
interquartile range. Uh, short i q r.
This mark, uh, is uh, the second quartile, uh,
minus one and a half times I Q R. And this mark, uh,
is the third quartile plus one and a half times i q r.
Here in this other chart we see this dot,
and these are outliers.
Here are more of them.
And the second thing,
why I showed you these transformations is that even that you will not do this
very frequently in this important to understand this concept,
that when you apply some function, you can change the distribution shape.
And also, uh, some statistical algorithms like regression for example,
make assumptions on the distribution of the input data and target variable,
which should have normal distributions, otherwise they behave, uh,
unexpectedly.
In our next video, we will learn how to explore categorical variables.