Correcting Skewed Data with Scipy and Numpy

Рет қаралды 6,209

Жыл бұрын

Skewed data can adversely affect your analysis and machine learning models. In this video, I demonstrate five methods for cleaning skewed data using the NumPy and SciPy modules. The methods include taking the square root, cube root, fourth root, log, and Yeo-Johnson transform. I also showcase the effectiveness of each method by summarizing the skewness of the data after each transformation with a bar plot.

Пікірлер: 27

@metinunlu_ 6 ай бұрын

Thank you for the video, subscribed! KZfaq needs more quality content like this.

@officialscience101 Жыл бұрын

the on-screen text is a great addition, Dr. P!

@CJP3 Жыл бұрын

🙏🏽, I’ll incorporate more in upcoming videos! Thanks for the feedback!

@user-gx6tn4wk7r 4 ай бұрын

Bro this is data science ASMR 🤤

@CJP3 4 ай бұрын

Hahaha I didn’t mean for it to be but glad you enjoyed it (I hope) 😂

@mushinart 10 ай бұрын

Outstanding explanation, professor

@CJP3 10 ай бұрын

Thank you so much!

@user-mm2uc6ye9x 2 ай бұрын

Amazing video I like it's structure: motivation, overview with examples, practical advices Thanks!

@CJP3 2 ай бұрын

Thanks for the feedback! I’ll do more of this style!

@nicolaslpf Жыл бұрын

Amazing video! I was creating a function for measuring the same you forgot to name log1p Wich is log of (x+1) really useful for right skewed data with values less than 1

@dannybee9068 Жыл бұрын

Thank you! That was helpful! So we basically can make the root of any power? Is there a drawbag for exploiting it , like keep increasing the n value for feature to the power of 1/n?

@CJP3 Жыл бұрын

Hi Danny! Context definitely matters. For analytical chemistry 1/n scaling is usually ok. a few downsides are that it makes the models less sensitive to potential outliers. Also its not suitable for certain distributions. Lastly, because 1/n scaling is non-linear, it can make data interpretation more difficult.

@thoniasenna2330 4 ай бұрын

SUBSCRIBED! What should one do before? Or, what's the correct order? - treating outliers, impute missing values, correct symmetry? Thanks Dr. P!

@CJP3 4 ай бұрын

You’re not going to like the answer 😂… it depends a lot on the application. It’s first best to be aware they exist and then evaluate their impact on your outcome. For example if you’re trying to determine outlier samples - then outlier msmts wouldn’t be so bad.. maybe. Or missing values could be useful depending on the application so instead of imputing maybe you engineer a new feature.

@CJP3 4 ай бұрын

Don’t unsubscribe after my answer! 😂 🤣

@pabloagogo1 Ай бұрын

This is interesting. If one corrects the original skewed data, via doing these kinds of transformations, in the context of linear regression or multiple linear regression, will that not change the interpretation of the original data. Curious to know.

@CJP3 28 күн бұрын

Perhaps, but that change may be for the better. I’d say it’s worth considering these transformation if you know you have skewed data. Many models especially linear models assume normally distributed variables. I usually build models with and without significant preprocessing and feature scaling/engineering.