Dealing with outliers

This post by Lisa Morgan discusses how to deal with data outliers in a Big Data environment. The management of outliers is valuable in any environment, of course, as it’s important to understand how to put outliers in the proper perspective. I have seen all too often a lot of attention paid to outliers (for example, in student evaluation of teaching), which can lead to a skewed understanding of what the entire corpus of data reveals. While Morgan argues that outliers should not be dismissed, it’s importance to not given them more value than they merit. I know, for example, how easy it is for instructors to fixate on one student comment, normally when it is of a critical nature, rather than focus on the general pattern that is revealed by the data as a whole. One needs to be careful to not spend a lot of energy  addressing outliers at the expense of the whole picture, especially since in some cases, no matter how hard you try, you simply cannot meet everyone’s needs all the time.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s