Teach you the cool pandas operations that 1% of people know

python tutorial column introduces the operation of pandas today.

pandas has a very powerful method, which is accessor. It can be understood as a property interface through which additional methods can be obtained. In fact, this is still very general. Let’s understand it through code and examples.

>>> pd.Series._accessors
{'cat', 'str', 'dt'}复制代码

Using the _accessors method for the Series data structure, we get 3 objects: cat, str, dt.

• .cat: is used for categorical data (Categorical data)
• .str: is used for character data (String Object data)
• .dt: Used for time data (datetime-like data)

Let’s take a look at how these three objects are used in turn.

Use of str object

Series data type: str string

# 定义一个Series序列
>>> addr = pd.Series([
...     'Washington, D.C. 20003',
...     'Brooklyn, NY 11211-1755',
...     'Omaha, NE 68154',
...     'Pittsburgh, PA 15211'
... ]) 

>>> addr.str.upper()
0     WASHINGTON, D.C. 20003
1    BROOKLYN, NY 11211-1755
2            OMAHA, NE 68154
3       PITTSBURGH, PA 15211
dtype: object

>>> addr.str.count(r'\d') 
0    5
1    9
2    5
3    5
dtype: int64复制代码

Instructions on the two methods of the above str object:

• Series.str.upper: Change all strings in Series to uppercase;
• Series.str.count: Change all characters in Series Count the number of strings;

In fact, it is not difficult to find that the usage of this usage is very similar to the operation of strings in Python. Yes, you can do such simple operations in pandas, but the difference is that you are operating an entire column of string data. Still based on the above data set, let’s look at another operation:

>>> regex = (r&#39;(?P<city>[A-Za-z ]+), &#39;      # 一个或更多字母
...          r&#39;(?P<state>[A-Z]{2}) &#39;        # 两个大写字母
...          r&#39;(?P<zip>\d{5}(?:-\d{4})?)&#39;)  # 可选的4个延伸数字
...
>>> addr.str.replace(&#39;.&#39;, &#39;&#39;).str.extract(regex)
         city state         zip
0  Washington    DC       20003
1    Brooklyn    NY  11211-1755
2       Omaha    NE       68154
3  Pittsburgh    PA       15211复制代码

Description of the two methods of the above str object:

• Series.str.replace:Replace the specified string in the Series;
• Series.str.extract: Extract the data information in the string through regular expressions;

This The usage is a bit complicated, because it is obvious that this is a chain usage. Replace " . " with "" through replace, which is empty, then use 3 regular expressions (corresponding to city, state, zip respectively) to extract the data through extract ,And changed from the original Series data structure to the DataFrame data structure.

Of course, in addition to the above usage, commonly used attributes and methods include .rstrip, .contains, split, etc. Let’s check the str attribute through the following code Complete list:

>>> [i for i in dir(pd.Series.str) if not i.startswith('_')]
['capitalize',
 'cat',
 'center',
 'contains',
 'count',
 'decode',
 'encode',
 'endswith',
 'extract',
 'extractall',
 'find',
 'findall',
 'get',
 'get_dummies',
 'index',
 'isalnum',
 'isalpha',
 'isdecimal',
 'isdigit',
 'islower',
 'isnumeric',
 'isspace',
 'istitle',
 'isupper',
 'join',
 'len',
 'ljust',
 'lower',
 'lstrip',
 'match',
 'normalize',
 'pad',
 'partition',
 'repeat',
 'replace',
 'rfind',
 'rindex',
 'rjust',
 'rpartition',
 'rsplit',
 'rstrip',
 'slice',
 'slice_replace',
 'split',
 'startswith',
 'strip',
 'swapcase',
 'title',
 'translate',
 'upper',
 'wrap',
 'zfill']复制代码

There are many attributes. For specific usage, if you are interested, you can explore and practice by yourself.

Use of dt object

Series data type: datetime

Because the data requires datetime type, the following uses pandas’ date_range() to generate a Group date datetime demonstrates how to perform dt object operations.

>>> daterng = pd.Series(pd.date_range('2017', periods=9, freq='Q'))
>>> daterng
0   2017-03-31
1   2017-06-30
2   2017-09-30
3   2017-12-31
4   2018-03-31
5   2018-06-30
6   2018-09-30
7   2018-12-31
8   2019-03-31
dtype: datetime64[ns]

>>>  daterng.dt.day_name()
0      Friday
1      Friday
2    Saturday
3      Sunday
4    Saturday
5    Saturday
6      Sunday
7      Monday
8      Sunday
dtype: object

>>> # 查看下半年
>>> daterng[daterng.dt.quarter > 2]
2   2017-09-30
3   2017-12-31
6   2018-09-30
7   2018-12-31
dtype: datetime64[ns]

>>> daterng[daterng.dt.is_year_end]
3   2017-12-31
7   2018-12-31
dtype: datetime64[ns]复制代码

The above three methods of dt are explained:

• Series.dt.day_name(): Judge the day of the week from the date;
• Series.dt.quarter: Determine the season from the date;
• Series.dt.is_year_end: Determine whether it is the end of the year from the date;

Other methods are also based on some transformations of datetime, and use transformations to view specific micro or macro dates.

Use of cat object

Series data type: Category

Before talking about the use of cat object, let’s talk about the Category data type. The effect is very powerful. Although we do not frequently run g data in memory, we always encounter situations where executing a few lines of code will wait for a long time. One benefit of using Category data is that can save time and space consumption. Let’s learn through several examples.

>>> colors = pd.Series([
...     'periwinkle',
...     'mint green',
...     'burnt orange',
...     'periwinkle',
...     'burnt orange',
...     'rose',
...     'rose',
...     'mint green',
...     'rose',
...     'navy'
... ])
...
>>> import sys
>>> colors.apply(sys.getsizeof)
0    59
1    59
2    61
3    59
4    61
5    53
6    53
7    59
8    53
9    53
dtype: int64复制代码

Above we used sys.getsizeof to display the memory usage. The number represents the number of bytes.
There is another way to calculate content occupancy: memory_usage(), which will be used later.

Now we map the unique values ​​of the colors above to a set of integers, and then look at the memory occupied.

>>> mapper = {v: k for k, v in enumerate(colors.unique())}
>>> mapper
{'periwinkle': 0, 'mint green': 1, 'burnt orange': 2, 'rose': 3, 'navy': 4}

>>> as_int = colors.map(mapper)
>>> as_int
0    0
1    1
2    2
3    0
4    2
5    3
6    3
7    1
8    3
9    4
dtype: int64

>>> as_int.apply(sys.getsizeof)
0    24
1    28
2    28
3    24
4    28
5    28
6    28
7    28
8    28
9    28
dtype: int64复制代码

Note: For the above integer value mapping, you can also use the simpler pd.factorize() method instead.

We found that the memory occupied above is half that of using the object type. In fact, this situation is similar to the internal principle of Category data type.

Difference in memory usage: The memory occupied by Categorical is proportional to the number of Categorical categories and the length of the data. On the contrary, the memory occupied by object is a constant multiplied by the data. length.

The following is a comparison of object memory usage and category memory usage.

>>> colors.memory_usage(index=False, deep=True)
650
>>> colors.astype('category').memory_usage(index=False, deep=True)
495复制代码

上面结果是使用object和Category两种情况下内存的占用情况。我们发现效果并没有我们想象中的那么好。但是注意Category内存是成比例的，如果数据集的数据量很大，但不重复分类（unique）值很少的情况下，那么Category的内存占用可以节省达到10倍以上，比如下面数据量增大的情况：

>>> manycolors = colors.repeat(10)
>>> len(manycolors) / manycolors.nunique() 
20.0

>>> manycolors.memory_usage(index=False, deep=True)
6500
>>> manycolors.astype('category').memory_usage(index=False, deep=True)
585复制代码

可以看到，在数据量增加10倍以后，使用Category所占内容节省了10倍以上。

除了占用内存节省外，另一个额外的好处是计算效率有了很大的提升。因为对于Category类型的Series，str字符的操作发生在.cat.categories的非重复值上，而并非原Series上的所有元素上。也就是说对于每个非重复值都只做一次操作，然后再向与非重复值同类的值映射过去。

对于Category的数据类型，可以使用accessor的cat对象，以及相应的属性和方法来操作Category数据。

>>> ccolors = colors.astype('category')
>>> ccolors.cat.categories
Index(['burnt orange', 'mint green', 'navy', 'periwinkle', 'rose'], dtype='object')复制代码

实际上，对于开始的整数类型映射，我们可以先通过reorder_categories进行重新排序，然后再使用cat.codes来实现对整数的映射，来达到同样的效果。

>>> ccolors.cat.reorder_categories(mapper).cat.codes
0    0
1    1
2    2
3    0
4    2
5    3
6    3
7    1
8    3
9    4
dtype: int8复制代码

dtype类型是Numpy的int8（-127~128）。可以看出以上只需要一个单字节就可以在内存中包含所有的值。我们开始的做法默认使用了int64类型，然而通过pandas的使用可以很智能的将Category数据类型变为最小的类型。

让我们来看一下cat还有什么其它的属性和方法可以使用。下面cat的这些属性基本都是关于查看和操作Category数据类型的。

>>> [i for i in dir(ccolors.cat) if not i.startswith('_')]
['add_categories',
 'as_ordered',
 'as_unordered',
 'categories',
 'codes',
 'ordered',
 'remove_categories',
 'remove_unused_categories',
 'rename_categories',
 'reorder_categories',
 'set_categories']复制代码

但是Category数据的使用不是很灵活。例如，插入一个之前没有的值，首先需要将这个值添加到.categories的容器中，然后再添加值。

>>> ccolors.iloc[5] = 'a new color'
# ...
ValueError: Cannot setitem on a Categorical with a new category,
set the categories first

>>> ccolors = ccolors.cat.add_categories(['a new color'])
>>> ccolors.iloc[5] = 'a new color'  
复制代码

如果你想设置值或重塑数据，而非进行新的运算操作，那么Category类型不是那么有用。

相关免费学习推荐：python教程（视频）

The above is the detailed content of Teach you the cool pandas operations that 1% of people know. For more information, please follow other related articles on the PHP Chinese website!