Using vectorization to replace loops in python

All programming languages ​​are inseparable from loops. So, by default, we start executing a loop whenever there is a repeating operation. But when we are dealing with large number of iterations (millions/billions of rows), using loops is a crime. You might be stuck for a few hours, only to realize later that it doesn't work. This is where implementing vectorization in python becomes very critical.

What is vectorization?

Vectorization is a technique for implementing (NumPy) array operations on data sets. Behind the scenes, it applies the operation to all elements of the array or series at once (unlike a "for" loop that operates one row at a time).

Next we use some use cases to demonstrate what vectorization is.

Find the sum of numbers

##使用循环
import time 
start = time.time()

 
# iterative sum
total = 0
# iterating through 1.5 Million numbers
for item in range(0, 1500000):
total = total + item


print('sum is:' + str(total))
end = time.time()

print(end - start)

#1124999250000
#0.14 Seconds

## 使用矢量化
import numpy as np

start = time.time()

# vectorized sum - using numpy for vectorization
# np.arange create the sequence of numbers from 0 to 1499999
print(np.sum(np.arange(1500000)))

end = time.time()

print(end - start)


##1124999250000
##0.008 Seconds

The execution time of vectorization is reduced by about 18 times compared to iteration using range functions. This difference becomes even more significant when using Pandas DataFrame.

Math Operations

In data science, when working with Pandas DataFrame, developers use loops to create new derived columns through mathematical operations.

In the example below we can see how easy it is to replace loops with vectorization for such use cases.

DataFrame is tabular data in the form of rows and columns.

We create a pandas DataFrame with 5 million rows and 4 columns filled with random values ​​between 0 and 50.

import numpy as np 
import pandas as pd 
df = pd.DataFrame(np.random.randint( 0 , 50 , size=( 5000000 , 4 )), columns=( 'a' , 'b' , 'c' , 'd ' )) 
df.shape 
# (5000000, 5)
 df.head()

Create a new column "ratio" to find the ratio of columns "d" and "c".

## 循环遍历
import time 
start = time.time() 

# 使用 iterrows 遍历 DataFrame 
for idx, row in df.iterrows(): 
# 创建一个新列
df.at[idx, 'ratio' ] = 100 * (row[ "d" ] / row[ "c" ]) 
end = time.time() 
print (end - start) 
### 109 秒

## 使用矢量化
start = time.time() 
df[ "ratio" ] = 100 * (df[ "d" ] / df[ "c" ]) 

end = time.time() 
print (end - start) 
### 0.12 秒

We can see significant improvements with the DataFrame, with the vectorized operation taking almost 1000 times faster compared to the loop in Python.

If-else statement

We have implemented many operations that require us to use "If-else" type logic. We can easily replace this logic with vectorized operations in python.

Let's see the following example to understand it better (we will use the DataFrame we created in use case 2):

Imagine that we want to create a data based on the existing column "a" Create a new column "e" with some condition on

## 使用循环
import time 
start = time.time() 

# 使用 iterrows 遍历 DataFrame 
for idx, row in df.iterrows(): 
if row.a == 0 : 
df.at[idx, 'e' ] = row.d 
elif ( row.a <= 25 ) & (row.a > 0 ): 
df.at[idx, 'e' ] = (row.b)-(row.c) 
else : 
df.at[idx, 'e' ] = row.b + row.c 

end = time.time() 

print (end - start) 
### 耗时：166 秒

## 矢量化
start = time.time() 
df[ 'e' ] = df[ 'b' ] + df[ 'c' ] 
df.loc[df[ 'a' ] <= 25 , 'e' ] = df [ 'b' ] -df[ 'c' ] 
df.loc[df[ 'a' ]== 0 , 'e' ] = df[ 'd' ]end = time.time()
打印（结束 - 开始）
## 0.29007707595825195 秒

The vectorized operation takes 600 times faster compared to a python loop using if-else statements.

Solving Machine Learning/Deep Learning Networks

Deep learning requires us to solve multiple complex equations, and there are millions and millions of equations to solve. The billion-row problem. Running loops to solve these equations in Python is very slow and vectorization is the best solution.

For example, calculate the y-values ​​for millions of rows in the following multiple linear regression equation:

We can vectorize instead of looping.

The values ​​of m1, m2, m3... are determined by solving the above equation using millions of values ​​corresponding to x1, x2, x3...

import numpy as np 
# 设置 m 的初始值
m = np.random.rand( 1 , 5 ) 

# 500 万行的输入值
x = np.random.rand( 5000000 , 5 )

## 使用循环
import numpy as np
m = np.random.rand(1,5)
x = np.random.rand(5000000,5)

total = 0
tic = time.process_time()

for i in range(0,5000000):
total = 0
for j in range(0,5):
total = total + x[i][j]*m[0][j] 

zer[i] = total 

toc = time.process_time()
print ("Computation time = "+ str ((toc - tic)) + "seconds" ) 

####计算时间 = 27.02 秒

## 矢量化
tic = time.process_time() 

#dot product
np.dot(x,mT) 

toc = time.process_time() 
print ( "计算时间 = " + str ((toc - tic)) + "seconds" ) 

####计算时间 = 0.107 秒

np.dot Implements vectorized matrix multiplication in the backend. It is 165 times faster compared to loops in Python.

Conclusion

Vectorization in python is very fast and should be preferred over loops whenever we are dealing with very large data sets.

# As you start implementing it over time, you will get used to thinking in terms of vectorization of your code.

The above is the detailed content of Using vectorization to replace loops in python. For more information, please follow other related articles on the PHP Chinese website!