NumPy | Replace NaN values with average of columns

Data visualization is one of the most important steps in machine learning and data analytics. 

Cleaning and arranging data is done by different algorithms. Sometimes in data sets, we get NaN (not a number) values that are unusable for data visualization. 

To solve this problem, one possible method is to replace NaN values with an average of columns. 

Given below are a few methods to solve this problem.   

• Using np.colmean and np.take 
• Using np.ma and np.where 
• Using Naive and zip 
• Using list comprehension and built-in functions
• Using zip()+lambda()

Let us understand them better with Python program examples:

Using np.colmean and np.take 

We use the colmean() method of the NumPy library to find the mean of columns. We then use the take() method to replace column mean (average) with NaN values.

Example:

# Python code to demonstrate
# to replace nan values
# with an average of columns

import numpy as np

# Initialising numpy array
ini_array = np.array([[1.3, 2.5, 3.6, np.nan], 
                      [2.6, 3.3, np.nan, 5.5],
                      [2.1, 3.2, 5.4, 6.5]])

# printing initial array
print ("initial array", ini_array)

# column mean
col_mean = np.nanmean(ini_array, axis = 0)

# printing column mean
print ("columns mean", str(col_mean))

# find indices where nan value is present
inds = np.where(np.isnan(ini_array))

# replace inds with avg of column
ini_array[inds] = np.take(col_mean, inds[1])

# printing final array
print ("final array", ini_array)

Output:

initial array [[ 1.3  2.5  3.6  nan]
 [ 2.6  3.3  nan  5.5]
 [ 2.1  3.2  5.4  6.5]]
columns mean [ 2.   3.   4.5  6. ]
final array [[ 1.3  2.5  3.6  6. ]
 [ 2.6  3.3  4.5  5.5]
 [ 2.1  3.2  5.4  6.5]]

Using np.ma and np.where 

We use the ma() method, which allows you to create a masked array where NaN values are masked out. We then use the where() method to replace the NaN values with column averages.

Example:

# Python code to demonstrate
# to replace nan values
# with average of columns

import numpy as np

# Initialising numpy array
ini_array = np.array([[1.3, 2.5, 3.6, np.nan],
                      [2.6, 3.3, np.nan, 5.5],
                      [2.1, 3.2, 5.4, 6.5]])

# printing initial array
print ("initial array", ini_array)

# replace nan with col means
res = np.where(np.isnan(ini_array), np.ma.array(ini_array,
               mask = np.isnan(ini_array)).mean(axis = 0), ini_array)   

# printing final array
print ("final array", res)

Output:

initial array [[ 1.3  2.5  3.6  nan]
 [ 2.6  3.3  nan  5.5]
 [ 2.1  3.2  5.4  6.5]]
final array [[ 1.3  2.5  3.6  6. ]
 [ 2.6  3.3  4.5  5.5]
 [ 2.1  3.2  5.4  6.5]]

Using Naive and zip 

We use Zip to pair up the elements from the unpacked arrays, effectively giving us pairs of (row, column) indices for each NaN value in the array. We then replace these values with column averages.

Example:

# Python code to demonstrate
# to replace nan values
# with average of columns

import numpy as np

# Initialising numpy array
ini_array = np.array([[1.3, 2.5, 3.6, np.nan],
                      [2.6, 3.3, np.nan, 5.5],
                      [2.1, 3.2, 5.4, 6.5]])

# printing initial array
print ("initial array", ini_array)

# indices where values is nan in array
indices = np.where(np.isnan(ini_array))

# Iterating over numpy array to replace nan with values
for row, col in zip(*indices):
    ini_array[row, col] = np.mean(ini_array[
           ~np.isnan(ini_array[:, col]), col])

# printing final array
print ("final array", ini_array)

Output:

initial array [[ 1.3  2.5  3.6  nan]
 [ 2.6  3.3  nan  5.5]
 [ 2.1  3.2  5.4  6.5]]
final array [[ 1.3  2.5  3.6  6. ]
 [ 2.6  3.3  4.5  5.5]
 [ 2.1  3.2  5.4  6.5]]

Using list comprehension and built-in functions

It first computes the column means using a list comprehension with the help of the filter and zip functions. Then, it replaces the NaN values in the array with the corresponding column means using another list comprehension with the help of the enumerate function. Finally, it returns the modified list.

Algorithm:

1. Compute the column means.
2. Replace the NaN values in the array with the corresponding column means using list comprehension and built-in functions.
3. Return the modified list.

def replace_nan_with_mean(arr):
    col_means = [sum(filter(lambda x: x is not None, col))/len(list(filter(lambda x: x is not None, col))) for col in zip(*arr)]
    for i in range(len(arr)):
        arr[i] = [col_means[j] if x is None else x for j, x in enumerate(arr[i])]
    return arr
arr=[[1.3, 2.5, 3.6, None],
     [2.6, 3.3, None, 5.5],
     [2.1, 3.2, 5.4, 6.5]]
print(replace_nan_with_mean(arr))

[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]

Using zip()+lambda()

Compute the column means excluding NaN values using a loop over the transposed array zip(*arr). Replace NaN values with column means using map() and lambda functions.

Algorithm

1. Initialize an empty list means to store the column means.
2. Loop over the transposed array zip(*arr) to iterate over columns.
3. For each column, filter out None values and compute the mean of the remaining values. If there are no remaining values, set the mean to 0.
4. Append the mean to the means list.
5. Use map() and lambda functions to replace None values with the corresponding column mean in each row of the array arr.
6. Return the modified array arr.

# initial array
arr = [[1.3, 2.5, 3.6, None],
       [2.6, 3.3, None, 5.5],
       [2.1, 3.2, 5.4, 6.5]]

# compute column means
means = []
for col in zip(*arr):
    values = [x for x in col if x is not None]
    means.append(sum(values)/len(values) if values else 0)

# replace NaN values with column means
arr = list(map(lambda row: [means[j] if x is None else x for j,x in enumerate(row)], arr))

# print final array
print(arr)

[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]