How to Remove Duplicate Objects from an Array in PowerShell?

Recently, while working on a PowerShell script, I got a requirement to remove duplicate objects from an array. I tried various methods, and in this tutorial, I will explain how to remove duplicate objects from an array in PowerShell using different methods with examples.

To remove duplicate objects from an array in PowerShell, you can use the Select-Object cmdlet with the -Unique switch. For example, $uniqueArray = $array | Select-Object -Unique will filter out duplicates from $array. For more complex objects with specific properties, use a hashtable to track and remove duplicates based on your chosen criteria.

Table of Contents

Remove Duplicate Objects from an Array in PowerShell

An array is a data structure that holds a collection of items. These items can be of any data type, including integers, strings, or even objects. PowerShell Arrays are zero-indexed, meaning the first element is at index 0.

Here’s an example of a simple array of objects in PowerShell:

# Define an array of objects with a 'Name' property
$people = @(
    @{Name = 'Alice'; Age = 30},
    @{Name = 'Bob'; Age = 35},
    @{Name = 'Alice'; Age = 30},
    @{Name = 'Charlie'; Age = 25}
)
$people

As you can see in the above PowerShell array, there are some duplicate objects, we will see how to remove those duplicate objects from the array in PowerShell using different methods.

Method 1: Using Select-Object with -Unique

One of the simplest ways to remove duplicates from an array in PowerShell is by using the Select-Object cmdlet with the -Unique switch. This method works well with simple data types like integers and strings. But you can also easily remove duplicate objects from an array in PowerShell by using these cmdlets.

Here’s how you can use it:

# Define an array of objects with a 'Name' property
$people = @(
    @{Name = 'Alice'; Age = 30},
    @{Name = 'Bob'; Age = 35},
    @{Name = 'Alice'; Age = 30},
    @{Name = 'Charlie'; Age = 25}
) | ForEach-Object { [PSCustomObject]$_ }

# Remove duplicates based on the 'Name' property
$uniquePeople = $people | Select-Object -Unique -Property Name

# Display the unique objects
$uniquePeople

This will filter out the duplicate objects based on the ‘Name’ property, leaving you with a unique list of people by their names.

Remove Duplicate Objects from an Array in PowerShell

Or you can also modify the PowerShell script like the below:

# Define an array of objects with 'Name' and 'Age' properties
$people = @(
    @{Name = 'Alice'; Age = 30},
    @{Name = 'Bob'; Age = 35},
    @{Name = 'Alice'; Age = 30},
    @{Name = 'Charlie'; Age = 25}
) | ForEach-Object { [PSCustomObject]$_ }

# Group objects by the 'Name' property and select the first instance of each group
$uniquePeople = $people | Group-Object -Property Name | ForEach-Object { $_.Group | Select-Object -First 1 }

# Display the 'Name' and 'Age' of each unique person
foreach ($person in $uniquePeople) {
    Write-Output "Name: $($person.Name), Age: $($person.Age)"
}

Method 2: Custom Comparison Script with Sort-Object

If you have an array of objects in PowerShell and you want to remove duplicates based on a specific property, you can use Sort-Object with a custom script to define uniqueness.

For example, if you have an array of custom objects with a Name property:

# Define an array of custom objects with a 'Name' property
$people = @(
    @{Name = 'John'; Age = 25},
    @{Name = 'Jane'; Age = 28},
    @{Name = 'John'; Age = 25},
    @{Name = 'Doe'; Age = 30}
) | ForEach-Object { [PSCustomObject]$_ }

# Remove duplicates based on the 'Name' property
$uniquePeople = $people | Sort-Object -Property Name -Unique

# Display the array with unique 'Name' values
$uniquePeople

In this case, the Sort-Object -Property Name -Unique command sorts the array by the Name property and removes duplicates.

Method 3: Using a Hashtable for Complex Duplicates

For more complex scenarios, such as removing duplicates based on multiple properties from a PowerShell array, you can use a hashtable to track uniqueness.

Here’s how you can do it:

# Define an array of custom objects with multiple properties
$people = @(
    @{Name = 'John'; Age = 25; City = 'New York'},
    @{Name = 'Jane'; Age = 28; City = 'London'},
    @{Name = 'John'; Age = 25; City = 'New York'},
    @{Name = 'Doe'; Age = 30; City = 'New York'}
) | ForEach-Object { [PSCustomObject]$_ }

# Use a hashtable to remove duplicates
$seen = @{}
$uniquePeople = $people | Where-Object { -not $seen.ContainsKey($_.Name + $_.Age) } | ForEach-Object {
    $seen[$_.Name + $_.Age] = $true
    $_
}

# Display the array with unique objects based on 'Name' and 'Age'
$uniquePeople

In this script, we create an empty hashtable $seen. We then filter the $people array with Where-Object, checking if the hashtable already contains a key that combines the Name and Age properties. If it doesn’t, we add that key to the hashtable and output the object, effectively filtering out duplicates.

You can see the output after I executed the above PowerShell script.

powershell remove duplicate objects from array

Conclusion

In this PowerShell tutorial, I have explained different methods of how to remove duplicate objects from an array in PowerShell. We checked each method with real examples.