First, you error is being caused by your While loop conditions. In your code you While loop will never exit, resulting in an out-of-index exception as the loop keeps iterating beyond the last rows in your grids.
I personally wouldn't use while loops in this scenario because you know up front how many times you want to loop based on the number of rows. It's easier to read the code if you use a for...each loop instead.
That said, to fix your code as you have it you need to clarify your While's condition so the loop can exit.
In your code you have the following While loop...
While DataGridView1.Rows.Count - 1
End While
Your While condition is basically saying 'While True' because you are not comparing the row count against anything. You want to check the row count is less than the cycle count (your j and i variables). To do this change your loops to something like this...
While i <= DataGridView1.Rows.Count - 1
End While
Here is your code but corrected so the While loops will exit...
If DataGridView1.Rows.Count > 2 Then
Dim count As Integer = 0
Dim i As Integer = 0
While i <= DataGridView1.Rows.Count - 1
Dim j As Integer = 1
While j <= DataGridView1.Rows.Count - 1
Dim str As String = DataGridView1.Rows(i).Cells("ColLedger").Value()
Dim str1 As String = DataGridView1.Rows(j).Cells("ColLedger").Value()
If str1 = str Then
count = count + 1
End If
j += 1
End While
i += 1
End While
If count > 0 Then
MsgBox(count)
End If
End If