PowerShell Search a Huge Disk or Log File for a String

You will likely get a memory error (System.OutOfMemoryException) if you use Get-Content to try to load a 550MB file into a string variable. We can using a System.IO.StreamRead in PowerShell to stream the data instead of doing one big massive load into memory. The script below allows you to search a huge file for a string. For example, in this case I had an OpenSSH SFTP log with 4 months of data in it. I wanted to find the date/time when a user last attempted to connect.

# Example from log file when a userid is matched 
#10464 2020-03-31 13:47:30.273 debug1: user Walmart matched 'User Walmart ' at line 134

Write-Host "Start" 

#$logFile = "c:\ProgramData\ssh\logs\sshd_log_Split_19_2020_03_30_to_03_31.log" 
#$logFile = "c:\ProgramData\ssh\logs\sshd_log_Split_20_2020_03_31_to_04_01.log" 
$logFile = "c:\ProgramData\ssh\logs\sshd_log_Split_21_2020_04_01B.log" 
#$logFile = "c:\ProgramData\ssh\logs\sshd_log_Split_22_2020_04_01_to_04_02.log" 

$ErrorActionPreference = "Stop"

$reader = new-object System.IO.StreamReader($logFile)
$count = 1
$lineNumber = 1 
while(($line = $reader.ReadLine()) -ne $null)
    if ($line.ToLower().Indexof("user walmart matched") -ge 0) 
        Write-Host $count $lineNumber $line 

Write-Host "End" 


Leave a Reply