Sometimes you have a huge large file. In my case, I encountered an OpenSSH log file that was 550 Megabytes. I will put some process in place so that it doesn’t grow so large. It contains four months of data with full debugging turned on.

In such a large file, it is hard to find or locate on specific user or date, so I wanted to split the file into smaller chunks.

For the $upperBound variable below, you can set to various values, such as 1GB, 100MB, 50KB etc…

We have to use the StreamReader. When I tried to do a “get-content”, I got an System.OutOfMemory exception.

On my production server, it took about 1 to 2 minutes for each 10MB file to get created. So it might run for quite some time.

<pre>
cls
$ErrorActionPreference = "Stop"
$upperBound = 10MB # calculated by Powershell
$ext = "log"
$rootName = "c:\ProgramData\ssh\logs\sshd_log_Split_"

$reader = new-object System.IO.StreamReader("c:\ProgramData\ssh\logs\sshd_Archive.log")
$count = 1
$fileName = "{0}{1}.{2}" -f ($rootName, $count, $ext)
while(($line = $reader.ReadLine()) -ne $null)
{
    Add-Content -path $fileName -value $line
    if((Get-ChildItem -path $fileName).Length -ge $upperBound)
    {
        ++$count
        $fileName = "{0}{1}.{2}" -f ($rootName, $count, $ext)
    }
}

$reader.Close()
</pre>

Base of code from Lee on StackOverflow