Using Powershell to to identify old/stale data in a Compellent SAN environment

One of the challenges of managing infrastructure at scale is keeping it in line with best practices. While it’s fairly easy to find and fix configuration drift in a small environment, the time required to do the same thing in a larger environment is daunting and impractical. This is the beauty of tools like Powershell. We can build automated processes to gather this information and report on it in a consistent fashion.

In my environment, I have a number of Dell-Compellent Storage Centers with a total capacity of approximately 1 PB. I wrote the following script to identify several conditions which are outside of what I consider “normal” configurations while handling known exceptions and minimizing false positive alerts. The script looks for the following items:

1) Any volumes which are not mapped to a server. Sometimes when a server is decommissioned, the SAN LUNs are retained for several days/weeks in case there is a need for quick recovery. However, it’s rare that anyone remembers to go back and clean these up. There are a few instances where an unmapped volume is accepted (like a template for a boot-from-SAN OS volume), so those are added to the exceptions list and not reported. This is pretty simple to capture. On each array, the script builds an array of all volume mappings and then runs another cmdlet to find any volumes that are not contained in that array.

2) View volumes. If you’re not familiar with Compellent’s terminology, these are recovery volumes built off a volume snapshot. While they provide great value for point-in-time recovery, when they are left in place for an extended period of time the source volume’s replay (snapshot) data after the recovery point is unable to expire off the array. For volumes with large change rates, this can cause excessive space consumption. As above, it’s fairly common for administrators to forget that these volumes need to be removed once data recovery is complete. This is a bit more challenging to gather as the get-scvolume cmdlet doesn’t have a property which details the volume type. Not to worry though, we just need to compare the SourceVolumeIndex and Index properties of each replay (technically it’s the data before the dash in the index property). If they don’t match, we use the SourceVolumeIndex property to identify the view volume.

Now, we move on to analyzing replays. This gets a bit trickier when we consider the impact of replicated volumes. In volume replication, any new replay generated on the source array gets sent to the target and any replay expired on the source will be expired on the target array. To evaluate whether a volume is a replication target, the script loops through all Storage Centers and builds an array of all replicated volumes which is referenced later. Building this as an array of objects with named properties, while it’s a few more lines of code initially, makes the dataset infinitely easier to query when needed.

3) Replays which have expired but still exist. Sometimes during a firmware upgrade when redundant controllers are in an unbalanced state, replays will not expire off the array correctly. When this happens, this must be handled by CoPilot support. To evaluate whether a replay falls into this category, the script determines what created it. If the CreateSource property is Application then the replay was created using the Java CLI or the Replay Manager Service on Windows (VSS snapshots). The newer versions of RMS (5.x and above) set all replays to “never expire” and manage that expiration at the application level. Knowing that, the script doesn’t bother to check the expiration date of any replay generated by an application, but it will note any replays created by an application that are older than 60 days, which in my environment might indicate an issue with an RMS job schedule.

One of the challenges in the expiration date analysis is that the ExpireTime property of the replay is a string and not a datetime. To convert the string to a datetime value, the script uses the [datetime]::ParseExact method which is a pretty cool tool in the .NET framework. As long as the string formatting is consistent, it works like a charm. Once the string is reformatted, it’s simply a matter of finding any dates that have passed where the replay still exists. In practice, I decided to only analyze replays which should have expired more than 23 hours before the script was run. Sometimes large replays which expired a few minutes before the script found them were still in the process of expiring and were really false positives.

4) Replays which are set to never expire. This is also pretty easy. The script just looks for an empty string value in the ExpireTime property where the replay wasn’t created by an external application. The script also identifies whether the volume is a replication target by looking for its Storage Center number and volume index in the array

5) Volumes with no replay profile. To facilitate data progression, all volumes should have a schedule for generating replays. This is easy to find using the ReplayProfiles property of the volume. The script omits volumes which are replication targets (they receive replays from the source volume) and any exceptions.

Handling the exceptions mentioned above is handled by importing the contents of a CSV file which simply has the Storage Center number and the volume index of all exception volumes. Again, importing this into an array of objects with named properties makes using this data very easy. As long as the first row in the CSV is a header row, it’s easy to create that array as well.

Once all of this data has been gathered, the script compiles it into an HTML formatted list and emails it. Really there’s no good reason for the HTML formatting other than making it easy to read but I hadn’t done anything like that in a while and “it seemed like a good idea at the time”.

Across my environment, I was able to reclaim over 40 TB of space based on information this script found. Definitely worth a few hours of keyboard time.

A note on setting this up to run as an automatic task…if you’re going to run this as a background process you need a way to pass credentials to the array. This is done in the script by reading the contents of a file and converting them to a secure string. If you do it this way rather than on demand, remember the following:

a) The file, once created, cannot be moved to a different folder or it will break.
b) The file must be created by the ID running the scheduled task (if using a service ID, that ID needs to log in on the server running the job to create the file).
c) Name the file something obscure. “MyPassword.txt” is probably an inviting target, just sayin’.
d) Since only the ID running the job needs access to the file, restrict permissions on that file once created.
e) Since this script is only reporting, it would be best practice to create an ID on each array that has limited privileges so the risk of a compromised password is reduced.

##########################################################            
#            
# Get-CompellentCleanupInfo.ps1            
#            
# Posted to practicalpowershell.com on 1/22/2012            
#            
##########################################################            

$user = "Reporter"            
$pw = ConvertTo-SecureString (Get-Content c:\powershell\supersecretfile.txt)            

$logdate = (get-date).tostring('yyyy-MM-dd')            
$LogFile = "C:\powershell\sancleanup\log-$logdate.txt"            

$ExceptionList = import-csv "c:\Powershell\SANCleanup\SANCleanupException.csv"            

#Enter a list of arrays to query as a comma delimited list of strings.            
$SANList = ('SAN1','SAN2')            

# Make sure the necessary snapins are loaded            
$compSnapinLoaded = $FALSE            
$currentSnapins = Get-PSSnapin            
# Check if we need to load the snapin            
foreach ($snapin in $currentSnapins)            
{            
    if ($snapin.Name -eq "Compellent.StorageCenter.PSSnapIn")            
    {$compSnapinLoaded = $TRUE}            
}            

if ($compSnapinLoaded -eq $FALSE)            
    {$null = Add-PSSnapin Compellent.StorageCenter.PSSnapIn}            

#Remove any output files generated the same day.            
if(test-path $logfile)            
{remove-item $logfile}            

#Pull in a list of exceptions from a CSV. These volumes will not be included.             
#Convert to array of objects using integers to make life easier later.            
#The header row is StorageCenter,Index            

$arrException = @()            
foreach($line in $exceptionlist)            
{            
[int]$sc = $line.StorageCenter            
[int]$index = $line.Index            
$objException = new-object system.object            
$objException | add-member -type NoteProperty -name StorageCenter -value $sc            
$objException | add-member -type NoteProperty -name Index -value $Index            
$arrException += $objException            
}            

$arrExceptionDetail = @()            
$arrRepl = @()            

#Build array of all replications to reference later.            
foreach($array in $SANList)            
{            
$SCConnect = Get-SCConnection -Hostname $array -user $user -pass $pw            
$repllist = get-scasyncreplication -connection $SCConnect            

    foreach($repl in $repllist)            
    {            
    $objRepl = new-object system.object            
    $objRepl | add-member -type NoteProperty -name SourceSC -value $repl.storagecenterserialnumber            
    $objRepl | add-member -type NoteProperty -name SourceIndex -value $repl.sourcevolumeindex            
    $objRepl | add-member -type NoteProperty -name TargetSC -value $repl.remotesystemindex            
    $objRepl | add-member -type NoteProperty -name TargetIndex -value $repl.remotevolumeindex            
    $arrRepl += $objRepl            
    }            
}            

#Begin the actual query for each array.            
foreach($array in $SANList)            
{            
#Set a flag to indicate whether any conditions exist that need to be addressed.            
$AllGood = 1            

write-output $array            
add-content $logfile "<h3>$array</h3>"
$SCConnect = Get-SCConnection -Hostname $array -user $user -pass $pw            

$ArraySN = (get-scstoragecenter -connection $SCConnect).StorageCenterSerialNumber            

#Build list of unmapped volumes            
$MapIndexList = @()            
$VolMaps = get-scvolumemap -connection $SCConnect            

#get list of all mappings, convert to array of integers            
foreach($VolMap in $VolMaps)            
{            
$MapIndexList += $VolMap.volumeindex            
}            

$ExcludeIndexList = @()            
foreach($exception in $arrexception)            
{            
if($exception.StorageCenter -eq $ArraySN)            
    {            
    $ExcludeIndexList += $exception.index            
    }            
}            

$UnmappedVolList = get-scvolume -connection $SCConnect |             
    where-object {($MapIndexList -notcontains $_.Index )-and ($ExcludeIndexList -notcontains $_.Index)}             

if($unmappedvollist.count -gt 0)            
    {            
    $AllGood = 0            
    add-content $logfile "<b><u>Unmapped volumes</b></u></br> "            
    foreach($unmappedvol in $unmappedvollist)            
        {            
        $vname = $unmappedvol.name            
        $vpath = $unmappedvol.logicalpath            
        $vsize = $unmappedvol.size            
        add-content $logfile "$vname ($vpath): $vsize<br/> "            
        }            
     add-content $logfile "<br/>"               
    }            

#Find view volumes and expired replays which still exist            
$viewvollist = @{}            
$expiredreplaylist = @()            
$permreplaylist = @()            

$VolReplays = get-screplay -connection $SCConnect            
foreach($VolReplay in $VolReplays)            
{            
$VRI = $volreplay.index            
$dash = $volreplay.index.indexof('-')            
$realindex = $volreplay.index.substring(0,$dash)            

if($volreplay.state -eq "Frozen")            
{            
    if($volreplay.expiretime -eq "" -and $volreplay.createsource -ne "Application")            
    {            
    $sourcevol = get-scvolume -connection $SCConnect -index $volreplay.sourcevolumeindex            

        if(@($arrRepl |             
            where-object {$_.TargetSC -eq $volreplay.storagecenterserialnumber `             -and $_.TargetIndex -eq $volreplay.sourcevolumeindex}).count -gt 0)            
        {            
        #this volume is a replication target            
        $volname = $sourcevol.name + " ***repl target***"            
        }            
        else            
        {$volname = $sourcevol.name}            

        if(@($arrexception | where-object {$_.StorageCenter -eq $volreplay.storagecenterserialnumber -and $_.Index -eq $volreplay.sourcevolumeindex}).count -eq 0)            
        {              
        $permreplaylist += ,@($volname,$sourcevol.logicalpath,$volreplay.createtime.tostring())            
        }            
    }            
    elseif($volreplay.expiretime -eq "" -and $volreplay.createsource -eq "Application" -and $volreplay.createtime -lt [datetime]::now.adddays(-60))            
    {            
    $sourcevol = get-scvolume -connection $SCConnect -index $volreplay.sourcevolumeindex            

        if(@($arrexception |            
             where-object {$_.StorageCenter -eq $volreplay.storagecenterserialnumber -and $_.Index -eq $volreplay.sourcevolumeindex}).count -eq 0)            
        {            
        $expiredreplaylist += ,@($sourcevol.index,$sourcevol.name,$sourcevol.logicalpath,$volreplay.createtime.tostring(),"VSS older than 60 days")            
        }            
    }            
    elseif($volreplay.expiretime -eq "" -and $volreplay.createsource -eq "Application" -and $volreplay.createtime -ge [datetime]::now.adddays(-60))            
    {            
    #ignore - these are VSS snaps newer than 60 days            
    }            
    else            
    {            
        try            
        {            
        $ExpireTime = [datetime]::ParseExact($volreplay.ExpireTime,"MM/dd/yyyy hh:mm:ss tt",$null)            
            if($expiretime -lt [datetime]::now.addhours(-23))            
            {            
            $sourcevol = get-scvolume -connection $SCConnect -index $volreplay.sourcevolumeindex            

            if(@($arrRepl | where-object {$_.TargetSC -eq $volreplay.storagecenterserialnumber -and $_.TargetIndex -eq $volreplay.sourcevolumeindex}).count -gt 0)            
                 {                           
                 #this volume is a replication target or on the exception list            
                 $volname = $sourcevol.name + " ***repl target***"            
                 }            
            else            
                {$volname = $sourcevol.name}            

            if(@($arrexception | where-object {$_.StorageCenter -eq $volreplay.storagecenterserialnumber -and $_.Index -eq $volreplay.sourcevolumeindex}).count -eq 0)            
                {            
                $expiredreplaylist += ,@($sourcevol.index,$volname,$sourcevol.logicalpath,$volreplay.createtime.tostring(),$expiretime)            
                }            
            }            
        }            
        catch            
        {            
        write-output "Error gathering info for $VRI"            
        If($volreplay.expiretime -eq $null){write-output "Replay expiration is null"}            
        elseif($volreplay.expiretime -eq ""){write-output "Replay expiration is an empty string"}            
        else{write-output "Unknown error"}            

        }            
    }            
}            

#this is where we determine if a volume is a view volume            
if($realindex -ne $volreplay.sourcevolumeindex)            
    {            
    $sourcevol = get-scvolume -connection $SCConnect -index $volreplay.sourcevolumeindex            
        if(!($viewvollist.containskey($sourcevol.name)) -and @($arrexception |             
            where-object {$_.StorageCenter -eq $sourcevol.storagecenterserialnumber -and $_.Index -eq $sourcevol.index}).count -eq 0)            
        {            
        $viewvollist.add($sourcevol.name,$sourcevol.logicalpath)            
        }            
    }            
}            

if($viewvollist.count -gt 0)            
{            
    $AllGood = 0            
    add-content $logfile "<b><u>View volumes</b></u><br/> "            

    foreach($viewvol in $viewvollist.keys)            
    {            
    $vname = $viewvol            
    $vpath = "$($viewvollist.$viewvol) `n"            
    add-content $logfile "$vname ($vpath)<br/> "            
    }            
    add-content $logfile "<br/>"              
}            

if($expiredreplaylist.count -gt 0)            
{            
    $AllGood = 0            
    add-content $logfile "<b><u>Expired replays that still exist</b></u><br/> "            
    foreach($expiredreplay in $expiredreplaylist)            
        {            
        $vindex = $expiredreplay[0]            
        $vname = $expiredreplay[1]            
        $vpath = $expiredreplay[2]            
        $createtime = $expiredreplay[3]            
        $expiretime = $expiredreplay[4]            
        add-content $logfile "[$vindex] $vname ($vpath) - Created: $createtime, Expire Time: $expiretime</br>"            
        }            
        add-content $logfile " "              
}            

if($permreplaylist.count -gt 0)            
{            
    $AllGood = 0            
    add-content $logfile "<b><u>Replays with no expiration date</b></u><br/> "            
    foreach($permreplay in $permreplaylist)            
        {            
        $vname = $permreplay[0]            
        $vpath = $permreplay[1]            
        $createtime = $permreplay[2]            
        add-content $logfile "$vname ($vpath) - Created: $createtime<br/> "            
        }            
        add-content $logfile " "              
}            

#Find volumes with no replay profile            

$arrNoReplayProf = @()            
$vollist = get-scvolume -connection $SCConnect |             
    where-object {$_.ReplayProfiles -eq ""}            

if($vollist.count -gt 0)            
{            

    foreach($vol in $vollist)            
    {               
    $Report=1            

         if(@($arrRepl | where-object {$_.TargetSC -eq $vol.storagecenterserialnumber -and $_.TargetIndex -eq $vol.index}).count -gt 0)            
         {$Report = 0}            

         if(@($arrexception | where-object {$_.StorageCenter -eq $vol.storagecenterserialnumber -and $_.Index -eq $vol.index}).count -gt 0)            
         {$Report = 0}            

        if($Report -eq 1)                      
        {            
        #this volume is neither a replication target or on the exception list            
        $vname = $vol.name            
        $vpath = $vol.logicalpath            
        $arrnoReplayProf += "$vname ($vpath) "            
        }            
    } #end for each vol in vollist            

    if($arrNoReplayProf.count -gt 0)            
    {            
    add-content $logfile "<b><u>Volumes with no replay profile (excludes active replication targets)</b></u><br/> "            
    $AllGood = 0            
    add-content $logfile $arrnoreplayprof            
    add-content $logfile " "              
    }            

} #end if vollist gt 0             

if($AllGood -eq 1)            
{            
add-content $logfile "<i>All volumes and replays appear normal</i><br/> "            
}            

add-content $logfile "<br/>"            

#Get any exception volumes and add to array.            

foreach($exception in $arrexception)            
{            
if($exception.StorageCenter -eq $ArraySN)            
    {            

    $arrexceptiondetail += get-scvolume -connection $SCConnect -index $exception.index            
    }            
}            

} #end for each array            

#List detail for exception volumes            
if($arrexceptiondetail.count -gt 0)            
{            
add-content $logfile "<b><u>Volumes listed as exceptions</b></u><br/> "            
foreach($exvol in $arrexceptiondetail)            
    {            
    $varray = $exvol.storagecenterserialnumber            
    $vname = $exvol.name            
    $vpath = $exvol.logicalpath            
    add-content $logfile "[$varray] $vname ($vpath)<br/>"            
    }            
}            

[string]$mailbody = get-content $logfile            

send-mailmessage -to you@yourcompany.com -subject "SAN cleanup script results" ` -smtpserver mail.yourcompany.com -from SANCleanup@yourcompany.com -body $mailbody -bodyashtml

About these ads