SnapMirror Initialize – Auto-Restart Powershell Script

I have a client that recently wanted to SnapMirror a 2TB volume to their DR array. Normally this wouldn’t be a hard task to tackle. In this instance the client was using a very unstable IPSec tunnel across their corporate Internet connection to their DR facility. Their Internet connectivity was already somewhat congested at certain times during the day, so adding a SnapMirror replication relationship to it wasn’t going to help matters. All of that aside, the client wanted to move forward with this so I quiesced (no pun intended) and decided to give this a shot.

We almost immediately ran into issues where the SnapMirror initialization would stop after a few hours due to a network connection error. The good news is that SnapMirror keeps regular checkpoints during the SnapMirror process allowing us to restart where it left off. We don’t have to completely start over from zero. A simple resubmission of the SnapMirror initialize command and things were off and running again. At least, for a few more hours before the relationship stopped again for the same reason.

This went on for a little over a day before I got fed up and decided to write a script to handle these regular restarts. NetApp has a very complete set of APIs that they publish through the use of a Powershell toolkit – the Data ONTAP Powershell Toolkit. The script I created would poll the destination NetApp SVM in the SnapMirror relationship and pull a full list of SnapMirror relationships. If any of the relationships where in an Unhealty state and Idle (indicating an incomplete initialization), I would resubmit the SnapMirror initialization command. Before and after submitting the command, I would pull status of the relationships. These outputs would be emailed to me whenever the relationship required a restart, letting me know the script was working. I inserted the script into a Task Scheduler job on a Windows server with access to the DR NetApp array and set it to run every 5 minutes. Below is the script:

#Pull in NetApp Data ONTAP Powershell Toolkit
Import-Module DataOnTap

function Connect-Filer([string]$filer, [string]$user, [string]$pass)
{
    # Create an encrypted string which the API will use
    $password = ConvertTo-SecureString $pass -AsPlainText –Force
    # Create a credential object in powershell.
    $cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $user,$password
    # Connect to the Filer using the credential created above.
    $connection =  Connect-NcController $filer -Credential $cred
} #end Connect-Filer

function Invoke-SnapmirrorRestart()
{
    #Query the Filer for the current SnapMirror Status and save it to a variable.
    $smStartStatus = Get-NcSnapmirror | Format-Table -Wrap -AutoSize -Property SourceLocation,DestinationLocation,Status,SnapshotProgress | Out-String

    $smStatus = Get-NcSnapmirror

    #Cycle through each SnapMirror relationship and determine state of each.  If state is Idle and Unhealthy, reinitialize.
    foreach($sm in $smStatus) {
        if(!$sm.IsHealthy) {
            if($sm.Status -eq "idle") {
                $smSource = $sm.SourceVserver + ':' + $sm.SourceVolume
                $smDest = $sm.DestinationVserver + ':' + $sm.DestinationVolume
                Invoke-NcSnapmirrorInitialize -Source $smSource -Destination $smDest
                $Output = $Output + 'Restarting Mirror of ' + $smSource + ' to ' + $smDest
            } #endif
        } #endif
    } #end foreach

    #Wait 30 seconds for the re-initialization to restart then grab status of the SnapMirror relationships for the status email to validate that everything restarted properly.
    Start-Sleep -s 30
    $smFinalStatus = Get-NcSnapmirror | Format-Table -Wrap -AutoSize -Property SourceLocation,DestinationLocation,Status,SnapshotProgress | Out-String

    #Only provide output if the SnapMirror relationship needs to be restarted
    if($Output) {
        $Output = $smStartStatus + $Output + $smFinalStatus
    } #endif

    return $Output
} #end Invoke-SnapmirrorRestart()

#Global Variables
$NetAppClusterIP = #<insert IP address or hostname of Cluster IP address>
$NetAppAdminUsername = #<insert username with admin rights to cluster or SVM>
$NetAppPassword = #<insert corresponding password>
$smtpServer = #<insert hostname or IP addresss of an open SMTP server>
$smtpFrom = "snapmirror-restart@mydomain.com"  #change to a valid from address
$smtpTo = "me@waflhouse.com"  #change to your email address
$messageSubject = "SnapMirror-Restart and Status" #change to fit your needs - subject of email

# Check NetApp Snapmirror status.
# Connect to the filer
Connect-Filer $NetAppClusterIP $NetAppAdminUsername $NetAppPassword

#Get SnapMirror Status.
$snapMirrorStatus = Invoke-SnapmirrorRestart
$messageBody = $snapMirrorStatus | Out-String

#Only send email if
if($snapMirrorStatus) {
    send-MailMessage -SmtpServer $smtpServer -To $smtpTo -From $smtpFrom -Subject $MessageSubject -Body $messageBody
} #endif

Kudos goes to Jason Edwards for inspiring this script with his post at: http://thoughts.stuart-edwards.info/index.php/programming/powershell/snapmirror-status-using-powershell

Leave a Comment Cancel reply