Multipathing on Filers

All the filers in our environment are multipathed. Requirement of Multipathing:

System requirements

Multipath storage for active/active configurations is available only on the following storage system models:

FAS900 series
FAS2020 and FAS2050 systems
FAS30xx series
FAS3140 and FAS3170
FAS6000 series
SA200
SA300
SA600

Active/active configuration-type requirements

Multipath Storage is available for the following types of active/active configurations:

· Standard active/active configurations
· Mirrored active/active configurations
· Stretch Metro Clusters

Your active/active configuration must be installed and fully operational. Your configuration testing should include successful failover and giveback.

Note: Fabric-attached MetroClusters have redundant disk paths by default; no special configuration is necessary.

Disk shelf requirements

Multipath Storage for active/active configurations is available for only the following combinations of disk shelves and modules:

DS14mk2 FC or DS14mk4 FC disk shelves with ESH2 or ESH4 modules
DS14mk2 AT disk shelves with RoHS-compliant AT-FCX modules


Best practice recommendation

If any loop in your active/active configuration is cabled for Multipath Storage, every loop should be cabled for Multipath Storage. This is the recommended best practice.

Note: If you have a mixed configuration in which some loops are cabled for Multipath Storage and some are not, the system displays a configuration error message when you boot the system or when a disk on a loop that is cabled for Multipath becomes single-pathed.

NFS performance issue

NFS ops increases and high performance in Filer.

Step 1 :

See below output. NFS OPS ~90000 (column 2 )

Begin: Tue Sep 09 17:52:30 GMT 2008 

 CPU   NFS  CIFS  HTTP   Total    Net kB/s   Disk kB/s     Tape kB/s Cache Cache  CP   CP Disk    FCP iSCSI   FCP  kB/s

                                  in   out   read  write  read write   age   hit time  ty util                 in   out

 61% 93233     0     0   93233 16978 17120      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 61% 94057     0     0   94057 17171 17170     24     24     0     0   >60  100%   0%  -    3%      0     0     0     0

 61% 93384     0     0   93384 16998 16997      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 60% 93020     0     0   93020 16932 16930      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 59% 87212     0     0   87212 24549 16171     32     32     0     0   >60  100%   0%  -    3%      0     0     0     0

 62% 89947     0     0   89947 23317 16618      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 61% 90205     1     0   90206 19157 16517      4      0     0     0   >60   99%   0%  -    1%      0     0     0     0

 59% 87641     0     0   87641 18877 16021   3728  22480     0     0   >60  100%  93%  Hf  33%      0     0     0     0

 61% 91401     0     0   91401 19355 16697    428   1012     0     0   >60  100%  13%  :    8%      0     0     0     0

 62% 92058     3     0   92061 19027 16943    392      8     0     0   >60  100%   0%  -

Step : 2

TIME: Tue Sep 09 18:00:02 GMT 2008TIME_DELTA: 07:33 (453s)nfsv3:nfs:nfsv3_read_latency:1426.57usnfsv3:nfs:nfsv3_read_ops:16/snfsv3:nfs:nfsv3_write_latency:267.78usnfsv3:nfs:nfsv3_write_ops:74/s

Step : 3

We can see 100 % of GETATTR request from the clients. SO most of the 90000 NFS OPS are for GETATTR . This can be due to your application behaviour orsome of your client is mis behaving in this environment.   
Server nfs V3: (43308280 calls)
null       getattr    setattr    lookup     access     readlink   read    
1 0%       43181988 100% 5302 0%    49543 0%   10529 0%   6 0%       3673 0% 
write      create     mkdir      symlink    mknod      remove     rmdir   
47953 0%   11 0%      1 0%       6 0%       0 0%       15 0%      1 0%    
rename     link       readdir    readdir+   fsstat     fsinfo     pathconf
1 0%       0 0%       5068 0%    3945 0%    129 0%     0 0%       108 0%  
commit     0 0%  

Step :4

192.168.4.101                 NFSOPS =  413629159 ( 1%)
192.168.4.95                  NFSOPS =  402901550 ( 1%)
192.168.4.78                  NFSOPS =  402401562 ( 1%)
192.168.4.79                  NFSOPS =  271381208 ( 1%)
192.168.4.83                  NFSOPS =  267008241 ( 1%)
192.168.4.102                 NFSOPS =  256459663 ( 1%)
192.168.4.66                  NFSOPS =  247692988 ( 1%)
192.168.4.84                  NFSOPS =  207956370 ( 0%)
192.168.4.100                 NFSOPS =  205229829 ( 0%)
192.168.4.80                  NFSOPS =  200581076 ( 0%)
192.168.4.97                  NFSOPS =  196879823 ( 0%)

Summary : Based on the perfstat we do not see any NFS latency on the filer end. CPU/DISK is normal on the filer but we do see high NFS OPS which looks

abnormal. Most of the NFS OPS are GETATTR calls and this can be either your application is malfunctioning or some of your client is malfunctioning. So checking 

your application or your clients is recomended. Also please check the client mount options and make sure you are not using "noac". Option "noac" will not 

cache the attribute information on the client end

Please also refer:

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb36582

SnapManager for VI 1.0 (Virtual Infrastructure)

In Vmware environments :

1. snapshots may not be granular enough.
2. vmware is best in NFS than ISCSI.
3. Traditional backups disadvantages.. (i) dont match your VI (ii) No cpu power left on the server.
4. Recommended.. only 4 snapshot copies a day.
   
5. how SMVI works.. (i) Backup initiated. (ii) Vm snapshot copy created. (iii) Snapmanager for VI triggers instant NetApp snapshot. (iv) vm snapshot removed.
6. only VMFS/NFS support. No RDM support.
   
7. Protocols - ISCSI, FCP, NFS.
8. snapmirror is integrated.
9. Backup scheduling and retention policies
10. SMVI only support netapp storage.
11. Efficiently meet the backup and DR needs of your ESX environment. (i) Leverages netapp snapshot tech (ii) provides a way to restore VM and data stores

Snap Manager

SnapManager provides rapid online backup and near instantaneous restoration of
Exchange databases by using online Snapshot™ technology that is part of the
Data ONTAP® software and integrating the Microsoft Exchange backup and
restore APIs and VSS. SnapManager can also leverage the SnapMirror
capabilities of storage systems to provide on-site or off-site SnapManager backup
set mirroring for disaster recovery.

Data management:

SnapManager supports the following data management
capabilities:
◆ Migrating Exchange databases and transaction logs to LUNs on storage
systems
◆ Backing up Exchange databases and transaction logs from LUNs on storage
systems
◆ Verifying the backed-up Exchange databases and transaction logs
◆ Managing the SnapManager backup sets
◆ Restoring Exchange databases and transaction logs from previously created
SnapManager backup sets

Data archival for long-term or remote storage of backups: You can
use SnapManager to create offline archives of Snapshot copies containing
Exchange backup sets for long term or remote backup storage. Three different
archive methods are supported:
◆ Manually initiated archival using NDMP (Network Data Management
Protocol) or the storage system’s dump command
◆ Manually initiated archival using a Windows backup utility
◆ Automatic archival using the Run Command After Operation feature with
your backup operation
The SnapVault software can be used to archive backup sets.

What Snap Manager does NOT do :


SnapManager 4.0 for Microsoft Exchange does not support the following uses:
◆ SnapManager does not support the capability to restore individual mailboxes
or public folders.
◆ SnapManager does not support SnapDrive versions prior to 4.2.1.
◆ SnapManager does not create or restore backups of Microsoft Exchange
databases that are stored on storage devices that are provided by companies
other than NetApp.
◆ SnapManager does not support Microsoft Windows 2000.
◆ SnapManager does not restore Microsoft Exchange Server 2003 and
Microsoft Exchange Server 2007 databases to an alternate location.
◆ SnapManager does not back up and restore Microsoft Exchange 5.5 and
Exchange 2000 databases.
◆ SnapManager is not supported for use on Windows Server 2003 IA-64
edition.

Ontap Upgrade 7.2.3 to 7.2.4 (Non disruptive Upgrade)

NetApp 3020 upgrade of Data OnTap 7.2.3 to Data OnTap 7.2.4

Performing a non-disruptive upgrade of our Network Appliance FAS 3020 (clustered filer configuration)

One of the benefits of having the clustered filers (FAS3020c) is that I can, in most cases, perform a system upgrade without having to disrupt services running on either system. The process is a little complex but well worth the payoff as in our environment I literally have thousands of students connecting to the storage at a time. Below is a slightly modified version of my notes from the upgrade (use at your own risk). I followed the directions from NetApp's upgrade guide. Although, I will note that their directions were not exact. I had differing outputs from commands at times which made me a little nervous. All in all the upgrade went pretty smooth and the systems have been running solid since.

Download from now.netapp.com under Download Software – Data OnTap – FAS 3020c

• new Shelf Firmware from now.netapp.com (all shelf firmware updates)
• new Disk Firmware from now.netapp.com (all disk firmware updates)
• newest release of Filer Firmware CFE 3.1
• newest GA Release of Data OnTap 7.2.3
• docs for Data OnTap 7.2.3

Copied and Made backups of files

• Mounted \\filerA\c$
• Mounted \\filerB\c$
• Made of backup of c$\etc\ folder on both systems (minus log files)
  - Copy to c$\backup\etc_8-24-2007
• From shelf zip file to the etc\shelf_fw on the both filerA and filerB
• From shelf zip file to the etc\shelf_fw on the both filerA and filerA
• From disk zip file to the etc\disk_fw on the both filerA and filerB
• From disk zip file to the etc\disk_fw on the both filerA and filerA

Shelf Firmware

• Login to the appliance console.
• Check current shelf firmware version ( > sysconfig -v )
• Enter Advanced privileges ( > priv set advanced )
• Start the update (> storage download shelf)
  - This will upgrade the shelf firmware on all the disk shelves in the system. (If you wish to only update the disk shelves attached to a specific adapter, enter storage download shelf adapter_number instead).
• Accept the update, Press y for yes and hit enter.
• To verify the new shelf firmware, ( > sysconfig -v )
• Exit Advanced privileges ( > priv set admin )

Disk Firmware

Disk firmware is automatically updated on reboot if there are updated files in the disk_fw folder. To keep the system from updating too many disks at once set or verify the following option.

• ( > options raid.background_disk_fw_update.enable)
  - if it is set to off, I recommend you change it to on

DataOnTap Update

1. Downloaded the newest General Deployment Release, in this case it was Data ONTAP 7.2.4.
2. Verified our system met all requirements for running the downloaded release, updates were required for Disk firmware and shelf firmware (which was done above)
3. Checked known problems and limitations of the new release to see if any would affect our environment.
4. Compared bug fixes from current version of OnTap 7.2.3 to new version of 7.2.4.
5. Downloaded newest documentation for 7.2.4

Update Procedure

With C$ mapped on both filers I ran the downloaded OS install (self extracting zip files) to the respective \etc directories. This is the first step and copies all the needed files over to the filers. Once completed, we perform the procedure below from the NOW upgrade guide for Windows Clients.

1. start the install on both systems ( > download )
2. Checked the cluster status ( > cf status ) to make sure cluster failover was enabled
3. Had filerB takeover services for filerA ( > cf takeover )
   - This causes filerA to reboot
4. During reboot of filerA hit ( ctrl-c ) to enter into maintenance mode
5. From maintenance mode type ( > halt ) to do a full reboot
6. Hit ( del ) during memory test to get to the CFE prompt
7. start the firmware update of the filer from the CFE> prompt using ( CFE> update_flash )
8. Now reboot, type ( bye ) at console after update was finished to reboot filerA
9. filerA is now in a …waiting for giveback state
10. Now to give services back to filerA we have to force it using ( > cf giveback –f ) from filerB
    - This is required since we are now on different version of DataOnTap between systems in the cluster.
11. Giveback successful, checked firmware and OS version on filerA using ( > sysconfig –v )
12. After checking services on both systems it's time to upgrade filerB
13. Have filerA take over the services of filerB ( > cf takeover –n )
14. Type ( > halt ) from filerB to reboot it
15. During reboot of filerB hit ( ctrl-c ) to enter into maintenance mode
16. From maintenance mode type ( > halt ) to do a full reboot
17. Hit ( del ) during memory test to get to the CFE prompt
18. start the firmware update of the filer from the CFE> prompt using ( CFE> update_flash )
19. Typed ( bye ) at console after update was finished to reboot filerB
20. filerB is now in a …waiting for giveback state
21. Now to give services back to filerB we have to force it using (> cf giveback –f ) from filerA
    - This is required since we are now on different version of DataOnTap between systems in the cluster.
22. Giveback successful, checked firmeware and os version on filerB using ( > sysconfig –v )
1. Both systems should now show the updated firmware and OnTap version 7.2.4
23. You should also notice that any out of date disk firmware is automatically updated.

My final steps were to test system connections

1. We use the following NetApp services: CIFS, FTP, HTTP, FCP via VMWARE. All worked fine. I Also checked our student websites and our web based FTP software that connects to the filer.
2. Checked Domain connection using cifs testdc ( filerA> cifs testdc )
   - appeared fine

Addt. Note: If you have MCCS cluster. Please make sure it is properly brought down before the reboot. Sometimes your luns doesn't come up.

Upgrading Shelf Firmware

To update the firmware without rebooting, during a maintenance window, complete the following:

1. Login to the storage system's console.

2. Type priv set advanced.

3. Enter storage download shelf. This will upgrade the shelf firmware on all the disk shelves in the system. (If you wish to only update the disk shelves attached to a specific adapter, enter storage download shelf adapter_number instead).

4. Press y for yes and hit enter.

5. To verify the new shelf firmware, run sysconfig -v.

VIFS on NetApp

Vifs (Virtual interfaces) are used for multi-homing and load balancing

A Single mode vif is for failovers only

Multi-mode vif is for load balancing

You can use a combination of multi-mode and single mode to be fully redundant and load balanced.

For more information on VIFS please check the NetApp now site.

Note: as of OnTap 7.2.3. there is a bug in Filerview with vifs and interfaces DO NOT modify interfaces or vifs using Filerview use command line and edit RC and hosts files to preserve settings.

To create a single mode vif

vif create single sample-vif0 e0a e0c

To configure an interface to use a vif

ifconfig sample-vif0 `hostname`-pub mediatype auto netmask 255.255.255.0 wins partner cluster-vif0

Make sure you reference the vif name not the interface name

To view all interfaces including vifs

Ifconfig –a

To view a single interface or vif (note you must view the vif in order to see ip address)

Ifconfig sample e0a

Ifconfig sample-vif0

To view all vifs status

vif status

To view single vif

vif status sample-vif0

To view statistics

vif stat sample-vif0

If you need to remove an interface from a vif

vif delete sample-vif0 e0c (This will remove e0c from sample-vif0)

To favor one interface in a VIF (this always be the one that’s used if up, if it ever goes down and then comes back up it will fail back to it)

vif favor e0a

To see more commands type the following at a command prompt.

Vif ?