Head Swap - FAS 3170 to FAS 6240

Overview:

1. Headswap 3170 to 6240
2. CIFS/NFS/iSCSI data
3. Cluster is SnapMirror source and destination

Systems:

Existing: FAS3170A TO New: FAS6240_1
Existing: FAS3170A TO New: FAS6240_2

Procedure:

1. Move Root Volume

a. NDMPcopy root volume ndmpcopy /vol/vol0 /vol/vol0_temp
b. Rename root volume once migrated
c. Make new vol root using vol options root
d. Reboot

2. Prepare for headswap

a. Modify root vol size to 250GB
b. Kick off ASUP
c. Make a copy of /etc directory
d. Capture LUN serial number info
e. Capture disk show –v and storage show disk -p
f. Verify hosts/applications connecting to iSCSI LUNs are shutdown
g. Verify CIFS and NFS protocols are shutdown
h. Verify SnapMirror is off
i. Disable cluster

3. Perform Headswap

a. Shutdown controllers
b. Verify all cards are added to new heads (check sysconfig guide for slot info)
c. Swap heads
d. Move FC and NIC cables from old head to new (using same FC port ID where available)
e. Boot heads

4. Configure New Heads

a. In maintenance mode, run disk show –v to verify all disks are seen
b. Reassign disks from old serial # to new serial #
c. Boot system and check aggr/vol/luns and connectivity
d. Add back old lun serial numbers using script
e. Modify rc file as necessary based on NIC port changes
f. Re-enable quotas if enabled
g. Disable ip fastpath
i. config.system.fast_ip.enable off
ii. options ip.fastpath.enable off
h. Customer to test and verify new environment
i. After full verification proceed to ontap upgrade

5. Upgrade OnTap

a. Upgrade ontap from 7.3.2 to 8.0.1
b. Enable cluster, test failover

6. Add New Shelves

a. Configure new SAS shelves and add to system
b. Cable ACP connections
c. Configure ACP

7. Final StepsCustomer to enable and resync SnapMirror relationships following day after headswap

Motherboard Replacement - Cluster

==Action Plan: Replacing Motherboard in cluster environment==

Goal: You will be replacing the motherboard for the system named XXX with serial number: XXX.

Impact: This is a non-disruptive action plan only if the correct parts are removed.

Please contact NetApp support immediately if there are any questions or concerns regarding this action plan.

System Downtime: 0 minutes

Plan Duration: 60 minutes

These steps assumes that the system with the faulty motherboard is up and running.

Laptop with console cable is required to perform this action plan.

BEGIN ACTION PLAN

On the partner head

1. If cluster is not in takeover, fail cluster over to the filer that does not have the bad motherboard.

filer2> cf takeover [wait for takeover to complete]

2. Gather the information about FC ports

filer2> partner fcadmin config [save the fc configuration information for later use in step 7]

On the head with bad motherboard

3. Replace motherboard

4. Move NVRAM card and any other additional cards from old motherboard to new one

5. Interrupt the boot process, at the firmware prompt, set the partner id variable;

LOADER> setenv partner-sysid

6. Boot into maintenance mode

7. Display the current setting of the FC ports (initiator vs target) and compare with output

from step 2.

filer1> fcadmin config

8. Set the correct FC port to target vs initiator.

filer1> fcadmin config -t target

AND

filer1> fcadmin config -t initiator

9. Verify FC port config and ensure that correct ports are set to initiator vs target:

filer1> fcadmin config

NOTE: The fcadmin config should be set the same as output from step 2.

10. Verify that you can see the correct amount of drives from maintenance mode.

11. Halt and boot into 'waiting for giveback

filer1> halt

LOADER> bye

On the partner head

12. Perform cf giveback

filer2> cf giveback

END ACTION PLAN

http://www.netapp-net2.com/documents/3/54/MB-FAS3040-70-SA300-006cfp.pdf

Flexshare

Flexshare - Dynamically prioritize storage traffice at the volume level.

Three Independent, Tunable Parameters

Each workload in your storage environment has a different level of importance to your business, so a "one-size-fits-all" approach to storage resource allocation doesn't make sense. FlexShare offers three parameters that can be configured independently on each volume to tune your storage to the needs of every application:

1. Relative priority. Assign a priority from "VeryLow" to "VeryHigh" to each volume. A higher priority gives a volume a greater percentage of available resources when a system is fully loaded. If higher priority applications aren't busy, lower priority applications can use available resources without limitation.

2. User versus system priority. Prioritize user workloads (application and end-user traffic) versus system work (backup, replication, and so on) or vice versa. For instance, give your OLTP application priority over SnapMirror®.

3. Cache utilization. Configure the cache to retain data in cache or reuse the cache depending on workload characteristics. Optimizing cache usage can significantly increase performance for data that is frequently read and/or written.

Filer> priority

The following commands are available; for more information

type "priority help "

delete off set show

help on

Filer> priority show

Priority scheduler is stopped.

Priority scheduler system settings:

io_concurrency: 8

Filer> priority on

Priority scheduler starting.

Filer> Wed Mar 17 14:46:38 EDT [Filer: wafl.priority.enable:info]: Priority scheduling is being enabled

Valid level and system options include:

1. VeryHigh
2. High
3. Medium
4. Low
5. VeryLow

Filer> priority set volume vol1 level=High

Filer> priority show volume -v vol1

Volume: vol1

Enabled: on

Level: High

System: Medium

Cache: n/a

Filer> priority delete volume vol2

Filer> priority show volume vol2

Unable to find priority scheduling information for 'vol2'

Below is a sample output of the FlexShare counters:

NetApp1*> stats show prisched

prisched:prisched:queued:0

prisched:prisched:queued_max:5

NetApp1*> stats show priorityqueue

priorityqueue:vol1:weight:76

priorityqueue:vol1:usr_weight:78

MULTIPATH made easy

Deducting Shelves

Not able to deduct newly attached shelves:

        slot 0: Fibre Channel Target Host Adapter 0c

        slot 0: Fibre Channel Target Host Adapter 0c

Make sure you enable the Fibre channel host Adapter.

fcadmin config -t initiator 0c

fcadmin config -t initiator 0d

you need a REBOOT for the changes to take effect and the disk shelves will be recognised by the filers.

Multipathing on Filers

All the filers in our environment are multipathed. Requirement of Multipathing:

System requirements

Multipath storage for active/active configurations is available only on the following storage system models:

FAS900 series
FAS2020 and FAS2050 systems
FAS30xx series
FAS3140 and FAS3170
FAS6000 series
SA200
SA300
SA600

Active/active configuration-type requirements

Multipath Storage is available for the following types of active/active configurations:

· Standard active/active configurations
· Mirrored active/active configurations
· Stretch Metro Clusters

Your active/active configuration must be installed and fully operational. Your configuration testing should include successful failover and giveback.

Note: Fabric-attached MetroClusters have redundant disk paths by default; no special configuration is necessary.

Disk shelf requirements

Multipath Storage for active/active configurations is available for only the following combinations of disk shelves and modules:

DS14mk2 FC or DS14mk4 FC disk shelves with ESH2 or ESH4 modules
DS14mk2 AT disk shelves with RoHS-compliant AT-FCX modules


Best practice recommendation

If any loop in your active/active configuration is cabled for Multipath Storage, every loop should be cabled for Multipath Storage. This is the recommended best practice.

Note: If you have a mixed configuration in which some loops are cabled for Multipath Storage and some are not, the system displays a configuration error message when you boot the system or when a disk on a loop that is cabled for Multipath becomes single-pathed.

NFS performance issue

NFS ops increases and high performance in Filer.

Step 1 :

See below output. NFS OPS ~90000 (column 2 )

Begin: Tue Sep 09 17:52:30 GMT 2008 

 CPU   NFS  CIFS  HTTP   Total    Net kB/s   Disk kB/s     Tape kB/s Cache Cache  CP   CP Disk    FCP iSCSI   FCP  kB/s

                                  in   out   read  write  read write   age   hit time  ty util                 in   out

 61% 93233     0     0   93233 16978 17120      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 61% 94057     0     0   94057 17171 17170     24     24     0     0   >60  100%   0%  -    3%      0     0     0     0

 61% 93384     0     0   93384 16998 16997      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 60% 93020     0     0   93020 16932 16930      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 59% 87212     0     0   87212 24549 16171     32     32     0     0   >60  100%   0%  -    3%      0     0     0     0

 62% 89947     0     0   89947 23317 16618      0      0     0     0   >60  100%   0%  -    0%      0     0     0     0

 61% 90205     1     0   90206 19157 16517      4      0     0     0   >60   99%   0%  -    1%      0     0     0     0

 59% 87641     0     0   87641 18877 16021   3728  22480     0     0   >60  100%  93%  Hf  33%      0     0     0     0

 61% 91401     0     0   91401 19355 16697    428   1012     0     0   >60  100%  13%  :    8%      0     0     0     0

 62% 92058     3     0   92061 19027 16943    392      8     0     0   >60  100%   0%  -

Step : 2

TIME: Tue Sep 09 18:00:02 GMT 2008TIME_DELTA: 07:33 (453s)nfsv3:nfs:nfsv3_read_latency:1426.57usnfsv3:nfs:nfsv3_read_ops:16/snfsv3:nfs:nfsv3_write_latency:267.78usnfsv3:nfs:nfsv3_write_ops:74/s

Step : 3

We can see 100 % of GETATTR request from the clients. SO most of the 90000 NFS OPS are for GETATTR . This can be due to your application behaviour orsome of your client is mis behaving in this environment.   
Server nfs V3: (43308280 calls)
null       getattr    setattr    lookup     access     readlink   read    
1 0%       43181988 100% 5302 0%    49543 0%   10529 0%   6 0%       3673 0% 
write      create     mkdir      symlink    mknod      remove     rmdir   
47953 0%   11 0%      1 0%       6 0%       0 0%       15 0%      1 0%    
rename     link       readdir    readdir+   fsstat     fsinfo     pathconf
1 0%       0 0%       5068 0%    3945 0%    129 0%     0 0%       108 0%  
commit     0 0%  

Step :4

192.168.4.101                 NFSOPS =  413629159 ( 1%)
192.168.4.95                  NFSOPS =  402901550 ( 1%)
192.168.4.78                  NFSOPS =  402401562 ( 1%)
192.168.4.79                  NFSOPS =  271381208 ( 1%)
192.168.4.83                  NFSOPS =  267008241 ( 1%)
192.168.4.102                 NFSOPS =  256459663 ( 1%)
192.168.4.66                  NFSOPS =  247692988 ( 1%)
192.168.4.84                  NFSOPS =  207956370 ( 0%)
192.168.4.100                 NFSOPS =  205229829 ( 0%)
192.168.4.80                  NFSOPS =  200581076 ( 0%)
192.168.4.97                  NFSOPS =  196879823 ( 0%)

Summary : Based on the perfstat we do not see any NFS latency on the filer end. CPU/DISK is normal on the filer but we do see high NFS OPS which looks

abnormal. Most of the NFS OPS are GETATTR calls and this can be either your application is malfunctioning or some of your client is malfunctioning. So checking 

your application or your clients is recomended. Also please check the client mount options and make sure you are not using "noac". Option "noac" will not 

cache the attribute information on the client end

Please also refer:

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb36582