objecvtive: Let us see how can we process iostat information and print in summary, the below
a) timestamp of the sample
b) Avg run queue for the sample from all the devices
c) Avg of await for the sample from all the devices
d) Avg of the %util for the sample from all the devices
e) print the max util% and print its corresponding device name,await,runqueue
Step 1) Decide the iostat format
1 Sample of iostat look like below...
12/15/2018 04:28:15 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 66.99 32.52 0.00 0.49
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 4117.00 0.00 3338.00 0.00 29892.00 0.00 17.91 1.35 0.41 0.24 78.80
sda 4065.00 0.00 3812.00 0.00 31612.00 0.00 16.59 1.54 0.40 0.24 89.90
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Step 2) process each sample seperately
#!/bin/ksh
if [ $# -ne 1 ]
then
echo "please give valid input file to process as command line arg"
exit 1
fi
grep "12/15/2018" $1 > tmp_stp1.out
while read line
do
i=`echo $line|cut -d ' ' -f 2-3`
if [ -z $j ]
then
j=$i
prl=$line
else
## in this step we break the file into working piece with only necessary content using sed & awk
##first break the file into 1 sample @ a time
##delete the timestamps used for sample break
##delete trailing line
sed -n "/$j/,/$i/p" $1|sed '/$^/d' > tmp_stp2.out
j=$i
k=`grep -n Device tmp_stp2.out|cut -d ':' -f 1`
l=`grep -n "$i" tmp_stp2.out|cut -d ':' -f 1`
## further trimdown here, keep only necessary device information
awk -v m=$k -v n=$l 'NR>m && NR<n{print}' tmp_stp2.out > tmp_stp3.out
## process here
awk '{OFS=","}$10<=.1{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "rangeupto100ms: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out > tmp_stp4.out
awk '{OFS=","}$10<=1 && $10>.1{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "range100to1000ms: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out >> tmp_stp4.out
awk '{OFS=","}$10<=10 && $10>1{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "range1to10s: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out >> tmp_stp4.out
awk '{OFS=","}$10>10{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "rangeabove10s: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out >> tmp_stp4.out
while read line2
do
echo $prl,$line2
done < tmp_stp4.out
prl=$line
fi
done < tmp_stp1.out
sed -n "/$j/,/$i/p" iostat_rpt3.out|sed "$i/d;$j/d"|sed -e :a -e '/./,$!d;/^\n*$/{$d;N;};/\n$/ba' > tmp_stp2.out
Step 3) Also there is a scope for examining the max util% and print its corresponding device name,await,runqueue
let us do this only for 100 to 1000ms range of ios which is predominant...
#!/bin/ksh
if [ $# -ne 1 ]
then
echo "please give valid input file to process as command line arg"
exit 1
fi
grep "12/15/2018" $1 > tmp_stp1.out
while read line
do
i=`echo $line|cut -d ' ' -f 2-3`
if [ -z $j ]
then
j=$i
prl=$line
else
## in this step we break the file into working piece with only necessary content using sed & awk
##first break the file into 1 sample @ a time
##delete the timestamps used for sample break
##delete trailing line
sed -n "/$j/,/$i/p" $1|sed '/$^/d' > tmp_stp2.out
j=$i
k=`grep -n Device tmp_stp2.out|cut -d ':' -f 1`
l=`grep -n "$i" tmp_stp2.out|cut -d ':' -f 1`
## further trimdown here, keep only necessary device information
awk -v m=$k -v n=$l 'NR>m && NR<n{print}' tmp_stp2.out > tmp_stp3.out
## process here
awk -v max=0 '{OFS=","}
$10<=1 && $10>.1
{cnt+=1;a+=$10;b+=$9;c+=$12}
{if(max<$12){max=$12; udvn=$1; uawt=$10; uqu=$9}}
END {printf "range100to1000ms: %d,%.2f,%.2f,%.2f,%.2f,%s,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),b/((cnt>0)?cnt:1),c/((cnt>0)?cnt:1),max,udvn,uawt,uqu}' tmp_stp3.out > tmp_stp4.out
while read line2
do
echo $prl,$line2
done < tmp_stp4.out
prl=$line
fi
done < tmp_stp1.out
One anamoly, if there is only 1 device found with matching record for the data selection, it prints the whole line as well, need to troubleshoot that further.
input:
12/15/2018 04:17:36 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.16 0.00 1.14 0.72 0.00 97.98
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.37 0.00 10.05 0.00 335.22 0.02 66.65 0.50 49.84 0.94 0.95
sda 0.26 0.00 10.08 0.00 335.11 0.00 66.50 0.50 49.50 0.98 0.99
sdc 1.02 0.16 1.12 0.53 26.50 2.73 35.42 0.01 6.05 2.92 0.48
scd0 0.00 0.00 0.01 0.00 0.04 0.00 8.00 0.00 94.11 94.11 0.10
12/15/2018 04:17:37 PM
Output:
12/15/2018 04:17:36 PM,range100to1000ms: 5,39900.00,0.20,0.50,0.99,sda,49.50,0.50
39900ms = 39.9secs avg wait time.
So the anamoly...
input:
12/15/2018 04:17:38 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.51 0.51 0.00 98.98
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 1.00 0.00 1.00 0.00 2.00 0.00 1.00 1.00 0.10
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
output:
12/15/2018 04:17:38 PM,sda 0.00 0.00 1.00 0.00 1.00 0.00 2.00 0.00 1.00 1.00 0.10 <<<<<<<<< this is that extra line.
12/15/2018 04:17:38 PM,range100to1000ms: 5,200.00,0.00,0.02,0.10,sda,1.00,0.00 <<<<<<<<< this is wrong as well, its not 5 devices.its just 1 device.
Need to check this further.
a) timestamp of the sample
b) Avg run queue for the sample from all the devices
c) Avg of await for the sample from all the devices
d) Avg of the %util for the sample from all the devices
e) print the max util% and print its corresponding device name,await,runqueue
Step 1) Decide the iostat format
1 Sample of iostat look like below...
12/15/2018 04:28:15 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 66.99 32.52 0.00 0.49
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 4117.00 0.00 3338.00 0.00 29892.00 0.00 17.91 1.35 0.41 0.24 78.80
sda 4065.00 0.00 3812.00 0.00 31612.00 0.00 16.59 1.54 0.40 0.24 89.90
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Step 2) process each sample seperately
#!/bin/ksh
if [ $# -ne 1 ]
then
echo "please give valid input file to process as command line arg"
exit 1
fi
grep "12/15/2018" $1 > tmp_stp1.out
while read line
do
i=`echo $line|cut -d ' ' -f 2-3`
if [ -z $j ]
then
j=$i
prl=$line
else
## in this step we break the file into working piece with only necessary content using sed & awk
##first break the file into 1 sample @ a time
##delete the timestamps used for sample break
##delete trailing line
sed -n "/$j/,/$i/p" $1|sed '/$^/d' > tmp_stp2.out
j=$i
k=`grep -n Device tmp_stp2.out|cut -d ':' -f 1`
l=`grep -n "$i" tmp_stp2.out|cut -d ':' -f 1`
## further trimdown here, keep only necessary device information
awk -v m=$k -v n=$l 'NR>m && NR<n{print}' tmp_stp2.out > tmp_stp3.out
## process here
awk '{OFS=","}$10<=.1{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "rangeupto100ms: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out > tmp_stp4.out
awk '{OFS=","}$10<=1 && $10>.1{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "range100to1000ms: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out >> tmp_stp4.out
awk '{OFS=","}$10<=10 && $10>1{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "range1to10s: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out >> tmp_stp4.out
awk '{OFS=","}$10>10{cnt+=1;a+=$10;b+=$9;c+=$11}END{printf "rangeabove10s: %d,%.2f,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),(b*1000)/((cnt>0)?cnt:1),(c*1000)/((cnt>0)?cnt:1)}' tmp_stp3.out >> tmp_stp4.out
while read line2
do
echo $prl,$line2
done < tmp_stp4.out
prl=$line
fi
done < tmp_stp1.out
sed -n "/$j/,/$i/p" iostat_rpt3.out|sed "$i/d;$j/d"|sed -e :a -e '/./,$!d;/^\n*$/{$d;N;};/\n$/ba' > tmp_stp2.out
Step 3) Also there is a scope for examining the max util% and print its corresponding device name,await,runqueue
let us do this only for 100 to 1000ms range of ios which is predominant...
#!/bin/ksh
if [ $# -ne 1 ]
then
echo "please give valid input file to process as command line arg"
exit 1
fi
grep "12/15/2018" $1 > tmp_stp1.out
while read line
do
i=`echo $line|cut -d ' ' -f 2-3`
if [ -z $j ]
then
j=$i
prl=$line
else
## in this step we break the file into working piece with only necessary content using sed & awk
##first break the file into 1 sample @ a time
##delete the timestamps used for sample break
##delete trailing line
sed -n "/$j/,/$i/p" $1|sed '/$^/d' > tmp_stp2.out
j=$i
k=`grep -n Device tmp_stp2.out|cut -d ':' -f 1`
l=`grep -n "$i" tmp_stp2.out|cut -d ':' -f 1`
## further trimdown here, keep only necessary device information
awk -v m=$k -v n=$l 'NR>m && NR<n{print}' tmp_stp2.out > tmp_stp3.out
## process here
awk -v max=0 '{OFS=","}
$10<=1 && $10>.1
{cnt+=1;a+=$10;b+=$9;c+=$12}
{if(max<$12){max=$12; udvn=$1; uawt=$10; uqu=$9}}
END {printf "range100to1000ms: %d,%.2f,%.2f,%.2f,%.2f,%s,%.2f,%.2f\n",cnt,(a*1000)/((cnt>0)?cnt:1),b/((cnt>0)?cnt:1),c/((cnt>0)?cnt:1),max,udvn,uawt,uqu}' tmp_stp3.out > tmp_stp4.out
while read line2
do
echo $prl,$line2
done < tmp_stp4.out
prl=$line
fi
done < tmp_stp1.out
One anamoly, if there is only 1 device found with matching record for the data selection, it prints the whole line as well, need to troubleshoot that further.
input:
12/15/2018 04:17:36 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.16 0.00 1.14 0.72 0.00 97.98
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.37 0.00 10.05 0.00 335.22 0.02 66.65 0.50 49.84 0.94 0.95
sda 0.26 0.00 10.08 0.00 335.11 0.00 66.50 0.50 49.50 0.98 0.99
sdc 1.02 0.16 1.12 0.53 26.50 2.73 35.42 0.01 6.05 2.92 0.48
scd0 0.00 0.00 0.01 0.00 0.04 0.00 8.00 0.00 94.11 94.11 0.10
12/15/2018 04:17:37 PM
Output:
12/15/2018 04:17:36 PM,range100to1000ms: 5,39900.00,0.20,0.50,0.99,sda,49.50,0.50
39900ms = 39.9secs avg wait time.
So the anamoly...
input:
12/15/2018 04:17:38 PM
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.51 0.51 0.00 98.98
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 1.00 0.00 1.00 0.00 2.00 0.00 1.00 1.00 0.10
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
output:
12/15/2018 04:17:38 PM,sda 0.00 0.00 1.00 0.00 1.00 0.00 2.00 0.00 1.00 1.00 0.10 <<<<<<<<< this is that extra line.
12/15/2018 04:17:38 PM,range100to1000ms: 5,200.00,0.00,0.02,0.10,sda,1.00,0.00 <<<<<<<<< this is wrong as well, its not 5 devices.its just 1 device.
Need to check this further.
