Assess process monitoring output in Linux to identify the root cause of the issue - with Gen AI assistance

 

Requirement: Launch a process monitoring command and process the output using a shell script & use an AI tool to document the flow

tools: Shell script, gemini & mermaidchart.com

This is part of a multipart series in identifying a root cause on a typical chicken-egg story kind of issue :)

YouTube: 

Step 1) Command which monitors the process:

You can launch the below command on a abnormally loaded machine; so you can use this output to assess the RCA. As going after top command every minute is challenging.

while true do echo "zzz ***$(date '+%a %b %e %T %Z %Y')" >> /tmp/psinfo2.out ps -aeo user,pid,ppid,pri,pcpu,pmem,vsize,rssize,wchan:42,s,start,cputime,command >> /tmp/psinfo2.out sleep 30 done

Step 2) Develop a script that is able to digest the above commands output and print it to a more readable info [preprocess]

#!/bin/ksh if [ $# -ne 1 ] then echo "please provide valid osstats file" exit 1 fi rm tmp_stp*.out grep "zzz" $1 > tmp_stp1.out unset j while read line do i=$(echo $line|cut -d ' ' -f 3-7) if ${j+"false"} then j=$i else ## in this step we break the file into working piece with only necessary content using sed & awk ##first break the file into 1 sample @ a time sed -n "/$j/,/$i/p" $1 >tmp_stp2.out; ##sort the file on field 11 to fix the start time issues like since "Jan 28" & for last 13:00:00 etc.. since both needs different processing. sort -nk11 tmp_stp2.out >tmp_stp3.out; csplit -z tmp_stp3.out /zzz/ '{*}' -f 'tmp_stp4_' >/dev/null 2>&1; for a in $(ls -tr tmp_st4_*) do sed -i '/zzz/d' $a; sed -i '/WCHAN/d' $a; sed -r -i '/^\s*$/d' $a; lncnt=$(cat $a|wc -l); if [ ${lncnt} -gt 0 ]; then chk1=$(head -1 $a|awk '{print $11}'); dt=$(date -d"$j" '+%d/%b/%Y %H:%M:%S'); if [[ "$chk1" =~ ^[a-zA-Z]+$ ]]; then while read line2 do tmpln=$(echo $line2|awk '{print $1","$2","$3","$4","$5","$6","$7","$8","$9","$10","$11"-"$12","$13}'); echo $dt","${tmpln} done <$a else while read line2 do tmpln=$(echo $line2|awk '{print $1","$2","$3","$4","$5","$6","$7","$8","$9","$10","$11","$12","$13}'); echo $dt","${tmpln} done <$a fi fi done j=$i fi done <tmp_stp1.out

Step 3) document the above code flow diagram and write code information (kind of readme) using GENAI

Here we will use gemini… I pasted the output from gemini below after several prompts to make it accurarte

graph TD A(Start) --> B(Argument count = 1?); B -- No --> C[Display error]; C --> D[Exit]; B -- Yes --> E[Remove temp files]; E --> F[Filter zzz to temp file 1]; F --> G[Init j]; G --> H[Loop: Read line from temp file 1]; H --> I[i = fields 3-7]; I --> J(j set?); J -- No --> K[j = i]; J -- Yes --> L[Extract lines between j and i to temp file 2]; L --> M[Sort temp file 2 by field 11 to temp file 3]; M --> N[Split temp file 3 at zzz to temp files]; N --> O[Loop: Iterate temp files]; O --> P[Remove zzz, WCHAN, blank lines]; P --> Q[Count lines]; Q --> R(Line count > 0?); R -- Yes --> S[Get field 11]; S --> T[Get date]; T --> U(Field is letters?); U -- Yes --> V[Loop: Read line]; V --> W[Format line with j and modified field 11 to temp file 5]; W --> X[End loop]; X --> Y[End conditional]; U -- No --> AA[Loop: Read line]; AA --> AB[Format line with j to temp file 5]; AB --> AC[End loop]; AC --> Y; R -- No --> Y; Y --> AD[End loop]; AD --> AE[j = i]; AE --> AF[End conditional]; AF --> H; AF --> AG[End]; X --> V; AC --> AA; AD --> O;

Step 4) Invoke the script developed

sh processosstat.sh <inputfile> ><processed_output.file>

sh processosstat_noop.sh /tmp/psinfo2.out >processed_ps2.out

Output:

[oracle@vcentos79-oracle-sa1 OSpsstat]$ head -10 processed_ps2.out 13/Feb/2025 22:44:42,chrony,920,1,19,0.0,0.0,117808,1852,poll_schedule_timeout,S,11:51:53,00:00:00,/usr/sbin/chronyd 13/Feb/2025 22:44:42,dbus,915,1,19,0.0,0.0,66456,2584,ep_poll,S,11:51:53,00:00:00,/usr/bin/dbus-daemon 13/Feb/2025 22:44:42,oracle,11558,11548,19,0.0,0.0,163364,2672,poll_schedule_timeout,S,11:56:04,00:00:00,sshd: 13/Feb/2025 22:44:42,oracle,11559,11558,19,0.0,0.0,115680,2208,do_wait,S,11:56:04,00:00:00,-bash 13/Feb/2025 22:44:42,oracle,930,1,19,0.2,1.0,1180400,41332,futex_wait_queue_me,S,11:51:53,00:00:14,/u01/app/oracle/product/21.3.0/ogg_home_1/bin/ServiceManager 13/Feb/2025 22:44:42,polkitd,929,1,19,0.0,0.2,613016,9176,poll_schedule_timeout,S,11:51:53,00:00:00,/usr/lib/polkit-1/polkitd 13/Feb/2025 22:44:42,postfix,1984,1979,19,0.0,0.1,89976,4108,ep_poll,S,11:51:59,00:00:00,qmgr 13/Feb/2025 22:44:42,root,1,0,19,0.0,0.1,128280,6972,ep_poll,S,11:51:45,00:00:02,/usr/lib/systemd/systemd 13/Feb/2025 22:44:42,root,10,2,39,0.0,0.0,0,0,rescuer_thread,S,11:51:45,00:00:00,[lru-add-drain] 13/Feb/2025 22:44:42,root,11,2,139,0.0,0.0,0,0,smpboot_thread_fn,S,11:51:45,00:00:00,[watchdog/0] [oracle@vcentos79-oracle-sa1 OSpsstat]$

Step 5) Now use your favourite tool include excel to perform analytics

This will be part 2 in our case.

Thank you!

Situation: Loosing a Directory by incorrect mount procedure (mountpath mixup!!!) -- fixup routine

 Situation: Loosing a Directory by incorrect mount procedure (mountpath mixup!!!)

YouTube Video:

Learning target: what should we do on such situation?

If we unmount the new fs, will we get our old fs back?

Test setup:

1. create /test directory

2. Ensure it is created on / filesystem

mkdir /test

3. create a subdirectory testsub in /test and touch a file called testfile

mkdir /test/testsub

[root@vcentos79-oracle-sa1 /]# mkdir /test

[root@vcentos79-oracle-sa1 /]# cd /test/

[root@vcentos79-oracle-sa1 test]# mkdir testdir

[root@vcentos79-oracle-sa1 test]# mv testdir testsub

[root@vcentos79-oracle-sa1 test]# touch testsub/testfile

[root@vcentos79-oracle-sa1 test]# ls -altr

total 4

dr-xr-xr-x. 25 root root 4096 Feb  5 00:14 ..

drwxr-xr-x.  3 root root   21 Feb  5 00:14 .

drwxr-xr-x.  2 root root   22 Feb  5 00:14 testsub

[root@vcentos79-oracle-sa1 test]# cd testsub

[root@vcentos79-oracle-sa1 testsub]# ls -altr

total 0

drwxr-xr-x. 3 root root 21 Feb  5 00:14 ..

-rw-r--r--. 1 root root  0 Feb  5 00:14 testfile

drwxr-xr-x. 2 root root 22 Feb  5 00:14 .

[root@vcentos79-oracle-sa1 testsub]# pwd

/test/testsub

[root@vcentos79-oracle-sa1 testsub]# pwd

/test/testsub

[root@vcentos79-oracle-sa1 testsub]# cd

[root@vcentos79-oracle-sa1 ~]# df -Th /test

Filesystem              Type  Size  Used Avail Use% Mounted on

/dev/mapper/centos-root xfs    17G  4.7G   13G  28% /

[root@vcentos79-oracle-sa1 ~]#

4. Now as if you didnt know that there is a subdirectory called testsub, let us create a new FS called /test (mountpath)

fdisk /dev/sde

Command (m for help): n

Partition type:

   p   primary (0 primary, 0 extended, 4 free)

   e   extended

Select (default p):

Using default response p

Partition number (1-4, default 1):

First sector (2048-2097151, default 2048):

Using default value 2048

Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151):

Using default value 2097151

Partition 1 of type Linux and of size 1023 MiB is set

Command (m for help): w

The partition table has been altered!

Command (m for help): t

Selected partition 1

Hex code (type L to list all codes): q

Changed type of partition 'Linux' to 'unknown'

Command (m for help): t

Selected partition 1

Hex code (type L to list all codes): 1

WARNING: If you have created or modified any DOS 6.xpartitions, please see the fdisk manual page for additionalinformation.

Changed type of partition 'Empty' to 'FAT12'

Command (m for help): t

Selected partition 1

Hex code (type L to list all codes): 8e

WARNING: If you have created or modified any DOS 6.xpartitions, please see the fdisk manual page for additionalinformation.

Changed type of partition 'FAT12' to 'Linux LVM'

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@vcentos79-oracle-sa1 ~]#

Disk /dev/sde: 1073 MB, 1073741824 bytes, 2097152 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x4d9dd83f

   Device Boot      Start         End      Blocks   Id  System

/dev/sde1            2048     2097151     1047552   8e  Linux LVM

[root@vcentos79-oracle-sa1 ~]# pvcreate /dev/sde1

  Physical volume "/dev/sde1" successfully created.

[root@vcentos79-oracle-sa1 ~]# vgcreate testvg /dev/sde1

  Volume group "testvg" successfully created

[root@vcentos79-oracle-sa1 ~]# lvcreate -L 1000 testvg -n testvg-test00

  Logical volume "testvg-test00" created.

lvdisplay

  --- Logical volume ---

  LV Path                /dev/testvg/testvg-test00

  LV Name                testvg-test00

  VG Name                testvg

  LV UUID                SIZBUy-FdtH-d6oV-KOFU-OUgg-zMan-H63Kqc

  LV Write Access        read/write

  LV Creation host, time vcentos79-oracle-sa1, 2025-02-05 00:25:24 +0000

  LV Status              available

  # open                 0

  LV Size                1000.00 MiB

  Current LE             250

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     8192

  Block device           253:10

[root@vcentos79-oracle-sa1 ~]# mkfs.xfs /dev/testvg/testvg-test00

meta-data=/dev/testvg/testvg-test00 isize=512    agcount=4, agsize=64000 blks

         =                       sectsz=512   attr=2, projid32bit=1

         =                       crc=1        finobt=0, sparse=0

data     =                       bsize=4096   blocks=256000, imaxpct=25

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

log      =internal log           bsize=4096   blocks=855, version=2

         =                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@vcentos79-oracle-sa1 ~]# lvdisplay|grep "LV Path"

...

  LV Path                /dev/testvg/testvg-test00

5. Mount it and see what happens.

a. You will loose the files

mount /dev/testvg/testvg-test00 /test

/dev/mapper/testvg-testvg--test00               xfs       997M   33M  965M   4% /test

[root@vcentos79-oracle-sa1 /]# cd /test

[root@vcentos79-oracle-sa1 test]# ls -altr

total 4

dr-xr-xr-x. 25 root root 4096 Feb  5 00:14 ..

drwxr-xr-x.  2 root root    6 Feb  5 00:25 .

[root@vcentos79-oracle-sa1 test]#

Recovery option:

6. Unmount the FS and you will see the old file coming back.

umount /test

7. List the files:

[root@vcentos79-oracle-sa1 /]# cd test

[root@vcentos79-oracle-sa1 test]# ls -altr

total 4

dr-xr-xr-x. 25 root root 4096 Feb  5 00:14 ..

drwxr-xr-x.  3 root root   21 Feb  5 00:14 .

drwxr-xr-x.  2 root root   22 Feb  5 00:14 testsub

[root@vcentos79-oracle-sa1 test]# cd testsub/

[root@vcentos79-oracle-sa1 testsub]# ls -altr

total 0

drwxr-xr-x. 3 root root 21 Feb  5 00:14 ..

-rw-r--r--. 1 root root  0 Feb  5 00:14 testfile

drwxr-xr-x. 2 root root 22 Feb  5 00:14 .

[root@vcentos79-oracle-sa1 testsub]#

8. Mount the /test again

[root@vcentos79-oracle-sa1 /]# mount /dev/testvg/testvg-test00 /test

[root@vcentos79-oracle-sa1 /]# cd /test

[root@vcentos79-oracle-sa1 test]# ls -altr

total 4

dr-xr-xr-x. 25 root root 4096 Feb  5 00:14 ..

drwxr-xr-x.  2 root root    6 Feb  5 00:25 .

[root@vcentos79-oracle-sa1 test]# cd

[root@vcentos79-oracle-sa1 ~]#

you loose it.

9. unmount you get the old content back again

[root@vcentos79-oracle-sa1 ~]# umount /test

[root@vcentos79-oracle-sa1 ~]# ls -altr /test/*/*

-rw-r--r--. 1 root root 0 Feb  5 00:14 /test/testsub/testfile

[root@vcentos79-oracle-sa1 ~]#

************** this closes the lab