LXC Containers on Fedora 21

HOWTO: Configure a LXC Linux Container on Fedora 21

First time i had encounter linux container some where in 2011 on SUSE 11, when my collage all at the time discussing about Solaris Zone. So decide to dig out some thing which look & feel like solaris zones in Linux.

Concept: 

Containers provide lightweight virtualization that lets you isolate processes and resources without the need to provide instruction interpretation mechanisms and other complexities of full virtualization.

Prerequisites:

1. Physical machine or a VM running Fedora 21.
2. Yum should be configured.
3. Enough System resources.
4. Update all packages to the latest versions available.

In my scenario: 

I am using MacBook Air as Physical Machine & for Virtulization using VMWare Fusion & on top of it Fedora 21 as guest OS, where i am about to play with LXC; You can see the beauty of Virtuliztaion, it always helping me to try the hell of the new technologies with minimum huddles & maximum outcome.

Screen Shot 2015-02-04 at 2.50.51 am

Install management libraries and utilities: 

Amits-MacBook-Air:~ amitvashist$ fed21
root@192.168.13.131's password:
Last login: Thu Jan 15 16:55:57 2015 from 192.168.13.1


[root@fedora21 ~]# yum history list 5
Failed to set locale, defaulting to C
Loaded plugins: langpacks
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
 5 | install libvirt libvirt- | 2014-12-31 10:11 | Install        |    6   
history list
[root@fedora21 ~]#


[root@fedora21 ~]#yum -y install libvirt-daemon-lxc libvirt-daemon-config-network

Launch libvirtd via systemd and ensure that it always comes up on boot. This step will also adjust firewalld for your containers and ensure that dnsmasq is serving up IP addresses via DHCP on your default NAT network.

[root@fedora21 ~]# systemctl start libvirtd.service
[root@fedora21 ~]# systemctl enable libvirtd.service
[root@fedora21 ~]# systemctl status network.service
[root@fedora21 ~]# systemctl start network.service
[root@fedora21 ~]# systemctl enable network.service

Now we ready to download / install container’s filesystem:

[root@fedora21 ~]# yum -y --installroot=/var/lib/libvirt/filesystems/fedora21 --releasever=21 --nogpg install systemd passwd yum fedora-release vim-minimal openssh-server procps-ng iproute net-tools dhclient

With the above mentioned step we have download the filesystem with the necessary packages to run a Fedora 21 container. We now need to tell libvirt about the container we’ve just created.

[root@fedora21 ~]# virt-install --connect lxc:// --name MyTestFedora21 --ram 512 --filesystem /var/lib/libvirt/filesystems/fedora21/,/

Screen Shot 2015-02-04 at 3.14.33 am

Now you container is up & running but in order to connected to the console of the container! We need to adjust some configuration files within the container to use it properly. Detach from the console with CTRL-].

Screen Shot 2015-02-04 at 3.16.10 am

So for now let’s stop the container so we can make some adjustments.

[root@fedora21 ~]# virsh -c lxc:// list
setlocale: No such file or directory
Id    Name                           State
----------------------------------------------------
61488 MyTestFedora21                 running
[root@fedora21 ~]#
[root@fedora21 ~]# virsh -c lxc:// shutdown MyTestFedora21
setlocale: No such file or directory
Domain MyTestFedora21 is being shutdown
[root@fedora21 ~]#

Let the container ready for production: 

  • Setup SELinux in permissive mode only for password modification, else getting:
[root@fedora21 ~]# chroot /var/lib/libvirt/filesystems/fedora21 /bin/passwd root
Changing password for user root.
New password:
Retype new password:
passwd: Authentication token manipulation error
[root@fedora21 ~]#
  • Setup the root password

Screen Shot 2015-02-04 at 3.35.06 am

  • Setup the required network configuration
[root@fedora21 ~]#cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network
NETWORKING=yes
EOF
[root@fedora21 ~]#cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=static
ONBOOT=yes
DEVICE=eth0
IPADDR=192.168.122.10
NETMASK=255.255.255.0
EOF

Launching the container by keeping finger crossed :

Launching sequence : 3 2 1 – Boom!!!

[root@fedora21 ~]# virsh -c lxc:// start MyTestFedora21
setlocale: No such file or directory
Domain MyTestFedora21 started
[root@fedora21 ~]#

Screen Shot 2015-02-04 at 3.50.22 am

Now login to container with new root password: 

Screen Shot 2015-02-04 at 3.51.17 am

Testing SSH Connectivity & it seems pretty good.

Screen Shot 2015-02-04 at 4.09.17 am

Happy learning 🙂 🙂

And Please do share your comments 🙂

Cheers!!!

Advertisements
Posted in Linux, Uncategorized, Virtualization | Tagged , , , , , , | Leave a comment

Linux Cluster Suite

Red Hat Cluster Suite Failover Cluster explained using HTTP as a failover service

Cluster:  A Group of two or more computers to perform a same task. Through this document we will explain about the Red Hat Cluster Implementation via Conga Project which has 2 important services running on Base Node and Cluster Nodes respectively.

LUCI: Luci is the service which gets installed on a separate base node which gives us an complete functionality via Admin Console in order to create / Configure / Manage our nodes in cluster.

RICCI: Ricci is the service known as Agent Service which gets installed on all the nodes in the cluster and it is because of this service via LUCI ADMIN they will become JOIN Clusters.

CMAN: Cluster Management: CMAN manages the quorum and cluster membership. A very important component (service) of Red Hat Cluster hence mandatory to run on each of the nodes

Fencing: Fencing is a mechanism of disconnecting a node from the cluster in case node has gone down faulty in order to avoid the data corruption and maintain data integrity.

Lock Management: Red Hat Cluster provides this lock management via known as DLM ( Distributed Lock Manager ). GFS uses locks from lock manager in order to synchronize their access to shared file system metadata.CLVM uses locks from lock manager in order to synchronize their updates to LVM Volumes and Volume Groups on a shared storage.

Cluster Configuration

Cluster Configuration file lies under /etc/cluster/cluster.conf and is an XML file Cluster Resources are defined under cluster configuration file like IP Address , Script , Red Hat Storage GFS 2 Maximum number of nodes supported in Red hat cluster deployment of GFS/GFS2 is 16

Through this document we shall explain the deployment of Apache Application in Red Hat Cluster

Hardware Requirements:

Server:For setting up 2 Node Cluster we may take 2 servers (Quad Core Quad CPU HP machine with minimum of 4GB RAM) and 1 Base node [Quad Core Quad CPU HP machine with minimum of 4GB RAM) on which we will have LUCI install.

Note : You can choose Virtual Nodes as well but there are certain limitation’s with VM Fencing.

Requirements

1.  IP Detailed Requirement:-

1. Server 1 - 2 Local IP for (bonding) HTTP server  + 1 IP for Cluster fencing
2. Server 2 - 2 Local IP for (bonding) HTTP server + 1 IP  for Cluster fencing 
3. Virtual IP - 1 virtual IP for HTTP ( Cluster IP )

2. Storage Requirement:-

a. One 200 GB SAN LUN for Database. (Depend upon requirement)

How to Configure Apache Cluster

Step 1: Firstly Install RHEL 5.6 on both the services respectively with custom packages.

Step 2 : Configure Network Bonding

Steps for creating bonding are as mentioned below: Create the bond interface file for the public network and save the file as                                        # vim /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=192.168.5.20 [This will be actual network IP address]
NETMASK=255.255.255.0
GATEWAY=192.168.5.1
USERCTL=no
BOOTPROTO=static
ONBOOT=yes

 

After creating bond0 file, modify eth0 and eth1 file respectively.

# vim /etc/sysconfig/network-scripts/ifcfg-eth0

Make sure you remove HW Address / IP Address / Gateway Information from eth0 and eth1 and add 2 important lines under those file:

# vim /etc/sysconfig/network-scripts/ifcfg-eth0
Make sure file read as follows for eth1 interface:
DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
# vim /etc/sysconfig/network-scripts/ifcfg-eth1
Make sure file read as follows for eth1 interface:
DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

Load bond driver/module

Edit vim /etc/modprobe.conf
Append following two lines:
alias bond0 bonding
options bond0 mode=balance-alb miimon=100
Save the file accordingly

Test configuration

First, load the bonding module, enter:
# modprobe bonding

Restart the networking service in order to bring up bond0 interface, enter:
# service network restart

Check with the below command whether Bonding is actually working or not.
# cat /proc/net/bonding/bond0

Step 3: Need to set the hostname on the Base node and other 2 nodes as well namely.

vim /etc/hosts
192.168.5.20 station20.example.com station20
192.168.5.10 station10.example.com station10
192.168.5.30 station30.example.com  station30

Step 4: Password Less authentication shall be among both the nodes namely station20.example.com and station30.example.com

Login on station20 and enter command as ssh-keygen
#ssh-copy-id  -i  /root/.ssh/id_rsa.pub station30

Login on station30 and enter command as ssh-keygen
#ssh-copy-id  -i /root/.ssh/id_rsa.pub station20

Step 5 : Set the yum repository on the base node [ station10 ] and other 2 servers

Step 6: Make sure IPTABLES and SELINUX are disabled on all the three machines

Step 7: Login on Base node and first install LUCI and cluster packages

#yum groupinstall  “ClusterStorage”  -y
#yum install luci*

Run command as #luci_admin init

Above command will generate a ssl certificate and asks for a password for user admin

Assign the password and it will come on # prompt stating we may login from URL as 

https://192.168.5.10:8084 via username as admin and password as redhat [assume we have given password as redhat]

#service luci restart && chkconfig luci on

Step 8: Login on other 2 nodes and first install

#yum groupinstall “ClusterStorage” –y 
#yum install ricci* -y 
#service ricci restart && chkconfig ricci on 

Once above all steps are done, we may need to login from https://192.168.5.10:8084 and start building our cluster.

Step 9: We will use Fencing Device as ILO while building this cluster hence will add the user id and password under the ILO configuration option available in BIOS mode and also set the IP address accordingly. This will be done on both the nodes accordingly.

Need to check whether manual fencing is working or not by logging on each node:

Step 10: Login on station20 and run command as :

#fence_ilo   -a  station30 -l admin  -p   redhat  -o reboot

Username as admin and Password as redhat which we have assigned inthe ILO configuration 

Login on station30 and run command as:
#fence_ilo  -a  station20  -l  admin  -p  redhat  -o reboot 

Step 11: Assign 200 GB Storage (LUN)  for Apache, 100 GB LUN should be visible in both server node1 and node2

Step 12: Create LVM on 200GB  LUN

Step 13: vim /etc/lvm/lvm.conf and set the locking_type=3 which makes LVM a Clustered aware file system

Step 14: Install Apache on both the servers respectively as node1 and node2

Step 15 : Configure the cluster, Login on https://192.168.5.10:8084

First Step which we need to do after Login via Luci console is to create a cluster

Click Cluster > Create a new cluster and add the node host name and password

Step 16 :Mention the Cluster Name as Cluster_01 and enter the both nodes name respectively with their password as shown in the screen shot as mentioned below

Step 17 :Click on view SSL finger print and it shall verify the finger print as mentioned in the screen shot as mentioned below

Redhat_Cluster_Conga_1

Step 18: Once we click on submit button it will INSTALL / REBOOT / CONFIGURE and JOIN the node in the cluster Redhat_Cluster_Conga_2Redhat_Cluster_Conga_3 Step 19: After Installation is successful we may login on each station20.example.com and station30.example.com and can check our Cluster status via clustat & cman_tool command

#clustat
Cluster Status for Cluster_01 @ Fri Jun 1 15:56:08 2012
Member Status: Quorate
Member Name                                                   ID   Status
------ ----                                                   ---- ------
station30.example.com                                             1 Online
station20.example.com                                             2 Online, Local
#cman_tool status 
Version: 6.2.0
Config Version: 1
Cluster Name: Cluster_01
Cluster Id: 25517
Cluster Member: Yes
Cluster Generation: 8
Membership state: Cluster-Member
Nodes: 2
Expected votes: 1
Total votes: 2
Quorum: 1
Active subsystems: 9
Flags: 2node Dirty
Ports Bound: 0 11 177
Node name: station20.example.com
Node ID: 2
Multicast addresses: 239.192.99.17
Node addresses: 192.168.5.20 

Step 20: Next step is to generate the fence key : For that click on Click on Cluster_01 and then click on Fence Option as mentioned in the screen shot Redhat_Cluster_Conga_4As mentioned in above screenshot tick mark on fence daemon and enter the node IP and click apply. Once we do that it will create fence_xvm.key under /etc/cluster folder   Step 21: Now we need to add the fence device and mention that fence device under each node. Since Fence device we have to add is a Non Shared Fencing Device, we shall create that fencing while adding a fence device under node itself Click on node and click on Manage Fence for this node Redhat_Cluster_Conga_5Once we click on fence device then we have to click on add a fence device for this node Redhat_Cluster_Conga_6Click on Add a fence device to this node & I am using HP ILO fencing here: Redhat_Cluster_Conga_7 Step 22: Few Setting required on ILO as well. To enter in ILO2 configuration reboot the server and wait for the prompt and press F8. First thing that we will configure is the IP address so you go to Network->DNS/DHCP as shown in the visual. ILO_Setting_1After that set DHCP Enabled to OFF. ILO_Setting_2From the main screen select Network->NIC and TCP/IP Set Network Interface Adapter to ON. Configure IP address, Subnet Mask and Gateway and press F10 to save the changes. ILO_Setting_3 Set Network Interface Adapter to ON. Configure IP address, Subnet Mask and Gateway and press F10 to save the changes. The next step is to change/create user account settings. From the main screen go to User->Add The next step is to change/create user account settings. From the main screen go to User->Add ILO_Setting_5Step 23: Click on Cluster then Failover Domain and Add Failover Domain Redhat_Cluster_Conga_8 Step 24: Format Clustered LVM with GFS2 file system

/dev/vg0/lv0 is an existing lvm
Create a file system on /dev/vg0/lvo
#mkfs.gfs2 -p lock_dlm Cluster_01:vg0 -j 3 /dev/vg0/lv0
#mkfs.gfs2: More than one device specified (try -h for help)

[root@station30 ~]# mkfs.gfs2 -p lock_dlm -t Cluster_01:vg0 -j 3 /dev/vg0/lv0
This will destroy any data on /dev/vg0/lv0.
It appears to contain a gfs filesystem.
Are you sure you want to proceed? [y/n] y
Device:                    /dev/vg0/lv0
Blocksize:                 4096
Device Size                0.48 GB (126976 blocks)
Filesystem Size:           0.48 GB (126973 blocks)
Journals:                  3
Resource Groups:           2
Locking Protocol:          "lock_dlm"
Lock Table:                "Cluster_01:vg0"
UUID:                      A4599910-69AF-5814-8FA9-C1F382B7F5E5

#mount /dev/vg0/lv0 /var/www/html/
#gfs2_tool df /dev/mapper/vg0-lv0

Step 24: Now we need to add the resources

1. Click on Add Resource and then select the IP
2. Then we need to add the GFS File system
3. Now we need to add the script
Redhat_Cluster_Conga_10

Step 25: Now Add a Service Group Redhat_Cluster_Conga_11Add resources in dependency order > IP > File System > script to run the service successfully. Start the Webby Service

#clusvcadm -r Webby -m station30.example.com

Redhat_Cluster_Conga_12 ******* If you interested in Qdisk Concept then follow the below steps ******** Quorum Disk: Just in case we have a 3 node cluster and out of 3, two of our nodes went down, then Cluster will not achieve quorum hence, cluster will not start, in order to start the cluster on a single node, we need max of 2 votes for 1 node, this quorum disk gives us that functionality of voting.

# mkqdisk -c /dev/qdisk-vg/qdisk-lv -l qdisk

Qdisk_2 Qdisk_3In the above mentioned screenshot; Setting UP  Qdisk configuration in Cluster

On all Nodes:
# /etc/init.d/qdiskd restart
# chkconfig qdiskd on

Qdisk_4

# cman_tool status

Qdisk_5 Final Setup to check your cluster is working as expected or not, i am going to power off Station30, where current my webby application is running. Expected behavior:  Webby application should be relocated to other cluster Node. I.E : station20 or station10; Station20 And Now I am going to power off my Station20 as well to check, whether my Qdisk configuration working as expected or not. Let’s figure crossed 🙂 🙂 Station30 Final cman_tool status to understand the voting  calculation: Final_Cman_Status Cheers!!!!

Posted in Clustering, Linux | Tagged , , , , | 1 Comment

How To Kill Defunct Or Zombie Process?

A “defunct” processes is also known as a “zombie” processes. A Zombie process is referred as dead process which is receding on your system though its completed executing. In one shot we can say its a dead processes which is still in RAM. This process will be in your process table and consuming your memory. Having more defunct process will consume your memory which intern slows your system. We have to kill the defunct process in order to free RAM and make system stable. Defunct processes are processes that have become corrupted in such a way that no longer  communicate (not really the right word, more like signal each other) with their parent or child process.
So kill the parent or child and 99% of the time (around here at least) the defunct process will go away! No parent or child, you’re out of luck, or look for a stuck automount.

 

Why defunct process are created?
Ans : When ever a process ends all the memory used by that process are cleared and assigned to new process but due to programming errors/bugs some processes are still left in process table. These are created when there is no proper communication between parent process and child process.

Linux_process_lifecycle

 

1. How to find a defunct process?
And : Grep defunct value in ps -ef output
#ps -ef | grep defunct

[root@amitvashist ~]# ps -ef | grep defunct | more
root      4801 29261  0 09:25 pts/5    00:00:00 grep defunct
root      6951     1  0 Dec30 ?        00:00:00 [bacula-sd] <defunct>

Or 

[root@amitvashist ~]# ps -el|grep Z
F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 Z  1000 10317     1 99  80   0 -     0 exit   ?        19:27:15 java <defunct>

2. How can I kill a defunct process?
And : Just use kill command
#kill defunct-pid

3. Still not able to kill?
Ans : Then use kill -9 to force kill that process
#kill -9 defunct-pid

[root@amitvashist ~]# kill -3 6951     {For nohop output. }
[root@amitvashist ~]# kill -9 6951

4. Still have an issue in killing it?
Ans : Then try to kill its parent id and then defunct.
#kill parent-id-of-defunct-pid
Then
#kill -9 parent-id-of-defunct-pid

[root@amitvashist ~]# kill -9 6951

5. Still having defunct?
Ans : If you still find defunct process eating up RAM then last and final solution is to reboot your machine(This is not preferred on production boxes).

6. To check all the current open files with that process ?
And: lsof in linux & pfile on solaris zone

[root@amitvashist ~]# lsof -p 6951  & pfile 6951
COMMAND    PID USER   FD   TYPE  DEVICE     SIZE    NODE NAME
bacula-sd 6951 root  cwd    DIR   253,0     4096 3801089 /root
bacula-sd 6951 root  rtd    DIR   253,0     4096       2 /
bacula-sd 6951 root  txt    REG   253,0  2110599  368004 /usr/local/sbin/bacula-sd
bacula-sd 6951 root  mem    REG   253,0    75284  389867 /usr/lib/libz.so.1.2.3
bacula-sd 6951 root  mem    REG   253,0    46680 3604521 /lib/libnss_files-2.5.so
bacula-sd 6951 root  mem    REG   253,0   936908  369115 /usr/lib/libstdc++.so.6.0.8

 

Posted in Linux, Monitoring, Uncategorized | Tagged | 2 Comments

Linux File System Recovery

Scenario: If our  file is about to reach 100% & we don’t have underlying LVM.

Solution: Moving data form disk 1 to New Disk { disk2 }

Please find below Step’s involve in this activity

Step1: Add a new Hard drive into the box if its VM; otherwise ask storage team for fresh LUN as per your requirement.

Step2: Run command : # echo “- – -“ > /sys/class/scsi_host/host0/scan { Note: host0,host1… so as per Environment }

Step3: Create new volume group structure.

# pvcreate /dev/sdc
# vgcreate rootvg /dev/sdc
# lvcreate –l 50%VG –n /dev/rootvg/lvol_var
# lvcreate –l 50%VG –n /dev/rootvg/lvol_opt
# mkfs.ext3 /dev/rootvg/lvol_var
# mkfs.ext3 /dev/rootvg/lvol_opt

Step4: Boot the node into Rescue mode

Insert the Linux DVD in cdrom & make the recommended setting into bios { boot from cd }
#chroot /mnt/sysimage
#lvm.static pvscan
#lvm.static vgscan or lvm vgscan -v
#lvm.static lvscan or lvm lvscan -v
#mkdir /tmp/{opt,var} –p

Step 5: Mounting the lvm in /tmp/ file system.

#mount –t ext3 /dev/rootvg/lvol_var /tmp/var
#mount –t ext3 /dev/rootvg/lvol_var /tmp/opt

Step 6: Moving Data form local file system to LVM.

 #nohup cp -r --preserve=mode,ownership,timestamps /var/* /tmp/var/ &
 #nohup cp -r --preserve=mode,ownership,timestamps /opt/* /tmp/opt/ &
 #ps –ef | grep cp

Step 7: After complication of successful copy modify your /etc/fstab

 # vim /etc/fstab
   /dev/rootvg/lvol_var /var ext3 defaults 1 2
   /dev/rootvg/lvol_opt /opt ext3 defaults 1 2
 # exit & exit

Step 8: Boot your system in normal mode & verify that everything is fine.

Step 9: IF everything is fine after modification then go single user or rescue mode one more time.

Step 10: chroot /mnt/sysimage

* Please make sure your lvm is not mounted at this moment
 # rm –rf /var/* & rm –rf /opt/*
 # run your lvm & check about data.
 # Exit & exit ;

Step 11: Boot your system into normal mode.

Error: if you are facing: unable touch /var/lock/subsys/lvm-monitor & all
Error: System stuck while booting up.

Solution: Boot your machine into Single user mode.

 Step1: chmod –R 777 /var/lock
 Step2: exit & exit

Note Points:-

 Always use Linux LVM for OS & other Partitions Layout it will provides various awesome features.
 • Resize volume groups online by absorbing new physical volumes (PV) or ejecting existing ones.
 • Resize logical volumes (LV) online by concatenating extents onto them or truncating extents from them.
 • Create read-only/write snapshots of logical volumes (LVM2).
 • Stripe whole or parts of logical volumes across multiple PVs, in a fashion similar to RAID 0.
 • Mirror whole or parts of logical volumes, in a fashion similar to RAID 1.
 • Move online logical volumes between PVs.
 • Split or merge volume groups in situ (as long as no logical volumes span the split). This can be useful when Migrating whole logical volumes to or from offline storage.
 • Take the backup of our current volumes layout & restore into fresh build machines for the same sort of multiple Node deployment.
 • It support Linux Native dm-multipathing & Emc power path for provide data redundancy for production environment.
Posted in File System, Linux, Uncategorized | Leave a comment

UNIX / LINUX : PS Commands & Real World UNIX Process Monitoring examples :) :)

“As Unix/Linux is a multitasking operating system, which means that more than one process can be active at once & to find out resource utilization. The PS/Top command on Linux is one of the most basic commands for viewing the processes running on the system. It provides a snapshot of the current processes along with detailed information like user id, cpu usage, memory usage, command name etc.”

1. List all Currently Running Processes (ps -ef, ps -aux)

$ ps -ef
 root     26551     5  0 Feb10 ?        00:03:41 [pdflush]
 root     26570     5  0 Feb10 ?        00:00:20 [pdflush]
 root     30344  3382  0 Feb21 ?        00:00:11 sshd: root@pts/14
 root     30365 30344  0 Feb21 pts/14   00:00:02 -bash
 root     30393  3382  0 Feb21 ?        00:00:10 sshd: root@pts/15

2. List process by user

$ ps -f -u www-data
UID        PID  PPID  C STIME TTY          TIME CMD
www-data  1329  1328  0 09:32 ?        00:00:00 nginx: worker process
www-data  1330  1328  0 09:32 ?        00:00:00 nginx: worker process
www-data  1332  1328  0 09:32 ?        00:00:00 nginx: worker process
www-data  1377  1372  0 09:32 ?        00:00:00 php-fpm: pool a.localhost


3. List process by Multiple user

$ ps -f -u amitvashist,postfix
 UID        PID  PPID  C STIME TTY          TIME CMD
 amitvashist   10078  7491  0 Mar09 ?        00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
 amitvashist   10082  7491  0 Mar09 ?        00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
 postfix  15677  7435  0 22:23 ?        00:00:00 pickup -l -t fifo -u
 postfix   7457  7435  0 Mar09 ?        00:00:00 qmgr -l -t fifo -u

4. Display process by Command

$ ps -C apache2
 PID TTY          TIME CMD
 2359 ?        00:00:00 apache2
 4524 ?        00:00:00 apache2

5. List the processes based on PIDs or PPIDs (ps -p, ps –ppid)

$ ps -f  -p 25009,7258,2426
 UID        PID  PPID  C STIME TTY          TIME CMD
 root      2426     4  0 Mar09 ?        00:00:00 [reiserfs/0]
 root      7258     1  0 Mar09 ?        00:00:00 /usr/sbin/nscd
 postfix  25009  7435  0 00:02 ?        00:00:00 pickup -l -t fifo -u

6. Sort process by cpu or memory usage

System administrators often want to find out processes that are consuming lots of memory or CPU. The sort option will sort the process list based on a particular field or parameter. Multiple fields can be specified with the “–sort” option separated by a comma. Additionally the fields can be prefixed with a “-” or “+” symbol indicating descending or ascending sort respectively.

$ ps aux --sort=-pcpu | head -5
 USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 root         1  2.6  0.7  51396  7644 ?        Ss   02:02   0:03 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
 root      1249  2.6  3.0 355800 30896 tty1     Rsl+ 02:02   0:02 /usr/bin/X -background none :0 vt01 -nolisten tcp
 root       508  2.4  1.6 248488 16776 ?        Ss   02:02   0:03 /usr/bin/python /usr/sbin/firewalld --nofork
$ ps aux --sort pmem
 USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
 root         1  0.0  0.0  1520  508 ?        S     2005   1:27 init
 inst  1309  0.0  0.4 344308 33048 ?      S     2005   1:55 agnt (idle)
 inst  2919  0.0  0.4 345580 37368 ?      S     2005  20:02 agnt (idle)
 inst 24594  0.0  0.4 345068 36960 ?      S     2005  15:45 agnt (idle)

7. List all threads for a particular process (ps -L)

You can get a list of threads for the processes. When a process hangs, we might need to identify the list of threads running for a particular process as shown below.

$ ps -C java -L -o pid,tid,pcpu,state,nlwp,args
 PID   TID %CPU S NLWP COMMAND
 16992 16992  0.0 S   15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
 16992 16993  0.0 S   15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
 16992 16994  0.0 S   15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006
 16992 16995  0.0 S   15 ../jre/bin/java -Djava.ext.dirs=../jre/lib/ext:../lib:../auto_lib -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5006

8. List elapsed wall time for processes (ps -o pid,etime=)

If you want the get the elapsed time for the processes which are currently running ps command provides etime which provides the elapsed time since the process was started, in the form [[dd-]hh:]mm:ss.

 $  ps -e -o pid,comm,etime
 PID COMMAND                                                                              ELAPSED
 10669 -bash                                                                           31-05:49:20
 9467 /usr/local/sbin/sshd                                                                01:05:30
 19817 /usr/lib/autofs/automountd                                                      45-21:45:26
 18518 /lib/svc/bin/svc.configd                                                        45-21:45:29
 19332 /usr/lib/ldap/ldap_cachemgr                                                     45-21:45:27

9. Display process hierarchy in a tree style

Many processes are actually forked out of some parent process, and knowing this parent child relationship is often helpful. The ‘–forest’ option will construct an ascii art style tree view of the process hierarchy.

 $ ps -f --forest -C apache2
 UID        PID  PPID  C STIME TTY          TIME CMD
 root      2359     1  0 09:32 ?        00:00:00 /usr/sbin/apache2 -k start
 amitvashist  4524  2359  0 10:03 ?        00:00:00  \_ /usr/sbin/apache2 -k start
 amitvashist  4525  2359  0 10:03 ?        00:00:00  \_ /usr/sbin/apache2 -k start
 amitvashist  4526  2359  0 10:03 ?        00:00:00  \_ /usr/sbin/apache2 -k start

10. Display child processes of a parent process

 $ ps -o pid,uname,comm -C apache2
 PID USER     COMMAND
 2359 root     apache2
 4524 amitvashist apache2
 4525 amitvashist apache2
 4526 amitvashist apache2

11. Turn ps into an realtime process viewer with watch.

$ watch -n 1 ‘ps -e -o pid,uname,cmd,pmem,pcpu –sort=-pmem,-pcpu | head -15’

Every 1.0s: ps -e -o pid,uname,cmd,pmem,pcpu --...  Sun Dec  1 18:16:08 2013
 PID USER     CMD                         %MEM %CPU
 3800 1000     /opt/google/chrome/chrome -  4.6  1.4
 7492 1000     /opt/google/chrome/chrome -  2.7  1.4
 3150 1000     /opt/google/chrome/chrome    2.7  2.5
 3824 1000     /opt/google/chrome/chrome -  2.6  0.6
Posted in Linux, Monitoring, Uncategorized | 3 Comments

Storage Migration

Storage Migration Activity:

Scenario: Storage Migration from EMC Clarion { Old Storage } to EMC V-Max { New Storage }.

We have asked storage team to allocate 4 new storage LUN's as per the below mentioned size.
 • 2 Lun’s of 5 GB for data base redo logs
 • 2 Lun’s of 200 GB for database

Scan for the newly added Storage, and get the consolidated device created for the different paths coming from the storage.

Before this you check the current devices on the server by running:

 [root@xxxxvm1 ~]#powermt display dev=all or multipath -ll

 (if power path is configured you will be having detailed output for all devices like emcpowera, emcpowerb or with dmmultipath
 you will be having detailed output for all devices like mpath15, mpath16 and so on.)

 [root@xxxxvm1 ~]#echo 1 > /sys/class/fc_host/host3/issue_lip
 [root@xxxxvm1 ~]#echo 1 > /sys/class/fc_host/host4/issue_lip
 [root@xxxxvm1 ~]#echo “- - -“ > /sys/class/scsi_host/host3/scan
 [root@xxxxvm1 ~]#echo “- - -“ > /sys/class/scsi_host/host4/scan
[root@xxxxvm1 ~]#powermt config

 After this you can check the new device created by running:

[root@xxxxvm1 ~]#powermt display dev=all

(here it will show all previous devices as well as the newly created device emcpowerX, where X=a,b,c…. the next
 character available)

Example:
Say from the previous step you may have discover a newly added device to the server is: emcpowerf

Check the Device as:
---------------------------------------------------------------------
 state=alive; policy=SymmOpt; priority=0; queued-IOs=0
---------------- Host --------------- - Stor - -- I/O Path - -- Stats ---
 HW Path I/O Paths Interf. Mode State Q-IOs Errors
-----------------------------------------------------------------------
 3 qla2xxx sdo FA 8cB active alive 0 0
 4 qla2xxx sdp FA 9cB active alive 0 0
[root@xxxxvm1 ~]#

Create physical volume on top of this new device:
root@xxxxvm1 ~]#pvcreate /dev/emcpowerf

Next add this PV to the existing volume group:
root@xxxxvm1 ~]#vgextend vgora /dev/emcpowerf

Now the new device will be part of the existing volume group.

Here we have two different approach to achieve the same storage migration

  • Using the safe  & simple  cp commands & in order to do that we need to create same disk layout before we start cp.
  • Either we can use lvm 2 functionality of pvmove & in order to do the we need to have same size or greater size new luns.

Example 1:  Explaining first Approach using copy command:

In the below mentioned screenshot you will see the above mentioned Lun’s which we have asked for.
Lun Details:-
Mpath11 & 12 of 200GB each
Mpath 9 & 10 of 5GB each

emc_disk_scan_1

 

 

 

 

 

 

 

 

 

 

 

 

Creating New Volume Layout for data migration :-

emc_scan_2

 

 

 

 

 

 

 

Mounting New Volume’s into /tmp directory

emc_disk_scan_3

 

 

 

 

 

 

 

Command use to migrate data:-

 # nohup cp -r --preserve=mode,ownership,timestamps /u001/. /root/tmp/vgora/u001 &
 # nohup cp -r --preserve=mode,ownership,timestamps /u002/. /root/tmp/vgora/u002 &

Finally Data has been migrated successfully & now database is running without reporting any issue.

emc_disk_scan_4

 

 

 

 

 

 

 

 

Example 2:  Explaining Second Approach using lvm – pvmove command:

Prepare the disk
First, you need to pvcreate the new disk to make it available to LVM. In this recipe we show that you don’t need to partition a disk to be able to use it.

# pvcreate /dev/emcpowerh  /dev/emcpoweri  /dev/emcpowerj  /dev/emcpowerk
pvcreate -- physical volume "/dev/emcpowerh" successfully created
pvcreate -- physical volume "/dev/emcpoweri" successfully created
pvcreate -- physical volume "/dev/emcpowerj" successfully created
pvcreate -- physical volume "/dev/emcpowerk" successfully created

Add it to the volume group
As developers use a lot of disk space this is a good volume group to add it into.

# vgextend vgora /dev/emcpowerh  /dev/emcpoweri  /dev/emcpowerj  /dev/emcpowerk
vgextend -- INFO: maximum logical volume size is 555.99 Gigabyte
vgextend -- doing automatic backup of volume group "vgora"
vgextend -- volume group "vgora" successfully extended

Move the data
Next we move the data from the old disk onto the new one. Note that it is not necessary to unmount the file system before doing this. Although it is highly recommended that you do a full backup before attempting this operation in case of a power outage or some other problem that may interrupt it. The pvmove command can take a considerable amount of time to complete and it also exacts a performance hit on the two volumes so, although it isn’t necessary, it is advisable to do this when the volumes are not too busy.

# pvmove /dev/emcpowera /dev/emcpowerh
pvmove -- moving physical extents in active volume group "vgora"
pvmove -- WARNING: moving of active logical volumes may cause data loss!
pvmove -- do you want to continue? [y/n] y
pvmove -- 249 extents of physical volume "/dev/emcpowera" successfully moved

Remove the unused disk
We can now remove the old dm-multipath device from the volume group.

# vgreduce vgora /dev/emcpowera 
vgreduce -- doing automatic backup of volume group "vgora"
vgreduce -- volume group "vgora" successfully reduced by physical volume:
vgreduce -- /dev/emcpowera

Finally Data has been migrated successfully & now database is running without reporting any issue.

Posted in File System, Linux, Uncategorized | 2 Comments

Differences between Ext2, Ext3 and Ext4 file systems

Differences between Ext2, Ext3 and Ext4 file systems

It is always a question in my mind what are the basic difference between ext2, ext3 and ext4 file systems & why we are only able create to four primary partition.

So here is the answer for you,

This article explains the following:

  • High level difference between these file systems.
  • How to convert from one file system type to another.
  • Difference between MBR & GPT Partition Table.

Ext2

  • Ext2 stands for second extended file system.
  • It was introduced in 1993. Developed by Rémy Card.
  • This was developed to overcome the limitation of the original ext file system.
  • Ext2 does not have journaling feature.
  • On flash drives, usb drives, ext2 is recommended, as it doesn’t need to do the over head of journaling.
  • Maximum individual file size can be from 16 GB to 2 TB
  • Overall ext2 file system size can be from 2 TB to 32 TB

Ext3

  • Ext3 stands for third extended file system.
  • It was introduced in 2001. Developed by Stephen Tweedie.
  • Starting from Linux Kernel 2.4.15 ext3 was available.
  • The main benefit of ext3 is that it allows journaling.
  • Journaling has a dedicated area in the file system, where all the changes are tracked.     When the system crashes, the possibility of file system corruption is less because of         journaling.
  • Maximum individual file size can be from 16 GB to 2 TB
  • Overall ext3 file system size can be from 2 TB to 32 TB with GPT.
  • We can convert a ext2 file system to ext3 file system directly (without backup/restore).
  • There are three types of journaling available in ext3 file system.
  1. Journal – Metadata and content are saved in the journal.
  2. Ordered – Only metadata is saved in the journal. Metadata are journaled                       only after writing the content to disk. This is the default.
  3. Writeback – Only metadata is saved in the journal. Metadata might be                            journaled either before or after the content is written to the disk.

Ext4

  • Ext4 stands for fourth extended file system.
  • It was introduced in 2008.
  • Starting from Linux Kernel 2.6.19 ext4 was available.
  • Supports huge individual file size and overall file system size.
  • Maximum individual file size can be from 16 GB to 16 TB with GPT
  • Overall maximum ext4 file system size is 1 EB (exabyte). 1 EB = 1024 PB (petabyte). 1 PB = 1024 TB (terabyte).
  • Directory can contain a maximum of 64,000 subdirectories (as opposed to 32,000 in ext3)
  • You can also mount an existing ext3 fs as ext4 fs (without having to upgrade it).
  • Several other new features are introduced in ext4: multiblock allocation, delayed allocation, journal checksum. fast fsck, etc. All you need to know is that these new features have improved the performance and reliability of the filesystem when compared to ext3.
  • Extents (basically automatic space allocation to avoid fragmentation)
  • In ext4, you also have the option of turning the journaling feature “off”.

A very good comparison from 2009 on linuxologist.com has a graph about write performance with 4Gb:

Iozone-1

Converting ext2 to ext3

I.E, if we are upgrading /dev/sda2 that is mounted as /home, from ext2 to ext3, do the following.

  1.   umount /dev/sda2
  2.   tune2fs -j /dev/sda2
  3.   mount /dev/sda2 /home

Note: You really don’t need to umount and mount it, as ext2 to ext3 conversion can happen on a live file system. But, I feel better doing the conversion offline.

Converting ext3 to ext4

I.E, If we are upgrading /dev/sda2 that is mounted as /home, from ext3 to ext4, do the following.

  1. umount /dev/sda2
  2. tune2fs -O extents,uninit_bg,dir_index /dev/sda2
  3. e2fsck -pf /dev/sda2
  4. mount /dev/sda2 /home

Again, try all of the above commands only on a test system, where you can afford to lose all your data.

Please follow the below mentioned like to know about : Difference between MBR & GPT Partition Table.

https://amitvashist.wordpress.com/2014/01/12/the-differences-between-mbr-and-gpt/

Aside | Posted on by | 1 Comment