So I received this 500 internal Server Error when trying to save my long posts on my WordPress 2.6.2.  In my case I have 1and1.com hosting and at first impressions, the problem appeared to be with MySQL and the time it took to save a post.  On top of this, my host doesn't allow access to the error_log files (HTTPD Error Log files) and only to the Access Log files (access_log) so I can't really see what the real issue is.  Here's how I solved the problem in my case:

I edited my .htaccess file in the root folder of my site to add this line to the end of it.  This directive tells the web server that files with .php extension should be handled by the PHP module:

"AddType x-mapp-php5 .php"

And the 500 Internal Server Error went away when I tried to save again.

Cheers!
TK.

Overview:

HTPC / Home Backup: MDADM, RAID6, LVM, XFS, CIFS and NFS

In this post we'll take a look at setting up a 4TB storage, HTPC, and active backup using basic components that can be bought from the local PC store and freely available software.   Here I set out to try and consolidate my storage to something central running RHEL Linux (Or a clone, such as the Scientific Linux that's freely available) that could also serve as a makeshift entertainment center in replacement of my current one.  In other words, upgrade my HTPC and turn it into an HTPC+B, if you will, with the following statistics:

• 4TB usable storage (6TB HDD - 2 HDD for RAID6).
• XFS Filesystem
• 317MB/s WRITES.
• 300 MB/s READS.
• A Wireless N card + 1GIG ETH in a bonding configuration. (Redundancy in case of failure)
• Scientific Linux 6.1 (RHEL 6.1).

So here is what a goal would be to achieve with this central storage server:

1. HTPC: Have it function as a media center.
2. Backup. (Single drives backup up to a RAID 6 setup within the same box)
3. File Server (NFS, CIFS, Mountable from my other windows systems etc)
4. Flash storage center for the most critical files.
5. (Optional) Authentication server using something like kerberos.

There are a few good posts out there that would suggest or argue against using RAID, but I still need a backup.  The other thing that I wanted to do is to find out how reliable this thing could be.  (As we'll see, it turned out to be quite an eye opener.)  A good read as well is the Triple Parity RAID and Beyond article.

So why not use a store baught backup solution?

That seamed like a good idea sometime back if it were not an episode a colleague of mine had with one of those.  Based on the noises and symptoms he describes as one having while he couldn't access his data was enough for me.  So this brings us to the most real question here:

What is a backup in the true sense given today's technology?

This is a loaded question and there's no right or wrong answer.  With today's estimates that neither RAID5 nor RAID6 would be enough in the upcoming years (see the link above), the word backup would appear to start to take on the meaning:  Storage media that would be available when needed to restore a copy from rather then storage that would hold a copy of my data. At first glance there doesn't appear to be a difference yet the first statement doesn't suggest a promise that the backup would be there at all.  YMMV or YOMV.

So what does that mean for RAID 6?  Let's illustrate with an example given the following to determine the likelihood of an unrecoverable data failure:

• A home laptop (contains one HDD)
• The HTPC+B setup (contains 6 HDD in a RAID 6 configuration)

If my laptop is being backed up to my HTPC+B server, the point at which one would loose data is when both the laptop HDD and 3 of the RAID 6 disks failed at the same time.  So in essence, a 4 disk failure at the same time which is highly unlikely.  So what this means is that, which would be more likely, when my laptop disk fails, my HTPC+B would likely still be around for me to restore from.  Likewise, if my backup fails, my PC would still be running along fine long enough to fix up my backup outage.  In either case, I would still have my data.

The flipside of the argument would suggest that the more hardware in a configuration, the more maintenance it would require over the same time frame.

Setup Summary:

A quick note.  The hardware used here is a combination of old and new and where price was right vs other options.  Other hardware should work just as well but your mileage may vary and my pocket book isn't that deep at all:

• LS ULTRA 600W ATX Power Supply (Reused from another PC)
• GA-890XA-UD3 Motherboard ( 333 - SATA 3, USB Power 3.0, USB 3.0).  6x SATA 3 for our software RAID 6.
• Seagate 1TB Green Barracuda  HDD x 6  ( ST31000520AS )
• ( Optional - Failed drive replacement ) Barracuda® Green SATA 6Gb/s 1.5TB Hard Drive ( ST1500DL003 )
  
• AMD RADEON HD 6450 1GB
  
• TP-LINK TL-WN821N (V2) Wireless USB stick.
• AMD Athlon(tm) 5200 Dual-Core Processor
• A Western Digital Caviar Green drive ( WD15EADS ) for the root FS.

For this setup, the motherboard is the most powerful item I got.  Lot's of room for upgrades here as a result.  I wasn't looking to break the bank and at about $50 per HDD in Aug of 2011, it made perfect sense, before a certain flood in Thailand.  I hope cheap doesn't transfer to the hardware but only time will tell.  All hooked up, it looked like this:

(  NOTE: The makeshift fan under the video card PCI card.  Turned out the heatsink of the AMD RADEON was getting too hot even though it is sold as a fanless GPU.  )

Once the hardware is hooked up, it's time for the software.  The steps below are the base commands I needed after a rather lengthy, trial and error setup of the HTPC+B:

Your array should now be ready.  However, before one is to put any data on the new storage, some brutal testing is in order.  We need to know that the array can withstand some hard hits and still tick before you can trust your files to it.

]]> http://www.microdevsys.com/WordPress/2012/04/02/linux-htpc-home-backup-mdadm-raid6-lvm-xfs-cifs-and-nfs/feed/ http://www.microdevsys.com/WordPress/2012/04/01/linux-persistent-naming-of-sata-devices-through-udev-rules/ http://www.microdevsys.com/WordPress/2012/04/01/linux-persistent-naming-of-sata-devices-through-udev-rules/#comments Mon, 02 Apr 2012 03:34:04 +0000 Tom http://www.microdevsys.com/WordPress/?p=1357 Here we'll look at obtaining persistent naming rules for SATA devices so we can reference these hardware devices through a constant name.  This is particularly useful when one needs a persistent name to a device through unique serial and model numbers that remains the same throughout reboots / restarts and shutdowns.  In this post, we'll [...]]]>

Here we'll look at obtaining persistent naming rules for SATA devices so we can reference these hardware devices through a constant name.  This is particularly useful when one needs a persistent name to a device through unique serial and model numbers that remains the same throughout reboots / restarts and shutdowns. 

In this post, we'll create six unique naming rules to uniquely identify those HDD we'll need for our RAID6 HTPC+B configuration.  Amongst other things, the rules would help to uniquely identify those devices that belong to the RAID6 array and those that don't to help prevent human error and remove confusion.

The first order of business is, of course, to plug in your hardware that you're going to create the rules for.  In this case, we'll be giving six Seagate HDD drives a unique name to help identify those that belong in a RAID6 array.  A summary of steps follows.  CAUTION: Before reading the below steps, do have a look at the various issues / problems encountered at the end of this article to get a general idea of what to expect:

This gave numerous UDEV rules with which we could customize our naming.

PROBLEMS

UDEV tweaking comes with it's own set of risks.  For this, you may wish to either have a backup CD or a live Linux CD ready running the OS type you're modifying.  Alternately, a good PXE installation off of another server is a good way to go too where you can simply plug an ethernet cable to the secondary server and boot off the network.

Cheers!
TK

PROBLEM

You've got a set of devices but their names, such as NIC / Network card names keep changing on each reboot or reconnect.  This is a continuation of Linux Networking: device eth0 does not seem to be present, delaying initialization.  where we resolved a similar issue due to naming but here we'll make it persistent using this simple naming system for Linux.  At the end of this very short post, we'll have a constant name for our physical device on reboots.  In particular, it is to make the line ip link set dev wlan1 name wlan0 permanent across reboots.

SOLUTION

To solve this, we'll use UDEV rules to give a certain device a persistent name.  We'll use the 70-persistent-net.rules file to do just that so our network interfaces keep coming up with the same name on reboots or replugging.

# cat /etc/udev/rules.d/70-persistent-net.rules

# PCI device 0×10ec:0×8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="1c:6f:35:3f:da:14", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# USB device 0×0cf3:0×1002 (usb) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:15:86:f1:65:d5", ATTR{type}=="1", KERNEL=="wlan*", NAME="wlan0"

#

There's several ways to get parameters for the devices you have.  Most notable is lspci that we can use here:

# lspci -s 03:00.0
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
#

Add some -v[v....] options for verbosity to see more details.   Another way, which is probably the most common, is to use netstat or ifconfig to get your MAC address and other parameters you can use with which to uniquely define your rules:

# netstat -anevi
Kernel Interface table
bond0     Link encap:Ethernet  HWaddr 1c:6f:35:3f:da:14 
          inet addr:192.168.0.8  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::1e6f:65ff:fe3f:fc14/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:53255 errors:0 dropped:0 overruns:0 frame:0
          TX packets:41224 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:28013914 (26.7 MiB)  TX bytes:4811379 (4.5 MiB)
 

Or ifconfig bond0 which would provide the exact same thing as above.  Another way still is to use udev itself to get more information on your device in this manner:

# udevadm info -a -p /sys/class/net/eth0 2>&1|grep -i "ATTR{address}"
    ATTR{address}=="1c:6f:35:3f:da:14"
#

Hope you found this useful.  Feel free to leave a reply!

Cheers!
TK

This is a quick and dirty AWK tutorial demonstrating the most common usages and features that one can use on a daily basis.  Three variants: AWK standalone, integrated into ksh, and one-liner are provided. 

On my system, AWK pointed to GAWK.  GAWK has an expanded command set but here we will only focus on specific to AWK.

AWK is a powerful language rarely used beyond printing a few columns but often can stack up better in performance, then other scripting languages.  AWK can also make for a relatively easy learn due to it's small set of functions.  This is similar to C without the complexities of such things as pointers:

[root@tom awk]# ./dfm.awk
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sdc2                29526     26568      1458  95% /
/dev/sdc5                19689     18226       463  98% /home
/dev/sdb1                 9538      9079       459  96% /mnt/c
[root@tom awk]# cat dfm.awk
#!/bin/awk -f

function dfm ( USAGEL ) {
        CMD="df -m";
        while (1) {
                rvar = ( ( CMD | getline ) > 0 );

                if ( rvar == 0 )                 
                        break;

                DUSE=$5;                                      
                sub(/[%]/, "", DUSE);    
                                                         

                if ( DUSE > USAGEL || $0 ~ /Filesystem/ )
                        print $0;
        }
}

BEGIN {
        dfm(85);
}

[root@tom awk]#

Below is a breakdown of the above commands.

The below script demostrates how to embed the above AWK script into a KSH script.  Notice how everything is the same with the exception of the parts in green which we add for KSH.  We'll add a twist to the below by creating a thread from the KSH function that completes the task 5 seconds after the main thread exits.

 

[root@tom awk]# cat dfm.ksh
#!/bin/ksh

function dfm {
        awk '
                function dfm ( USAGEL ) {
                        CMD="df -m";
                        while (1) {
                                rvar = ( CMD | getline );

                                if ( rvar == 0 )                        # If this is 0
                                        break;

                                DUSE=$5;                          
                                sub(/[%]/, "", DUSE);
                                                                  

                                if ( DUSE > USAGEL || $0 ~ /Filesystem/ )
                                        print $0;
                        }
                }

                BEGIN {
                        SCMD="sleep 5";
                        system(SCMD);
                        dfm(85);
                        close(SCMD);
                }
        ';
}

function main {
        dfm &;
}
main;
 

A breakdown of the components:

The thread created in the above sleeps for 5 seconds but the execution of the main function continues, due to use of the & after the dfm function.  After sleeping 5 seconds, the thread exits and prints the results:

# ./dfm.ksh
#  (press enter a few times or run ps -aux|grep dfm.ksh to verify the thread is still there.)
#
#
# Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sdc2                29526     26569      1457  95% /
/dev/sdc5                19689     18226       463  98% /home
/dev/sdb1                 9538      9079       459  96% /mnt/c
#

 

The below line is a one liner that does the same.  It can simply be pasted to the CLI and ran.  The result is the same in a more compact form:

[root@tom awk]# df -m|awk '{ DUSE=$5; sub(/[%]/, "", DUSE); if ( DUSE > 85 || $0 ~ /Filesystem/ ) print; }'
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sdc2                29526     26569      1457  95% /
/dev/sdc5                19689     18226       463  98% /home
/dev/sdb1                 9538      9079       459  96% /mnt/c
[root@tom awk]#

Performance of a script can be tested using the date function as in the below example:

# date +"%s:%N"; df -m|awk '{ DUSE=$5; sub(/[%]/, "", DUSE); if ( DUSE > 85 || $0 ~ /Filesystem/ ) print; }'; date +"%s:%N";
1327730034:862739185
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sdc2                29526     26569      1457  95% /
/dev/sdc5                19689     18226       463  98% /home
/dev/sdb1                 9538      9079       459  96% /mnt/c
1327730034:873508735
#

Doing the math on the nanoseconds, we can see the code above took approx 10.7ms to run to produce the output or essentially just a little over one time slice to complete.

UPDATE: Wed April 25nd 2012

Here's another handy example that someone has asked me about recently.  The equivalent of the below is the UNIX comm command available on most distributions.  Here's the equivalent in awk that the reader can tweak to suit their tastes:

# cat scmp.awk
#!/bin/awk -f

function scmp ( SERVERF, HOSTF ) {
        FIL=SERVERF;
        while (1) {
                if ( FIL != "" )        rvar = ( ( getline < FIL ) > 0 );
                if ( rvar == 0 ) break;
                SVR[$1]=1;
        }

        FIL=HOSTF;
        while (1) {
                if ( FIL != "" )        rvar = ( ( getline < FIL ) > 0 );

                if ( rvar == 0 ) break;
                if ( SVR[$1] == 1 ) SVR[$1]++; else SVR[$1]=-1;
        }

        for ( key in SVR ) {
                if ( SVR[key] == 2 )    printf("%12s %12s %12s\n", key, key, "=");
                if ( SVR[key] == -1 )   printf("%12s %12s %12s\n", " ", key, ">")
                if ( SVR[key] == 1 )    printf("%12s %12s %12s\n", key, " ", "<")
        }
}

BEGIN {
        scmp( ARGV[1], ARGV[2] );
}

And the output is:

# ./scmp.awk server.txt host.txt
       host8                         <
                    host9            >
      host10                         <
       host1                         <
                    host2            >
       host3        host3            =
       host4        host4            =
                    host5            >
                    host6            >
       host7                         <
#
#
# cat server.txt
host1
host10
host3
host4
host7
host8
# cat host.txt
host2
host3
host4
host5
host6
host9
#

Cheers!
TK

Just happened to have forgotten a password for one of my user accounts on my workstation and remembered a handy little thing one can do to reset a pass.

If you have root access, one can simply regenerate the MD5 hash with:

# openssl passwd -1 -salt mysalt
Password:
$1$mysalt$JyKGOJR1343sJ7N91hXVI/
#

# ftp localhost
Connected to localhost (127.0.0.1).
220 Welcome to the Crazy Town .  If you are not an authorized user, please logout now!
Name (localhost:root): tom
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> quit
221 Goodbye.
#

and replace the existing hash string in proftpd or any other service with what ever is entered as a pass above.  This should help to reset your pass.

Cheers!
TK

In this post we'll configure network booting through PXE.  PXE stands for Preboot eXecution Environment and has been available for Network Devices (or NIC's) for some time.  Such a network boot setup has many uses in particular in OS setup and deployment.  In this post I'll introduce the steps to quickly set one up.  The only thing required is some patience, persistance and a second NIC (NIC also known as Network or Network Interface Card) card on the second PXE server together with a network cable.  Some of the benefits are:

1. The target environment where the OS will be installed does not need any external storage devices such as USB or CD/DVD drives.
2. There is no CD burning.  It is all software based.
3. Most motherboards and computers support PXE today.
4. You are not limited to a single file size limit in place on some USB devices.
5. There's no OS size restrictions.  The amount of data available to the boot software is only limited by the amount of free hard disk space on the PXE server.

These are just a few advantages that PXE has and the cost of setup, as we will see, is only time and a little patience.

For this post, I'll simply list the configuration files in place to be used.  I'll highlight in each the section that can represent the addition of a new entry to the boot menu with a blue color.  The highlighted section would represent the steps to take when adding a new OS to boot.  All items in BLUE below represent the changes that would be needed to add a new OS to the choices after this PXE setup is complete.

Here we also have the option of booting Windows 7 but that will be left for another topic.

PXE SETUP

We'll begin by setting up your Linux workstation as a PXE boot server.

Check for required packages.	Run these or install the latest DHCP, XINETD, HTTPD and TFTP packages for your environment.  Most should already come with your distribution.  If not, below are some examples to use: # rpm -qa|grep -i dhcp dhcp-4.1.0p1-13.fc12.i686 # rpm -aq|grep -i tftp tftp-server-0.49-5.fc12.i686 tftp-0.49-5.fc12.i686 # rpm -aq|grep xinetd xinetd-2.3.14-32.fc13.i686 # rpm -aq|grep httpd httpd-2.2.15-1.fc13.i686
Install a second network card and configure it.	PXE will need to connect to your server using ethernet.  To that end, you need a second network card and a straight through cable.  For this step you'll need to install the second network card (Preferrably 100Mbits or higher).  The above links also contains instructions on debugging in case you are having problems.  In the end, this is what the configuration we will need looks like and what the contents of ifcfg-eth1 should be before proceeding to the next steps (if you already have ifcfg-eth1 create ifcfg-eth2 instead): # ifconfig eth1 eth1  Link encap:Ethernet  HWaddr 00:02:E3:17:E1:47            inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0           inet6 addr: fe80::202:e3ff:fe17:e147/64 Scope:Link           UP BROADCAST MULTICAST  MTU:1500  Metric:1           RX packets:569836 errors:0 dropped:0 overruns:0 frame:0           TX packets:799910 errors:6 dropped:0 overruns:6 carrier:6           collisions:0 txqueuelen:1000           RX bytes:206818639 (197.2 MiB)  TX bytes:1090171901 (1.0 GiB)           Interrupt:17 Base address:0×4000 # cd /etc/sysconfig/network-scripts # cat ifcfg-eth1 # National Semiconductor Corporation DP83815 (MacPhyter) Ethernet Controller DEVICE=eth1 ONBOOT=yes BOOTPROTO=none IPADDR=192.168.1.1 NETMASK=255.255.255.0 HWADDR=00:02:E3:17:E1:47 NM_CONTROLLED=no USERCTL=no IPV6INIT=yes TYPE=Ethernet DNS1=192.168.0.1 NOTE: It is assumed here that eth0 is your primary NIC you already have working and can browse the web with.  This tends to be handy.
Setup DHCP on the second NIC.	Below is the configuration we are going to use so the DHCP server can assign IP addresses to new machines that we will connect: # cat /etc/dhcp/dhcpd.conf # # DHCP Server Configuration file. #   see /usr/share/doc/dhcp*/dhcpd.conf.sample #   see 'man 5 dhcpd.conf' # ddns-update-style interim; ignore client-updates; subnet 192.168.1.0 netmask 255.255.255.0 {         # Set the default gateway / netmask / broadcast to be used by the PC clients         option routers                  192.168.1.1;         option broadcast-address        192.168.1.255;         option subnet-mask              255.255.255.0;         default-lease-time 43200;         max-lease-time 86400;         allow booting;         allow bootp;         default-lease-time 21600;         max-lease-time 43200;         group {                 next-server 192.168.1.1;                 # This is the pxe bootloader file                 filename "pxelinux.0";                 # One host block per client. This network only has one.                 host stb1 {                         hardware ethernet 00:11:5b:1c:4b:21;                         fixed-address 192.168.1.2;                         filename "FCSystem";                         option root-path "/tftpboot";                                                                                         # filename "\\Boot\\WINSystem";                 }                 host stb2 {                         hardware ethernet 90:e6:ba:c6:7d:35;                         fixed-address 192.168.1.2;                         filename "FCSystem";                         option root-path "/tftpboot";                 }                 host stb3 {                         hardware ethernet 00:0c:6e:53:9f:a6;                         fixed-address 192.168.1.2;                         filename "FCSystem";                         option root-path "/tftpboot";                 }                 host stb4 {                         hardware ethernet 1c:6f:65:3f:fc:14;                         fixed-address 192.168.1.2;                         filename "FCSystem";                         option root-path "/tftpboot";                 }         } }   Issue: # service dhcpd restart There are a few important sections in each area: The hardware ethernet address, is the MAC address of the remote machine that we will connect to this server.  You will need this from the remote server you are configuring.  If unclear on how to obtain this, it could be retrieved either from the first boot attempt: the MAC may show up on the PXE server /var/log/messages file on first attempt, or from the boot up screens of the remote machine.   fixed-address: The address that will be assigned to the remote machine when it attempts to connect.   FCSystem is, in essence, the base boot system that will be loaded over the network.  It is the kick start boot file chain link that will allow booting everything else.  This file can be copied from the following files on your system.  It can be renamed to whatever you like: # ll /usr/share/syslinux/pxelinux.0 5965619 -rw-r–r– 1 root root 16466 Jan 12  2010 /usr/share/syslinux/pxelinux.0 # ll /usr/share/syslinux/gpxelinux.0 5966129 -rw-r–r– 1 root root 75564 Jan 12  2010 /usr/share/syslinux/gpxelinux.0 # The gpfxlinux.0 is the open source gPXE boot loader and what we used here.  I simply copied it as and to /tftpboot/FCsystem.   The /tftpboot folder is where all the boot files reside in and where the boot environment will kick things off of.  It can be any path of your naming choice but per convention, /tftpboot is used.   Ensure allow bootp and allow booting is also defined as per above.
The /tftpboot menu files.	Below is a sample of how I setup mine on my own system: # cat /tftpboot/fcboot.msg (Optional) ▒splash.lss ————————————————————————-  -  To install/upgrade F14(64) type 0f'f14-64'07.  -  To install/upgrade Scientific Linux (64) type 0f's6.1-64'07.  -  To install/upgrade Ubuntu 11.04 (64) type 0f'u11.04-64'07.  -  To install/upgrade Ubuntu 10.04 (i386) type 0f'u10.04-i386'07.  -  To install/upgrade Solaris 10 type 0f's10'07.  -  To install/upgrade Windows 7 type 0f'w7'07. ————————————————————————-  -  You may add additional options such as 0f'askmethod'07 or     0f'text'07 or 0f'rescue'07 or 0f'dd'07.  For example     0f'f11 dd xfs text askmethod'07. —————————————————————————— 02/F1-Main/F2-HTTP/F3-Final/F4-Lending/F5-Work 1/F6-Work 2/F7-Friend/07 # cat /tftpboot/pxelinux.cfg/default # prompt 1 default vesamenu.c32 timeout 600 # display fcboot.msg menu background splash.jpg menu title PXE Network Installation Menu (F1 for Help & Details) menu color border 0 #ffffffff #00000000 menu color sel 7 #ffffffff #ff000000 menu color title 0 #ffffffff #00000000 menu color tabmsg 0 #ffffffff #00000000 menu color unsel 0 #ffffffff #00000000 menu color hotsel 0 #ff000000 #ffffffff menu color hotkey 7 #ffffffff #ff000000 menu color scrollbar 0 #ffffffff #00000000 F1 vesamenu.c32 F2 fcboot.msg F3 http.msg F4 final.msg F5 lending.msg F6 work-1.msg F7 work-2.msg F8 friend.msg label memtest         menu label Fedora 14 (64) System Memory Test         kernel f14-64/isolinux/memtest label s6.1-64         menu label Scientific Linux (RHEL) 6.1 (64)         kernel s6.1-64/isolinux/vmlinuz         append initrd=s6.1-64/isolinux/initrd.img label f14-64         menu label Fedora 14 (64) (Standard)         kernel f14-64/isolinux/vmlinuz         append initrd=f14-64/isolinux/initrd.img label u10.04-i386         menu label Ubuntu 10.04 (i386)         kernel u10.04-i386/casper/vmlinuz         append initrd=u10.04-i386/casper/initrd.lz label u11.04-64         menu label Ubuntu 11.04 (64)         kernel u11.04-64/casper/vmlinuz         append initrd=u11.04-64/casper/initrd.lz LABEL w7         MENU LABEL Windows 7 (Beta / Pre release)         KERNEL Boot/startrom.0   Explanation for a few things is in order: These files define the boot menu that will be seen on the remote server.  here we will use the vesamenu.c32, which is a more GUI oriented rather then text oriented.  Just looks better.  :)  The alternative, currently commented, is the fcboot.msg file if a text menu is preferred. The paths are relative to the /tftpboot folder.   The boot menu, when loaded, will look like this:
The HTTPD server httpd.conf file.	The httpd.conf file should contain the locations the installation can retrieve the entire ISO image installation files once selections have been made: # cat /etc/httpd/conf/httpd.conf|tail -n50 . . . . <Directory /tftpboot/RHEL4-22B>    Options Indexes    AllowOverride None </Directory> Alias /rhel4-22b /tftpboot/RHEL4-22B <Directory /tftpboot/f14-64>    Options Indexes    AllowOverride None </Directory> Alias /f14-64 /tftpboot/f14-64 <Directory /tftpboot/s6.1-64>    Options Indexes    AllowOverride None </Directory> Alias /s6.1-64 /tftpboot/s6.1-64 <Directory /tftpboot/u11.04-64>    Options Indexes    AllowOverride None </Directory> Alias /u11.04-64 /tftpboot/u11.04-64 <Directory /tftpboot/u10.04-i386>    Options Indexes    AllowOverride None </Directory> Alias /u10.04-i386 /tftpboot/u10.04-i386 In order for the above new definitions to take effect, you'll need to restart the Apache HTTPD daemon: # service httpd restart Then test by typing the links below into a browser: http://192.168.1.1/s6.1-64/ OR http://192.168.0.1/s6.1-64/ Either link should produce a list of files for the distribution.  (You'll need to mount the ISO if nothing shows up.  See further down for this.  The below links even worked for me so would be a good test for your installation if everything is wroking correctly.)   Parent Directory   -   EFI/ 27-Jul-2011 16:52 -   GPL 27-Jul-2011 16:53 18K   Packages/ 27-Jul-2011 16:31 -   RPM-GPG-KEY-sl 27-Jul-2011 16:53 1.8K   RPM-GPG-KEY-sl3 27-Jul-2011 16:53 1.9K   RPM-GPG-KEY-sl4 27-Jul-2011 16:53 1.9K   RPM-GPG-KEY-sl5 27-Jul-2011 16:53 1.9K   RPM-GPG-KEY-sl6 27-Jul-2011 16:53 1.9K   TRANS.TBL 27-Jul-2011 17:00 2.8K   images/ 27-Jul-2011 16:53 -   isolinux/ 27-Jul-2011 16:52 -   repodata/ 27-Jul-2011 16:34 -   Check the /var/log/httpd/ folder and files for any error messages if it is not working for you.  Potentially, the firewall, if you have one, may be blocking content.  Adjust it accordingly.  You may also need to define which ports and interfaces the Apache HTTPD server listens on: Change /etc/httpd/conf/httpd.conf FROM Listen 80 TO Listen 80 Listen 192.168.1.1:80 By default, httpd may already listen on all interfaces defined.
Check the /etc/xinetd.d/tftp config file.	Ensure that your tftp xinetd file is setup correctly.  By default, it should already be defined as per below or similar so no changes should be necessary.  Below is a sample configuration that worked with the setup in this post.  Adjust accordingly to your tastes: # cat /etc/xinetd.d/tftp # default: off # description: The tftp server serves files using the trivial file transfer \ #       protocol.  The tftp protocol is often used to boot diskless \ #       workstations, download configuration files to network-aware printers, \ #       and to start the installation process for some operating systems. service tftp {         disable = no         socket_type             = dgram         protocol                = udp         wait                    = yes         user                    = root         server                  = /usr/sbin/in.tftpd         server_args             = -s /tftpboot -vvvvvvvv –verbosity 10 -m /etc/tftpd.map         per_source              = 11         cps                     = 100 2         flags                   = IPv4 } To ensure the file is taken into effect, issue: # service xinetd restart Pay close attention to the output messages to ensure everything has restarted and is working correctly.
Download and mount the ISO's you defined menu options for above.	For this step, I created a short script to mount my ISO's from various sources.  I mounted them all under the /tftpboot but stored the ISO's elsewhere on the system: # cat /tftpboot/mount.ksh #!/bin/ksh mount -o loop /Files/Ubuntu/ubuntu-11.04-desktop-amd64.iso /tftpboot/u11.04-64/ mount -o loop /home/Files/u10.04.i386/ubuntu-10.04-desktop-i386.iso /tftpboot/u10.04-i386 mount -o loop /home/Files/F14-64/Fedora-14-x86_64-DVD.iso /tftpboot/f14-64/ mount -o loop /home/Files/S6.1/SL-61-x86_64-2011-07-27-Install-DVD.iso /tftpboot/s6.1-64 You'll need to create each of the mount points (BOLD BLACK) to each of the downloaded ISO's (BOLD GREEN) you would like to use.  All are mounted using the loopback device (-o loop).
Add .msg files to make for a user friendly informative menu. (Optional)	Below is a sample of a .msg file, /tftpboot/http.msg.  The ^ NNN characters are special codes that highlight.  They can be generated by pressing the CTRL (see below): ^L Once you start the installation, you can choose ^O0f'HTTP'^O07 as the install method. Doing so the installation will prompt you for network parameters.  Use the following: ^O0fIP            : 192.168.1.2^O07 ^O0fMASK          : 255.255.255.0^O07 ^O0fGATEWAY       : 192.168.1.1^O07 ^O0fFIRST DNS     : 192.168.1.1^O07 Following this, the system will ask you for the location of the files.  Use the following: ^O0fHOST          : 192.168.1.2^O07 ^O0fLOCATION      : /<FOLDER>^O07 where ^O0f<FOLDER>^O07 is the same option that you selected on the opening screen such as ^O0f'fc4'^O07 or ^O0f'rhel4-22'^O07 —————————————————————————— ^O02/F1-Main/F2-HTTP/F3-Final/F4-Lending/F5-Work 1/F6-Work 2/F7-Friend/^O07 CUSTOMIZING THE .MSG FILES There are a couple of characters in the .msg files you may be interested in (Press and hold all three down together or press O while holding ctrl-v down): CTRL-V-O CTRL-V-L The ctrl-v-o is followed by '07' or '0f' etc.  In fcboot.msg, to display the splash.lss file, use: ^L ^Xsplash.lss (The rest of the message here) . . This will create a customized boot menu for you with an image.
On the remote machine, enable PXE boot in the boot menu.	Visit your PC or Motherboard manual to enable PXE or pre boot execution on the machine.
Connect and test.	Ensure your new PC/Server is connected on it's primary NIC to the secondary NIC on your newly configured PXE server. When given a choice where to obtain installation files from, select HTTP then use one of the HTTP URL's you defined above, and tested with, in the httpd.conf part of the configuration.

Inevitably, there is always problems with initial setup or any setup.  The next page deals with some things you may encounter and their solutions. (Work In Progress)

Cheers!
TK

So you've lost the password to the root account.  Worst, you've had a boot password and it was the same.  Here are some instructions on how to get back into your system:

Create a user on another Unix / Linux operating system.

# useradd resetrootpass

then set the password on the new user:

# passwd resetrootpass

once password is set, edit the /etc/shadow file and copy the encrypted password text:

# cat /etc/shadow
resetrootpass:$1$4K5.hqbH$lKPzvk8qbmBaK9TJi0p0:11675:101:88888:3:::

to the /etc/shadow root's password field on your OS where you've forgotten your password too (You may need to use a boot CD like a Linux Live CD to mount the system with then mount the partition to get access to the /etc/ folder).

Once you replace the root's encrypted string with the new one you created above for an arbitrary user, you should be able to boot and login with the new password.

Cheers!
TK

This post is designed as a blind test, meaning its sole purpose is for testing and learning LVM functionality.  In this process we will work with an existing LVM that contains live data to grow, shrink and move it's contents around.  Before we do anything, a disclaimer is in order:

DISCLAIMER: Aside from the disclaimer already present on the site at the bottom, I do not recommend you to follow these instructions.  The exception exists where you wish to learn and test functionality on non PRODUCTION environments (ie meaning DEVELOPMENT or QUALITY ACCEPTANCE testing environments).  Using these steps puts you at risk of loosing your data.  BACKUP YOUR DATA BEFORE PROCEEDING.  By reading the below and following any steps, you are agreeing to undertake any and all risks associated with the resulting consequences of following the steps on this site and post below.

Having said that, let's begin.  First, we'll begin by getting some information and marking our LVM for change.  The goal here is to eventually resize this partition that spans the entire 1.5TB to something much smaller (around 200GB) that we can copy to a smaller part of the hard drive to free space for a new installation and VG.  NOTE: Only one hard drive will be used here to do this with:

Before we begin, please note the version of LVM we are using here:

  LVM version:     2.02.83(2)-RHEL6 (2011-03-18)
  Library version: 1.02.62-RHEL6 (2011-03-18)
  Driver version:  4.20.6

The main goal, is to resize and shape our LVM to make room for another installation.  The images below illustrate the goals for this post.  The final image represents the setup, including a resized copy (backup) of the original VG VolGroup called VG VGEntertain sitting on /dev/sda3 at the end of the partition:

If your version differs, these steps may not work the same way for you.  The steps:

1. fdisk -l (Just root partitions should be listed)
2. lvm vgchange -a y
3. fdisk -l (You should see the volume groups listed now.)
4. Run ll /dev/VolGroup/ to locate and identify your partitions.  Note the one you want to resize based on the fdisk and ll output above.
5. To check the partition, you may want to mount it first using these steps:
   su
   lvm vgchange -a y
   lvm lvdisplay
   mkdir /mnt/root
   mount /dev/VolGroup/lv_root /mnt/root
   
   This is good for any data recovery etc.
6. In this case I used it to verify that this is an ext4 partition from the /mnt/root/etc/fstab file.
7. After verifying the usage on the partition from last boot to it, I ran

   resize2fs -p /dev/VolGroup/lv_root 32G

   from 500GB.  This partition was only 3% full anyway and I needed the extra space.  This step may take a while.

8. Next type:

   lvm lvresize /dev/VolGroup/lv_root –size 32G

   This step should be quick but will give you a warning message before you resize.
    

9. Run and record the output of lvm pvs and lvm lvs.  I booted with PXE so could scp the files to the server using scp lvm.pvs.lvs.txt 192.168.0.1:/home/Files/HTPC or equivalent for you.
    
10. At this point we need to manually 'defragment' by moving the other partitions next to each other in the volume group.  We'll start with the swap.  We'll delete it then recreate it.
     
11. I'll recreate the swap in this manner to move it to the beginning right next to the root partition:
    lvm lvremove /dev/VolGroup/lv_swap
    lvm lvcreate –size 4G –name lv_swap VolGroup
    mkswap /dev/VolGroup/lv_swap

12. There is another volume in my volume group LogVol02 where /home resides.  I've got some files in there.  It's 1TB and has about 100G of files on it.  I'll extend this volume then shrink it down to a smaller size which should line it up with the above resized volumes.  The steps are in reverse, somewhat, of the shrinking steps:
    lvm lvextend -L+500G /dev/VolGroup/LogVol02

13. This however, gave me an error saying I don't have that much space to extend by.  So I have to use a smaller number.
    Insufficient free space: 128000 extents needed, but only 116792 available.
     
14. This message is telling us exactly how much we can extend it with:
    500G/128000 * 116792=456.21875=NewSize
     
15. I rerun the command like this:
    lvm lvextend -L+456218M /dev/VolGroup/LogVol02

    Rounding up size to full physical extent 445.53GiB
    Extending logical volume LogVol02 to 1.32 TiB
    Logical volume LogVol02 successfully resized

     

16. Now we need to extend the filesystem in this LV (You can get the actual size of this volume from fdisk -l):

    resize2fs -p /dev/VolGroup/LogVol02 1450G

    But I got:

    The containing partition (or device) is only 353992704 (4k) blocks.
    You requested a new size of 380108800 blocks.
    
    Doing the math 353992704 * 4096 = 1449.95.  Dividing this 3 times by 1024 yields 1350.375G (or 1382784M).  I'll use M instead with this number.  But it wanted me to run e2fsck on it first.

17. Resized the LV using:
    
    resize2fs -p /dev/VolGroup/LogVol02 1382784M

    This step is fairly quick but the next one isn't.
     

18. Now resize the file system back down before we also resize the LV down to 500G.  (I have about 115GB of storage there anyway.  This should give us about 1TB to work with):

    resize2fs -p /dev/VolGroup/LogVol02 500G

    This may take a while.
     

19. Next resize the LV to 500G:

    lvm lvresize /dev/VolGroup/LogVol02 –size 500G

    Now all your volumes should be resized and lined up.  We should be able to resize the VG down to 538G now.  You can mount the /dev/VolGroup/LogVol02 on /mnt/mr or another folder of your choosing then issuing df -h to check the size.
     

20. Now I run:

    lvm pvresize /dev/sda2 –setphysicalvolumesize 500G (Really should be 538G)

    This tells me:

    /dev/sda2: cannot resize to 127999 extents as 137216 are allocated.
    0 physical volmes(s) resized / 1 physical volume(s) not resized

    This calculated for me the numbers that I will need.  However, I still couldn't extend even with sizes of 700G which told me that my attempt to grow then shrink didn't work as planned.  My LogVol02 was now sitting in between two free areas only only shrunk around it's center so to speak.
     

21. We can see the above locations by issuing:

    lvm pvdisplay –maps

    which shows:

    – Physical Segments –
    Physical extent 0 to 8191
    Logical volume          /dev/VolGroup/lv_root
    Logical extents          0 to 8191

    Physical extent 8192 to 9215
    Logical volume          /dev/VolGroup/lv_swap
    Logical extents          0 to 1023

    Physical extent 9216 to 126007
    FREE

    Physical extent 126008 to 254007:
    Logical volume          /dev/VolGroup/LogVol02
    Logical extents          0 to 127999

    Physical extent 254008 to 357648:
    FREE

    So my outermost extent is 254008.

22. So what I'll do is shrink the LogVol02 down to a smaller size (Mounting and checking with df -h showed only 119G usage on it) instead of 500G so it is easier to move, shrink the PV to what ever the last used extent was, create another PV group in the now free space on the disk, move the LogVol02 LV to the new PV, remove the old PV and recreate it, then move the old LV back and remove the new PV created earlier in this step.  All this should line things up.  So here we go:

    resize2fs -p /dev/VolGroup/LogVol02 150G  (fsck if it asks you too)
    lvm lvresize /dev/VolGroup/LogVol02 –size 150G
    Check usage and space (Was at 85% after above): mount /dev/VolGroup/LogVol02 /mnt/mr; cd mr; df -h;  
    lvm pvdisplay –maps  (Last FREE starts at 164408 instead of 254008, which is good)  LV LogVol2 is now 0 to 38399 extents.
    lvm pvresize /dev/sda2 –setphysicalvolumesize 643G   ( 150G / 38399 * 164407.  Complained about 642G )  Don't care about precision here.  Just need to shrink to make room for a new VG.  Only 164408 to 164606 is left at the end.  This is tiny in comparison.

    The next step here was to create a new VG on the same PV:

    lvm vgcreate BackupVG /dev/sda2

    But that gave me:

    Physical volume '/dev/sda2' is already in volume group 'VolGroup'
    Unable to add physical volume '/dev/sda2' to volume group 'BackupVG'

    This was odd but could not find anything conclusive until I read this:

    "PVs and LVs cannot be shared between or span different VGs" - http://en.wikipedia.org/wiki/Logical_volume_management
    
    This basically says I can't have two VG's on the same PV in this case, on the the same /dev/sda2.  So I have to resize the partition, then create a new partition on the free space, then a VG on that then move the LV to the VG on the new PV. (This is certainly turning into an exercise)

    Now we determine the new end sector for our partition.  Well need to verify numbers and calculate it:

    lvm pvs –units g                     ( VolGroup PSize / PFree is 643.00g / 457.00g )
    lvm pvs –units s                     ( VolGroup PSize / PFree is 1348460544S / 958390272S )
    parted /dev/sda unit s print   ( Start / End / Size for the lvm listead as 409663s / 2930272064s / 2929862402s.  It is of type Primary and the flags are lvm)

    Because the VG is a whole unit, not fragmented, it's start to end should be exactly the sectors highlighted in green.  So the next end sector should be the green and blue number (1348870207+2042(safety margin) = 1348872249 ).  There is the extra free space in the middle we are trying to get rid of but that is included in the total VG size, which is good.  If we really need that 1MB later, we can extend it.  

    WARNING: These steps will remove /dev/sda2 partition on sda and recreate it.  If you did not get the end sector correct, this can truncate your partition resulting in DATA LOSS.  There is NO backing out from this.  Make sure you made a backup of your partitions earlier.

    parted /dev/sda rm 2
    parted /dev/sda mkpart primary 409663s 1348872249s
    parted /dev/sda set 2 lvm on
    parted /dev/sda print
    fdisk -l /dev/sda

    Done.  Now check with parted /dev/sda unit s print and the end size should match exactly the number you specified above when recreating it.

    At this point check the FS using e2fsck -f /dev/VolGroup/LogVol02 to ensure things are still fine.

    Now I'm at the point where I have /dev/sda1 and /dev/sda2 and have about 500GB of free space to create another partition /dev/sda3.

    At this point I'll boot up into the system to ensure things still can be booted up as well.  This is the ultimate test.  I'll return to the PXE boot on the second reboot.  The system booted up fine.

    Now we need to make another partition and VG to where we will move LogVol02.  To do this, create a new partition /dev/sda3 then create the VG on that.

    parted /dev/sda "mkpart primary ext4 1348462588s -1s"   (Need the " because shell tries to interpret the - )
    parted /dev/sda set 3 lvm on
    lvm vgcreate BVolGroup /dev/sda3 (But this failed as I had to reboot after creating the above partitions before making VG's)
    lvm pvcreate /dev/sda3
    lvm vgcreate BVolGroup /dev/sda3    (This would create the PV but since we're doing this manually…..)

    Now we should be able to move the LV from /dev/sda2.  The way we do this is not with LVM but rsync (or dd).  dd should be quicker but can leave empty unusable space that could cause more work. This is because we cannot move LV's between VG's according to LVM rules.  We could move LV's to other PV's within the same VG (A VG can span multiple PV's) but that is not our setup here.

    Unfortunately, it occurred to me that since we can't move LV's between VG's, and I have more then 400G space in the middle of my VolGroup, I should be able to create a new LV, rsync between them, then rename the new one.  This should do the trick:

    lvm lvcreate –name /dev/VolGroup/LogVol01 –size 150G
    lvm pvdisplay –maps (verify the volumes LogVol01 and LogVol02 are identical in extents)
    mke2fs -t ext4 /dev/VolGroup/LogVol01
    mkdir /mt/amr /mnt/bmr
    mount /dev/VolGroup/LogVol02 /mnt/amr
    mount /dev/VolGroup/LogVol01 /mnt/bmr
    cd bmr
    rsync -avh /mnt/amr/* /mnt/bmr
    
    Since the copy procedure target and source are using the same physical disk, the write/read speeds will be halved.

    umount /mnt/amr; umount /mnt/bmr
    lvm lvremove /dev/VolGroup/LogVol02
    lvm lvrename /dev/VolGroup/LogVol01 /dev/VolGroup/LogVol02

    Let's see how everything looks like now before we resize the VG / PV:

    lvm pvdisplay –maps
    lvm pvs –units g                     ( VolGroup PSize / PFree is 643.00g / 457.00g )
    lvm pvs –units s                     ( VolGroup PSize / PFree is 1348460544S / 958390272S )
    parted /dev/sda unit s print   ( Start / End / Size for the lvm on /dev/sda2 listead as 409663s / 1348872249s / 1348462587s.  It is of type Primary and the flags are lvm)

    Finally, our LV's show to be lined up in the VG, with the free space showing up between extents 47616 and 164606.  We now need to resize the VG / PV as we did earlier above, but to a now smaller size.  So again, here are the calculations:

    lvm pvresize /dev/sda2 –setphysicalvolumesize 187G   (150G / 38399 * 47616 = 186.  It complained about 186 as there is a fractional remainder)

    Check again with:

    lvm pvs –units s                     ( VolGroup PSize / PFree is 392159232S / 2088906S )
    parted /dev/sda unit s print   ( Start / End / Size for the lvm on /dev/sda2 listead as 409663s / 1348872249s / 1348462587s.  It is of type Primary and the flags are lvm)

    Now we can shrink our /dev/sda2 partition to it's minimal size (409663s + 392159232S + 2048(safety margin)):

    parted /dev/sda rm 2
    parted /dev/sda mkpart primary 409663s 392570943s
    parted /dev/sda set 2 lvm on
    parted /dev/sda print
    fdisk -l /dev/sda
    
    Remove the backup volume we created earlier:

    lvm vgremove /dev/BVolGroup
    lvm pvremove /dev/sda3

    And let's test with a restart.  After the reboot, we'll move the entire PV / VG to partition  /dev/sda3 in preperation for installing Scientific Linux (Another RHEL 6.1 clone like CentOS but with OpenAFS and Kerberos pre included).

    lvm lvresize /dev/VolGroup/LogVol02 -l+255 (to resize to the maximum VG size as only 255 extents were left available at the end.)
    lvm pvsize –maps (Should now show no free extents.)

    There should now be ample 1.5TB - 187GB of free space on the physical device /dev/sda3.  We'll now proceed to move everything to /dev/sda3 in the next step.

    This is how things look now visually:

     

23. Now we'll create /dev/sda3 at the end of the disk with the exact same size as /dev/sda2.  Then we'll pvmove from 2 to 3:

    parted /dev/sda unit s print free
    parted /dev/sda mkpart primary 2538115887s 2930277167s (Based on results of above command and START +1 for an exact size of 392161281s as /dev/sda2)
    parted /dev/sda set 3 lvm on
    parted /dev/sda unit s print free
    lvm pvs
    

24. Now we create the PV:

    lvm pvcreate /dev/sda3
    
    You will notice that after we create the above PV, it's size may be greater then the actual size reported by parted:

    Before:
    PV         VG       Fmt  Attr PSize   PFree 
    /dev/sda2  VolGroup lvm2 a-   187.00g      0
    

    After:
    PV         VG       Fmt  Attr PSize   PFree 
    /dev/sda2  VolGroup lvm2 a-   187.00g      0
    /dev/sda3           lvm2 a-   754.07g 754.07g
     

25. Notice the large PSize of 754.07g that doesn't match the 392161281s sector size of /dev/sda3 (Which should be equal to /dev/sda2).  This is why we need to resize the PV to it's actual size before we extend anything:

    pvresize –test -v –setphysicalvolumesize 392161281s /dev/sda3
    pvresize -v –setphysicalvolumesize 392161281s /dev/sda3  (If everything is ok from the above step.)
    
    PV         VG       Fmt  Attr PSize   PFree 
    /dev/sda2  VolGroup lvm2 a-   187.00g      0
    /dev/sda3            lvm2 a-   187.00g 187.07g
    
    (Double check with lvm pvs –units s and resize again if necessary to match /dev/sda2.) 
     

26. Now we should be ready to extend the VG:
    
    lvm vgextend VolGroup /dev/sda3

27. Finally, we do a pvmove from 2 to 3.  Before you do, ensure that after the vgextend, both /dev/sda2 and /dev/sda3 print the same size by using lvm pvs –units s:

    lvm pvmove /dev/sda2 /dev/sda3
    lvm pvs -o +tags (To check the PV's in the VG VolGroup)

    The above will take some time since /dev/sda2 is around 187GB in this case:
     

28. Once the move is complete and without errors, reduce the VG by /dev/sda2:

    lvm vgreduce VolGroup /dev/sda2

    Mount and check manually.  lvm vgdisplay should also show half the space again.
     

29. Now remove /dev/sda2 PV:

    lvm pvremove /dev/sda2
     

30. Mount to /mnt/mr check manually and reboot to ensure it can be read fine:

    ERROR: Mounting on /mnt/mr worked just fine but after the reboot, nothing mounted and subsequent lvm pvs, lvm vgs and lvm lvs all resulted in:

     

31. So my data was toast at which point I learned the very important commands the various online wiki's never informed me about: vgcfgbackup and vgcfgrestore.  Unfortunately, I HAD NOT USED THE BACKUP.

    WARNING: My system was toast at this point and I had no way to recover the data because I did not have the LVM restore files.

    NOTE: Please read the post LVM: Recovering a lost volume.  The below steps actually overwrote the VolGroup definitions as they were stored on the disk.  I was not aware that the VG definitions were stored at the head of the partition before attepting step 32.  DO NOT DO  
     

32. A BAD ATTEMPT TO RECOVER.  THIS STEP WILL PERMANENTLY DESTROY THE DATA ON THE DISK.  TRY ONLY IF YOU ARE PRACTICING.

    Here, we will try to recover our lost data by rebuilding the config backup we should have done earlier.  We will try to rebuild the backup config manually from screen printouts we took earlier.  First, we will recreate the basic structure we had earlier.  Then, we'll modify this backup config with the same UUID's as in the original configuration.  

    lvm pvcreate /-u x0O0xW-TAXH-Vrz2-g299-pJm6-Mj5g-srkOBH –setphysicalvolumesize 47871e –norestorefile /dev/sda3

    lvm vgcreate VolGroup /dev/sda3

    lvm lvcreate –extents 8192 –name lv_root VolGroup
    lvm lvcreate –extents 1024 –name lv_swap VolGroup
    lvm lvcreate –extents 38655 –name LogVol02 VolGroup

    We will now copy the VG backup from /etc/lvm/backup/VolGroup to our remote host so we can adjust the UUID's from the screen prints we took earlier.  (Copy and paste is cumbersome on the recovery console or I simply do not know an easier way.)

    scp /etc/lvm/backukp/VolGroup root@192.168.1.1:/recovery/

    Once we adjust the VolGroup backup file, we will use the vgcfgrestore option to recreate it with updated UUID numbers.

    Please see the link below for recovering at this point the proper way:

    LVM: Recovering a lost volume.
     

33. Now that we have recovered the LVM, I'll be sure to take backups of /etc/lvm/backup/<VOLUME GROUP> to any remote host.  This way data can be recovered when needed instead of trying to recover the hard way. 
     
34. After the above recovery, the partition is now in an extended state over /dev/sda2 and /dev/sda3, two partitions apart by about 1TB of data.  Having tried the faulty pvmove method above, we won't retry it here.  Instead we will shrink the partition back to /dev/sda2 and create a backup partition on /dev/sda3 of the same size as /dev/sda2.  Then we will rsync / scp data between the two instead.  Should be safer.
     
35. Here is the list of steps:

    #  lvm pvcreate /dev/sda3
    #  lvm vgcreate VGEntertain /dev/sda3
    #  lvm lvs –help
    #  lvm lvs –units s
    LV        VG          Attr   LSize      Origin Snap%  Move Log Copy%  Convert
    LogVol02  VolGroup    -wI-a- 316661760S                                     
    lv_root   VolGroup    -wI-a-  67108864S                                     
    lv_swap   VolGroup    -wI-a-   8388608S                                     

    #  lvm lvcreate -L316661760S -n oLogVol02 VGEntertain
    #  lvm lvcreate -L67108864S -n olv_root VGEntertain
    #  lvm lvcreate -L8388608S -n olv_swap VGEntertain
    #  lvm lvs –units s

      LV        VG          Attr   LSize      Origin Snap%  Move Log Copy%  Convert
      oLogVol02 VGEntertain -wi-a- 316661760S                                     
      olv_root  VGEntertain -wi-a-  67108864S                                     
      olv_swap  VGEntertain -wi-a-   8388608S                                     
      LogVol02  VolGroup    -wI-a- 316661760S                                     
      lv_root   VolGroup    -wI-a-  67108864S                                     
      lv_swap   VolGroup    -wI-a-   8388608S                                     
     
    # lvm vgchange VGEntertain -a y
    #  mke2fs -ext4 /dev/VGEntertain/oLogVol02
    #  mke2fs -t ext4 /dev/VGEntertain/oLogVol02
    #  mount /dev/VolGroup/LogVol02 /mnt/mr
    #  mkdir /mnt/rr
    #  mount /dev/VolGroup/lv_root /mnt/rr
    #  mkdir /mnt/sr
    #  mount /dev/VolGroup/lv_swap /mnt/sr
    #  mkswap /dev/VGEntertain/olv_swap
    #  mke2fs -t ext4 /dev/VGEntertain/olv_root
    #  df -T

    Filesystem    Type   1K-blocks      Used Available Use% Mounted on
    /dev         tmpfs     2026840       224   2026616   1% /dev
    none         tmpfs      256000    129212    126788  51% /tmp
    /dev/loop0
              squashfs      128896    128896         0 100% /mnt/runtime
    /dev/mapper/VolGroup-LogVol02
                  ext4   154818540 124239716  22714504  85% /mnt/mr
    /dev/mapper/VolGroup-lv_root
                  ext4    33027952  13555808  17794424  44% /mnt/rr
     

36. At this point, knowing that my VG's and LV's were recovered, I rebooted to my old installation to take advantage of the GUI and Konsole to copy things between the two VG's I created a number of mount points and mounted each VG:

    # mkdir /mnt/omr /mnt/orr /mnt/osr

    # mount /dev/VGEntertain/oLogVol02 /mnt/omr
    # mount /dev/VGEntertain/olv_root /mnt/orr
    # mount /dev/VGEntergain/olv_swap /mnt/osr

    # mount /dev/VolGroup/LogVol02 /mnt/mr
    # mount /dev/VolGroup/lv_root /mnt/rr
    # mount /dev/VolGroup/lv_swap /mnt/sr
     

37. Next is to sync the two volumes together.  For this I used rsync and went to sleep.  The process completed overnight:

    EX: rsync -avh <SOURCE> <TARGET>

    # rsync -avh /mnt/mr /mnt/omr

    Repeat the same thing for the other mounted folders.
     

38. Now we should also take a backup of the boot partition.  For this again we can use dd.  The below steps highlight the process.  The most important thing below is that the two blue parts below have to match and the two green parts below have to match once you copy things over.  Ideally, you want to save the new dd image file in the backup volume so it is not destroyed when we install the new OS:

    # parted /dev/sda unit s print free
    Model: ATA WDC WD15EADS-32R (scsi)
    Disk /dev/sda: 2930277168s
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos

    Number  Start        End          Size         Type     File system  Flags
     1      63s          409662s      409600s      primary  ext4         boot
     2      409663s      392570943s   392161281s   primary               lvm
            392570944s   2538115886s  2145544943s           Free Space
     3      2538115887s  2930277167s  392161281s   primary               lvm

    #
    # dd if=/dev/sda1 bs=512 count=409600 of=./boot-partition.sda1
    409600+0 records in
    409600+0 records out
    209715200 bytes (210 MB) copied, 9.41159 s, 22.3 MB/s
    # ll
    total 204824
    3407876 drwx——  2 root      root           4096 Mar  9  2010 lost+found
    3407874 drwx——  4 root      root           4096 Jun 13  2010 .Trash-0
    3407875 drwxr-xr-x 20 root      root           4096 Aug 31 21:57 Files
    3407877 drwx—— 47 videouser videouser      4096 Sep  2 14:01 videouser
     262145 drwxr-xr-x 10 root      root           4096 Sep 24 01:39 ..
          2 drwxr-xr-x  6 root      root           4096 Sep 25 21:27 .
         11 -rw-r–r–  1 root      root      209715200 Sep 25 21:27 boot-partition.sda1
    # parted /dev/sda unit b print free
    Model: ATA WDC WD15EADS-32R (scsi)
    Disk /dev/sda: 1500301910016B
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos

    Number  Start           End             Size            Type     File system  Flags
     1      32256B          209747455B      209715200B      primary  ext4         boot
     2      209747456B      200996323327B   200786575872B   primary               lvm
            200996323328B   1299515334143B  1098519010816B           Free Space
     3      1299515334144B  1500301910015B  200786575872B   primary               lvm
    

39. The last thing you want to check is if saving the boot partition above worked properly by mounting it using the loopback device on some free directory:

    # mount /mnt/omr/boot-partition.sda1 /mnt/orr/ -o loop

    If all went according to plan, you should be able to browse the newly created dd out file from the mountpoint /mnt/orr in above example.   
     

40. At this point we are ready to install any new OS over the original VG's and PV's since a backup already exists of them.

    IMPORTANT: Don't forget to take a backup of the VG groups as defined from /etc/lvm/backup/VolGroup and /etc/lvm/backup/VGEntertain per the above.

Cheers!
TK

It may become necessary to recover or rebuild an LVM in case either a backup /etc/lvm/backup/<VOLGROUP> is not available or becomes lost or other conditions arise causing loss of system volume data.  What might otherwise appear destroyed, deleted or inaccessible, LVM comes with a few backup options in case of disaster.  Here are the recovery steps I took to recover previously lost data without either a backup and using only a few open source tools:

NOTE: Do not attempt these steps unless you have a firm understanding of UNIX / LINUX.  The concepts here are relatively advanced.  Also ensure you have a backup of the data that will be affected by your actions below.  Also take note that your mileage may vary and that this may or may not necessarily work for you.  Use with caution. 

Before we proceed any further, the LVM version used here is:

  LVM version:     2.02.83(2)-RHEL6 (2011-03-18)
  Library version: 1.02.62-RHEL6 (2011-03-18)
  Driver version:  4.20.6

If your version differs, the below steps may not work for you.

The first step to take is to grab the start sectors of the partitions that used to hold your LVM.  The Linux LVM holds the equivalent of /etc/lvm/backup/<VOLGROUP> just in case of loss to allow for recovery later, as we will be doing here. To do this, invoke dd and grab 255, 512 byte sectors from the start of, in this case, /dev/sda2:

# dd if=/dev/sda2 bs=512 count=255 skip=1 of=./sda2.lvm

This will create a binary file that I found only best to use a hex editor to view with.  I used Okteta to get a look at the file in HEX notation, simply because it was installed on my secondary system at the time.  I used it to identify start and end of blocks I need to get the correct block from the file.  The start and end sectors of a block begin with the 0 ASCII character: 00.  Here is how the file looked like in Okteta, showing start and end markers for the LVM PV's, VG's, and LV's definition block:

Okteta Linux LVM - START of LVM Definition.

Okteta Linux LVM - END of LVM Definition.

Okteta Linux LVM - END of LVM Definition (alternate example).

Please note the end point actually marked by the green in the second image, NOT the red.  They are marked on a 00 byte boundary.  I did not find the START and END as easily descernable in VI.  I expect Okteta, or any other HEX editor for that matter,  to better indicate the starting and ending points.  Your mileage may vary.  The recovered file looked like this (sda2-lvm-final.lvm):

VolGroup {
        id = "l6D50I-m8tf-2iPJ-ef1R-O9kf-kH2v-7SslkB"
        seqno = 51
        status = ["RESIZEABLE", "READ", "WRITE"]
        flags = []
        extent_size = 8192
        max_lv = 0
        max_pv = 0
        metadata_copies = 0

        physical_volumes {

                pv0 {
                        id = "AR2cbu-Zlxw-8nR0-423o-20pa-3g3l-Te3gcI"
                        device = "/dev/sda2"

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 392161281
                        pe_start = 384
                        pe_count = 47871
                }

                pv1 {
                        id = "1tycCU-hDC0-USvN-fuS8-gdgd-QIHq-UHcb0Y"
                        device = "/dev/sda3"

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 392161281
                        pe_start = 2048
                        pe_count = 47871
                }
        }

        logical_volumes {

                lv_root {
                        id = "b3XsIw-JC3l-cIEU-njSP-1NbY-kPaO-gvMJda"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 8192

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pv1", 0]
                        }
                }

                lv_swap {
                        id = "KBdB2L-j1ZS-8iA6-Uilh-43G1-NKxX-eE06FW"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 1024

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pv1", 8192]
                        }
                }

                LogVol02 {
                        id = "beOzB5-znIO-SD44-bkgr-aT9F-gb6N-9Ayyia"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 38655

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pv1", 9216]
                        }
                }
        }
}
# Generated by LVM2 version 2.02.83(2)-RHEL6 (2011-03-18): Sun Sep 11 05:51:56 2011

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "localhost.localdomain" # Linux localhost.localdomain 2.6.32-131.0.15.el6.x86_64 #1 SMP Sat May 21 10:27:57 CDT 2011 x86_64
creation_time = 1315720316      # Sun Sep 11 05:51:56 2011

After trying to apply the restored VolGroup sda2-lvm-final.lvm file, received the message:

# lvm vgcfgrestore -f sda2-lvm-final.lvm VolGroup
Couldn't find device with uuid AR2cbu-Zlxw-8nR0-423o-3g3l-Te3gcI.
Couldn't find device with uuid 1tycCu-hDC0-USvN-fuS8-gdgd-QIHq-UHcb0Y.
Cannot restore Volume Group VolGroup with 2 PVs marked as missing.
Restore failed.
#

This is because in order to restore, the vgcfgrestore commands needs to see the same PV's (including matching UUID's) on the system as existed before.  At this point, I haven't created them yet.

We'll need to PV create on /dev/sda2 and /dev/sda3 as per the VolGroup recovered file above and assign the same UUID's as in our restore file.  (The alternate might be to change the VolGroup recovery backup file and assign the new PV UUID's to the VolGroup restore file before applying.  We will not try that here however so I cannot vouch for the success of this method.)

Before we can restore the VolGroup, we need to do so to PV's with the same UUID's.  For this we create teh PV's first:

# lvm pvcreate -u $(cat sda2-lvm-final.lvm |grep 1tycCU|awk '{ gsub(/"/, "", $0) } { print $3 }') /dev/sda3 –restorefile sda2-lvm-final.lvm
# lvm pvcreate -u $(cat sda2-lvm-final.lvm |grep AR2cbu|awk '{ gsub(/"/, "", $0) } { print $3 }') /dev/sda2 –restorefile sda2-lvm-final.lvm

Followed by:

# lvm vgcfgrestore -f sda2-lvm-final.lvm VolGroup
Restored volume group VolGroup
#

Unfortunately, in my case, this didn't work.  I can't say I didn't expect that entirely because I had earlier done lvm pvmove from /dev/sda2 to /dev/sda3.  Next I tried the same thing but this time attempting to recover the volume from /dev/sda3.  This one I had less hope of recovering because I had previously unsuccessfully tried to recreate the LVM structure on /dev/sda3 that I thought I had lost.  Unfortunately unknown to me at the time, I did NOT know about this type of recovery process.  But I degress and now on to the show.

Below is the derived VolGroup definition I could derive from my former installation from the beginning of the partition using the means already explained above:

VolGroup {
        id = "l6D50I-m8tf-2iPJ-ef1R-O9kf-kH2v-7SslkB"
        seqno = 5
        status = ["RESIZEABLE", "READ", "WRITE"]
        flags = []
        extent_size = 8192
        max_lv = 0
        max_pv = 0
        metadata_copies = 0

        physical_volumes {

                pv0 {
                        id = "x0O0xW-TAXH-Vrz2-g299-pJm6-Mj5g-srkOBH"
                        device = "/dev/sda3"

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 392161281
                        pe_start = 2048
                        pe_count = 47871
                }
        }

        logical_volumes {

                lv_root {
                        id = "b3XsIw-JC3l-cIEU-njSP-1NbY-kPaO-gvMJda"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 8192

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pv0", 0]
                        }
                }

                lv_swap {
                        id = "KBdB2L-j1ZS-8iA6-Uilh-43G1-NKxX-eE06FW"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 1024

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pv0", 8192]
                        }
                }

                LogVol02 {
                        id = "beOzB5-znIO-SD44-bkgr-aT9F-gb6N-9Ayyia"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 38655

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pv0", 9216]
                        }
                }
        }
}
# Generated by LVM2 version 2.02.83(2)-RHEL6 (2011-03-18): Mon Sep 12 05:04:15 2011

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "localhost.localdomain" # Linux localhost.localdomain 2.6.32-131.0.15.el6.x86_64 #1 SMP Sat May 21 10:27:57 CDT 2011 x86_64
creation_time = 1315803855      # Mon Sep 12 05:04:15 2011

Again we mimick the steps taken above to try and recover the lost LVM:

1. lvm pvcreate -u $(cat sda3-lvm-final-3.lvm |awk '{ if ( $0 ~ /x0O0xW/ ) { gsub(/"/,"",$3); print $3 } }') /dev/sda3 –restorefile sda3-lvm-final-3.lvm
2. lvm vgcfgrestore -f sda3-lvm-final-3.lvm VolGroup
3. lvm vgscan
4. lvm pvscan
5. lvm vgchange VolGroup -a y
6. lvm pvs
7. lvm vgs
8. lvm lvs

The last three commands being the verification.  All showed LV's, PV's and VG's listead.  Now to mount the volumes to test.  Unfortunately:

# mount /dev/VolGroup/lv_root /mnt/mr
mount: you must specify the filesystem type
#

Adding the -vvvvv to the above command reveals that the mounter couldn't see an OS type from the head of the volume it appears:

mount: types: "(null)"
mount: opts:  "(null)"

At this point it looked like I was really in trouble.  So I decided to see where my data was really located.  For this I used dd:

# dd if=/dev/sda2 bs=512 count=100000 skip=1 of=./sda2.dd.lvm
# dd if=/dev/sda3 bs=512 count=100000 skip=1 of=./sda3.dd.lvm

Then scp'd the files to my server and analyzed them using:

# strings sda2.dd.lvm | more
# strings sda3.dd.lvm | more

This quickly revealed where the problem looked to be.  I could see my file names off the old partition on sda2 but NOT on sda3.  I had remembered what I did that lead to my predicament which was:

# pvcreate /dev/sda3
# vgextend VolGroup /dev/sda3
# pvmove /dev/sda2 /dev/sda3
# vgreduce VolGroup
# pvremove /dev/sda2

There is also free space between /dev/sda2 and /dev/sda3, about 1TB o f free space if not more.  I had hoped the pvmove will move everything between partitions as advertised but unfortunately, it did not.  But at least now I could confirm that my data was still on /dev/sda2 and nothing was on /dev/sda3.  So I had to work with any remaining LVM configs off of /dev/sda2 if I have any hope of recovering my data.  So I edited the smaller dd output earlier for sda2 and recreated any and all available definitions I could find and started reimplementing from the earliest moving backwards:

And this time, mount worked on /dev/VolGroup/LogVol02 and I could see my files:

# mount /dev/VolGroup/LogVol02 /mnt/mr
# du -sh /mnt/mr
119G           /mnt/mr
#

But I wanted /dev/VolGroup/lv_root to be visible.  So off I go to the next available definition, inspired by my success above and try that.  This time I used the third last one:

VolGroup {
        id = "l6D50I-m8tf-2iPJ-ef1R-O9kf-kH2v-7SslkB"
        seqno = 47
        status = ["RESIZEABLE", "READ", "WRITE"]
        flags = []
        extent_size = 8192
        max_lv = 0
        max_pv = 0
        metadata_copies = 0

        physical_volumes {

                pv0 {
                        id = "AR2cbu-Zlxw-8nR0-423o-20pa-3g3l-Te3gcI"
                        device = "/dev/sda2"

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 392161281
                        pe_start = 384
                        pe_count = 47871
                }

                pv1 {
                        id = "1tycCU-hDC0-USvN-fuS8-gdgd-QIHq-UHcb0Y"
                        device = "/dev/sda3"

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 392161281
                        pe_start = 2048
                        pe_count = 47871
                }
        }

        logical_volumes {

                lv_root {
                        id = "b3XsIw-JC3l-cIEU-njSP-1NbY-kPaO-gvMJda"
                        status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 8192

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pvmove0", 0]
                        }
                }

                lv_swap {
                        id = "KBdB2L-j1ZS-8iA6-Uilh-43G1-NKxX-eE06FW"
                        status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 1024

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pvmove0", 8192]
                        }
                }

                LogVol02 {
                        id = "beOzB5-znIO-SD44-bkgr-aT9F-gb6N-9Ayyia"
                        status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 38655

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pvmove0", 9216]
                        }
                }

                pvmove0 {
                        id = "2D2d4C-tTFL-IJLu-CDX8-Ud6l-juav-7Xh9on"
                        status = ["READ", "WRITE", "PVMOVE", "LOCKED"]
                        flags = []
                        allocation_policy = "contiguous"
                        segment_count = 3

                        segment1 {
                                start_extent = 0
                                extent_count = 8192

                                type = "mirror"
                                mirror_count = 2
                                extents_moved = 0

                                mirrors = ["pv0", 0,"pv1", 0]
                        }
                        segment2 {
                                start_extent = 8192
                                extent_count = 1024

                                type = "mirror"
                                mirror_count = 2
                                extents_moved = 0

                                mirrors = ["pv0", 8192,"pv1", 8192]
                        }
                        segment3 {
                                start_extent = 9216
                                extent_count = 38655

                                type = "mirror"
                                mirror_count = 2
                                extents_moved = 0

                                mirrors = ["pv0", 9216,"pv1", 9216]
                        }
                }
        }
}
# Generated by LVM2 version 2.02.83(2)-RHEL6 (2011-03-18): Sun Sep 11 02:55:09 2011

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "localhost.localdomain" # Linux localhost.localdomain 2.6.32-131.0.15.el6.x86_64 #1 SMP Sat May 21 10:27:57 CDT 2011 x86_64
creation_time = 1315709709      # Sun Sep 11 02:55:09 2011

This time, I could see and mount all my partitions again:

# mount /dev/VolGroup/LogVol02 /mnt/mr
# mount /dev/VolGroup/lv_root /mnt/lv_root
# du -sh /mnt/mr
119G           /mnt/mr
# du -sh /mnt/lv_root
13G              /mnt/lv_root
#

And this is why I got my partitions back and working on /dev/sda2.  I did notice one thing however.  As soon as I restored the above definition the hard drive light on the disk of the target server stayed on as if it was copying something.  Looking I noticed kblockd using some CPU while the hard drive light was on.  So I checked what is doing disk I/O using this command:

# for ikh in $(ls /proc/*/io); do echo "[$ikh]: "$(cat $ikh 2>/dev/null)|awk '{ if ( $0 ~ /rchar/ && $3 != 0 ) print $1" "$3; }'; done

Which, when ran twice, reveals which processes show a delta on Disk I/O:

[/proc/1/io]: 388814
[/proc/1000/io]: 24292
[/proc/1004/io]: 673413398
[/proc/12569/io]: 49573
[/proc/12583/io]: 30445
[/proc/12592/io]: 170451
[/proc/67/io]: 107712397
[/proc/773/io]: 7780
[/proc/774/io]: 261734
[/proc/775/io]: 2412006
[/proc/776/io]: 26700
[/proc/778/io]: 1777975
[/proc/78/io]: 115248
[/proc/794/io]: 440762
[/proc/807/io]: 16012
[/proc/836/io]: 7767303
[/proc/84/io]: 58436892
[/proc/93/io]: 4478177
[/proc/self/io]: 1956
 

Checking processes, this quickly revealed the processes:

root      1004  0.0  0.0 108524  2120 tty1     S    Sep17   0:08 bash
root     23238  0.0  0.0  20204  1252 ?        S    02:32   0:00 hald-addon-input: Listening on /dev/input/event3

With 1004 belonging to my current shell.  So the only thing that I could deduce, that when I restored the LVM config, the pvmove defined in it, retriggered an lvm pvmove /dev/sda2 /dev/sda3 again.  I will let this one finish but I won't vgreduce anymore once it's done, but instead check with dd exactly where it moved things before I reduce and shrink my partition. 

Having let the activity run for a while, I noted that even when it was done, I still could not reduce the VolGroup by /dev/sda3 at all:

# lvm vgreduce VolGroup /dev/sda3
Physical volume /dev/sda3 is still in use.

So I decided to tweak the last VolGroup definition by removing the move blocks or amending them like this by editing everything in red out (Now this should work since we verified above that /dev/sda3 has NO data):

VolGroup {
        id = "l6D50I-m8tf-2iPJ-ef1R-O9kf-kH2v-7SslkB"
        seqno = 47
        status = ["RESIZEABLE", "READ", "WRITE"]
        flags = []
        extent_size = 8192
        max_lv = 0
        max_pv = 0
        metadata_copies = 0

        physical_volumes {

                pv0 {
                        id = "AR2cbu-Zlxw-8nR0-423o-20pa-3g3l-Te3gcI"
                        device = "/dev/sda2"

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 392161281
                        pe_start = 384
                        pe_count = 47871
                }

                pv1 {
                        id = "1tycCU-hDC0-USvN-fuS8-gdgd-QIHq-UHcb0Y"
                        device = "/dev/sda3"

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 392161281
                        pe_start = 2048
                        pe_count = 47871
                }
        }

        logical_volumes {

                lv_root {
                        id = "b3XsIw-JC3l-cIEU-njSP-1NbY-kPaO-gvMJda"
                        status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 8192

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pvmove0", 0]
                        }
                }

                lv_swap {
                        id = "KBdB2L-j1ZS-8iA6-Uilh-43G1-NKxX-eE06FW"
                        status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 1024

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pvmove0", 8192]
                        }
                }

                LogVol02 {
                        id = "beOzB5-znIO-SD44-bkgr-aT9F-gb6N-9Ayyia"
                        status = ["READ", "WRITE", "VISIBLE", "LOCKED"]
                        flags = []
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 38655

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = ["pvmove0", 9216]
                        }
                }

                pvmove0 {
                        id = "2D2d4C-tTFL-IJLu-CDX8-Ud6l-juav-7Xh9on"
                        status = ["READ", "WRITE", "PVMOVE", "LOCKED"]
                        flags = []
                        allocation_policy = "contiguous"
                        segment_count = 3

                        segment1 {
                                start_extent = 0
                                extent_count = 8192

                                type = "mirror"
                                mirror_count = 2
                                extents_moved = 0

                                mirrors = ["pv0", 0,"pv1", 0]
                        }

                        segment2 {
                                start_extent = 8192
                                extent_count = 1024

                                type = "mirror"
                                mirror_count = 2
                                extents_moved = 0

                                mirrors = ["pv0", 8192,"pv1", 8192]
                        }

                        segment3 {
                                start_extent = 9216
                                extent_count = 38655

                                type = "mirror"
                                mirror_count = 2
                                extents_moved = 0

                                mirrors = ["pv0", 9216,"pv1", 9216]
                        }
                }
        }
}

# Generated by LVM2 version 2.02.83(2)-RHEL6 (2011-03-18): Sun Sep 11 02:55:09 2011

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "localhost.localdomain" # Linux localhost.localdomain 2.6.32-131.0.15.el6.x86_64 #1 SMP Sat May 21 10:27:57 CDT 2011 x86_64
creation_time = 1315709709      # Sun Sep 11 02:55:09 2011

Sure enough, when the above edited file was reimplemented with vgcfgrestore, the volume was mountable and this time showed enough free space on /dev/sda3 by which to reduce it:

  PV /dev/sda2   VG VolGroup   lvm2 [187.00 GiB / 0    free]
  PV /dev/sda3   VG VolGroup   lvm2 [187.00 GiB / 187.00 GiB free]
  Total: 2 [373.99 GiB] / in use: 2 [373.99 GiB] / in no VG: 0 [0   ]

And this time vgreduce worked successfully, and I was back on /dev/sda2 about to attempt a mirror instead then a vgreduce and pvremove.

MORAL OF THE STORY: 
Consider using rsync or scp between volumes and define new ones manually before you copy over.  You may get better mileage that way.  Also, most importantly, take backup of the LVM config before you reboot.  If that fails and you still get yourself into such a mess, then at least bookmark this site. 

Cheers!
TK