Discussion:
Prepare for the worst, Howto backup the backuppc-server
(too old to reply)
Kurt Tunkko
2008-07-09 22:23:07 UTC
Permalink
Hi,

we have two servers in two offices that are running backuppc.
Since I have setup a (Soft-)RAID1 (mirroring) the server should be fine
if one of its disk dies.

Now I would like to make sure that if in case of fire or
<INSERT-SCENERIO-HERE> I still have a backup of the server on an
external harddrive that is locked away.

Unfortunately I don't know where to start to make sure that I can do a
baremetal recovery of the backuppc server.

My Idea was todo a full diskimage of the server and then to (r)sync
daily to the external harddrive. This harddrive will then be stores
away when everybody is out of office.

What do you think may be the right approach?

Also I would like to hear if and how you made bare metal recovery's with
backuppc(s data).

- Kurt
dan
2008-07-10 00:05:26 UTC
Permalink
add a third member drive to your software raid so you have a 3 device
raid1. setup a script for when you hotplug the drive to automatically add
it to the raid1 and another script to disconnect it from the raid(by serial
number or UUID or something unique to each disk).

you can access the disk on any linux system with the md module loaded.

you can even do this across a WAN link to home or another office with iscsi
on the remote side, then you just have that iscsi device as a member of the
raid1 and make a script to detach and reattach the iscsi drive when you want
a backup. additionally, you can have an email sent to you when a rebuild
completes, or even a text message so you know the sync is done.

if you leave a USB device or a remote iscsi disk connected to your raid1, it
will certainly slow performance down so remember to detach the backup drive
Post by Kurt Tunkko
Hi,
we have two servers in two offices that are running backuppc.
Since I have setup a (Soft-)RAID1 (mirroring) the server should be fine
if one of its disk dies.
Now I would like to make sure that if in case of fire or
<INSERT-SCENERIO-HERE> I still have a backup of the server on an
external harddrive that is locked away.
Unfortunately I don't know where to start to make sure that I can do a
baremetal recovery of the backuppc server.
My Idea was todo a full diskimage of the server and then to (r)sync
daily to the external harddrive. This harddrive will then be stores
away when everybody is out of office.
What do you think may be the right approach?
Also I would like to hear if and how you made bare metal recovery's with
backuppc(s data).
- Kurt
-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Kurt Tunkko
2008-07-10 07:50:30 UTC
Permalink
Post by dan
add a third member drive to your software raid so you have a 3 device
raid1. setup a script for when you hotplug the drive to automatically
add it to the raid1 and another script to disconnect it from the raid(by
serial number or UUID or something unique to each disk).
Great idea,

if I understand it correctly, iSCSI will allow me to add a harddrive via
TCP/IP to my RAID. Sounds nice, but as far as I can see (after a quick
research) this will cost arround 700 Eur (ex: Allnet NAS RAID System 5x
SATA - ALL6600 iSCSI)?

My idea is to use the harddrives we have available after we changed from
backing to external harddrives to one backuppc server.

Questions:

1) Is is also possible to add a 3rd harddrive via USB or Firewire to the
RAID1 and then using your idea to detach and reattach the harddrive?

2) When I attach the harddrive to my RAID 1, is md smart enough to just
transfer the changes to the attached drive or will it make a full sync?
(sorry for this newbie question)
Post by dan
additionally, you can have an email sent
to you when a rebuild completes, or even a text message so you know the
sync is done.
That sounds interesting, can you point me to a good place to start
reading about how to implement this?

Thanks for your helpfull idea using the already available RAID1.

Best regards

- Kurt
Carl Wilhelm Soderstrom
2008-07-10 12:53:04 UTC
Permalink
Post by Kurt Tunkko
My Idea was todo a full diskimage of the server and then to (r)sync
daily to the external harddrive. This harddrive will then be stores
away when everybody is out of office.
Rsync scales poorly to millions of files; the memory requirements are too
high and you'll run the box out of memory very easily.
Post by Kurt Tunkko
What do you think may be the right approach?
For the moment, the proposed solution of making/breaking a mirror seems to
be the favored one.
Post by Kurt Tunkko
Also I would like to hear if and how you made bare metal recovery's with
backuppc(s data).
I've done it a bunch of times; at least on Linux boxen. (Windows is
obviously much more difficult due to the registry and the lack of a good
tool to rewrite the bootloader). The basic idea of it is:

boot bare metal machine with knoppix or some other rescue distro
BackupPC_tarCreate | netcat to baremetal machine | untar
rewrite the bootloader
reboot and you're done

It works pretty much the same way you'd clone a machine.
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
dan
2008-07-11 00:18:19 UTC
Permalink
1) Is is also possible to add a 3rd harddrive via USB or Firewire to the
RAID1 and then using your idea to detach and reattach the harddrive?

yes, that is a very good method. I suggest you try to do esata but usb or
firewire will work. esata is just much faster.

2) When I attach the harddrive to my RAID 1, is md smart enough to just
transfer the changes to the attached drive or will it make a full sync?
(sorry for this newbie question)

it is smart enough to only read identical blocks on the disk and only write
changed blocks. you still have to wait, at a minimum, the time it takes to
do an entire sequential read of the disk plus the time to write the changes
to the disk.

other stuff
iscsi:
you do not have to have any kind of high end fancy iscsi SAN, you can make
an iSCSI target with a regular linux machine. iSCSI is not terribly hard to
setup, but you would need some sore of dedicated equipment, either a small
server or a stand alone device.

AoE: another very very handy solution is ATA-over-Ethernet. It is
something like iscsi but is not a protocol on top of IP, it is a protocal on
the same level as IP. That means that you cannot route it between networks
which could be considered a negative but the BIG positive is that you need
ZERO IP configuration.

When you make an AoE server, you export disks with a "shelf" + "slot"
topology that is carried over to each machine that sees the AoE.

For example. on your linux server, you do AoE with a program called vblade
and you make a disk vblade 0 0 which is shelf 0 slot 0. you can then do
another disk that is shelf 0 slot 1. on another server you can do a shelf 1
slot 0 and so on.

on every single machine that can see the AoE disks(because they have the aoe
module running) they all see the same drives with the same ID.
/dev/etherd/e0.0 for shelf0slot0 /dev/etherd/e0.1 for shelf0slot1.

on your backuppc server, you can add /dev/etherd/e0.1 to the mirror, you
could even add two AoE disks or more to the mirror if you desired. You can
even export AoE disks(with vblade) that are just disk images.

The beauty of this system is that the backuppc server can be on a server
rack and all the hot swapping of drives can be done on a linux machine
somewhere else. this happens buy exporting a USB drive on the linux machine
via AoE and having backuppc watch for the AoE disk and running the
appropriate mdadm script. If you really wanted to get creative, you could
run linux in a vmware vm and tie a specific USB port to that vm, now you can
export the USB disk over AoE on linux, while running WinXP on your desktop.
ubuntu server with AoE can run in a 128MB of ram vm quite well.

AoE works without IP overhead so is very fast. near-wire speeds on 100Mb
ethernet or even gigabit.

Maybe I'm crazy, I don't know!
Kurt Tunkko
2008-07-14 00:44:17 UTC
Permalink
Hello Dan & ...,

back from a 2 weeks vacation time to implemend your ideas about how to
backup the backuppc-server.

I'll try to add a 3rd harddrive to my RAID1-setup.

As far I understand you also mentioned two other possibilities:

1) builting a iSCSI-device by using another pc. I didn't know that this
is possible, but doing while browsing the web, I found:

"Don't put your iSCSI SAN on your regular IP network, either. There's
plenty of traffic running over that, and iSCSI is bandwidth-intensive.
Also consider whether your servers have enough CPU power to handle
iSCSI. Unwrapping and reassembling iSCSI packets can take a lot of CPU
time."

Since I didn't found a beginner-friendly step-by-step guide, I think I
have to wait until I gain more experience in order to try to setup an
own iSCSI-device.

2) AoE - ATA-over-Ethernet.
Sounds really interesting but also seems to be harder to implement than
adding another disk to my raid-system.
Post by dan
Maybe I'm crazy, I don't know!
uhm .. maybe you just know much much more than I know about linux,
server-administration :-)

As soon as I have succesfully added more disks to my raid I'll update
the wiki and will also your ideas (iSCSI, AoE).


- Kurt
dan
2008-07-22 18:05:30 UTC
Permalink
AOE is quite simple. on ubuntu you can build a target like this

have a spare drive/partition/lvm!
sudo apt-get install vblade
vblade 0 0 eth0 /dev/sdc

thats it! now you are exporting a device on shelf 0, slot 0 (vblade 0 0)

on another machine
sudo apt-get install aoetools
modprobe aoe

you should now see /dev/etherd/e0.0 (this is the shelf 0, slot 0 drive from
the other machine)

done!

a 'client' can see any aoe device on the network, without even having an IP
address. IP is not used at all on aoe.

now on the 'client' machine, you can
"mkfs.ext3 /dev/etherd/e0.0 && mkdir /mnt/e0.0 && mount /dev/etherd/e0.0
/mnt/e0.0"

or better yet, add it to your raid array with mdadm.
Post by Kurt Tunkko
Hello Dan & ...,
back from a 2 weeks vacation time to implemend your ideas about how to
backup the backuppc-server.
I'll try to add a 3rd harddrive to my RAID1-setup.
1) builting a iSCSI-device by using another pc. I didn't know that this
"Don't put your iSCSI SAN on your regular IP network, either. There's
plenty of traffic running over that, and iSCSI is bandwidth-intensive.
Also consider whether your servers have enough CPU power to handle
iSCSI. Unwrapping and reassembling iSCSI packets can take a lot of CPU
time."
Since I didn't found a beginner-friendly step-by-step guide, I think I
have to wait until I gain more experience in order to try to setup an
own iSCSI-device.
2) AoE - ATA-over-Ethernet.
Sounds really interesting but also seems to be harder to implement than
adding another disk to my raid-system.
Post by dan
Maybe I'm crazy, I don't know!
uhm .. maybe you just know much much more than I know about linux,
server-administration :-)
As soon as I have succesfully added more disks to my raid I'll update
the wiki and will also your ideas (iSCSI, AoE).
- Kurt
-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Kurt Tunkko
2008-07-22 19:09:07 UTC
Permalink
Hey Dan,
Post by dan
AOE is quite simple.
:-o ... wait, not THAT simple ... simple is something like: throw in
coin here -> get a beer :-)

I just think that I don't fully understand the concept of AOE and how I
can apply your setup in order to back our server to a 3rd harddrive that
is NOT in the same room like our server.

As far as I understand your idea, I would setup an other-machine that
will have one harddrive/parition configured as AOE.

BackupPC-Server -> network -> other-machine -> AOE-Disk

have I understand this right - since we have lots of spare machines
available there wouldn't be a problem setting up another one.

BUT I don't get how the backuppc-server "knows" where the AOE disk is?
sounds like magic.

If it's not to offtopic I would like to ask if you can add some more
newbie-friendly information how this is working, maybe using the names
backuppc-server (that wants to add the AOE-device to its RAID) and
aoeserver (that has the real drive built in).

thanks a lot - I'm looking forward to make some testing

- Kurt
Post by dan
[...] on ubuntu you can build a target like this
have a spare drive/partition/lvm!
sudo apt-get install vblade
vblade 0 0 eth0 /dev/sdc
thats it! now you are exporting a device on shelf 0, slot 0 (vblade 0 0)
on another machine
sudo apt-get install aoetools
modprobe aoe
you should now see /dev/etherd/e0.0 (this is the shelf 0, slot 0 drive
from the other machine)
done!
a 'client' can see any aoe device on the network, without even having an
IP address. IP is not used at all on aoe.
now on the 'client' machine, you can
"mkfs.ext3 /dev/etherd/e0.0 && mkdir /mnt/e0.0 && mount /dev/etherd/e0.0
/mnt/e0.0"
or better yet, add it to your raid array with mdadm.
Jon Craig
2008-07-22 22:56:17 UTC
Permalink
Post by Kurt Tunkko
:-o ... wait, not THAT simple ... simple is something like: throw in
coin here -> get a beer :-)
mmm, Beeeeer!

AOE stands for "ATA over Ethernet" and is a means of building a poor
mans Storage Area Network (SAN). Look at
"http://en.wikipedia.org/wiki/Ata_over_ethernet" for the details.

At one time the world was flat, and we used ATA disks to make it spin.
In particular, we used Parallel ATA (PATA). These drives used a
ribbon cable and transferred data down the wires a bit per wire (hence
parallel). Then Christopher Columbus set out and discovered the new
world, Serial ATA. In this world data was transferred down a single
wire serially. The beauty of this was that by using serial
communications the length of the wire could be drastically increased
(literally from inches to miles).

What AOE does is replace the serial interface normally found within
the box and extends it across the network (which is itself a serial
communications link). Now when your dealing with a interface in a box
you have a straight forward point-to-point link so you don't require
any discovery mechanisms. When you introduce the network a number of
protocol related things get added to allow a provider to advertise
resources and consumers the ability to discover resources. This puts
the "simple" into the equation. Once the AOE serves up a device it
looks to the Linux kernel like its just another locally attached disk.
All of the standard mkfs/lvm/etc utilities can be used to integrate
it into your environment.

Caveats - It is normally recommended that you have a separate network
interface for AOE devices and it should be a minimum of 100MB and even
better would be GigE. Disk IO is notoriously chatty and can
disrupt/overburden a shared network interface.
--
Jonathan Craig
dan
2008-07-23 13:31:14 UTC
Permalink
to the question of how and why backuppc server would 'know' where the aoe
disk are, aoe is actually a protocol like like IP. it is only visible on
the same network, as in no routing possible, period. the aoe module in the
kernel looks at the local network and can see all the local aoe nodes
because they are so close** meaning they look like they are all in the same
room so it is quite easy to find the guy standing next to you :)

that is the big advantage to aoe, as well as being such a low level protocal
that it skips TCP/IP overhead and can give near-wire speeds. I can pull
11.7MB/s off 12.5MB/s 100Mb ethernet and 116MB/s off gigabit, which is about
as good as it gets.

keep in mind that these need to be on the same physical network, not on a
network across the country. if you need to get further than that, you need
to use iSCSI.

iSCSI is a bit more complicated, but it is not so bad either. with iSCSI
you just need to make each target have a specific name in a FQDN type
format. You can put that in DNS if you like but you can also tell an
initiator aka 'client' the IP address of the disk target so you dont have to
mess with DNS. What is nice about iSCSI is that it is routable. and it also
is using scsi packets. aoe is using ATA packets and while iscsi is using
scsi packets, scsi packets being a bit more efficient in format meaning that
though aoe can have a bit higher thoroughput, iscsi can do a bit better on
I/O.
Post by Kurt Tunkko
Hey Dan,
Post by dan
AOE is quite simple.
:-o ... wait, not THAT simple ... simple is something like: throw in
coin here -> get a beer :-)
I just think that I don't fully understand the concept of AOE and how I
can apply your setup in order to back our server to a 3rd harddrive that
is NOT in the same room like our server.
As far as I understand your idea, I would setup an other-machine that
will have one harddrive/parition configured as AOE.
BackupPC-Server -> network -> other-machine -> AOE-Disk
have I understand this right - since we have lots of spare machines
available there wouldn't be a problem setting up another one.
BUT I don't get how the backuppc-server "knows" where the AOE disk is?
sounds like magic.
If it's not to offtopic I would like to ask if you can add some more
newbie-friendly information how this is working, maybe using the names
backuppc-server (that wants to add the AOE-device to its RAID) and
aoeserver (that has the real drive built in).
thanks a lot - I'm looking forward to make some testing
- Kurt
Post by dan
[...] on ubuntu you can build a target like this
have a spare drive/partition/lvm!
sudo apt-get install vblade
vblade 0 0 eth0 /dev/sdc
thats it! now you are exporting a device on shelf 0, slot 0 (vblade 0 0)
on another machine
sudo apt-get install aoetools
modprobe aoe
you should now see /dev/etherd/e0.0 (this is the shelf 0, slot 0 drive
from the other machine)
done!
a 'client' can see any aoe device on the network, without even having an
IP address. IP is not used at all on aoe.
now on the 'client' machine, you can
"mkfs.ext3 /dev/etherd/e0.0 && mkdir /mnt/e0.0 && mount /dev/etherd/e0.0
/mnt/e0.0"
or better yet, add it to your raid array with mdadm.
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Kurt Tunkko
2008-07-23 23:57:30 UTC
Permalink
Success story :-)
Post by dan
AOE is quite simple. on ubuntu you can build a target like this
Using VMWare I was able to setup another Linux machine and got AoE
working in less than 30min!

I added the AoE-disk/partition to the RAID1 on the backuppc-server (also
a VM for testing purposes) and it was accepted like a 'normal' drive,
very nice.

Thank you for everyone mentioning and teaching me more about AoE, I hope
I can assemble everything to a nice wiki-page.

Summing up some information from the last posts I see the following
different solutions to backup the backuppc-server (I haven't look at
iSCSI yet):

1) Backup using RAID1 and AoE
Setup another machine, that is located at another place and is connected
via Gigabit LAN to the backuppc server (that means I need a 2nd NIC for
the server). The new machine is 'offering' one of it's drives via AoE -
this AoE-drive will become member of the RAID1.
Since the RAID will be spread across two machines that can be located at
different locations I get an additional security.

Question
========
What is the minimal hardware I can use to built this AoE server? Pentium
III and a Gigabit-Ethernet-Card (if this is possible on older hardware)?

As Dan suggested it should be possible to create a VM (to offer AoE) and
tie a special USB port to it that is connected to an external harddrive.
This VM can be run on another (even Windows-based) machine.
I guess its still good to add a 2nd network card to this machine (and
the BackupPC server in order to reduce traffic on LAN, especially if
you're also offering other services like samba on the server).

Question
========
Just to make sure: since AoE is not IP-based, does that mean that I can
NOT connect the machine offering AoE to my server if I have something
like a hub/switch between them?

Recovery plan:
If the backuppc-server dies, I would use the spare RAID-disk to get the
server back online - as far as I understand I need todo some manual
stuff in order to get the RAID bootable again, since the bootloader will
not installed on the spare drive?!

+ easy to implement
+ no human interaction needed
+ having a full mirrored disk of the backuppc server
- need another machine (maybe also running 24/7?) - maybe not if using a
VM, need to do further testing
- probably a 2nd network connection between these two machines to keep
the LAN free from extra traffic
- probably less perfomance because disk activity will be transfered via
network
- backups are not really offsite but at two different locations/machines

----------------------------

2) Backup using a RAID1 on the backuppc-server with a detachable spare-drive
In order to have a backup that can also be taken offsite, I would make
sure to put /var/lib/backuppc on a RAID1. If I add the 3rd disk I need
to make sure to stop BackupPC and unmount the RAID.
Then add the 3rd disk to the array and wait until the data is synced.
Then I stop BackupPC again, unmount the drive and can put it away on a
secure place.

Question
========
I guess I need more information than everything that is in
/var/lib/backuppc to restore the BackupPC server. Probably /etc/backuppc
and information the disk-layout. True?

Recovery plan:
In case I loose some of my clients AND the BackupPC server:
1. Bare metal recovery of the backupPC server with a diskimage (without
/var/lib/BackupPC since that was stored on the RAID1)
2. Recreate the RAID with the spare disk (that has been stored offsite).
3. Since BackupPC should be running now on the new backuppc server I
could restore (on the backuppc server and also the clients) everything
that has been done between the last diskimage-backup and the last sync
of the spare disk.

+ having daily/weekly offsite backups
- sounds more complicated
- need more human interaction
- backuppc has to be stopped when swapping disks
- maybe less perfomance when disks will be resynced after attaching the
spare drive


Kind regards

- Kurt
Adam Goryachev
2008-07-24 00:35:25 UTC
Permalink
Post by Kurt Tunkko
Success story :-)
Post by dan
AOE is quite simple. on ubuntu you can build a target like this
1) Backup using RAID1 and AoE
Setup another machine, that is located at another place and is connected
via Gigabit LAN to the backuppc server (that means I need a 2nd NIC for
the server). The new machine is 'offering' one of it's drives via AoE -
this AoE-drive will become member of the RAID1.
Since the RAID will be spread across two machines that can be located at
different locations I get an additional security.
As mentioned earlier, you can also backup using ndb (Network Block
Device) over a truly remote TCP/IP connection, almost the same as AoE
(except you need to specify the hostname/IP and port number to import
the drive from). This allows for the case where you want a true
"off-site" storage location without a bridged VPN.

Regards,
Adam
Tino Schwarze
2008-07-24 09:32:55 UTC
Permalink
Post by Adam Goryachev
Post by Kurt Tunkko
Success story :-)
Post by dan
AOE is quite simple. on ubuntu you can build a target like this
1) Backup using RAID1 and AoE
Setup another machine, that is located at another place and is connected
via Gigabit LAN to the backuppc server (that means I need a 2nd NIC for
the server). The new machine is 'offering' one of it's drives via AoE -
this AoE-drive will become member of the RAID1.
Since the RAID will be spread across two machines that can be located at
different locations I get an additional security.
As mentioned earlier, you can also backup using ndb (Network Block
Device) over a truly remote TCP/IP connection, almost the same as AoE
(except you need to specify the hostname/IP and port number to import
the drive from). This allows for the case where you want a true
"off-site" storage location without a bridged VPN.
drbd seems to do rather well in such cases. AFAIK, it's derived from
nbd.

Tino.
--
"What we nourish flourishes." - "Was wir nähren erblüht."

www.craniosacralzentrum.de
www.forteego.de
Kurt Tunkko
2008-07-14 22:25:27 UTC
Permalink
Hello,

following the ideas mentioned by Dan and Carl Wilhelm to add a 3rd
harddrive to my existing RAID1 and always swap one drive to backup my
backuppc-server I came up with the following solution:

*** How to use Software RAID1 to get a 3rd drive for offline storage ***
This has been tested with Ubuntu Server (in VMWare)

1. Existing RAID1 (/dev/md0) containing of two drives (/dev/sda1 and
/dev/sdb1). Check if everything is fine:

# sudo mdadm --query --detail /dev/md0

2. Connect my 3rd harddrive (dev/sdc1) to my machine and add it to the
RAID1 as spare-drive:

# sudo mdadm --add /dev/md0 /dev/sdc1

3. Mirror partition tables from one of the existing disks:

# sudo sfdisk –d /dev/sda | sfdisk /dev/sdc

4. Disable one of the the two harddrives and remove it from the RAID:

# sudo mdadm --fail /dev/md0 /dev/sdb1
# sudo mdadm --remove /dev/md0 /dev/sdb1

5. The RAID-software will start to resync the data to the attached spare
drive. You can check how far the process is:

# sudo mdadm --query --detail /dev/md0

6. The harddrive that has been removed from the array can now be locked
away to have an additional backuo in case of <INSERT-SCENARIO-HERE>


*** Further Reading ***

Most information has been taken from two locations:

1) Setting up Software RAID in Ubuntu Server
http://tinyurl.com/24x25z

2)
Chapter 6. Detecting, querying and testing
http://tinyurl.com/26ua9q


*** Improvement needed - automate steps ***

After the first steps, I would like to see the following improvements in
order to get a more userfriendly and automated offline storage backup.

1) I would like to automate the process above as soon as I attach the
3rd harddrive to my system.

2) I would like to get an email as soon as resync has been completed.

3) Will it harm my harddrives if I use hot-swapable harddrive-cases?

4) Is it possible to sync the data to the 3rd harddrive without removing
one of the others. This would be the best approach since I can just use
an external disk in this case.

With the method described above I need to make two full syncs in case I
have two not-hot-swapable-harddrives:

1. attach 3rd harddrive & add to RAID as spare
2. remove 2nd harddrive from RAID
3. resync
4. add 2nd harddrive to RAID as spare
5. remove 3rd harddrive
6. resync (even when no data has changed in between)

Hope to get some ideas how to automate everything - I know it can be
done :-)

Best regards from Berlin

- Kurt
Holger Parplies
2008-07-14 23:34:27 UTC
Permalink
Hi,
Post by Kurt Tunkko
[...]
*** How to use Software RAID1 to get a 3rd drive for offline storage ***
This has been tested with Ubuntu Server (in VMWare)
[...]
2. Connect my 3rd harddrive (dev/sdc1) to my machine and add it to the
# sudo mdadm --add /dev/md0 /dev/sdc1
# sudo sfdisk ???d /dev/sda | sfdisk /dev/sdc
apart from something having been mangled ("???"), I tend to wonder why you
need root permission to read the partition table of /dev/sda but not to write
it to /dev/sdc ;-). You might, of course, have relaxed the permissions on
/dev/sdc, but I'd spare an extra 'sudo' for the howto ;-).

Oh, and I wonder even more why you are first adding the *partition* to the
RAID array and then modifying the *partition table* defining it. If I were
the RAID subsystem, I'd revoke your root privileges for doing that ...
Post by Kurt Tunkko
# sudo mdadm --fail /dev/md0 /dev/sdb1
# sudo mdadm --remove /dev/md0 /dev/sdb1
I believe the original idea is *not* to temporarily cripple your RAID but
rather to add a third disk (three way mirror). I'm not sure if you can do that
after initial creation of the array, but the man page suggests it should be
possible "on kernels which provide necessary support". So you would have
something like

4. Add new HDD to raid:

# sudo mdadm --grow --raid-devices=3 /dev/md0
# sudo mdadm --add /dev/md0 /dev/sdc1

and --fail and --remove only /dev/sdc1, *never* /dev/sd[ab]1.

I'm afraid I don't have any spare HDDs around to test this right now ...
Post by Kurt Tunkko
[...]
3) Will it harm my harddrives if I use hot-swapable harddrive-cases?
As long as they don't get *too* hot ... why should it? Whether or not
hot-swapping actually works seems to depend on the driver for the hardware
controller (at least for SATA).

Regards,
Holger
Les Mikesell
2008-07-15 00:35:10 UTC
Permalink
Post by Holger Parplies
I believe the original idea is *not* to temporarily cripple your RAID but
rather to add a third disk (three way mirror). I'm not sure if you can do that
after initial creation of the array, but the man page suggests it should be
possible "on kernels which provide necessary support". So you would have
something like
# sudo mdadm --grow --raid-devices=3 /dev/md0
If you are starting from scratch, you can create a raid with 3 devices
and specify one as "missing". I haven't tried changing the number
after creation, but it is supposed to work.
Post by Holger Parplies
# sudo mdadm --add /dev/md0 /dev/sdc1
and --fail and --remove only /dev/sdc1, *never* /dev/sd[ab]1.
If you want the filesystem to be clean, you should momentarily stop
backuppc and unmount the array before failing and removing the extra mirror.
Post by Holger Parplies
I'm afraid I don't have any spare HDDs around to test this right now ...
Post by Kurt Tunkko
[...]
3) Will it harm my harddrives if I use hot-swapable harddrive-cases?
As long as they don't get *too* hot ... why should it? Whether or not
hot-swapping actually works seems to depend on the driver for the hardware
controller (at least for SATA).
Yes, some SATA controllers will let the kernel notice the new device and
some won't.
--
Les Mikesell
***@gmail.com
Kim Pedersen
2008-10-06 22:40:41 UTC
Permalink
Hi everyone,

I have been following the discussions about using Linux md raid1 as a
way of creating an offline copy of the pool filesystem, which method Les
Mikesell seems to be a big proponent of.

I am about to try it out myself on an installation, where users will
connect an external harddrive to the BackupPC server and sync the drive
to the raid1 array using mdadm, and later disconnect it again for
offline/offsite storage.

As this is something that will happen on a regular basis, I am looking
to automate it as much as practical (and possible) by writing a script
for the umount/fail/remove and recognition of the external drives.

Might there be someone on the list who have taken a shot at this already
and be willing to share the code to get me jump started?


Regards,

Kim Pedersen
dan
2008-10-07 05:12:25 UTC
Permalink
Your best bet here is to use udev rule for the device

whenever the drive gets plugged in it should run the script

[SCRIPT1]
add the device to the array so it can sync.

launch a script in the background that watches the sync and when it is 100%
complete, remove the disk from the array. (use while and awk probably)
eject the disk
notify you via email.

make sure you are generous with `sleep` commands. some things take a few
seconds to settle so a 30 second wait here or there is a good thing.


I will certainly give any help I can but I really suggest that you read this
on writing udev rules http://reactivated.net/writing_udev_rules.html
also, a quick while loop using awk to pull out the specific field you need
for the while. this shouldnt be terribly hard.
eject /dev/sdb is pretty straightforward

it will be very helpfull for you to learn to do this so i suggest you put as
much effort into this as you can before getting any help.

good luck.
Post by Kim Pedersen
Hi everyone,
I have been following the discussions about using Linux md raid1 as a
way of creating an offline copy of the pool filesystem, which method Les
Mikesell seems to be a big proponent of.
I am about to try it out myself on an installation, where users will
connect an external harddrive to the BackupPC server and sync the drive
to the raid1 array using mdadm, and later disconnect it again for
offline/offsite storage.
As this is something that will happen on a regular basis, I am looking
to automate it as much as practical (and possible) by writing a script
for the umount/fail/remove and recognition of the external drives.
Might there be someone on the list who have taken a shot at this already
and be willing to share the code to get me jump started?
Regards,
Kim Pedersen
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Rob Owens
2008-10-08 16:06:26 UTC
Permalink
Post by dan
Your best bet here is to use udev rule for the device
Careful! This will sync *any* drive that you plug in, won't it? Or is
there a way to have a udev rule that applies to only a drive with a
specific UUID?

-Rob
Post by dan
whenever the drive gets plugged in it should run the script
[SCRIPT1]
add the device to the array so it can sync.
launch a script in the background that watches the sync and when it is
100% complete, remove the disk from the array. (use while and awk probably)
eject the disk
notify you via email.
make sure you are generous with `sleep` commands. some things take a
few seconds to settle so a 30 second wait here or there is a good thing.
I will certainly give any help I can but I really suggest that you read
this on writing udev rules http://reactivated.net/writing_udev_rules.html
also, a quick while loop using awk to pull out the specific field you
need for the while. this shouldnt be terribly hard.
eject /dev/sdb is pretty straightforward
it will be very helpfull for you to learn to do this so i suggest you
put as much effort into this as you can before getting any help.
good luck.
Hi everyone,
I have been following the discussions about using Linux md raid1 as a
way of creating an offline copy of the pool filesystem, which method Les
Mikesell seems to be a big proponent of.
I am about to try it out myself on an installation, where users will
connect an external harddrive to the BackupPC server and sync the drive
to the raid1 array using mdadm, and later disconnect it again for
offline/offsite storage.
As this is something that will happen on a regular basis, I am looking
to automate it as much as practical (and possible) by writing a script
for the umount/fail/remove and recognition of the external drives.
Might there be someone on the list who have taken a shot at this already
and be willing to share the code to get me jump started?
Regards,
Kim Pedersen
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win
great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
<http://moblin-contest.org/redirect.php?banner_id=100&url=/>
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
------------------------------------------------------------------------
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
------------------------------------------------------------------------
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
********************************************************

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. If you are not the addressee, any disclosure, reproduction,
copying, distribution, or other dissemination or use of this transmission in
error please notify the sender immediately and then delete this e-mail.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted lost, destroyed, arrive late or
incomplete, or contain viruses.
The sender therefore does not accept liability for any errors or omissions
in the contents of this message which arise as a result of e-mail
transmission. If verification is required please request a hard copy
version.

********************************************************
Nils Breunese (Lemonbit)
2008-10-08 17:42:00 UTC
Permalink
Post by Rob Owens
Post by dan
Your best bet here is to use udev rule for the device
Careful! This will sync *any* drive that you plug in, won't it? Or is
there a way to have a udev rule that applies to only a drive with a
specific UUID?
I believe it's possible to have udev rules for specific devices, yes.

Nils Breunese.
dan
2008-10-08 18:47:06 UTC
Permalink
yes, UDEV can do an action for a device by UUID or serial number or label
etc etc.
Post by Nils Breunese (Lemonbit)
Post by Rob Owens
Post by dan
Your best bet here is to use udev rule for the device
Careful! This will sync *any* drive that you plug in, won't it? Or is
there a way to have a udev rule that applies to only a drive with a
specific UUID?
I believe it's possible to have udev rules for specific devices, yes.
Nils Breunese.
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Kim Pedersen
2008-10-10 23:48:41 UTC
Permalink
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
Thanks for the udev pointers everone - had a hectic work week with an
even more crazy one coming up, but I'll see if I can't sneak in some
time over the week for a first jab at it.<br>
<br>
Will let post back how it goes<br>
<br>
<br>
Regards,<br>
<br>
Kim Pedersen<br>
<br>
<br>
<br>
dan wrote:
<blockquote
cite="mid:***@mail.gmail.com"
type="cite">
<div dir="ltr">yes, UDEV can do an action for a device by UUID or
serial number or label etc etc.<br>
<br>
<div class="gmail_quote">On Wed, Oct 8, 2008 at 11:42 AM, Nils
Breunese (Lemonbit) <span dir="ltr">&lt;<a moz-do-not-send="true"
href="mailto:***@lemonbit.com">***@lemonbit.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">Rob Owens wrote:<br>
<br>
&gt; dan wrote:<br>
&gt;&gt; Your best bet here is to use udev rule for the device<br>
&gt;&gt;<br>
&gt;<br>
&gt; Careful! &nbsp;This will sync *any* drive that you plug in, won't it?
&nbsp;Or<br>
&gt; is<br>
&gt; there a way to have a udev rule that applies to only a drive with a<br>
&gt; specific UUID?<br>
<br>
</div>
I believe it's possible to have udev rules for specific devices, yes.<br>
<font color="#888888"><br>
Nils Breunese.<br>
</font>
<div>
<div class="Wj3C7c"><br>
-------------------------------------------------------------------------<br>
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge<br>
Build the coolest Linux based applications with Moblin SDK &amp; win
great prizes<br>
Grand prize is a trip for two to an Open Source event anywhere in the
world<br>
<a moz-do-not-send="true"
href="http://moblin-contest.org/redirect.php?banner_id=100&amp;url=/"
target="_blank">http://moblin-contest.org/redirect.php?banner_id=100&amp;url=/</a><br>
_______________________________________________<br>
BackupPC-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:BackupPC-***@lists.sourceforge.net">BackupPC-***@lists.sourceforge.net</a><br>
List: &nbsp; &nbsp;<a moz-do-not-send="true"
href="https://lists.sourceforge.net/lists/listinfo/backuppc-users"
target="_blank">https://lists.sourceforge.net/lists/listinfo/backuppc-users</a><br>
Wiki: &nbsp; &nbsp;<a moz-do-not-send="true"
href="http://backuppc.wiki.sourceforge.net" target="_blank">http://backuppc.wiki.sourceforge.net</a><br>
Project: <a moz-do-not-send="true"
href="http://backuppc.sourceforge.net/" target="_blank">http://backuppc.sourceforge.net/</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<pre wrap="">
<hr size="4" width="90%">
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK &amp; win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
<a class="moz-txt-link-freetext" href="http://moblin-contest.org/redirect.php?banner_id=100&url=/">http://moblin-contest.org/redirect.php?banner_id=100&amp;url=/</a></pre>
<pre wrap="">
<hr size="4" width="90%">
_______________________________________________
BackupPC-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:BackupPC-***@lists.sourceforge.net">BackupPC-***@lists.sourceforge.net</a>
List: <a class="moz-txt-link-freetext" href="https://lists.sourceforge.net/lists/listinfo/backuppc-users">https://lists.sourceforge.net/lists/listinfo/backuppc-users</a>
Wiki: <a class="moz-txt-link-freetext" href="http://backuppc.wiki.sourceforge.net">http://backuppc.wiki.sourceforge.net</a>
Project: <a class="moz-txt-link-freetext" href="http://backuppc.sourceforge.net/">http://backuppc.sourceforge.net/</a>
</pre>
</blockquote>
<br>
</body>
</html>

Kurt Tunkko
2008-07-15 23:05:30 UTC
Permalink
Hello Holker,
Post by Holger Parplies
# sudo sfdisk -d /dev/sda | sfdisk /dev/sdc
apart from something having been mangled ("???"), I tend to wonder why you
need root permission to read the partition table of /dev/sda but not to write
it to /dev/sdc ;-). You might, of course, have relaxed the permissions on
/dev/sdc, but I'd spare an extra 'sudo' for the howto ;-).
I took the information above from the page:

Setting up software RAID in Ubuntu Server
http://advosys.ca/viewpoints/2007/04/setting-up-software-raid-in-ubuntu-server/

" Once system as been rebooted with the new unformatted replacement
drive in place, some manual intervention is required to partition the
drive and add it to the RAID array.
The new drive must have an identical (or nearly identical) partition
table to the other. You can use fdisk to manually create a partition
table on the new drive identical to the table of the other, or if both
drives are identical you can use the “sfdisk” command to duplicate the
partition. For example, to copy the partition table from the second
drive “sdb” onto the first drive “sda”, the sfdisk command is as follows:

sfdisk –d /dev/sdb | sfdisk /dev/sda"

I don't know if it's possible to add a 3rd drive to the RAID, that
hasn't got the right partitions on it :-?
Post by Holger Parplies
I believe the original idea is *not* to temporarily cripple your RAID but
rather to add a third disk (three way mirror).
you're right I changed my setup and have now a three way raid, so that I
can unplugg one drive and keep it as offline backup.
Post by Holger Parplies
I'm not sure if you can do that after initial creation of the array,
but the man page suggests it should be possible "on kernels which
provide necessary support".
On Ubuntu Server I was able to add a third harddisk to the array and get
it synced. After sync had been completed I can remove the 3rd drive from
the array and lock it away for offline storage.

Change a 2-disk RAID1 to a 3-disk RAID:
# sudo mdadm --grow --raid-devices=3 /dev/md0

Add a 3rd drive to the existing 2disk RAID1:
# sudo mdadm --add /dev/md0 /dev/sdc1
-> spare will be rebuilded

Remove 3rd disk from RAID
# sudo mdadm --fail /dev/md0 /dev/sdb1
# sudo mdadm --remove /dev/md0 /dev/sdb1

While this approach is much better than the one I suggested yesterday it
still leads to some Questions:

1) Is there a way to add the 3rd drive to RAID1 as soon as it will be
connected to the system (External harddrive that is oonnected via usb2)?
Or more generally: Can I run a script when an external storage device is
connected via usb?

2) Do I need to resize my RAID after removing the 3rd harddrive with
# sudo mdadm --grow --raid-devices=2 /dev/md0
Are there any problems when the RAID will be used in clean, but
degraded state?

- Kurt
Chris Robertson
2008-07-15 23:58:52 UTC
Permalink
Post by Kurt Tunkko
Hello Holker,
Post by Holger Parplies
# sudo sfdisk -d /dev/sda | sfdisk /dev/sdc
apart from something having been mangled ("???"), I tend to wonder why you
need root permission to read the partition table of /dev/sda but not to write
it to /dev/sdc ;-). You might, of course, have relaxed the permissions on
/dev/sdc, but I'd spare an extra 'sudo' for the howto ;-).
Setting up software RAID in Ubuntu Server
http://advosys.ca/viewpoints/2007/04/setting-up-software-raid-in-ubuntu-server/
" Once system as been rebooted with the new unformatted replacement
drive in place, some manual intervention is required to partition the
drive and add it to the RAID array.
The new drive must have an identical (or nearly identical) partition
table to the other. You can use fdisk to manually create a partition
table on the new drive identical to the table of the other, or if both
drives are identical you can use the “sfdisk” command to duplicate the
partition. For example, to copy the partition table from the second
sfdisk –d /dev/sdb | sfdisk /dev/sda"
I don't know if it's possible to add a 3rd drive to the RAID, that
hasn't got the right partitions on it :-?
Post by Holger Parplies
I believe the original idea is *not* to temporarily cripple your RAID but
rather to add a third disk (three way mirror).
you're right I changed my setup and have now a three way raid, so that I
can unplugg one drive and keep it as offline backup.
Post by Holger Parplies
I'm not sure if you can do that after initial creation of the array,
but the man page suggests it should be possible "on kernels which
provide necessary support".
On Ubuntu Server I was able to add a third harddisk to the array and get
it synced. After sync had been completed I can remove the 3rd drive from
the array and lock it away for offline storage.
# sudo mdadm --grow --raid-devices=3 /dev/md0
# sudo mdadm --add /dev/md0 /dev/sdc1
-> spare will be rebuilded
Remove 3rd disk from RAID
# sudo mdadm --fail /dev/md0 /dev/sdb1
# sudo mdadm --remove /dev/md0 /dev/sdb1
While this approach is much better than the one I suggested yesterday it
1) Is there a way to add the 3rd drive to RAID1 as soon as it will be
connected to the system (External harddrive that is oonnected via usb2)?
Or more generally: Can I run a script when an external storage device is
connected via usb?
Yes. http://ubuntuforums.org/showthread.php?t=502864
Post by Kurt Tunkko
2) Do I need to resize my RAID after removing the 3rd harddrive with
# sudo mdadm --grow --raid-devices=2 /dev/md0
Personally, I wouldn't bother, unless it made my monitoring software
throw a false error.
Post by Kurt Tunkko
Are there any problems when the RAID will be used in clean, but
degraded state?
I doubt it. The mirror is going to be active with two drives, so writes
might even be faster.
Post by Kurt Tunkko
- Kurt
Chris
Holger Parplies
2008-07-16 01:43:15 UTC
Permalink
Hi,
Post by Kurt Tunkko
# sudo sfdisk -d /dev/sda | sfdisk /dev/sdc
[...]
Setting up software RAID in Ubuntu Server
http://advosys.ca/viewpoints/2007/04/setting-up-software-raid-in-ubuntu-server/
" [...]
For example, to copy the partition table from the second
sfdisk ???d /dev/sdb | sfdisk /dev/sda"
my point is that your pipeline consists of the two commands

sudo sfdisk -d /dev/sda
sfdisk /dev/sdc

connected by a pipe. It's not as though 'sudo' somehow magically applies to
the pipe and not only its own parameters. You need either

sudo sh -c "sfdisk -d /dev/sda | sfdisk /dev/sdc"

or rather

sudo sfdisk -d /dev/sda | sudo sfdisk /dev/sdc

Similarly, you can

echo 'backuppc hold' | sudo dpkg --set-selections

because you don't need root privileges to echo 'backuppc hold', and my
favorite for I/O-redirection with root privileges is

some-command-that-generates-output | sudo dd of=/root/output.file

Sorry, that was off-topic.
Post by Kurt Tunkko
I don't know if it's possible to add a 3rd drive to the RAID, that
hasn't got the right partitions on it :-?
You're not adding a drive to the RAID, you're adding a partition, i.e. a block
device consisting of at least N blocks, where N is the size of the smallest
partition currently in the array. You usually choose identically sized
partitions to avoid wasting space. If you've got identical disks with only a
single partition, you can probably use the disk (/dev/sd[abc]) instead of a
partition (/dev/sd[abc]1) and put the first track to good use. It might be
desirable to have the partitions of one RAID array in the same place on all
disks (identical R/W speed, presuming identical disks) or it might not. You've
got a lot of freedom.
Of course the partition needs to exist and be large enough though.
Post by Kurt Tunkko
[...]
While this approach is much better than the one I suggested yesterday it
1) Is there a way to add the 3rd drive to RAID1 as soon as it will be
connected to the system (External harddrive that is oonnected via usb2)?
Or more generally: Can I run a script when an external storage device is
connected via usb?
Yes. Are you sure you want that Windoze-type-oh-no-feeling when you plug a USB
stick with valuable data into the computer and it automatically fills it with
the first few GBs of your BackupPC pool? :) Not sure if sfdisk will allow the
partition table to exceed the device size though ...

man udev && good luck
Post by Kurt Tunkko
2) Do I need to resize my RAID after removing the 3rd harddrive with
# sudo mdadm --grow --raid-devices=2 /dev/md0
Are there any problems when the RAID will be used in clean, but
degraded state?
I haven't had any so far with one-way-mirrors. You might get an occasional
warning e-mail. Otherwise, I wouldn't expect any difference between
non-degraded two-way and degraded three-way mirrors with two operational
disks.

Finally, you might take a look at the mdadm --write-mostly option. Sounds
as if it were meant for cases like this one.

Regards,
Holger
Adam Goryachev
2008-07-16 02:00:23 UTC
Permalink
Post by Holger Parplies
Hi,
Post by Kurt Tunkko
While this approach is much better than the one I suggested yesterday it
1) Is there a way to add the 3rd drive to RAID1 as soon as it will be
connected to the system (External harddrive that is oonnected via usb2)?
Or more generally: Can I run a script when an external storage device is
connected via usb?
Yes. Are you sure you want that Windoze-type-oh-no-feeling when you plug a USB
stick with valuable data into the computer and it automatically fills it with
the first few GBs of your BackupPC pool? :) Not sure if sfdisk will allow the
partition table to exceed the device size though ...
You should be able to recognise the UUID or similar of the drive, to
ensure that it is the drive you want to deal with (or one of the
multiple drives you want to deal with).

Also, remember that you *should* stop backuppc and umount the raid1
array before you remove the 3rd drive from the array. Otherwise, the
copy you have will be in an in-consistent state, and so you may have
some random data corruption.

Oh, and maybe throw in a "sync" or two just for fun....

Regards,
Adam

- --
Adam Goryachev
Website Managers
www.websitemanagers.com.au
Holger Parplies
2008-07-18 14:52:10 UTC
Permalink
Hi,
Post by Adam Goryachev
[...]
Post by Holger Parplies
Are you sure you want that Windoze-type-oh-no-feeling when you plug a USB
stick with valuable data into the computer and it automatically fills it
with the first few GBs of your BackupPC pool? :) Not sure if sfdisk will
allow the partition table to exceed the device size though ...
You should be able to recognise the UUID or similar of the drive, to
ensure that it is the drive you want to deal with (or one of the
multiple drives you want to deal with).
yes, and I appreciate the simplicity of simply plugging in the drive and
everything happening automatically. *But* we are speaking of administrative
operations here, not a fancy end-user interface. There's always the chance
you want to plug in the (correct, recognized) drive for a different reason,
such as checking its SMART status, copying the image, accessing a previous
version of your BackupPC pool, reading the serial number or because you got
drives confused and think it's your transportable photo collection. Chances
are, you shouldn't be doing that on the BackupPC server. Chances are, you
should be aware of what will happen. But still: extensive, potentially
destructive operations happening automatically without a chance of user
interaction is something that belongs in the world of Windoze. Linux is
becoming more like that fast enough already, so I won't let this chance pass
to argument against it :-).

Of course there's also the chance of you simply plugging the drive in at an
inappropriate moment, when you wouldn't want the sync to happen. You might
notice it when logging in to call the script to start the sync, but you won't
log in just to check, if you went to the trouble of automating the sync in the
first place, will you?

Just because you *can* automate something, that doesn't necessarily mean it's
a good idea to do it. Then again, under some circumstances it may well be.

Regards,
Holger
dan
2008-07-22 21:34:46 UTC
Permalink
Definitely do a 3 device raid with 2 always on devices and a 3rd swappable.

As far as an automated script to sync the drives, I say DONT DO IT! your
best bet is to make a script that adds the drive to the mirror set, waits
for it to sync completely, and then removes it from the set. then you can
fire that off manually.

*ALTERNATIVE*
if you get an external drive with a button on it that is used to launch a
quick backup or something, you should be able to get to that button as it
would be a usb device on the system and you could write a script that
watches that device for a certain code(the one for the button being pressed)
as well as makes sure it is the only copy of the script running and have the
run on the hotplug of the drive, and killed on unplug. you could also do
the same thing with a button on a serial port or on usb or something or even
a keyboard combination on the console(Ctrl+Alt+Meta+B)
Hi,
Adam Goryachev wrote on 2008-07-16 12:00:23 +1000 [Re: [BackupPC-users]
Post by Adam Goryachev
[...]
Post by Holger Parplies
Are you sure you want that Windoze-type-oh-no-feeling when you plug a
USB
Post by Adam Goryachev
Post by Holger Parplies
stick with valuable data into the computer and it automatically fills
it
Post by Adam Goryachev
Post by Holger Parplies
with the first few GBs of your BackupPC pool? :) Not sure if sfdisk
will
Post by Adam Goryachev
Post by Holger Parplies
allow the partition table to exceed the device size though ...
You should be able to recognise the UUID or similar of the drive, to
ensure that it is the drive you want to deal with (or one of the
multiple drives you want to deal with).
yes, and I appreciate the simplicity of simply plugging in the drive and
everything happening automatically. *But* we are speaking of administrative
operations here, not a fancy end-user interface. There's always the chance
you want to plug in the (correct, recognized) drive for a different reason,
such as checking its SMART status, copying the image, accessing a previous
version of your BackupPC pool, reading the serial number or because you got
drives confused and think it's your transportable photo collection. Chances
are, you shouldn't be doing that on the BackupPC server. Chances are, you
should be aware of what will happen. But still: extensive, potentially
destructive operations happening automatically without a chance of user
interaction is something that belongs in the world of Windoze. Linux is
becoming more like that fast enough already, so I won't let this chance pass
to argument against it :-).
Of course there's also the chance of you simply plugging the drive in at an
inappropriate moment, when you wouldn't want the sync to happen. You might
notice it when logging in to call the script to start the sync, but you won't
log in just to check, if you went to the trouble of automating the sync in the
first place, will you?
Just because you *can* automate something, that doesn't necessarily mean it's
a good idea to do it. Then again, under some circumstances it may well be.
Regards,
Holger
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Les Mikesell
2008-07-23 18:25:59 UTC
Permalink
Post by Holger Parplies
Chances are, you
should be aware of what will happen. But still: extensive, potentially
destructive operations happening automatically without a chance of user
interaction is something that belongs in the world of Windoze. Linux is
becoming more like that fast enough already, so I won't let this chance pass
to argument against it :-).
But this is the backuppc list where you might not be running Linux... I
still think zfs with it's incremental send/receive operation might be
the way to go to duplicate changes in the archive filesystem, but
haven't been able to do any testing myself yet.
Post by Holger Parplies
Of course there's also the chance of you simply plugging the drive in at an
inappropriate moment, when you wouldn't want the sync to happen. You might
notice it when logging in to call the script to start the sync, but you won't
log in just to check, if you went to the trouble of automating the sync in the
first place, will you?
Just because you *can* automate something, that doesn't necessarily mean it's
a good idea to do it. Then again, under some circumstances it may well be.
Another alternative that seems like it should work is to plug the
external drive into some other machine on a fast lan, export the
partition via iscsi, then add the iscsi device to the raid set. Really
the same scenario, but you wouldn't have to walk over to the server room
or keep the server at your desk.
--
Les Mikesell
***@gmail.com
dan
2008-07-25 17:13:12 UTC
Permalink
i have tested the send and receive functionality of zfs on openbsd. the
problem is that it sends the entire fileset, block by block. this is not
going to work for remote replication within a reasonable timeframe.

rsync is about the only option. i know it does not scale well with file
count but it still does the best job of syncing up filesystems remotely.

the best solution here is to get with the rsync team and see what can be
done about the memory usage. maybe an option to write the file list to a
temp file or maybe compress the file list in memory.

I do remote rsyncs for a large fileset with millions of files. It does
work, reliably even, though i have 4GB of ram available on both computers.

I find that raid5 is quite evil for backuppc and/or rsync. though this is
already known on this list, maybe the degree of evil is not known. raid5
will more than double the write time of an rsync transfer. one might think
that this would be hidden behing the low bandwidth of a WAN connection but
it is not because these are not parrallel transactions, they are serial so
those I/O delays cause a waiting period before the next chunk is downloaded.

so, dont use raid5. do use raid1+0.

I have also tried some cluster filesystems recently and lets just put all of
those on the 'dont use' list.

I have done massive amounts of testing and trial and error and have found
that this is the best setup for my needs, and purhaps many people's needs

a raid set with raid1+0 running reiserfs on linux -or- zfs on bsd or solaris
with hardware raid. zfs is dramatically faster because it is much much
better at cacheing I/O than any filesystem I have tested. also, I find that
exporting a zfs snapshot via iscsi and mounting that on the remote machine
and doing the rsync *local* from the iscsi mount to the local backuppc share
is MUCH(maybe 20-30%) faster that just regular rsync as iscsi is much better
at handling I/O than rsync. use many spindles in raid0! pair the drives in
raid1 and get as many spindles as possible. raid0 will do a good job of
spreading I/O(raid5 does NOT). if using zfs, do NOT use compression, and do
not do a raidz or raidz2 as you will have a parity penalty though it is much
smaller than a regular raid5. if using bsd or solaris, use a hardware raid
card(a good one) and run zfs on top of that, again dont use zfs' raid
functionalty or you will get a parity penalty.
Post by Les Mikesell
Post by Holger Parplies
Chances are, you
should be aware of what will happen. But still: extensive, potentially
destructive operations happening automatically without a chance of user
interaction is something that belongs in the world of Windoze. Linux is
becoming more like that fast enough already, so I won't let this chance
pass
Post by Holger Parplies
to argument against it :-).
But this is the backuppc list where you might not be running Linux... I
still think zfs with it's incremental send/receive operation might be
the way to go to duplicate changes in the archive filesystem, but
haven't been able to do any testing myself yet.
Post by Holger Parplies
Of course there's also the chance of you simply plugging the drive in at
an
Post by Holger Parplies
inappropriate moment, when you wouldn't want the sync to happen. You
might
Post by Holger Parplies
notice it when logging in to call the script to start the sync, but you
won't
Post by Holger Parplies
log in just to check, if you went to the trouble of automating the sync
in the
Post by Holger Parplies
first place, will you?
Just because you *can* automate something, that doesn't necessarily mean
it's
Post by Holger Parplies
a good idea to do it. Then again, under some circumstances it may well
be.
Another alternative that seems like it should work is to plug the
external drive into some other machine on a fast lan, export the
partition via iscsi, then add the iscsi device to the raid set. Really
the same scenario, but you wouldn't have to walk over to the server room
or keep the server at your desk.
--
Les Mikesell
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Les Mikesell
2008-07-25 18:03:10 UTC
Permalink
Post by dan
i have tested the send and receive functionality of zfs on openbsd. the
problem is that it sends the entire fileset, block by block. this is
not going to work for remote replication within a reasonable timeframe.
You have to do that once, but I thought you could do one snapshot, send
it for the initial snapshot copy, then subsequently do other snapshots
and incremental sends to be received into the remote snapshot copy. The
question is, how efficient are those incremental sends when backuppc has
made a run that didn't add a lot of new data but made lots of new
hardlinks?
Post by dan
rsync is about the only option. i know it does not scale well with file
count but it still does the best job of syncing up filesystems remotely.
the best solution here is to get with the rsync team and see what can be
done about the memory usage. maybe an option to write the file list to
a temp file or maybe compress the file list in memory.
I do remote rsyncs for a large fileset with millions of files. It does
work, reliably even, though i have 4GB of ram available on both computers.
I haven't seen this work even locally with several hundred gigs in the
archive filesystem.
Post by dan
I have done massive amounts of testing and trial and error and have
found that this is the best setup for my needs, and purhaps many
people's needs
Did you measure the size of the zfs incremental send when done from a
snapshot where a previous snapshot had already been sent? So far I
haven't been able to boot any opensolaris based system on the boxes
where I'd like to test. I might eventually try it with freebsd.

Maybe a zfs snapshot copied to a local external drive or sent to an
external drive on another machine on the LAN, then carried offsite would
work if the incremental send is not efficient.
--
Les Mikesell
***@gmail.com
dan
2008-07-25 21:04:27 UTC
Permalink
the problem with zfs snapshots is that you will have to be very careful
about organizing the data. you are taking a snapshot of a live filesystem
but what does it depend on? and when you send it to the other machine is
that dependancy met? what if you miss sending a snapshot?

The problem here is automating the process. I have toyed with the idea of
doing a script that would do an md5 and timestamp on each file, write that
to a log and then do the same on the remote system. then compare the two
systems and use rsync on each item. It is something like rsync already does
but uses a file on the local filesystem to write the transfer list and spawn
a seperate process for each file to be transfered and do this sequentially.
you could also mix in a little in-line bzip2 or 7z compression to improve
speed on slow links and even allow a certain number of processes to run in
paralell to utilize multiple cpus

I am currently syncing over 240GB with rsync without issues. the sync takes
about 1-1.5 hours with most of that time being spent making the file list.
this does work well, you just have to be patient and avoid the temptation to
kill rsync because you think it is dead.
Post by dan
i have tested the send and receive functionality of zfs on openbsd. the
problem is that it sends the entire fileset, block by block. this is not
going to work for remote replication within a reasonable timeframe.
You have to do that once, but I thought you could do one snapshot, send it
for the initial snapshot copy, then subsequently do other snapshots and
incremental sends to be received into the remote snapshot copy. The
question is, how efficient are those incremental sends when backuppc has
made a run that didn't add a lot of new data but made lots of new
hardlinks?
rsync is about the only option. i know it does not scale well with file
Post by dan
count but it still does the best job of syncing up filesystems remotely.
the best solution here is to get with the rsync team and see what can be
done about the memory usage. maybe an option to write the file list to a
temp file or maybe compress the file list in memory.
I do remote rsyncs for a large fileset with millions of files. It does
work, reliably even, though i have 4GB of ram available on both computers.
I haven't seen this work even locally with several hundred gigs in the
archive filesystem.
I have done massive amounts of testing and trial and error and have found
Post by dan
that this is the best setup for my needs, and purhaps many people's needs
Did you measure the size of the zfs incremental send when done from a
snapshot where a previous snapshot had already been sent? So far I haven't
been able to boot any opensolaris based system on the boxes where I'd like
to test. I might eventually try it with freebsd.
Maybe a zfs snapshot copied to a local external drive or sent to an
external drive on another machine on the LAN, then carried offsite would
work if the incremental send is not efficient.
--
Les Mikesell
Les Mikesell
2008-07-25 22:21:43 UTC
Permalink
Post by dan
the problem with zfs snapshots is that you will have to be very careful
about organizing the data. you are taking a snapshot of a live
filesystem but what does it depend on?
If I understand the process right, you make a zfs snapshot first, so
what you send isn't from a live filesystem. Then you send that and at
the other end receive it into a snapshot that you could promote to the
filesystem if you wanted (but you don't have to).
Post by dan
and when you send it to the other
machine is that dependancy met? what if you miss sending a snapshot?
The only thing I'd expect to be a problem would be if you failed partway
through the send. But, you should be able to snapshot, receive to that,
and revert on a failure.
Post by dan
The problem here is automating the process.
Someone must have done this already. If it is sent over ssh, you should
know if it completes or not.
Post by dan
I have toyed with the idea
of doing a script that would do an md5 and timestamp on each file, write
that to a log and then do the same on the remote system. then compare
the two systems and use rsync on each item.
I don't think you need any of that - just snapshot and retry until the
send of that snapshot succeeds.
Post by dan
I am currently syncing over 240GB with rsync without issues. the sync
takes about 1-1.5 hours with most of that time being spent making the
file list. this does work well, you just have to be patient and avoid
the temptation to kill rsync because you think it is dead.
I've let rsync run for days and never got the first copy completed.
--
Les Mikesell
***@gmail.com
Daniel Denson
2008-07-28 13:43:33 UTC
Permalink
i think you missed the point on sending zfs snapshots. *if* the remote
filesystem is in sync with the snapshot you have then it should work,
but what if that is not the case? if the remote filesystem is not
synced with any specific snapshot, then there is no base level target to
snapshot from so the new smaller snapshot cannot be created because
common sync point is not known. you would first have to make sure that
the remote filesystem matched a snapshot on the local system and then
proceed if it does, if it does not then what? run a 100+GB complete
sync over a 1.5Mb link? there is the problem, what the remote system
falls out of sync because of a deleted snapshot or missed sync? how
will you check to see if the remote filesystem and local snapshot are
the same?

I think the intent of the zfs send is just to migrate a filesystem from
one system to another. I know that this is along those lines but it is
just different enough that i think zfs is a bit lacking for remote
syncs. I'm sure it can be done but I'm not convinced that it would be
very reliable or headache free because of a big potential for a very big
bandwidth penalty when the systems drift out of sync.

as far as rsync in concerned, i think that you need to have a ton of ram
and a fast CPU to make large fileset transfers work, which I have. I
doubt a 1GB p3 1Ghz is going to cut it.

I have another idea I will post in a new message as I think it is off of
this subject, please read on about BITTORRENT. thanks
Post by Les Mikesell
Post by dan
the problem with zfs snapshots is that you will have to be very
careful about organizing the data. you are taking a snapshot of a
live filesystem but what does it depend on?
If I understand the process right, you make a zfs snapshot first, so
what you send isn't from a live filesystem. Then you send that and at
the other end receive it into a snapshot that you could promote to the
filesystem if you wanted (but you don't have to).
Post by dan
and when you send it to the other machine is that dependancy met?
what if you miss sending a snapshot?
The only thing I'd expect to be a problem would be if you failed
partway through the send. But, you should be able to snapshot,
receive to that, and revert on a failure.
Post by dan
The problem here is automating the process.
Someone must have done this already. If it is sent over ssh, you
should know if it completes or not.
Post by dan
I have toyed with the idea of doing a script that would do an md5 and
timestamp on each file, write that to a log and then do the same on
the remote system. then compare the two systems and use rsync on
each item.
I don't think you need any of that - just snapshot and retry until the
send of that snapshot succeeds.
Post by dan
I am currently syncing over 240GB with rsync without issues. the
sync takes about 1-1.5 hours with most of that time being spent
making the file list. this does work well, you just have to be
patient and avoid the temptation to kill rsync because you think it
is dead.
I've let rsync run for days and never got the first copy completed.
Les Mikesell
2008-07-28 15:10:51 UTC
Permalink
Post by Daniel Denson
i think you missed the point on sending zfs snapshots. *if* the remote
filesystem is in sync with the snapshot you have then it should work,
but what if that is not the case?
Don't do the next one until the previous has succeeded. All you need is
a status returned for the transfer and some simple scripting for that.
Post by Daniel Denson
if the remote filesystem is not
synced with any specific snapshot, then there is no base level target to
snapshot from so the new smaller snapshot cannot be created because
common sync point is not known.
Don't discard your previous snapshots at either end until the update
succeeds.
Post by Daniel Denson
you would first have to make sure that
the remote filesystem matched a snapshot on the local system and then
proceed if it does, if it does not then what? run a 100+GB complete
sync over a 1.5Mb link?
Don't let that happen. If an incremental doesn't succeed, revert the
target and start that update over again.
Post by Daniel Denson
there is the problem, what the remote system
falls out of sync because of a deleted snapshot or missed sync? how
will you check to see if the remote filesystem and local snapshot are
the same?
If the incremental update status says it succeeds, I'd expect it to have
actually succeeded. I thought that zfs had its own internal
checksumming concepts to ensure that it is consistent. In any case, if
you felt you had to do a check you could probably do an 'rsync -n' on
the pool and each pc directory separately. You don't need to
reconstruct the hardlinks in this case, just verify that the contents
exist and match - so you don't take the hit of having to get all the
filenames loaded in one run.
Post by Daniel Denson
I think the intent of the zfs send is just to migrate a filesystem from
one system to another.
Then why would it have an incremental mode? The real question is
whether it is efficient enough in handling hardlinks to make it worth
scripting the updates to make sure the updates are applied consistently.
Post by Daniel Denson
as far as rsync in concerned, i think that you need to have a ton of ram and a fast CPU to make large fileset transfers work, which I have. I doubt a 1GB p3 1Ghz is going to cut it.
Hmmm, you probably need a 64-bit OS too, so you can use that ram in one
process.
--
Les Mikesell
***@gmail.com
Daniel Denson
2008-07-28 15:14:56 UTC
Permalink
I am just using 32bit ubuntu with 4GB(3.4GB available) and it is working
nicely.

as far as rsync in concerned, i think that you need to have a ton of
ram and a fast CPU to make large fileset transfers work, which I
have. I doubt a 1GB p3 1Ghz is going to cut it. Hmmm, you probably
need a 64-bit OS too, so you can use that ram in one process.
Continue reading on narkive:
Loading...