[BackupPC-users] Windows bare metal restore

[Chris Bennett]

Hi All,

I'm hoping to catch the eye of some clever combined BackupPC/Windows
admins that can tell me what I'm trying to do will not work, or can
help me to understand why what I'm doing isn't working... :)

In a test environment, I'm trying to perform a file-level backup of a
Windows 2003 R2 server using BackupPC, along with any additional
metadata required, in order to perform a bare metal restore in the
future.

The backup procedure I'm using is:
- Cygwin 1.7, rsync 3.0.6, OpenSSH on Windows server
- backup via rsync method from BackupPC
- dump of file acl's using fileacl and subinacl
- ntbackup of systemstate to c:\systemstate.bkf
- performing VSS snapshot and backing the logical mapped drive via
backuppc

Recovery procedure tested so far:
- new VM/hard disk
- partition / make active / NTFS formate via Win2003 Recovery Console
- copy data via Linux rescue cd to NTFS volume (from backuppc
recovered zip file)
- re-apply filesystem permissions via a second virtual machine (map
the VMWare .vmdk file to a working Win2k3 machine, then restore
acl's via fileacl or subinacl (tried both))
- run fixboot / fixmbr from Windows recovery console

Booting the system results in:
STOP:c0000218 {Registry File Failure}
The registry cannot load the hive (file):
\SystemRoot\System32\Config\DEFAULT
or its log or alternate.
It is corrupt, absent, or not writable.

I've also tested a straight cygwin/rsync between VSS snapshot, and a second
drive accessible in the source Windows machine, and applying ACL's, installing
boot loader, and then trying to boot from that second drive, with the same
results.

Is this a case of not possible? I've seen enough references to performing an
ASR restore followed by a file-level for data, and that is a possibility for my
final solution, but I'd like to konw if a purely file-level + required metadata
is possible.

I'm yet to work out how to perform a system-state restore without an already
working system too - that may be another way to make the system bootable after the above restore procedure.

I do know that a Repair Install of Win2k3 R2 does bring the system back to
life, but that seems so invasive on the file-level restore.

Any responses / advice / feedback would be much appreciated.

Regards,

Chris Bennett
cgb

[Jeffrey J. Kosowsky]

Post by Chris Bennett
Hi All,
I'm hoping to catch the eye of some clever combined BackupPC/Windows
admins that can tell me what I'm trying to do will not work, or can
help me to understand why what I'm doing isn't working... :)
In a test environment, I'm trying to perform a file-level backup of a
Windows 2003 R2 server using BackupPC, along with any additional
metadata required, in order to perform a bare metal restore in the
future.
- Cygwin 1.7, rsync 3.0.6, OpenSSH on Windows server
- backup via rsync method from BackupPC
- dump of file acl's using fileacl and subinacl
- ntbackup of systemstate to c:\systemstate.bkf
- performing VSS snapshot and backing the logical mapped drive via
backuppc
- new VM/hard disk
- partition / make active / NTFS formate via Win2003 Recovery Console
- copy data via Linux rescue cd to NTFS volume (from backuppc
recovered zip file)
- re-apply filesystem permissions via a second virtual machine (map
the VMWare .vmdk file to a working Win2k3 machine, then restore
acl's via fileacl or subinacl (tried both))
- run fixboot / fixmbr from Windows recovery console
STOP:c0000218 {Registry File Failure}
\SystemRoot\System32\Config\DEFAULT
or its log or alternate.
It is corrupt, absent, or not writable.
I've also tested a straight cygwin/rsync between VSS snapshot, and a second
drive accessible in the source Windows machine, and applying ACL's, installing
boot loader, and then trying to boot from that second drive, with the same
results.
Is this a case of not possible? I've seen enough references to performing an
ASR restore followed by a file-level for data, and that is a possibility for my
final solution, but I'd like to konw if a purely file-level + required metadata
is possible.
I'm yet to work out how to perform a system-state restore without an already
working system too - that may be another way to make the system bootable after the above restore procedure.
I do know that a Repair Install of Win2k3 R2 does bring the system back to
life, but that seems so invasive on the file-level restore.
Any responses / advice / feedback would be much appreciated.
Regards,
Chris Bennett
cgb

I'm very interested in this topic since I have been spending a lot of
time developing my "shadowmountrsync" routines which attempt to
automate the backup part of your approach outlined above. So far I
have robustly implemented startup of shadows and backup of acls (using
subinacl and/or getfacl).

Couple of questions:
1. What is 'fileacl' and how is it better/worse than subinacl?
(which seems to backup all the ACL info (vs. the limited getfacl
capabilities)

2. What does the "ntbackup of systemstate" add?
If I understand that better, I would be happy to add it as an
additional option to shadowmountrsync.

Please keep me informed on your progress on getting bare-metal
restores to work. I am very interested in getting this all working.

[Chris Bennett]

Hi Jeff,

Post by Jeffrey J. Kosowsky
I'm very interested in this topic since I have been spending a lot of
time developing my "shadowmountrsync" routines which attempt to
automate the backup part of your approach outlined above. So far I
have robustly implemented startup of shadows and backup of acls (using
subinacl and/or getfacl).

Great to hear from you - I was going to mention your script in my
first post, as it's a fantastic contribution to the problem I'm trying
to solve. The funny thing is I spent the better part of a weekend
working out how it all works _before_ I found your script, but what
you've put together has been a good reference for me nonetheless.

I'm primarily trying to do this with rsync over ssh, so the details
are a little different, but the 'at' job schedule to work around the
passphrase ssh login problem was a very clever idea (that I had not
found elsewhere).

Post by Jeffrey J. Kosowsky
1. What is 'fileacl' and how is it better/worse than subinacl?
(which seems to backup all the ACL info (vs. the limited getfacl
capabilities)

Good question - it's a tool I found in 2007 when I last had a look at
doing more then file-level backups with backuppc.

http://www.gbordier.com/gbtools/fileacl.asp

I had not heard/seen of subinacl or getfacl until I looked at your
work (hint: I'm not a Windows admin.. :) )

I'm willing to use whatever works, and fileacl claims save/restore
acl's comprehensively.

I've tried both fileacl and subinacl with the same results.

I'm wondering what isn't restored by a file-level restore, permission
restore, and then a fixboot / fixmbr. Or whether it's something to do
with the change of 'hard disks' (in my case VMWare vmdk files). Or
whether the on-disk block location of some files is stored somewhere
and that breaks on the restore. Coming from a Linux background, I
don't know why it has to be so hard to understand how to perform a
full system recovery without forking out $$.. :)

Post by Jeffrey J. Kosowsky
2. What does the "ntbackup of systemstate" add?
If I understand that better, I would be happy to add it as an
additional option to shadowmountrsync.

Well, I don't know the details as it's all a little abstract, but
reading:

http://technet.microsoft.com/en-us/library/cc785306(WS.10).aspx

gives some hints on what it covers, and I'm just used to seeing it be
done with other backup tools. For some of it's components backed up
(e.g. AD), I think additional massaging is done on backup, and when
restored, additional unrolling/replaying of files onto the system
instead of just file-level recovery. But I could be way off there.

Post by Jeffrey J. Kosowsky
Please keep me informed on your progress on getting bare-metal
restores to work. I am very interested in getting this all working.

My major aim is to do a backup/restore out of band without recovering
a base-os installed. Barring that, I want to work out what the
minimum amount of work is required before I can restore from my
backuppc files / permission dump and have the system functioning as it
was at the time of backup.

I hope someone out there with the right Windows knowledge can provide
some additional answers to the questions we are asking.

Regards,

Chris Bennett
cgb

[Jeffrey J. Kosowsky]

Post by Chris Bennett
Hi Jeff,

Post by Jeffrey J. Kosowsky
I'm very interested in this topic since I have been spending a lot of
time developing my "shadowmountrsync" routines which attempt to
automate the backup part of your approach outlined above. So far I
have robustly implemented startup of shadows and backup of acls (using
subinacl and/or getfacl).

Great to hear from you - I was going to mention your script in my
first post, as it's a fantastic contribution to the problem I'm trying
to solve. The funny thing is I spent the better part of a weekend
working out how it all works _before_ I found your script, but what
you've put together has been a good reference for me nonetheless.

Thanks for the kind words...
And I too am a newbie on Windows -- and frustrated by the complexity
of what is required to get a full metal restore. Between the
complexity of ntfs (with so many extra non-POSIX features) and the
many special Windows files (like the registry hives), it is so much
harder than Linux where you can do something as easy as 'cp -a /' or
'rsync' or 'tar' or any of dozens of other ways to get a full backup.
Even worse, I haven't seen any good documentation of what needs to be
done to back up an NTFS share and/or what needs to be done to properly
back up the special Windows files that lie top of it.

As an aside, note that in addition to the things you mention, there
are elements of the NTFS filesystem itself (even before you get to the
Windows OS) that are not properly copied with 'rsync' (and hence also
not with BackupPC). This includes things like reparse points (of which
there are several varieties that differ across OS versions) which are
only partially implemented in cygwin and may not be backed up and/or
restored properly. Secondly, there are non-POSIX elements like NTFS
alternate data streams (which are pretty cool but not consistent with
POSIX so intentionally not supported by cygwin -- I had a thread over
on cygwin on this).

So, the bottom line is that a 100% reliable under all possible
circumstances bare-metal backup using *nix-like tools may be almost
impossible. That being said much of the above weirdness is not used if
at all on standard Windows installations (for example junctions (a
type of reparse point) were not really used prior to Vista and the
version used in Vista is reportedly supported at least to some degree
in cygwin). Alternate data streams are not even supported by Windows
explorer and are not used in any common applications (though they are
a great friend to rootkit writers since they can hide streams in files
that look zero length).

Post by Chris Bennett

Post by Jeffrey J. Kosowsky
1. What is 'fileacl' and how is it better/worse than subinacl?
(which seems to backup all the ACL info (vs. the limited getfacl
capabilities)

Good question - it's a tool I found in 2007 when I last had a look at
doing more then file-level backups with backuppc.
http://www.gbordier.com/gbtools/fileacl.asp
I had not heard/seen of subinacl or getfacl until I looked at your
work (hint: I'm not a Windows admin.. :) )
I'm willing to use whatever works, and fileacl claims save/restore
acl's comprehensively.
I've tried both fileacl and subinacl with the same results.

I too only heard about 'subinacl' after first implementing 'getfacl'
and then finding access weirdness that I couldn't address with
'getfacl' which led to me realize that there are other attributes out
there not covered by 'getfacl' -- I had several posts on this on the
cygwin list. getfacl only covers the POSIX acl's. This deficiency led
me to 'subinacl'

I would recommend 'subinacl' since it is supported by Microsoft and
seems to get out all the ACL details (and also has some nice options
to make batch changes for things like moving systems, changing
machine/domain names, and changing SIDs).

One thing to be careful about is that 'subinacl' like vshadow and
dosdev can't be run directly from SSH (where USERNAME=SYSTEM). The
problem is insidious in that 'subinacl' seems to run fine without
error but all file ownership is replaced by:

/owner =nt authority\system

Interestingly, 'getfacl' was hacked by the cygwin folks to get around
that limitation so it works even directly from SSH though as I noted
above it misses the non-POSIX acl details.

This requires a slight change from the posted version of my
'shadowmountrsync' script to make sure that 'subinacl' is run after
the 'at' recursion. I hacked the change on my personal copy but have
not had the time to do it right on the posted script (and there is a
subtlety for the case where you ssh but don't want to use shadows)

Post by Chris Bennett
I'm wondering what isn't restored by a file-level restore, permission
restore, and then a fixboot / fixmbr. Or whether it's something to do
with the change of 'hard disks' (in my case VMWare vmdk files).

I seem to recall that changing disks requires some fiddling.

Post by Chris Bennett
Or
whether the on-disk block location of some files is stored somewhere
and that breaks on the restore. Coming from a Linux background, I
don't know why it has to be so hard to understand how to perform a
full system recovery without forking out $$.. :)

Me too -- it's amazing how hard it seems to be just to understand what
needs to be done for a full backup let alone implement and test
it. And in my opinion such complexity and obscurity just makes
mistakes and bugs more likely even if you do finally implement it all.

Post by Chris Bennett

Post by Jeffrey J. Kosowsky
2. What does the "ntbackup of systemstate" add?
If I understand that better, I would be happy to add it as an
additional option to shadowmountrsync.

Well, I don't know the details as it's all a little abstract, but
http://technet.microsoft.com/en-us/library/cc785306(WS.10).aspx
gives some hints on what it covers, and I'm just used to seeing it be
done with other backup tools. For some of it's components backed up
(e.g. AD), I think additional massaging is done on backup, and when
restored, additional unrolling/replaying of files onto the system
instead of just file-level recovery. But I could be way off there.

I read about ntbackup of systemstate a while back. But I seem to
recall that one of the key things it backs up are the registry hives
which I do with shadow copy. So I wonder what the 'systemstate' adds
to a full shadow copy (plus full subinacl acl backup). Specifically,
the page you reference says:
Windows XP Professional: The System State data includes only the
registry, COM+ Class Registration database, files under Windows
File Protection, and boot files."
Note that the COM+ Class Registration database is stored under:
C:\Windows\Registration
which seems to be backed up correctly under my shadow script.
So, I'm not sure what if anything it adds (at least in the non-Domain
controller non-Cluster setting).

Post by Chris Bennett

Post by Jeffrey J. Kosowsky
Please keep me informed on your progress on getting bare-metal
restores to work. I am very interested in getting this all working.

My major aim is to do a backup/restore out of band without recovering
a base-os installed. Barring that, I want to work out what the
minimum amount of work is required before I can restore from my
backuppc files / permission dump and have the system functioning as it
was at the time of backup.

That would be my goal too.

Post by Chris Bennett
I hope someone out there with the right Windows knowledge can provide
some additional answers to the questions we are asking.

I hope so but I wouldn't hold my breath.

One alternative that I have been thinking about (but have neither the
time nor knowledge to do) would be to adapt 'ntfsclone' with the
'--metadata' and '--save-image' flags. The first flag just backs up
the NTFS metada but not the file data and the second flag helps save
image space. According to the manpage when compressed, the metada is
typically 1-8MB. Customarily, it is used for forensics, but I wonder
whether it could be combined with the raw file data from BackupPC to
yield a complete byte-level accurate backup.

One problem with ntfsclone is that as written, you need to run it from
Linux and mount the drive... but conceptually the idea of having
another program to back up the metada byte-by-byte while using the
pooling of BackupPC to save the file data sounds like the ideal. In
particular, accurate cloning of the metadata would make sure that you
captured all the subtleties of the filesystem while pooling maximizes
the efficiency of the data backup.

[Leen Besselink]

Hi,

(only just now discovered this thread)

Post by Jeffrey J. Kosowsky
I read about ntbackup of systemstate a while back. But I seem to
recall that one of the key things it backs up are the registry hives
which I do with shadow copy. So I wonder what the 'systemstate' adds
to a full shadow copy (plus full subinacl acl backup). Specifically,
Windows XP Professional: The System State data includes only the
registry, COM+ Class Registration database, files under Windows
File Protection, and boot files."
C:\Windows\Registration
which seems to be backed up correctly under my shadow script.
So, I'm not sure what if anything it adds (at least in the non-Domain
controller non-Cluster setting).

I'm someone that completely agrees on the subject of Windows and backup and
the complexity of it. It's one of the reasons I would never setup
Windows as a server
(if I think prevent it).

I'll tell you what I think about how the system state and some other
things in
Windows work.

The registry (among other things) holds information about a number of
things:
- driveletter and partitions/filesystems mapping information
- information about drivers to load at startup

Then somewhere else (I forgot right now) it also keeps information about
machine-id,
domain (member ?)-id and thus also the machine-id/localusername-mapping.

To be honest, I don't really know how these are mapped to NTFS (in
unix/linux it's
id's in the filesystem, name/id mappings in passwd, etc.).

Also the filesystem has an ID as wel, just like you have 'blkid' in Linux.

So to boot a windows-system you need to have a few things setup right:
- information about hardware drivers, especially the HD-controller and
maybe some other motherboard chips
- the driver needs to be installed
- the right drive-mapping, which has several methods, which possible
even changed
over the years (from NT4 to Windows 2008R2), it used to be the
partitions. But it
may now also use some filesystem id (as you can add a 'signature' to the
filesystem).
I'm not sure.

The idea about system-state-backup I think was, you create such a
backup, then
on an other machine to a new installation and then do a of the restore
system-state-backup.

The new installation would create the proper information in the registry
about
hardware-drivers, drive-letter mappings and filesystem-ids, etc. And
then you restore
the system-state-backup and get all the other registry settings back.

Maybe you also first need to install all the applications on the system
so their
settings also get overwritten ? I don't know. Maybe you install the
after the steps
above.

'fun'


A few months ago I was trying to create a rsync with VSS-capabilities
( http://consolejunky.net/cwrsync-vss/ ) build in, I however have been
really busy
with other things. Not even took the time to update the README.

My lastest binary did work on all the windows-versions I could easily
support:
- does NOT work: windows 2000 would not be supported, it doesn't have VSS
- does work: windows XP has an old version of VSS
- does work: all 32-bit versions of windows vista/2003 and newer, I skip
the 'junction-points'
- does NOT work: all 64-bit versions of windows, I didn't think I could
create a usable
build-enviroment, but I recently found ming64 might be the answer (so
building on Linux).

I don't know if that's useful to you folks.

What I did last time with a windows machine with a bad harddisk (but
still kind of running,
including windows), I plug in the new harddisk on the same machine
create the new
partitions and filesystems and xcopy with acl information on the new
disk and open the
registry on the new harddisk and load the hive in a registry editor and
change the drive
mapping so it can find the right C:, etc.

Plug the new disk on the right cable and run:
http://www.sysint.no/nedlasting/mbrfix.htm

I hope this adds something useful to the discussion.

Have a nice weekend,
Leen.

[Jeffrey J. Kosowsky]

Post by Chris Bennett
Hi Jeff,

Post by Jeffrey J. Kosowsky
I'm very interested in this topic since I have been spending a lot of
time developing my "shadowmountrsync" routines which attempt to
automate the backup part of your approach outlined above. So far I
have robustly implemented startup of shadows and backup of acls (using
subinacl and/or getfacl).

Great to hear from you - I was going to mention your script in my
first post, as it's a fantastic contribution to the problem I'm trying
to solve. The funny thing is I spent the better part of a weekend
working out how it all works _before_ I found your script, but what
you've put together has been a good reference for me nonetheless.

Thanks for the kind words...
And I too am a newbie on Windows -- and frustrated by the complexity
of what is required to get a full metal restore. Between the
complexity of ntfs (with so many extra non-POSIX features) and the
many special Windows files (like the registry hives), it is so much
harder than Linux where you can do something as easy as 'cp -a /' or
'rsync' or 'tar' or any of dozens of other ways to get a full backup.
Even worse, I haven't seen any good documentation of what needs to be
done to back up an NTFS share and/or what needs to be done to properly
back up the special Windows files that lie top of it.

As an aside, note that in addition to the things you mention, there
are elements of the NTFS filesystem itself (even before you get to the
Windows OS) that are not properly copied with 'rsync' (and hence also
not with BackupPC). This includes things like reparse points (of which
there are several varieties that differ across OS versions) which are
only partially implemented in cygwin and may not be backed up and/or
restored properly. Secondly, there are non-POSIX elements like NTFS
alternate data streams (which are pretty cool but not consistent with
POSIX so intentionally not supported by cygwin -- I had a thread over
on cygwin on this).

So, the bottom line is that a 100% reliable under all possible
circumstances bare-metal backup using *nix-like tools may be almost
impossible. That being said much of the above weirdness is not used if
at all on standard Windows installations (for example junctions (a
type of reparse point) were not really used prior to Vista and the
version used in Vista is reportedly supported at least to some degree
in cygwin). Alternate data streams are not even supported by Windows
explorer and are not used in any common applications (though they are
a great friend to rootkit writers since they can hide streams in files
that look zero length).

Post by Chris Bennett

Post by Jeffrey J. Kosowsky
1. What is 'fileacl' and how is it better/worse than subinacl?
(which seems to backup all the ACL info (vs. the limited getfacl
capabilities)

Good question - it's a tool I found in 2007 when I last had a look at
doing more then file-level backups with backuppc.
http://www.gbordier.com/gbtools/fileacl.asp
I had not heard/seen of subinacl or getfacl until I looked at your
work (hint: I'm not a Windows admin.. :) )
I'm willing to use whatever works, and fileacl claims save/restore
acl's comprehensively.
I've tried both fileacl and subinacl with the same results.

I too only heard about 'subinacl' after first implementing 'getfacl'
and then finding access weirdness that I couldn't address with
'getfacl' which led to me realize that there are other attributes out
there not covered by 'getfacl' -- I had several posts on this on the
cygwin list. getfacl only covers the POSIX acl's. This deficiency led
me to 'subinacl'

I would recommend 'subinacl' since it is supported by Microsoft and
seems to get out all the ACL details (and also has some nice options
to make batch changes for things like moving systems, changing
machine/domain names, and changing SIDs).

One thing to be careful about is that 'subinacl' like vshadow and
dosdev can't be run directly from SSH (where USERNAME=SYSTEM). The
problem is insidious in that 'subinacl' seems to run fine without
error but all file ownership is replaced by:

/owner =nt authority\system

Interestingly, 'getfacl' was hacked by the cygwin folks to get around
that limitation so it works even directly from SSH though as I noted
above it misses the non-POSIX acl details.

This requires a slight change from the posted version of my
'shadowmountrsync' script to make sure that 'subinacl' is run after
the 'at' recursion. I hacked the change on my personal copy but have
not had the time to do it right on the posted script (and there is a
subtlety for the case where you ssh but don't want to use shadows)

Post by Chris Bennett
I'm wondering what isn't restored by a file-level restore, permission
restore, and then a fixboot / fixmbr. Or whether it's something to do
with the change of 'hard disks' (in my case VMWare vmdk files).

I seem to recall that changing disks requires some fiddling.

Post by Chris Bennett
Or
whether the on-disk block location of some files is stored somewhere
and that breaks on the restore. Coming from a Linux background, I
don't know why it has to be so hard to understand how to perform a
full system recovery without forking out $$.. :)

Me too -- it's amazing how hard it seems to be just to understand what
needs to be done for a full backup let alone implement and test
it. And in my opinion such complexity and obscurity just makes
mistakes and bugs more likely even if you do finally implement it all.

Post by Chris Bennett

Post by Jeffrey J. Kosowsky
2. What does the "ntbackup of systemstate" add?
If I understand that better, I would be happy to add it as an
additional option to shadowmountrsync.

Well, I don't know the details as it's all a little abstract, but
http://technet.microsoft.com/en-us/library/cc785306(WS.10).aspx
gives some hints on what it covers, and I'm just used to seeing it be
done with other backup tools. For some of it's components backed up
(e.g. AD), I think additional massaging is done on backup, and when
restored, additional unrolling/replaying of files onto the system
instead of just file-level recovery. But I could be way off there.

I read about ntbackup of systemstate a while back. But I seem to
recall that one of the key things it backs up are the registry hives
which I do with shadow copy. So I wonder what the 'systemstate' adds
to a full shadow copy (plus full subinacl acl backup). Specifically,
the page you reference says:
Windows XP Professional: The System State data includes only the
registry, COM+ Class Registration database, files under Windows
File Protection, and boot files."
Note that the COM+ Class Registration database is stored under:
C:\Windows\Registration
which seems to be backed up correctly under my shadow script.
So, I'm not sure what if anything it adds (at least in the non-Domain
controller non-Cluster setting).

Post by Chris Bennett

Post by Jeffrey J. Kosowsky
Please keep me informed on your progress on getting bare-metal
restores to work. I am very interested in getting this all working.

My major aim is to do a backup/restore out of band without recovering
a base-os installed. Barring that, I want to work out what the
minimum amount of work is required before I can restore from my
backuppc files / permission dump and have the system functioning as it
was at the time of backup.

That would be my goal too.

Post by Chris Bennett
I hope someone out there with the right Windows knowledge can provide
some additional answers to the questions we are asking.

I hope so but I wouldn't hold my breath.

One alternative that I have been thinking about (but have neither the
time nor knowledge to do) would be to adapt 'ntfsclone' with the
'--metadata' and '--save-image' flags. The first flag just backs up
the NTFS metada but not the file data and the second flag helps save
image space. According to the manpage when compressed, the metada is
typically 1-8MB. Customarily, it is used for forensics, but I wonder
whether it could be combined with the raw file data from BackupPC to
yield a complete byte-level accurate backup.

One problem with ntfsclone is that as written, you need to run it from
Linux and mount the drive... but conceptually the idea of having
another program to back up the metada byte-by-byte while using the
pooling of BackupPC to save the file data sounds like the ideal. In
particular, accurate cloning of the metadata would make sure that you
captured all the subtleties of the filesystem while pooling maximizes
the efficiency of the data backup.

[Les Mikesell]

Post by Jeffrey J. Kosowsky
One problem with ntfsclone is that as written, you need to run it from
Linux and mount the drive... but conceptually the idea of having
another program to back up the metada byte-by-byte while using the
pooling of BackupPC to save the file data sounds like the ideal. In
particular, accurate cloning of the metadata would make sure that you
captured all the subtleties of the filesystem while pooling maximizes
the efficiency of the data backup.

Maybe someone could come up with the minimal system that you'd have to
clone in a clonezilla image to get the magic filesystem stuff right,
then modify the clonezilla restore script to drop that in, then pull the
latest backup from backuppc via ssh and restore on top of it. Likewise
for linux it would be nice to have backuppc store enough information
about the filesystem that clonezilla (or a similar live CD) could
reproduce the formatting and then restore the files.

[Adam Goryachev]

Post by Les Mikesell

Post by Jeffrey J. Kosowsky
One problem with ntfsclone is that as written, you need to run it from
Linux and mount the drive... but conceptually the idea of having
another program to back up the metada byte-by-byte while using the
pooling of BackupPC to save the file data sounds like the ideal. In
particular, accurate cloning of the metadata would make sure that you
captured all the subtleties of the filesystem while pooling maximizes
the efficiency of the data backup.

Maybe someone could come up with the minimal system that you'd have to
clone in a clonezilla image to get the magic filesystem stuff right,
then modify the clonezilla restore script to drop that in, then pull the
latest backup from backuppc via ssh and restore on top of it. Likewise
for linux it would be nice to have backuppc store enough information
about the filesystem that clonezilla (or a similar live CD) could
reproduce the formatting and then restore the files.

Have been meaning to jump in on this conversation, but anyway, better
late than never...

In one instance, I've done the following:
Using the version of rsync which automatically uses vss to backup open
files (I'm not sure the exact details, but they have been posted to the
list previously), I do daily backups using rsyncd to backuppc.

On the actual backuppc system, there is an additional HDD which is not
used for linux/backuppc/etc. The hardware of this machine is identical
to the main windows server being backed up above.

At a single point in time, we did a installation of windows on this
extra HDD on the backuppc machine. I think the install was stopped at
the second windows reboot (ie, I don't think we actually
finished/completed the install).

Using partimage I took an image of this partition and saved it on the
backuppc drive.

The windows server has two partitions (cdrive and ddrive rsyncd shares).

Each night, we run a script which does the following (on the backuppc
server, the live server is only accessed by backuppc - rsyncd for backups):
1) Make sure the windows 'cdrive' is not mounted
2) restore the partimage file to the partition
3) mount the partition in linux
4) backuppc restore the most recent backup to the cdrive using
BackupPC_tarCreate .... | tar -xf -
5) umount the 'cdrive'
6) Check the 'ddrive' is not mounted
7) mkntfs the 'ddrive' partition
8) mount 'ddrive'
9) backuppc restore the most recent backup to the ddrive using
BackupPC_tarCreate .... | tar -xf -
10) umount the ddrive

In our testing so far, everything seems to work perfectly, except we
need to "re-activate" MS Windows within 3 days of booting the machine.
Generally we just do our testing without re-activating windows, and also
we don't want to re-activate too many times or it may not work when we
need it to.

I realise this doesn't save all the permissions/ACL's/etc, but it
produces a fully functional machine complete with MS Exchange server.
(We also do a MS Exchange backup which is backed up into backuppc in
case the exchange DB is corrupted or was not in a reliable state when we
happened to backup).

From the sounds of it, adding the subinacl backup should be useful in my
situation to preserve the additional information, though it doesn't seem
to be important to get the system up and running again and to preserve
the data.

Please see the script attached, and let me know if you have any
suggestions for improvement.

Regards,
Adam

[Chris Bennett]

Post by Adam Goryachev
Have been meaning to jump in on this conversation, but anyway, better
late than never...

Definitely, given the additional information you've contributed.. :)

Post by Adam Goryachev
Each night, we run a script which does the following (on the backuppc
1) Make sure the windows 'cdrive' is not mounted
2) restore the partimage file to the partition
3) mount the partition in linux
4) backuppc restore the most recent backup to the cdrive using
BackupPC_tarCreate .... | tar -xf -
5) umount the 'cdrive'
6) Check the 'ddrive' is not mounted
7) mkntfs the 'ddrive' partition
8) mount 'ddrive'
9) backuppc restore the most recent backup to the ddrive using
BackupPC_tarCreate .... | tar -xf -
10) umount the ddrive
In our testing so far, everything seems to work perfectly, except we
need to "re-activate" MS Windows within 3 days of booting the machine.
Generally we just do our testing without re-activating windows, and also
we don't want to re-activate too many times or it may not work when we
need it to.

That's great to know that _something_ works. I'm actually surprised
ACL's were not the biggest of your problems. As Les suggested, and
you've confirmed, this looks like a good grounding to connect the
dots.

Does your cdrive backup exclude any particular files (other than
'normal' excludes like pagefile.sys etc)?

I'm wondering if the reason your method works with a file level
restore over a fully functioning mininmal install, is some kind of
dependancy on block-level location for some important data? And if
that is the case, no amount of fiddling out-of-band will work unless
you preserve block location for whatever those important files are..

e.g. why doesn't a new NTFS file system, file restore, fixboot/fixmbr
"just work"? What else is missing?

Post by Adam Goryachev
Please see the script attached, and let me know if you have any
suggestions for improvement.

I'll have a play and see if I can replicate your results, as well as
try and tie in permissions. I suspect Jeff will be similarly
interested in having a look at this.

Thanks again for sharing :)

Regards,

Chris Bennett
cgb

[Adam Goryachev]

Post by Chris Bennett

Post by Adam Goryachev
Each night, we run a script which does the following (on the backuppc
1) Make sure the windows 'cdrive' is not mounted
2) restore the partimage file to the partition
3) mount the partition in linux
4) backuppc restore the most recent backup to the cdrive using
BackupPC_tarCreate .... | tar -xf -
5) umount the 'cdrive'
6) Check the 'ddrive' is not mounted
7) mkntfs the 'ddrive' partition
8) mount 'ddrive'
9) backuppc restore the most recent backup to the ddrive using
BackupPC_tarCreate .... | tar -xf -
10) umount the ddrive
In our testing so far, everything seems to work perfectly, except we
need to "re-activate" MS Windows within 3 days of booting the machine.
Generally we just do our testing without re-activating windows, and also
we don't want to re-activate too many times or it may not work when we
need it to.

That's great to know that _something_ works. I'm actually surprised
ACL's were not the biggest of your problems. As Les suggested, and
you've confirmed, this looks like a good grounding to connect the
dots.
Does your cdrive backup exclude any particular files (other than
'normal' excludes like pagefile.sys etc)?

I don't exclude *any* files at all. The only errors I see from the
backup are (about 70 per night):
Remote[1]: rsync: send_files failed to open "System Volume
Information/{d8d52fa7-cfa7-11de-bd0f-000fb5d47709}{3808876b-c176-4e48-b7ae-04046e6cc752}"
(in ddrive): Device or resource busy (16)

As mentioned though, I *AM* using the VSS version of rsyncd:
Remote[2]: Using VSS volume:
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy97

Post by Chris Bennett
I'm wondering if the reason your method works with a file level
restore over a fully functioning mininmal install, is some kind of
dependancy on block-level location for some important data? And if
that is the case, no amount of fiddling out-of-band will work unless
you preserve block location for whatever those important files are..
e.g. why doesn't a new NTFS file system, file restore, fixboot/fixmbr
"just work"? What else is missing?

Yep, which is what I initially tried with no success... I suspect
something very early in the boot process is causing the problem. One
thing I thought of/considered is what would happen if we used grub as
the boot loader.... Never tried it though, as this network/setup is
about 1.5 hours away, so I never go on site...

Post by Chris Bennett

Post by Adam Goryachev
Please see the script attached, and let me know if you have any
suggestions for improvement.

I'll have a play and see if I can replicate your results, as well as
try and tie in permissions. I suspect Jeff will be similarly
interested in having a look at this.

Sounds good, I'd be happier to have someone else able to replicate my
work, and if you sort out the permissions part it would also be nice to
incorporate that also.

Regards,
Adam

- --
Adam Goryachev
Website Managers
www.websitemanagers.com.au

[Chris Bennett]

Post by Adam Goryachev
Sounds good, I'd be happier to have someone else able to replicate my
work, and if you sort out the permissions part it would also be nice to
incorporate that also.

Excellent - thanks for the additional information. I'll communicate
back any test results and perhaps we'll be closer to restoring Windows
systems as easily as a Linux system :)

Regards,

Chris Bennett
cgb

[Jeffrey J. Kosowsky]

Post by Les Mikesell

Post by Jeffrey J. Kosowsky
One problem with ntfsclone is that as written, you need to run it from
Linux and mount the drive... but conceptually the idea of having
another program to back up the metada byte-by-byte while using the
pooling of BackupPC to save the file data sounds like the ideal. In
particular, accurate cloning of the metadata would make sure that you
captured all the subtleties of the filesystem while pooling maximizes
the efficiency of the data backup.

Maybe someone could come up with the minimal system that you'd have to
clone in a clonezilla image to get the magic filesystem stuff right,
then modify the clonezilla restore script to drop that in, then pull the
latest backup from backuppc via ssh and restore on top of it. Likewise
for linux it would be nice to have backuppc store enough information
about the filesystem that clonezilla (or a similar live CD) could
reproduce the formatting and then restore the files.

Agreed 100%. That's exactly what I was so roughly trying to outline in
my full post. There would also be the additional (and potentially not
trivial) work to get clonezilla to run on an active windows
system. I'm not sure if it can read all the NTFS metada from a
shadow copy but there has to be some way to do it since Norton Ghost
is able to clone a live system using some type of shadow copy.

I wish we had more Windows power users in this group, since I would
(and have been) willing to commit effort to this project but I am
anything but a filesystem or Windows guru.

[Chris Bennett]

There would also be the additional (and potentially not trivial)
work to get clonezilla to run on an active windows system. I'm not
sure if it can read all the NTFS metada from a shadow copy but there
has to be some way to do it since Norton Ghost is able to clone a
live system using some type of shadow copy.

You may have realised this already, but the previous suggestion was to
just have a minimal clone of an equivalent base Windows system - it
doesn't have to be a clone of the server you want to restore.

Regards,

Chris Bennett
cgb

[Les Mikesell]

Post by Chris Bennett

There would also be the additional (and potentially not trivial)
work to get clonezilla to run on an active windows system. I'm not
sure if it can read all the NTFS metada from a shadow copy but there
has to be some way to do it since Norton Ghost is able to clone a
live system using some type of shadow copy.

You may have realised this already, but the previous suggestion was to
just have a minimal clone of an equivalent base Windows system - it
doesn't have to be a clone of the server you want to restore.

Yes, the idea would be to have a minimal base image of the OS that you
would not have to update, or at least not frequently, along with a
bare-metal re-installer that you would boot from CD. But, I'm not
sure if anyone knows exactly what needs to be in this image for
Windows or whether you have to update it after installing windows
updates.

[Bob Weber]

Post by Les Mikesell

There would also be the additional (and potentially not trivial)
work to get clonezilla to run on an active windows system. I'm not
sure if it can read all the NTFS metada from a shadow copy but there
has to be some way to do it since Norton Ghost is able to clone a
live system using some type of shadow copy.

Yes, the idea would be to have a minimal base image of the OS that you
would not have to update, or at least not frequently, along with a
bare-metal re-installer that you would boot from CD. But, I'm not
sure if anyone knows exactly what needs to be in this image for
Windows or whether you have to update it after installing windows
updates.

I have cloned Windows systems (win 2k to XP) by using sysrescucd. First
I would use ntfsresize to re-size the file system to the smalist disk I
would restore to. This is not necessary if you will always use the same
disk size or larger. Then I would copy the first 100 megs or so (not
really sure how much is necessary but this amount allways worked) using
dd. If the destination disk was a different size I would use fdisk to
resize the c: partition to the size of the new disk making sure the
partition always started at the same block as the original. If the size
is the same I would use fdisk to write the the original partition table
back so the kernel would know the disk partition table changed (this
saves a reboot since the dd copy process copies a new partition table
also). Next I would use ntfsclone to clone the whole ntfs file system.
ntfsclone just copies the data and directory structure so it is
usually pretty fast. If the disk is bigger than the original I would
use ntfsresize to resize the ntfs file system to the new partition.

This seems pretty complicated but I used a script to automate the
process. I used this at a local school system to create classroom
images saved to Linux server that I could later use to populate all the
computers in a class. The 2 files for each image were compressed with
gzip and transferred with ssh to and from the Linux server. Check out
the documentation that comes with ntfsclone to see examples.

...Bob

[Jeffrey J. Kosowsky]

Post by Bob Weber
I have cloned Windows systems (win 2k to XP) by using sysrescucd.

What is sysrescucd?

Post by Bob Weber
First I would use ntfsresize to re-size the file system to the
smalist disk I would restore to. This is not necessary if you will
always use the same disk size or larger. Then I would copy the
first 100 megs or so (not really sure how much is necessary but
this amount allways worked) using dd. If the destination disk was
a different size I would use fdisk to resize the c: partition to
the size of the new disk making sure the partition always started
at the same block as the original. If the size is the same I would
use fdisk to write the the original partition table back so the
kernel would know the disk partition table changed (this saves a
reboot since the dd copy process copies a new partition table
also). Next I would use ntfsclone to clone the whole ntfs file
system. ntfsclone just copies the data and directory structure so
it is usually pretty fast. If the disk is bigger than the original
I would use ntfsresize to resize the ntfs file system to the new
partition.

Not sure why you need to use 'dd' before ntfsclone. I thought that
ntfsclone was supposed to clone the entire ntfs filesystem at a low
level -- analogous to Norton Ghost.

If you are just trying to copy over the partition table then why not
just export that using something like 'sfdisk -d /dev/sdx'.

If you are trying to copy over the MBR, then you would just need to
use dd on the first 512 bytes (if memory servers me).

If I am mistaken, I would be interested in knowing what part of an NTFS is not
copyable by just using ntfsclone (and any sources for additional info
would be most helpful.)

Thanks

Post by Bob Weber
This seems pretty complicated but I used a script to automate the
process. I used this at a local school system to create classroom
images saved to Linux server that I could later use to populate all the
computers in a class. The 2 files for each image were compressed with
gzip and transferred with ssh to and from the Linux server. Check out
the documentation that comes with ntfsclone to see examples.
...Bob
--
This message has been scanned for viruses and dangerous content
by MailScanner4.66/SA 3.2.3/Bingo, and is believed to be clean.
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[Bob Weber]

*...Bob*

Post by Jeffrey J. Kosowsky

Post by Bob Weber
I have cloned Windows systems (win 2k to XP) by using sysrescucd.

What is sysrescucd?

Post by Bob Weber
First I would use ntfsresize to re-size the file system to the
smalist disk I would restore to. This is not necessary if you will
always use the same disk size or larger. Then I would copy the
first 100 megs or so (not really sure how much is necessary but
this amount allways worked) using dd. If the destination disk was
a different size I would use fdisk to resize the c: partition to
the size of the new disk making sure the partition always started
at the same block as the original. If the size is the same I would
use fdisk to write the the original partition table back so the
kernel would know the disk partition table changed (this saves a
reboot since the dd copy process copies a new partition table
also). Next I would use ntfsclone to clone the whole ntfs file
system. ntfsclone just copies the data and directory structure so
it is usually pretty fast. If the disk is bigger than the original
I would use ntfsresize to resize the ntfs file system to the new
partition.

Not sure why you need to use 'dd' before ntfsclone. I thought that
ntfsclone was supposed to clone the entire ntfs filesystem at a low
level -- analogous to Norton Ghost.
If you are just trying to copy over the partition table then why not
just export that using something like 'sfdisk -d /dev/sdx'.

I tried to clone the disks this way by just making a partition table and
then using ntfsclone and using fixmrb to restore the mbr. Windows
wouldn't boot. I didn't go into a lot of trouble to see exactly what
needed to be copied. Maybe the programmers at Norton have access to the
internal workings of windows that us users don't have so they can
program ghost to copy the appropriate data. Copying 50 to 100 megs of
data was sufficient to fool windows into thinking it was on the same
disk. The file after compression was only about 8 megs (from a 50 meg
dd of the disk) and this is nothing space wise compared to the several
gigs of data that ntfsclone would copy. I just assumed that windows
stored a disk serial number or other data that it could use to assure
itself that the installation it was running on was legal. The ntfsclone
program probably overwrote data that was past the beginning of the C:
partition but still in the first 50 megs of the disk. I just used the
50 megs number since I didn't want to have to calcilate the exact amount
to copy up to the C: partition for each disk geometry. Also, sometimes
there would be a "diagnostic" partition that was usually less then the
amount I would copy so it was included in this "boot" image I would make.

The machines I was cloning are identical since we would purchase several
classroom worth of computers at one time so the rest of the hardware
would look the same to windows. So this method might not work if the
hardware is not the same as in the case of buying a new replacement
machine. It does work nicely in restoring a machine after a disk
failure. I have done this several times at home when my wife's machine
and my laptop had failed disks.

Post by Jeffrey J. Kosowsky
If you are trying to copy over the MBR, then you would just need to
use dd on the first 512 bytes (if memory servers me).
If I am mistaken, I would be interested in knowing what part of an NTFS is not
copyable by just using ntfsclone (and any sources for additional info
would be most helpful.)

ntfsclone apparently copies the entire file system correctly since I
assume that it overwrote all file system data after the beginning of the
C: partition that was copied from my dd boot image of the first 50 megs
of the disk. I have used ntfsresize successfully many times to re-size
ntfs file systems to new disk partition sizes and fdisk to resize
partitions on destination disks that are larger than the image I was
cloning from. So these programs know what they are doing when copying
or modifying a ntfs file system or partition table. My "good enough for
government work" way of solving the no boot problem seems awkward but it
has worked for me. I mainly use Linux, so these tools have allowed me
to clone and save images of Window system without having to use Windows
and buy propriety programs. I use backuppc to daily backup the user data.

Post by Jeffrey J. Kosowsky
Thanks

Post by Bob Weber
This seems pretty complicated but I used a script to automate the
process. I used this at a local school system to create classroom
images saved to Linux server that I could later use to populate all the
computers in a class. The 2 files for each image were compressed with
gzip and transferred with ssh to and from the Linux server. Check out
the documentation that comes with ntfsclone to see examples.
...Bob
--
This message has been scanned for viruses and dangerous content
by MailScanner4.66/SA 3.2.3/Bingo, and is believed to be clean.
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[Les Mikesell]

Post by Bob Weber

Post by Jeffrey J. Kosowsky
If I am mistaken, I would be interested in knowing what part of an NTFS is not
copyable by just using ntfsclone (and any sources for additional info
would be most helpful.)

ntfsclone apparently copies the entire file system correctly since I
assume that it overwrote all file system data after the beginning of the
C: partition that was copied from my dd boot image of the first 50 megs
of the disk. I have used ntfsresize successfully many times to re-size
ntfs file systems to new disk partition sizes and fdisk to resize
partitions on destination disks that are larger than the image I was
cloning from. So these programs know what they are doing when copying
or modifying a ntfs file system or partition table. My "good enough for
government work" way of solving the no boot problem seems awkward but it
has worked for me. I mainly use Linux, so these tools have allowed me
to clone and save images of Window system without having to use Windows
and buy propriety programs. I use backuppc to daily backup the user data.

Clonezilla does a good job of cloning windows systems as long as the target as
the same size or larger than the source, so you could probably look through its
scripts to see what it uses besides ntfsclone.

[l***@consolejunky.net]

Post by Les Mikesell

Post by Bob Weber

Post by Jeffrey J. Kosowsky
If I am mistaken, I would be interested in knowing what part of an NTFS is not
copyable by just using ntfsclone (and any sources for additional info
would be most helpful.)

ntfsclone apparently copies the entire file system correctly since I
assume that it overwrote all file system data after the beginning of the
C: partition that was copied from my dd boot image of the first 50 megs
of the disk. I have used ntfsresize successfully many times to re-size
ntfs file systems to new disk partition sizes and fdisk to resize
partitions on destination disks that are larger than the image I was
cloning from. So these programs know what they are doing when copying
or modifying a ntfs file system or partition table. My "good enough for
government work" way of solving the no boot problem seems awkward but it
has worked for me. I mainly use Linux, so these tools have allowed me
to clone and save images of Window system without having to use Windows
and buy propriety programs. I use backuppc to daily backup the user data.

Clonezilla does a good job of cloning windows systems as long as the target as
the same size or larger than the source, so you could probably look through its
scripts to see what it uses besides ntfsclone.

add this to the list of possible Windows imaging issues:

http://anandtech.com/storage/showdoc.aspx?i=3691

[Carl Wilhelm Soderstrom]

Post by l***@consolejunky.net
http://anandtech.com/storage/showdoc.aspx?i=3691

Thanks for the link, that's an informative article.