I am trying to do a backup from Linux to USB Pen Drive with rsync each day and I take home the Pen Drive at night. I am running the following command but it looks like its update every file every day?

rsync -y -r -vv ~/Documents/ /media/PenDrive/xxxx-home/

Can someone please tell me if this can be done better I used to just tar of my ~/Documents and cp the file to my Pen Drive but my SA said this would be better but I dont think it is..

and sometimes I am getting this message

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

rsync –modify-window 1 -a –no-o –no-p –no-g ~/Documents/ /media/PenDrive/uftwf-home/

--modify-window 1 because fat "represents times with a 2-second resolution"
--no-o because vfat does not support owners
--no-g because vfat does not support groups
--no-p because vfat does not support permissions

Check more discussion of this question.

I’m currently configuring an Ubuntu Server system and would like to perform routine backups using rsync. The goal is to have the right files backed up so that I can use the backup to create an identical system running somewhere else without a hitch (identical ssh, apache, mysql, etc).

Is rsync the right tool for the job? If so, which directories would I absolutely need to include in the backup?

If you have to ask “What should I back up?”, then the only answer I can give is:

Everything.

Rsync might work for your situation, but it’s missing many critical features found in real backup tools.

One thing to consider that would ease the bring-up of new machines is to use a configuration management product like Puppet or Chef. Using these to deploy your servers ensures that your systems’ configurations are in a known-good and reproducible state.

Check more discussion of this question.

We are using robocopy with MIR switch to backup our Microsoft Access Databases, i.e., .accdb, and .mdb files. Interesting thing with Access database files is that you can open a database, add a few records into a table, close the db and the size of the database file will not change.
Will robocopy know that he has to copy this file next time when he does the backup?

Thanks

Changing the file not necessarily leads to a change in the file’s size, but typically (i.e. unless a sufficiently privileged process has changed that) the following always happens:

• the “last modified” timestamp of the file is updated to the current timestamp
• the “archive” bit is touched (if it was not set, it is set by the OS automatically)

Robocopy would compare the “last modified” timestamp of the destination with that of the source and copy when these two differ (this behavior might be tweaked according to your requirements using the /xn and /xo switches). Robocopy would only honor (and reset) the archive file attribute when you use /m switch – see the robocopy documentation for details.

Check more discussion of this question.

We are currently destroying our old DLT tape (some are 30 years old…), and we are amazed by the length of the band inside.

We would know what is it length ? It looks infinite ! (+1)

Each generation of DLT was different:

• DLT III is about 365m
• DLT IV is about 557m
• SDLT I is about 558m
• SDLT II is about 630m

Check more discussion of this question.

I backup my data every day on tape using one drive DAT HP Storageworks DAT 160. I use one tape for every day and I turn them weekly. Every monday I check one tape randomly recover some files saved on it.

I know that when data is saved on tape, the driver and backup software check data integrity, but I wonder if a manual check of some data saved has a sense or not. I re-use these tapes many times and I would be sure data are safe.

Check your backp softtware for a validation routine.

In general, restoring to SEE THE CONTENT is ridiculous – you can trust checksums to work well enough that a file that does not show an error contains the same content that you had at backup. You basically waste a LOT of time.

Tape integrity checks should be a semi (need to replace the tapes in the drive) automatic function of your backup software.

Check more discussion of this question.

When a SMART check on a disk reports a bad sector, it is important to be able to identify the file that has the bad sector – and restore it from backups. Below, I show how I did this for my Linux/ext3 VMWARE server – but does anyone know if this can be done for Windows/NTFS?

Here’s how I did it for Linux/ext3: I first asked the drive to do a hardware surface scan (below the OS level, with the on-drive SMART circuits):

vserver:~# smartctl -t long /dev/sdc

I looked at the results:

vserver:~# smartctl -a /dev/sdc
...
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       9
...
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     27679         591363172

So, one sector was already marked bad, 9 were marked for replacing from the “staging” sector space. More importantly, the first logical block address (LBA) that is unreadable, was 591363172.

I found the partition (and the offset inside it) that this number “translated” to:

vserver:~# fdisk -lu /dev/sdc
Device Boot      Start         End      Blocks   Id  System
/dev/sdc1           32   976773119   488386544   83  Linux

The partition started at sector 32. So, the bad sector was…

vserver:~# bc -l
591363172-32+1
591363141

…at an offset of 591363141 sectors from the beginning of the partition.

Now I could find which file was “hosed”:

vserver:~# tune2fs -l /dev/sdc1 | grep Block\ size
Block size:               4096

The block size of this EXT3 filesystem was 4096 bytes, so the bad sector destroyed this block in the filesystem:

vserver:~# bc -l
591363141*512/4096
73920392.62500000000000000000

And the block number (73920392) corresponded into this file:

vserver:~# debugfs
debugfs 1.41.3 (12-Oct-2008)
debugfs:  open /dev/sdc1
testb 73920392
debugfs:  testb 73920392
Block 73920392 marked in use
debugfs:  icheck 73920392
Block           Inode number
73920392        18472967
debugfs:  ncheck 18472967
Inode           Pathname
18472967        /path/to/filewithbadsector

And I restored that file from my backups.

Is there an equivalent procedure I can follow for Windows/NTFS?

I know you have an NTFS FS, and run windows on that FS.
I don’t know if you “could” boot a live Linux to work on that driver or not.

If you can boot Linux from CD or USB,
you can use ntfsprogs. look at –

ntfscluster ntfsinfo 

I believe ntfscluster tell you what file a particular cluster stores. I hope this puts you in the right direction.

Check more discussion of this question.

I’m trying to use Windows Server Backup on my Windows Server 2012 box to backup my Hyper-V VMs. I created a backup job but each job ends with some “Access is denied” errors.

One of my VMs (Linux Server) is backed up properly. All others (one Windows 8, one Linux) are not (or at least it seems that way from the looks of the log file below).

How can I solve this problem?

Here’s the log I’m getting:

Error in backup of D:\ during read: Error [0x80070005] Access is denied.
Application backup
Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
   Component: C435964E-C07A-4958-BA73-A04C6583280F
   Caption     : Backup Using Saved State\Alter Server
   Logical Path: 
   Error           : 8078010E
   Error Message   : Copy of the files failed.   Detailed Error  : 80070005
   Detailed Error Message : (null)Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
   Component: E780F138-9676-42FB-821C-4561B9B263DC
   Caption     : Backup Using Child Partition Snapshot\Windows 8
   Logical Path: 
   Error           : 8078010E
   Error Message   : Copy of the files failed.   Detailed Error  : 80070005
   Detailed Error Message : (null)Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
   Component: Host Component
   Caption     : Host Component
   Logical Path: 
   Error           : 8078010E
   Error Message   : Copy of the files failed.   Detailed Error  : 80070005
   Detailed Error Message : (null)

It seems that Windows Server Backup can’t handle Hyper-V VMs on ReFS file systems. I solved the problem by changing the file system to NTFS, and now everything works fine.

Check more discussion of this question.

I’m running a restore from a previous backup of my FreeBSD system, and run into trouble when restoring / (excluding mount-points). The problem is that /rescue fills the / partition (to the extent that I can’t restore my entire / partition) by taking up more place than it originally did.

I suppose it’s because /rescue contains links and not files, and that files, not links, are restored into my / partition. I’ve tried restoring with both tar and rsync:

( cd /mybak/ ; tar --one-file-system -cvf - . ) | ( cd /newroot/ ; tar -xpf - . )

and

/usr/local/bin/rsync -va --delete --one-file-system  /mybak/ /newroot/

Both methods lead to my problem. What can I do to properly restore (or initially backup) the / partition, including /rescue, so that the restore doesn’t take up more disk space than the original?

As you surmised, the problem here is /rescue doesn’t contain many files – it contains one file (inode) with many names (hard links). As a result when backing up or restoring /rescue you need to use software that’s aware of hard links.

For rsync this means the -H flag.

For tar this should not require any special magic: BSD tar (and GNU tar) are smart enough to know what a hard link is.
For other backup programs, you need to check the documentation.

If you’re using the regular Unix tools to make backups of your system (and grabbing whole filesystems) you may be better off using dump and restore. Aside from the ability to use UFS snapshots, there are other advantages to the more traditional utilities.

Check more discussion of this question.

I’m looking for a suitable program to copy files from one linux machine to another one. The program should be able to do authentication but it should not do encryption. The reason behind the latter is the lack of CPU power to do the encryption.

I copy backups from ~70 machines to a single backup server simultaneously. The single server is an HP Proliant DL360 G7, with 10 Gbps ethernet connection and an FC storage backend that can do 4 Gbps. Through FTP I can write ~400MB/sec to the storage (that’s about what I want) but through ssh with arcfour I can only do ~100MB/sec while having 100% CPU usage. That’s why I want file transfers not to be encrypted.

The alternatives that I found not really suitable:

• rcp: no authentication, forget it
• FTP: making the authentication “secure” (at least preventing plain-text password exchange) is possible but not really easy and I haven’t found a method to force any FTP daemon to encrypt the control channel (for the authentication) and not to encrypt the data channel (for data transfers)
• SCP/SFTP: in farely recent ssh(d) implementations you can’t turn off encryption. The best you can do is to use the arcfour cypher for the encryption but it sill uses too much CPU power for my needs.
• rsync over ssh: same problems as with SCP/SFTP.
• plain rsync: from the documentation of rsyncd: “The authentication protocol used in rsync is a 128 bit MD4 based challenge response system. This is fairly weak protection, though (with at least one brute-force hash-finding algorithm publicly available), so if you want really top-quality security, then I recommend that you run rsync over ssh.” It’s a no-go.

Is there a protocol/program that can do exactly what I want?

(A big plus would be if it could work on windows as well and/or if it would support rsync-stlye copying/synchronization (e.g. copy only the differences).)

You tried doing what I normally do, using a lower-weight encryption algorithm (like arcfour). When that becomes the bottleneck, the next approach I use is disabling ssh encryption entirely.

One approach for this is to use hpn-ssh as your transport. My normal application for this is SSH copies across high-bandwidth, long-distance links (e.g. a 10GbE link between Chicago and New York with 16ms latency). hpn-ssh allows tuning of TCP windows there, but also has the option of disabling encryption entirely. That may help in your case.

Also see: Why is my rsync so slow?

Check more discussion of this question.

all:

I bought a personal linode server without the backup service.

All the data on it is totally personal and not that important. But for sure, I still don’t want them to be lost.

I want to know, without the backup service, which level of data protection should I expect?

Do they have in-place RAID or some other replication facilities for my data?

How do you think, without the backup service, the probability of a data loss would be?

Thanks.

Without backups, you have no data protection. RAID is for fault tolerance and availability in the event of a disk failure, not data protection. If you have a RAID 1 mirror and you accidentally delete or corrupt data, RAID isn’t going to protect you.

Check more discussion of this question.