Linux Cpanel Backup to Amazon S3
In this article i will explain how to take cpanel backup to amazon s3(with backup rotation enabled). The step by step procedure is explained below,
Step1) Activate an account in s3. You will get an access key and secret_key after the activation.
You can create a new s3 account by following the url,
Step2) Install s3 client for linux. The package name is “s3cmd-1.0.0-4.1”.
root@heuristics:~# apt-get install s3cmd
On redhat or centos based machines(using rpm packages), you can install “s3cmd” as follows,
Alternatively, you can download it from the url pasted below:
Step3) Configure s3 client using the command,
root@heuristics:~# s3cmd --configure
It will ask for the access key and secret key that we got during our account activation. This process reports failure, if we provide the wrong key values. Once this step is completed, the configuration will be stored inside the file “/root/.s3cfg”.
During configuration you will be asked whether to enable encryption or not. Enabling encryption will improve the security of transfer but will make the upload a little bit slower.
Step4) We need to create buckets in s3 for storing the backup.
eg: creating a bucket named “Backup_daily”,
root@heuristics:~# s3cmd mb s3://Backup_daily
For additional options refer the url,
Step5) Enable daily backup from WHM. Refer the url pasted below for reference,
If backup is already configured, then we can know the location of the backup using the command,
BACKUPDIR /backup
root@heuristics:~#
Inside “/backup” there will be another directory named “cpbackup”, which will be holding the daily,weekly and monthly backup’s. In my case,
./ ../ daily/ monthly/ weekly/
root@heuristics:~#
Step6) Create log directories,
root@heuristics:~#
Step7) Write a script to automate the backup, and save it as “/root/dailybackup.sh” . In the script pasted below, the backup rotation degree is set as 3(“DEGREE=3” , line16). This means that, 3 days old backup will be deleted automatically. You can increase this backup retention period by adjusting the “DEGREE” variable in line16.
#!/bin/bash ##Notification email address _EMAIL=your_email@domain.com ERRORLOG=/var/log/backuplogs/backup.err`date +%F` ACTIVITYLOG=/var/log/backuplogs/activity.log`date +%F` ##Directory which needs to be backed up SOURCE=/backup/cpbackup/daily ##Name of the backup in bucket DESTINATION=`date +%F` ##Backup degree DEGREE=3 #Clear the logs if the script is executed second time :> ${ERRORLOG} :> ${ACTIVITYLOG} ##Uploading the daily backup to Amazon s3 /usr/bin/s3cmd -r put ${SOURCE} s3://Backup_daily/${DESTINATION}/ 1>>${ACTIVITYLOG} 2>>${ERRORLOG} ret2=$? ##Sent email alert msg="BACKUP NOTIFICATION ALERT FROM `hostname`" if [ $ret2 -eq 0 ];then msg1="Amazon s3 Backup Uploaded Successfully" else msg1="Amazon s3 Backup Failed!!\n Check ${ERRORLOG} for more details" fi echo -e "$msg1"|mail -s "$msg" ${_EMAIL} ####################### ##Deleting backup's older than DEGREE days ## Delete from both server and amazon ####################### DELETENAME=$(date --date="${DEGREE} days ago" +%F) /usr/bin/s3cmd -r --force del s3://Backup_daily/${DELETENAME} 1>>${ACTIVITYLOG} 2>>${ERRORLOG}
Step8) Grant execute privilege for the script and schedule it to run everyday,
root@heuristics:~# cp -p /root/dailybackup.sh /etc/cron.daily/
root@heuristics:~#
NOTE:
Or if you wish to start the amazon s3 backup script right after the cpanel backup process, then create a cpanel post backup hook named “/scripts/postcpbackup” with the following contents,
system(“/root/dailybackup.sh”);
The post backup hook will start the amazon s3 backup script right after every cpanel backup completion.
In case of disasters we can download the backup from the bucket using the same s3cmd tool.
root@heuristics:~# s3cmd -r get s3://Backup_daily/2011-02-32 restore
Perfect!
I will test on my cPanel server.
Thank you.
Really great article, thanks. Three questions if you don’t mind.
1) In my Daily directory on my VPS I actually have separate tar files for each account on my server (done via the normal cpbackup). So does the process you detail above simply move each of these tarred files separately onto S3, or does it tar them all up into one file and move them in one go? I’m also then thinking ease of use of then restoring just one account if needs be.
2) If I also take Weekly and Monthly WHM backups, do I have to setup identical Weekly and Monthly scripts as above to subsequently upload these weekly and monthly tar files to S3 too?
3) With the DEGREE days, my server already overwrites the daily back up with the latest one each time (so I only have the latest daily to hand). But I want it so that Amazon S3 should hold all 7 days worth of daily backups before they start to be deleted. How can I do it so that only DEGREE days applies to Amazon S3?
Thanks in advance
Hi DonkeyBoy,
Thanks for your feedback:)
Replies to your queries are pasted below,
>> 1) In my Daily directory on my VPS I actually have separate tar files for each account on my server (done via the normal cpbackup). So does the process you detail above simply move each of these tarred files separately onto S3, or does it tar them all up into one file and move them in one go? I’m also then thinking ease of use of then restoring just one account if needs be.
The backup script I wrote would just simply copy the contents inside daily backup folder to amazon s3. It won’t compress the daily backup folder during this process. Also if you want to restore a single domain in case of emergency, you just have to download the corresponding compressed backup file.
You can use the following command to download backup of a single account.
eg:
s3cmd get s3://bucketname/2011-03-02/cpmove-username.tar.gz
>> 2) If I also take Weekly and Monthly WHM backups, do I have to setup identical Weekly and Monthly scripts as above to subsequently upload these weekly and monthly tar files to S3 too?
Yes, you need to setup different backup scripts. All you need to modify is the source(“SOURCE”) directory location at the script(and the scheduling time also).
————–
##Directory which needs to be backed up
SOURCE=/backup/cpbackup/daily
————–
>> 3) With the DEGREE days, my server already overwrites the daily back up with the latest one each time (so I only have the latest daily to hand). But I want it so that Amazon S3 should hold all 7 days worth of daily backups before they start to be deleted. How can I do it so that only DEGREE days applies to Amazon S3?
The ‘DEGREE’ variable in script applies only to s3 backup and doesn’t in any way effect regular cpanel backup’s. If you want to hold 7days backup in amazon s3, then consider changing the vaulue of ‘DEGREE’ from 3 to 7.
I have written another article mentioning how to mount an amazon s3 bucket as a local drive. You can create three buckets for daily,weekly and monthly. And then mount these buckets as local drive, later you can use “rsync” to copy over all
the backup’s inside cpanel backup directory to the mounted s3 buckets(you can automate this usning bash scripts also).
For more details refer the url pasted below,
+++++++++++++++++++++++++
+++++++++++++++++++++++++
Thanks for your comprehensive reply! I’ve now implemented S3 transfer on my VPS – thanks to your really useful instructions, my perseverance and a big thanks to my hosting provider!!
I found out there were some additional steps or caveats which I thought I’d mention for others:
Step 2) Before I could install, I had to download the appropriate repo into /etc/yum.repos.d
So for instance I’m running CentOS 5, and this was `wget http://s3tools.org/repo/CentOS_5/s3tools.repo `
Then to install I had to `yum install s3cmd` not apt-get. Maybe just useful to mention that different OS’s might need different commands
Step 3) It also asked for an encryption key – which I had no idea, so I simply made up a long random string. Seems to work fine. Also it was asking for GPG program which I left as default.
Step 7) This was the biggie as simply copy/pasting the script used curly double quotes rather than straight quotes. Even if I copy/pasted into a text editor first. Obviously this totally didn’t work to start off with until all instances were replaced.
Other than that, all seems to be good!! Thanks very much.
thanks Donkeyboy! that did it for me.
Hi,
When I try and run the script, I get the following error:
pico dailybackup.sh
root@server [~]# ./dailybackup.sh
./dailybackup.sh: line 31: [: -eq: unary operator expected
date: extra operand `+%F’
Line 31 is specifically: if [ $ret2 -eq 0 ];then
I’m struggling to debug the script to work out why this has occurred?
The second error : date: extra operand `+%F’ is in reference to this line – sure it’s something simple like backticks but I just can’t get it working….
DELETENAME=$(date –date=”${DEGREE} days ago” +%F)
Hi Tom,
Could you please check the return value saved in line 25 ?
Specifically,
=========
ret2=$?
=========
Try to put an echo statement after this line. It will show the return value of the above statement.
I have written another article in which the steps to mount an s3 bucket as a local drive is specified. If interested you can refer it. The url is pasted below,
=========
=========
Once the s3 bucket is mounted as a local drive, then you can use rsync to copy over the backup files:)
I get the following when I run the scripts. BTW, I’m running on CentOS 5 and I replaced the smart quotes with straight.
/etc/cron.daily/dailybackup.sh: line 19: ${ERRORLOG}: ambiguous redirect
/etc/cron.daily/dailybackup.sh: line 20: ${ACTIVITYLOG}: ambiguous redirect
/etc/cron.daily/dailybackup.sh: line 23: ${ACTIVITYLOG}: ambiguous redirect
/etc/cron.daily/dailybackup.sh: line 29: [: -eq: unary operator expected
date: invalid date `ate=3 days ago’
/etc/cron.daily/dailybackup.sh: line 47: ${ACTIVITYLOG}: ambiguous redirect
I should have mentioned that line 29 is
if [ $ERRORFLAG -eq 0 ];then
The ambigous redirect lines are all of the lines with the mentioned variables in them.
Any help would be greatly appreciated. Thanks!
Hi John,
Thanks for notifying me about the ambigous line. Previously I used the /scripts/pkgacct to backup the accounts and the return value of this backup script was saved in ERRORFLAG variable. Sometime before I removed this /scripts/pkgacct method from the script and instead configured backup from the WHM. The “/scripts/pkgacct” line was removed from the script, but forget to remove the ERRORFLAG variable from the script. I have made the corrections now:)
You can also checkout the following url to mount s3 bucket as a local filesystem in linux. I believe this will be easy to use than the backup script.
———————
———————
Thanks a lot for this, I found it extremely helpful.
I am having one small error with the email notification. It doesn’t seem to be working and the script ends with this error:
./dailybackup.sh: line 28: NOTIFICATION: command not found
./dailybackup.sh: line 31: syntax error near unexpected token `)’
./dailybackup.sh: line 31: `msg1=”Backup completed
successfully:)\nBackup’s are stored in ‘
I appreciate any direction.
Hi redstormj,
It seems that you forget to put the NOTIFICATION string inside double quotes. The entire line should be like this:
================
msg=”BACKUP NOTIFICATION ALERT FROM `hostname`”
================
Regarding the error with line 31, please paste the following entry in line 31 :
=============
msg1=”Amazon s3 Backup Uploaded Successfully”
=============
Thank you for the quick reply. It seems like the copy paste of the quotes doesn’t come out right. I manually replaced the quotes one by one and I am now running the script again to see if it works well this time.
Thank you.
After running the scripts, the backups were transferred but it ended on this error:
date: extra operand `+%F’
This might be a silly question but I have to ask, since there is no configuration parameter in regards of at which time cpanel runs the backups everyday, how does one makes sure that the s3 script is only processed AFTER cpanel has generated the backups for a given day and not before?
Hi redstormj,
Usually in the cronjob of root user the cpanel backup script is scheduled to run every midnight. You can create a cpanel post backup( /scripts/postcpbackup ) hook to execute the amazon s3 backup script. This script needs to be written in perl, and I think you can use the perl system call function to execute this s3 backup script.
Ref:
————
https://forums.cpanel.net/f49/executing-script-after-main-cpbackup-has-finished-145129.html
————-
Thank you. I can make sense of how that works, I’m just not sure exactly what it is that I have to do to have the s3 script run at that point.
I appreciate your efforts regardless.
Making the hook for after the backup has ran was simpler than I though. Here is what you need:
create a file:
/scripts/postcpbackup
give this file execute permission (755)
Insert this in the file:
#!/usr/bin/perl
system(“/root/dailybackup.sh”);
that’s it. Nothing fancy, it will just execute your s3 script after the cpanel backup has completed. I hope that helps someone.
I’m still having issues with this line:
DELETENAME=$(date –date=”${DEGREE} days ago” +%F)
It keeps throwing an error:
date: extra operand `+%F’
Ok, after a little reading on the web about the date command I was able to figure it out. Here is the correct line:
DELETENAME=$(date –date=”${DEGREE} days ago” +%F)
It appears that the format in this website is changing the actual code. The correction is simple, where it says $(date –date
that dash should be a double dash (as in the minus sign)
so it should be – – (without the space in between them).
Unless I am missing something are you guys storing this data unencrypted? If so thats a pretty big security problem.
s3cmd has an encrypted method for transferring. It is slower than the regular, but it is there none the less. This is what I am currently using for file transfer to s3. When configuring it, it will ask you if you want to enable that or not.
I installed s3cmd per the directions, the config tested successfully. I am able to put files on s3. I then downloaded your script. It works awesome when I run it manually, but as soon as I copy the backup script to /etc/cron.daily, I get the error ERROR: /.s3cfg: No such file or directory.
I tried copying the .s3cfg file to the /etc/cron.daily folder, but it still doesn’t seem to get recognized when cron runs it, but again, works fine if done manually. I’m looking at the script again to see if there is a config I missed, but I thought I’d check here and see too.
Hi Scott,
Instead of moving the script to “/etc/cron.daily” folder, you can create a cpanel post backup hook which will automatically call the s3 upload script. This has been mentioned by redstormj in comment posted on “June 20, 2011 at 8:40 pm”.
Excellent! I tried this last night and it worked beautifully!
Someone else had noted the double dash too; when I copy/pasted this script, I had to modify –date and –force to have double dashes.
Thanks for the great script!
I had this problem too:
ERROR: /.s3cfg: No such file or directory.
It’s because it’s looking for the .s3cfg file in the directory one higher than root.
Creating a symlink to connect .s3cfg to the one in root worked for me.
See here for more:
http://sourceforge.net/projects/s3tools/forums/forum/618865/topic/3941168
Great, it seems to work well. Really fast and easy to implements.
I’m having issues with the script.
I’ve changed the curly quotes to normal but when I try to run the script with
sh dailybackup.sh
I get errors
: command not found 5:
: command not found 8;
: command not found 11;
: command not found 14;
: command not found 21;
: command not found 25;
: command not found 28;
dailybackup.sh: line 44 syntax error: unexpected end of file
could you tell me why this is so?
ok , this was a windoze thing.
I copied and pasted the script into geany on linux and now those errors don’t happen but I get new errors now
Error: not enough parameters for command ‘put’
dailybackup.sh: line 24: s3://mydailybackc3/2011-09-04/ no such file or directory
Error: not enough parameters for command ‘del’
dailybackup.sh line 45: s3://mydailybackupc3/2011-09-01: no such file or directory
right. looks like something is happening now. it appears to be because I am more monkey than human :-z
I had line breaks in the script for put and del
I manually ran dailybackup.sh and it’s now waiting on the command line, it’ll probably take a while because I have around 10 accounts that would need to be transferred.
I’ll go for lunch and hope it shows it worked when I get back 🙂
sorry to clutter up your comments. i hope I wont need to come back and ask more dumb questions
If, god forbid, someone roots you… how do you protect your files in your bucket from being deleted?
Hi Tommy,
Unfortunately we can only push backups to amazon s3. They haven’t yet provided the feature of pulling data or backup from servers and then keeping them in buckets. You can raise this query in amazon s3’s community forum @ https://forums.aws.amazon.com/forum.jspa?forumID=24
Yes, this is a valid concern!
You’d need to look at setting up Group/User IAM policies on the S3 buckets and then only allow S3:PutObject. This will have the unfortunate side-effect of not permitting S3:DeleteObject, so the last line of the script above will no longer automatically delete objects of a certain age. This will mean you’ll have to go in manually and delete older backup directories yourself – but surely this is a minor inconvenience compared to the worse-case scenario!
Then if you ever need to restore an object, simply temporarily add S3:GetObject into the IAM policy, pull down the file and then remove that permission again afterwards.
Depending on what you’re using this for, for ultimate protection, you might also want to look at enabling versioning on the bucket to prevent malicious overwrites and maybe even only allowing connections from a particular IP address.
Hello Guys,
I’m running a dedicated server, and I’m not very good at SSH, the server runs CentOS5.7 and apt-get doesn’t exist for some reason :S
Can any help me a little, so I don’t break anything, and I really need some kind of backup of all my files. Because atm, it takes automatically backup, then I need to download from my /backup folder, and then reupload to S3 which is a long process with my upload.
Best regards,
Lucas
Hi Lucas,
In centos the package manager is named “yum”(not apt-get). You need to install s3cmd using yum. This has been mentioned by Donkeyboy in comment #4 dated “May 18, 2011 at 1:04 pm”.
Thank you! Got it to work, but now I’m wondering, is there a way, when I already is backing all user data up, those backups is performed Sunday night,
Can I then make another ‘backup’ that only takes backup of all the databases? Per user? And store it in /backup folder :S
Thank you
/root/dailybackup.sh: line 30: date : command not found
Line 30 =
DELETENAME=$(date –date=”${DEGREE} days ago” +%F)
According to network activity my entire backup was uploaded to S3 however in S3 there is nothing. I would think this error wouldn’t prevent the data from actually being there?
(Backup shows in S3 now, so it’s just fixing that error MSG)
Hi guys,
I got s3cmd installed but when I try to do a configure I get following message:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the following lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Problem: Err: unsupported locale setting
S3cmd: 1.0.0
Traceback (most recent call last):
File “/usr/bin/s3cmd”, line 2006, in ?
main()
File “/usr/bin/s3cmd”, line 1700, in main
preferred_encoding = locale.getpreferredencoding() or “UTF-8”
File “/usr/lib64/python2.4/locale.py”, line 417, in getpreferredencoding
setlocale(LC_CTYPE, “”)
File “/usr/lib64/python2.4/locale.py”, line 381, in setlocale
return _setlocale(category, locale)
Error: unsupported locale setting
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please report the above lines to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Well my script seems to only work if I run it manually. I followed the instructions for adding it to the daily CRON to the “T”. ANd after 2 days nothing in my S3 bucket.
When I ran the .sh file manually it worked.
Please advise on what I should check.
From Error Log:
ERROR: /.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using –configure parameter to create one.
ERROR: /.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using –configure parameter to create one.
I guess I should state that: “/root/.s3cfg” exists.
NEVERMIND — PLEASE DELETE MY COMMENTS — DON’T POST — I GOT IT!
SAME ISSUE AS ABOVE
Hey, This is fantastic, but will the code understand a DEGREE of any number of days? so if i wanted to keep 3 months having 90???
Thanks in advance
Rob
Hi Rob,
If you want to keep 3 months backup, then set the DEGREE to 3 and then change the line number 40 in script to the one pasted below,
DELETENAME=$(date --date="${DEGREE} month ago" +%F)
The above line of code will provide the date which is DEGREE months before.
eg:
cherubium@heuristics:~$ date +%F
2011-11-05
cherubium@heuristics:~$ date --date="3 month ago" +%F
2011-08-05
cherubium@heuristics:~$
I used this: http://fiddyp.co.uk/cpanel-backups-to-amazon-s3/ (links here)
How can the /monthly and /weekly (and maybe even /daily) directories be auto-deleted after uploading to S3?
Hi Cliffpaulick,
What I’ve done to make daily, weekly, and monthly backup, is to make 3 different scripts, 1 for daily, 1 for weekly, 1 for monthly, each script is linking to their own bucket etc backupdaily, backupweekly, backupmonthly on S3. And it works great!.
In case anyone else has issues getting the backup script to work, I’m on a windows machine and I had to paste into geedit and save as linux script (.sh) with linux line endings set for it to work. Also has some rogue formatting issues such as appearing, so check your pasted version carefully against what appears on this page.
I then used filezilla to upload to the server.
Terminal editors confuse the bejesus out of me.
I successfully set up the daily backup process, using the postcpbackup hook to kick off the script. I also have a weekly backup. What would be the best way to trigger a weeklybackup.sh so it doesn’t run at the same time as the daily backup?
Research tells me that I cannot change the time of the weekly backup using WHMs croncfg.
Anyone?
HI,
I got the script to work and it’s communicating to S3 but gives me a 403 ACCESS ERROR once the dailybackup.sh is run. Does anyone know what this means and how to fix it??
Any help would be appreciated…
Thanks,
J.J.
Hi,
This has been working so great for me for a few months, after we then changed timezone, all the deleting of the files on amazon isn’t happening, and I found out the bad way.. A huge bill because of 2.3TB data stored.
When I run the date command that checks if the weekly backups is older than 28 days, returns the correct date, etc “2012-03-11” without the quotes.. But it doesn’t delete the files on amazon, even when that folder exists.
I could need some help here.
Hi,
This was working great, until we changed timezone here in Denmark – For some reason it ain’t deleting the files anymore from amazon s3, the date command, that checks if a file is older than 28 days, for my weekly backup, returns the correct date – the cronjob is run at 4am in the morning, so it shouldn’t be the reason.
I found out the bad way, that it wasn’t working.. Recently we got a bill from amazon for 2.3TB storage.
Hi,
I’m facing a problem now, My backups has been working great for around 4 months, and it did delete the files too when they got too old.. After we changed timezone in Denmark and Netherlands, it hasn’t been deleting the files.. Eventho that the backup script has been running 4am in the morning. – When I do the date –date=’28 days ago’, ‘+%F’ it returns the correct date.
Does anyone know whats going on?
same problem here.
How much is your bill for S3 each month using this method?
I’m using this method, and I do daily, weekly and monthly backups this way, all sent to Amazon S3 – Currently I store 51 gigs, which is 4.49$ a month.
I’ve decided to use reduced redundancy, because it’s a little cheaper.
Hey guys,
Now that Glacier is out is there a similar way to change it over to that?
Rob
hello
i’m having that error
[cpbackup] Executing user defined post backup script (/usr/local/cpanel/scripts/postcpbackup).
syntax error at /usr/local/cpanel/scripts/postcpbackup line 2, near “system(.”
Execution of /usr/local/cpanel/scripts/postcpbackup aborted due to compilation errors.
If some one desires expert view on the topic of running a blog after that
i recommend him/her to go to see this weblog, Keep up the pleasant work.
wowwwww…..great script….
This is a topic that is near to my heart… Thank you!
Exactly where are your contact details though?
Very very thanks, 100%!!
Hi, this script works great with the “legacy” backup, but how would I get it to work with the new backup system that was put in place with cpanel 11.38?
Good Article!
Hello,
Iam using cpanel and on a shared Linux hosting wondering if i can also be able to backup my website to my amazon s3 account and if yes, how do i install the script? Do i install it to my website or at my amazon s3 account? Sorry to ask such but am not familiar with script language used but am very desperate to use it on my website.
The steps on how to install it will be highly appreciated
I will be happy to hear from you
Kind regards,
Ronnie