• Backup script tuning

    From paulie420@1337:3/129 to All on Thu Sep 15 16:01:03 2022
    I know I need to get [a lot] better at bash scripting, but I'm trying to learn how to go from green level just up a notch, first. :P I'm using rsync to backup

















































































































































































































































































































    external HDDs to another HDD, and writing results info to a backup_info.txt file after rsync runs.

    The first iteration just wrote 'successfully completed' blah blah blah - but I wanted to add in checking the rsync error codes and write either 'success' or 'this was the error code' and here's what I came up with in my first attempt at

















































































































































































































































































































    checking exit codes. Do things l00k right?

    -----
    paulie420@pbs:~$ cat backup.sh
    #!/bin/bash

    #Backup 3 external HDDs to 16TB Drive
    retVal=$?
    rsync -av --delete /mnt/Beers4TB/ /mnt/datastore/PBS_backup/Beers4TB
    if [ retVal -ne 0 ]; then
    echo "There was an rsync error backing up Beers4TB:" >> ~/backup_info.txt
    echo $retVal >> ~/backup_info.txt
    else
    echo $(date) >> ~/backup_info.txt
    echo "/mnt/Beers4TB -> /mnt/datastore/PBS_Backup/Beers4TB COMPLETE!" >> ~/backup_info.txt
    fi
    rsync -av --delete /mnt/Backup4TB/ /mnt/datastore/PBS_backup/Backup4TB
    if [ retVal -ne 0 ]; then
    echo "There was an rsync error backing up Backup4TB:" >> ~/backup_info.txt
    echo $retVal >> ~/backup_info.txt
    else
    echo $(date) >> ~/backup_info.txt
    echo "/mnt/Backup4TB -> /mnt/datastore/PBS_Backup/Backup4TB COMPLETE!" >> ~/backup_info.txt
    fi
    rsync -av --delete /mnt/Backup6TB/ /mnt/datastore/PBS_backup/Backup6TB
    if [ retVal -ne 0 ]; then
    echo "There was an rsync error backing up Backup6TB:" >> ~/backup_info.txt
    echo $retVal >> ~/backup_info.txt
    else
    echo $(date) >> ~/backup_info.txt
    echo "/mnt/Backup6TB -> /mnt/datastore/PBS_Backup/Backup6TB COMPLETE!" >> ~/backup_info.txt
    fi
    exit $retVal
    -----



    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 2022/07/15 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (1337:3/129)
  • From paulie420@1337:3/129 to paulie420 on Thu Sep 15 17:56:33 2022
    my first attempt at checking exit codes. Do things l00k right?

    Well, I've tested and I'm getting err0rs. :/

    ./backup.sh: line 6: [: retVal: integer expression expected
    ./backup.sh: line 12: [: retVal: integer expression expected
    ./backup.sh: line 18: [: retVal: integer expression expected

    Lines 6, 12 & 18 are;

    if [ retVal -ne 0 ]; then

    Spot my issue? I ran some test scripts using the above, I thought, and both exits were detected.

    (complete script again;)
    paulie420@pbs:~$ cat backup.sh
    #!/bin/bash

    #Backup 3 external HDDs to 16TB Drive
    retVal=$?
    rsync -av --delete /mnt/Beers4TB/ /mnt/datastore/PBS_backup/Beers4TB
    if [ retVal -ne 0 ]; then
    echo "There was an rsync error backing up Beers4TB:" >> ~/backup_info.txt && echo $retVal >> ~/backup_info.txt
    else
    echo $(date) >> ~/backup_info.txt && echo "/mnt/Beers4TB -> /mnt/datastore/PBS_Backup/Beers4TB COMPLETE!" >> ~/backup_info.txt
    fi
    rsync -av --delete /mnt/Backup4TB/ /mnt/datastore/PBS_backup/Backup4TB
    if [ retVal -ne 0 ]; then
    echo "There was an rsync error backing up Backup4TB:" >> ~/backup_info.txt && echo $retVal >> ~/backup_info.txt
    else
    echo $(date) >> ~/backup_info.txt && echo "/mnt/Backup4TB -> /mnt/datastore/PBS_Backup/Backup4TB COMPLETE!" >> ~/backup_info.txt
    fi
    rsync -av --delete /mnt/Backup6TB/ /mnt/datastore/PBS_backup/Backup6TB
    if [ retVal -ne 0 ]; then
    echo "There was an rsync error backing up Backup6TB:" >> ~/backup_info.txt && echo $retVal >> ~/backup_info.txt
    else
    echo $(date) >> ~/backup_info.txt && echo "/mnt/Backup6TB -> /mnt/datastore/PBS_Backup/Backup6TB COMPLETE!" >> ~/backup_info.txt
    fi
    echo "-----" >> ~/backup_info.txt
    exit $retVal


    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 2022/07/15 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (1337:3/129)
  • From MeaTLoTioN@1337:1/101 to paulie420 on Fri Sep 16 10:58:21 2022
    On 15 Sep 2022, paulie420 said the following...

    my first attempt at checking exit codes. Do things l00k right?

    Well, I've tested and I'm getting err0rs. :/

    Try setting retVal after running the rsync, $? is only populated on completion either successfully "0" or failure "1-127" of any command.

    So, you need really;

    rsync blah blah
    retVal=$?

    do some stuff...

    if [[ $retVal -gt 0 ]]; then
    # some stuff happened but it failed
    else
    # some stuff happened and succeeded
    fi

    ---
    |14Best regards,
    |11Ch|03rist|11ia|15n |11a|03ka |11Me|03aTLoT|11io|15N

    |07ÄÄ |08[|10eml|08] |15ml@erb.pw |07ÄÄ |08[|10web|08] |15www.erb.pw |07ÄÄÄ¿ |07ÄÄ |08[|09fsx|08] |1521:1/158 |07ÄÄ |08[|11tqw|08] |151337:1/101 |07ÂÄÄÙ |07ÄÄ |08[|12rtn|08] |1580:774/81 |07ÄÂ |08[|14fdn|08] |152:250/5 |07ÄÄÄÙ
    |07ÄÄ |08[|10ark|08] |1510:104/2 |07ÄÙ

    --- Mystic BBS v1.12 A47 2021/12/13 (Linux/64)
    * Origin: thE qUAntUm wOrmhOlE, rAmsgAtE, uK. bbs.erb.pw (1337:1/101)
  • From MeaTLoTioN@1337:1/101 to paulie420 on Fri Sep 16 10:59:39 2022
    On 15 Sep 2022, paulie420 said the following...

    my first attempt at checking exit codes. Do things l00k right?

    Well, I've tested and I'm getting err0rs. :/
    ./backup.sh: line 6: [: retVal: integer expression expected
    ./backup.sh: line 12: [: retVal: integer expression expected
    ./backup.sh: line 18: [: retVal: integer expression expected


    Also, you're defining use of /bin/bash as your shebang, but using posix if statements.

    if using bash, if should be;

    if [[ $var -gt 0 ]]; then

    if using posix, if should be;

    if [ $var -gt 0 ]; then


    Hope this helps.

    ---
    |14Best regards,
    |11Ch|03rist|11ia|15n |11a|03ka |11Me|03aTLoT|11io|15N

    |07ÄÄ |08[|10eml|08] |15ml@erb.pw |07ÄÄ |08[|10web|08] |15www.erb.pw |07ÄÄÄ¿ |07ÄÄ |08[|09fsx|08] |1521:1/158 |07ÄÄ |08[|11tqw|08] |151337:1/101 |07ÂÄÄÙ |07ÄÄ |08[|12rtn|08] |1580:774/81 |07ÄÂ |08[|14fdn|08] |152:250/5 |07ÄÄÄÙ
    |07ÄÄ |08[|10ark|08] |1510:104/2 |07ÄÙ

    --- Mystic BBS v1.12 A47 2021/12/13 (Linux/64)
    * Origin: thE qUAntUm wOrmhOlE, rAmsgAtE, uK. bbs.erb.pw (1337:1/101)
  • From paulie420@1337:3/129 to MeaTLoTioN on Fri Sep 16 07:01:12 2022
    Try setting retVal after running the rsync, $? is only populated on Me>
    completion either successfully "0" or failure "1-127" of any command.

    So, you need really;

    rsync blah blah
    retVal=$?

    do some stuff...

    if [[ $retVal -gt 0 ]]; then
    # some stuff happened but it failed
    else

    Also, you're defining use of /bin/bash as your shebang, but using posix
    if statements.

    if using bash, if should be;

    if [[ $var -gt 0 ]]; then

    if using posix, if should be;

    if [ $var -gt 0 ]; then

    Ah - ok, understood; thank you for both of your replies. I'm sure I can iron it

















































































































































































































































































































    out with the info you provided. The guide I read must not have mentioned to setup using #!/bin/bash, and I was just used to doing so. Derp.



    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 2022/07/15 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (1337:3/129)
  • From Ganiman@1337:3/170 to paulie420 on Thu Nov 24 12:06:41 2022
    A little late to the discussion here, but figured I'd drop a comment.

    I use Bacula for all of my backups at home, and have used it to back up entire corporate enterprises. It will do everything you need - rotating your backups, restoring, messagine, reporting, etc - its a very good and fully featured open source backup utility. It backs up all the games I host, databases, etc. I think you like Debian and there are certainly packages for it. Bacula has 3 components to it: manager, storage daemon (where your backups go) and file daemon (the client).

    If you decide to take the plunge and try it out, I'm happy to help with questions. Just wanted to mention it. Learning bash scripting is fun too, but I












































































































































































































































    find can be cumbersome to manage when you deal with multiple systems you want to back up to a central location.

    Take care,

    ---
    Ganiman
    bbs.madetoraid.com:[2323/2222]

    --- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
    * Origin: Maid to Raid BBS (1337:3/170)
  • From paulie420@1337:3/129 to Ganiman on Thu Nov 24 12:26:46 2022
    I use Bacula for all of my backups at home, and have used it to back up entire corporate enterprises. It will do everything you need - rotating your backups, restoring, messagine, reporting, etc - its a very good and fully featured open source backup utility. It backs up all the games I host, databases, etc. I think you like Debian and there are certainly packages for it. Bacula has 3 components to it: manager, storage daemon (where your backups go) and file daemon (the client).

    Nice - I went with PBS in the end, but will take a peek at Bacula - it sounds light and full featured. Thanks for sharing. :P



    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 2022/07/15 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (1337:3/129)
  • From deon@1337:2/101 to Ganiman on Fri Nov 25 11:13:18 2022
    Re: Re: Backup script tuning
    By: Ganiman to paulie420 on Thu Nov 24 2022 12:06 pm

    Howdy,

    I use Bacula for all of my backups at home, and have used it to back up entire corporate enterprises. It will do everything you need - rotating your

    it. Bacula has 3 components to it: manager, storage daemon (where your backups go) and file daemon (the client).

    I think the days of having a "server" and "manager" for back up are coming to an end.

    I use restic - and it can send it's backup data to multiple destinations, but I











































































































































































































































    just point it at an S3 bucket (on my NAS with minio), and it can backup files or anything sent to STDIN. (It can do also purge old backups based on a few conditions.)

    Together with cron, I have scheduling and a little bash foo, it pauses my docker containers, backups them up (well the data files part) and resumes them.











































































































































































































































    And a bit more foo, it dumps the databases directly ito it via STDIN.

    Thought I'd share :)


    ...ëîåï
    --- SBBSecho 3.15-Linux
    * Origin: I'm playing with ANSI+videotex - wanna play too? (1337:2/101)
  • From Bucko@1337:3/102 to paulie420 on Thu Nov 24 20:53:25 2022
    On 24 Nov 2022, paulie420 said the following...

    Nice - I went with PBS in the end, but will take a peek at Bacula - it sounds light and full featured. Thanks for sharing. :P


    Another to look at is VEEAM Free Agent, I use that for my Linux machines, it automatically backs up my Linux machines at specific times, rotates backups, and can restore a full machine or individual files very easily. I was using Active Backup for Business on my Synology NAS to backup all of my machines both











































































































































































































































    Windows and Linux but every time the kernal on Linux changes it breaks ABB so I











































































































































































































































    have stopped using that for my Linux machines. Veeam works real well for me and











































































































































































































































    I don't have to worry about backups at all.

    AL

    ... My software never has bugs. It just develops random features...

    --- Mystic BBS v1.12 A48 2022/07/15 (Linux/64)
    * Origin: The Wrong Number Family Of BBS' - Wrong Number ][ (1337:3/102)
  • From Ganiman@1337:3/170 to deon on Thu Nov 24 21:11:10 2022
    I think the days of having a "server" and "manager" for back up are
    coming to an end.

    I don't think I agree with this. Almost anything on-prem will still use this model and its not really going away. If you run things on cloud platforms, backups are just snapshots or are provided by other cloud native services, but for on-prem this is still very much the thing to do. Custom bash scripts that rely on cron jobs is not scalable to many nodes and gets very ugly to manage especially if you have more than one person resposible for it.

    You can customize bacula to stop your docker containers and do any pre or post scripts/commands.

    Backups are necessary. You will always need something to manage them, to schedule them, to point at storage devices/accounts, etc. Your bash scripts are











































































































































































































































    just recreating this. By all means, do it your way if it works for you, but to say this is "coming to an end" is inaccurate.

    ---
    Ganiman
    bbs.madetoraid.com:[2323/2222]

    --- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
    * Origin: Maid to Raid BBS (1337:3/170)
  • From Ganiman@1337:3/170 to Ganiman on Thu Nov 24 21:12:39 2022
    Figured I'd mention, if you are shipping backups off site, consider using Glacier instead of S3. Its really designed for that kind of use case and will save you some money especially if you almost never restore from your backups.

    ---
    Ganiman
    bbs.madetoraid.com:[2323/2222]

    --- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
    * Origin: Maid to Raid BBS (1337:3/170)
  • From deon@1337:2/101 to Ganiman on Fri Nov 25 16:46:32 2022
    Re: Re: Backup script tuning
    By: Ganiman to deon on Thu Nov 24 2022 09:11 pm

    Howdy,

    I think the days of having a "server" and "manager" for back up are coming to an end.

    are just recreating this. By all means, do it your way if it works for you, but to say this is "coming to an end" is inaccurate.

    Actually, I understand that you have a view that you dont agree with me. I work











































































































































































































































    in the business of protecting enterprise data and have done for many years, so I do think it is accurate (hence why I mentioned it).

    I see what clients are asking for, and the legacy "backup server" and "backup client" is not as popular as it was 20+ years ago.

    When I look at alternative backup technologies, I *can* see that the "backup server" is less of a requirement today than it was, because there now exists new storage technologies that change the architecture and cost of having said architecture, but still deliver on the requirements.

    True, enterprise scheduling, as is scalability is a requirement, and there are strong technologies in that space as well, that organisations are using anyway outside of "backup and recovery".

    Let see how this pans out...



    ...ëîåï
    --- SBBSecho 3.15-Linux
    * Origin: I'm playing with ANSI+videotex - wanna play too? (1337:2/101)
  • From Roon@1337:1/112 to paulie420 on Fri Nov 25 13:26:06 2022
    Hello paulie420,

    24 Nov 22 12:26, you wrote to Ganiman:

    I use Bacula for all of my backups at home, and have used it to
    back up entire corporate enterprises. It will do everything you
    need - rotating your backups, restoring, messagine, reporting,
    etc - its a very good and fully featured open source backup
    utility. It backs up all the games I host, databases, etc. I
    think you like Debian and there are certainly packages for it.
    Bacula has 3 components to it: manager, storage daemon (where
    your backups go) and file daemon (the client).

    Nice - I went with PBS in the end, but will take a peek at Bacula - it sounds light and full featured. Thanks for sharing. :P

    i'm using safekeep (safekeep.sf.net) which combines rdiff-backup and rsync. lightweight solution :)

    Regards,
    --
    dp

    telnet://bbs.roonsbbs.hu:1212 <<=-

    ... 3:21pm up 7 days, 9:43:03, load: 78 processes, 280 threads.
    --- GoldED/2 1.1.4.7+EMX
    * Origin: Roon's BBS - Budapest, HUNGARY (1337:1/112)
  • From Ganiman@1337:3/170 to deon on Fri Nov 25 14:17:18 2022
    Actually, I understand that you have a view that you dont agree with me.
    I work in the business of protecting enterprise data and have done for many years, so I do think it is accurate (hence why I mentioned it).

    I see what clients are asking for, and the legacy "backup server" and "backup client" is not as popular as it was 20+ years ago.

    When I look at alternative backup technologies, I *can* see that the "backup server" is less of a requirement today than it was, because
    there now exists new storage technologies that change the architecture
    and cost of having said architecture, but still deliver on the requirements.

    True, enterprise scheduling, as is scalability is a requirement, and
    there are strong technologies in that space as well, that organisations are using anyway outside of "backup and recovery".

    I too support IT enterprises and have done so for the last ~25 years. I agree that things have changed, but they have also very much stayed the same. We don't use tapes anymore and "off site" is now "the cloud" and not Iron Mountain











































































































































































































































    (although they probably still exist too). You specifically mentioned the the model of a manager/files/storage has gone/is going away, while at the same time











































































































































































































































    explaining that your bash scripts and cron jobs are doing the exact same thing.











































































































































































































































    So I still feel that what you said is inaccurate - its not wrong, its just not entirely correct. You still need *something* to schedule your backups, something to fetch what you want to back up and a place to put them. That does not go away, and tools like Bacula and others fit very well into the modern version of this believe it or not.

    I doubt your customers are saying they don't want an old model or they specifically want a new model of backups - they just want their backups to work











































































































































































































































    and have come to you so they don't have to think about it, and are probably not











































































































































































































































    being very specific about how that is done. If they are being that specific about their backup strategy... yikes, tough customers.

    By using your own bash scripts and cron jobs to achieve the same goal, you are missing out on a lot of very valuable tools that come along with it - the reporting, messaging/alerting, and a number of other features that can be useful to even home users would have to be re-written and you're collection of scripts is just going to fill the same hole that software like bacula fills. I like getting emails when jobs fail and I like having a dashboard to show me status, storage capacity, etc - bash scripts aren't going to do all
    of that extra stuff without a whole lot of time and work.

    Even when I write terraform to do large deployments, I'm still setting schedules, rotations, vaults, etc for my backups - its the same things being configured that bacula does, just in the cloud, and still using them to recover











































































































































































































































    when things go bad (which honestly is extremely rare). Sometimes this is in the











































































































































































































































    form of snapshots even, which is really not traditional, but still circles back











































































































































































































































    to the same methodology.

    Am I going to use Bacula to backup my dev team's kubernetes clusters? Nope, never, but I'm not using bash scripts and cron jobs either. Am I going to use it to backup on-prem production or Mystic or the games I run and the dozens of other things I do for fun, even docker containers? Yes.. absolutely - I'm not dealing with straight bash scripts and cron jobs for those tasks anymore. I've done it and I'd rather not reinvent the wheel again. Could I use my NAS with a built in backup feature? Probably, but its not really any different.

    A lot of people around these echoes seem be running their own "cloud" at home, and they don't have the tools that Azure or AWS provides for backups. They might not have a NAS with a built in feature. I still hold the opinion that straight bash scripts vs a mature tool to do the job is going to be more of a headache for most people, especially if you're not the only one managing it. Even if you are using xen-orchestra or something else to automate your snapshots, a proper backup tool will still provide value, and of course can still ship your backups to S3(?)/Glacier or an Azure storage vault/blob.

    I think if we are trying to share information with people who are learning, that it is good to be fair about these topics. Custom bash scripts and cron jobs can work, but lets not say that they are replacing the more mature tools or that they are going away in favor of cron jobs and scripts.

    We are a linux shop, but we moved from AWS to Azure in the last couple of years











































































































































































































































    and Azure has some good documentation on exactly this - backing up on-prem to the cloud, etc. The pictures are still backup/restore. I had to double check before responding to make sure I wasn't losing it because you said 'outside of "backup and recovery"'. I don't love Microsoft and am not an Azure fan really, but if their docs are still saying it so will I.

    I hope that all makes sense to you. I'm just sharing my own experience being involved in this for so long.

    ---
    Ganiman
    bbs.madetoraid.com:[2323/2222]

    --- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
    * Origin: Maid to Raid BBS (1337:3/170)
  • From MegaBrutal@1337:2/104 to paulie420 on Sun Dec 18 00:55:04 2022
    Hello,

    Not sure if the question is still actual, but here are some quick notes:

    1. Try to write POSIX compliant scripts those don't depend on Bash. Use checkbashisms to check whether you are using any Bash-only features. Then you can safely use the #!/bin/sh shebang. (Unless you really need some Bash features, but then you should declare that your script requires Bash with the #!/bin/bash shebang. Unlikely you'll need Bash for a backup script)

    2. Use shellcheck to validate your code. It helps you point out gotchas you might not think of.

    3. This is just my taste, but for FS I need regular backups from, I'd prefer
    to use BTRFS, then my script would make a snapshot of the subvolume and transfer it to another disk via the btrfs send command. Not only it would be more error-prone(as an entire subvolume would be transferred differentially
    (in case you use -p to specify your previous snapshot to use as a base) which either fails or succeeds, but you could save previous states of your files snapshotted at specific points in time without using significantly more disk space.

    4. If I can't use BTRFS but need differential archives, I prefer to use DAR.

    Ask if you want me to elaborate or anyone disapproves. ;)


    ~ MegaBrutal (new on this BBS)

    --- Mystic BBS v1.12 A46 2020/08/26 (Linux/64)
    * Origin: Sent from Al's Geek Lab 1337:2/104 (1337:2/104)
  • From paulie420@1337:3/129 to MegaBrutal on Sat Dec 17 12:14:50 2022
    1. Try to write POSIX compliant scripts those don't depend on Bash. Use checkbashisms to check whether you are using any Bash-only features.
    Then you can safely use the #!/bin/sh shebang. (Unless you really need some Bash features, but then you should declare that your script
    requires Bash with the #!/bin/bash shebang. Unlikely you'll need Bash
    for a backup script)

    2. Use shellcheck to validate your code. It helps you point out gotchas you might not think of.

    3. This is just my taste, but for FS I need regular backups from, I'd prefer to use BTRFS, then my script would make a snapshot of the
    subvolume and transfer it to another disk via the btrfs send command.
    Not only it would be more error-prone(as an entire subvolume would be transferred differentially (in case you use -p to specify your previous snapshot to use as a base) which either fails or succeeds, but you could save previous states of your files snapshotted at specific points in
    time without using significantly more disk space.

    4. If I can't use BTRFS but need differential archives, I prefer to use DAR.


    While my backups are being handled via another method now, I appreciate you explaining some tips. I've always used /bin/bash, but only b/c thats how I've been taught. I'm still lingering on EXT4, but I know... I need to take the dive





















































































































































































































    into something newer. One day.

    Thanks for the writeup, I've extracted the text and it'll remain in my helper folder.



    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 2022/07/15 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (1337:3/129)