We backup over the weekend our full backups to a backup server with a large EBS volume. Once this is complete we have a second step to invoke a powershell script remotely on the backup server that uses Rclone to sync the backups to S3. Perhaps this is a daft idea - and I'd be more than willing to hear what other suggestions you have.
However, there are a few problems with this. Firstly, it takes soooo long - and at scale this isn't going to work. Secondly when the powershell script invokes on the backup server, if the rclone stops working for whatever reason the SQL job just assumes it has completed successfully. So my question is, is there a way for the backup job to be "aware" of whether the rclone S3 sync has completed successfully or not - and obviously if it fails then the job is considered a failure and notifies accordingly.
Any ideas or suggestions?
thanks.