Overview

  • Sign up to Google Cloud Platform
  • Install gsutil command
  • Use gsutil to create our API credentials
  • Create our Nearline Bucket
  • Do an initial sync using gzutil’s rsync method

Sign up and initial configuration

Sign up to Google Cloud Platform at https://cloud.google.com/

You should be redirected to console.developers.google.com, your first step is to create a project. I’ve called mine Linux Backup.

Don’t create any storage buckets or additional credentials yet, we’ll create them via gsutil.

Install gsutil

See Also: https://cloud.google.com/storage/docs/gsutil_install

My instructions install gsutil to /usr/local/gsutil, change this path to any path you prefer.

# wget https://storage.googleapis.com/pub/gsutil.tar.gz
# tar xzf gsutil.tar.gz -C /usr/local/
# echo 'PATH=$PATH:/usr/local/gsutil' > /etc/profile.d/gsutil.sh

Configure gsutil

See Also: https://cloud.google.com/storage/docs/gsutil_install#authenticate

Running the gsutil config will provide a URL, open this URL in a browser and login using the Google account you’d like to use for storage and billing. You’ll be asked to authorise the request and once completed the website will give you a token to copy and paste back into gsutil. It’s straight forward.

# gsutil config
This command will create a boto config file at /root/.boto containing
your credentials, based on your responses to the following questions.
Please navigate your browser to the following URL:
https://accounts.google.com/o/oauth2/<snip>
In your browser you should see a page that requests you to authorize access to Google Cloud Platform APIs and Services
on your behalf. After you approve, an authorization code will be displayed.

Enter the authorization code: <some key>

Please navigate your browser to https://cloud.google.com/console#/project,
then find the project you will use, and copy the Project ID string from the
second column. Older projects do not have Project ID strings. For such projects,
click the project and then copy the Project Number listed under that project.

What is your project-id? symmetric-index-<some id>

Boto config file "/root/.boto" created. If you need to use a proxy to
access the Internet please see the instructions in that file.

Create Backups

Create a bucket, see gsutil help mb to see a complete list of options, such as specifying bucket region.

# gsutil mb -c nearline gs://<bucket_name>

Now, perform your initial rsync

# gsutil -m rsync -r /<directory> gs://<bucket_name>

The -m option runs a parallel rsync

For future backups, use the -q option to hide all output but errors, this is useful for cron, so it will only email if an error occurs.

# gsutil -qm rsync -r /<directory> gs://<bucket_name>
``

Faster CRC32 Checksums
======================

Note, by default it's likely rsync will use a slow method to calcualte CRC32 checksums. For a faster method it's
recommended to

$ gsutil ver -l | grep crcmod ```

If the output shows compiled crcmod: False, then install the compiled module by following the instructions in gsutil help crc32c - which essentially uses pip to install crcmod32.