Using Google Cloud Platform's Nearline storage for cheap Linux backups.
- Sign up to Google Cloud Platform
gsutilto create our API credentials
- Create our Nearline Bucket
- Do an initial sync using
gzutil’s rsync method
Sign up and initial configuration
Sign up to Google Cloud Platform at https://cloud.google.com/
You should be redirected to console.developers.google.com, your first step is to create a project. I’ve called mine
Don’t create any storage buckets or additional credentials yet, we’ll create them via
My instructions install gsutil to /usr/local/gsutil, change this path to any path you prefer.
# wget https://storage.googleapis.com/pub/gsutil.tar.gz # tar xzf gsutil.tar.gz -C /usr/local/ # echo 'PATH=$PATH:/usr/local/gsutil' > /etc/profile.d/gsutil.sh
gsutil config will provide a URL, open this URL in a browser and login using the Google account you’d like
to use for storage and billing. You’ll be asked to authorise the request and once completed the website will give you a
token to copy and paste back into
gsutil. It’s straight forward.
# gsutil config This command will create a boto config file at /root/.boto containing your credentials, based on your responses to the following questions. Please navigate your browser to the following URL: https://accounts.google.com/o/oauth2/<snip> In your browser you should see a page that requests you to authorize access to Google Cloud Platform APIs and Services on your behalf. After you approve, an authorization code will be displayed. Enter the authorization code: <some key> Please navigate your browser to https://cloud.google.com/console#/project, then find the project you will use, and copy the Project ID string from the second column. Older projects do not have Project ID strings. For such projects, click the project and then copy the Project Number listed under that project. What is your project-id? symmetric-index-<some id> Boto config file "/root/.boto" created. If you need to use a proxy to access the Internet please see the instructions in that file.
Create a bucket, see
gsutil help mb to see a complete list of options, such as specifying bucket region.
# gsutil mb -c nearline gs://<bucket_name>
Now, perform your initial rsync
# gsutil -m rsync -r /<directory> gs://<bucket_name>
-m option runs a parallel rsync
For future backups, use the
-q option to hide all output but errors, this is useful for cron, so it will only email if
an error occurs.
# gsutil -qm rsync -r /<directory> gs://<bucket_name> `` Faster CRC32 Checksums ====================== Note, by default it's likely rsync will use a slow method to calcualte CRC32 checksums. For a faster method it's recommended to
$ gsutil ver -l | grep crcmod ```
If the output shows
compiled crcmod: False, then install the compiled module by following the instructions in
help crc32c - which essentially uses
pip to install