Batch Ship Data to Google Team Drive

A few weeks ago, I wrote a brief bit about capturing webcam images. This post expands on the initial process: specifically, what to do with the images once procured.

The first few times we captured webcam images for a time-lapse video, I'd just let the process run and, when the capture period was complete, zip up the whole thing, download it, and send wherever necessary. While that works just fine as a one-off, it didn't work out so well when we had multiples running either in sequence or simultaneously. Technically it still worked just fine, but keeping track of the details was just unnecessary work.

Enter...The Google

I'd been playing around with the Google Developer Console for a time, thinking about ways to leverage the various Google REST APIs to automate stuff. This seemed like the perfect option, especially as we'd just started really using Google Team Drives (now Shared Drives) across the enterprise. We'll let the sharing and access stuff be handled by the drive managers, and use a "service" account to upload the files via the Drive API. Ultimately, using this process totally removes the IT folks (me) from the management process, once set up (and to decommission). I don't have to be in the middle of sending data to folks, because I don't add value to the process. Sure there's some setup and tear-down, but those are simple and can be done as necessary (not necessarily on demand).

While the code I've displayed below uses Team/Shared Drives, it will also work for standard Google Drive folders as well (with minor modification).

Requirements

This was all built out in a standard LAMP environment, though Apache and MySQL are not required.

Further, you'll need to create a developer project and obtain the proper credentials from the Google Developer Console. I used a specific service account with key, but there are ways to do this process tied directly to an individual account (OAuth with Google). Using a specific service account requires the Team/Shared Drive (or destination folder) to be shared with the service account for write access.

Lastly, for any of this to work in PHP, you'll need to obtain the Google PHP API Client Library via your preferred method.

Creating the Connection

Once the libraries, configuration, and account credentials are all ready to go, it's more or less a matter of invoking the client:

<?php
require_once '/google/vendor/autoload.php';
putenv('GOOGLE_APPLICATION_CREDENTIALS=/path/to/google/api/credentials.json');
/**
* Returns an authorized API client.
* @return Google_Client the authorized client object
*/
$client = new Google_Client();
$client->setApplicationName('YourAppNameHere');
$client->setDeveloperKey('YourDeveloperKeyHere');
$client->useApplicationDefaultCredentials();
$client->addScope(Google_Service_Drive::DRIVE);
$service = new Google_Service_Drive($client);
?>

This block will initiate everything with your details and give you a $service for the Google Drive API. Pretty simple!

Preparing and Sourcing Files and Destinations

We have to specify the source and destination for the files. Keep in mind that this particular process was obtaining files with a date string coded into each file name. More details on why this was A Thing at the end of this post.

// Destination Team Drive ID and Local File Path
$teamDriveId = 'YourTeamDriveIDHere';
$localFilePath = '/path/to/localfiles';
// Find Yesterday's Files (files are named "sc-<mm>-<dd>-<yy>-<hh>-<ss>.jpg")
$yesterday = date("m-d-y", strtotime("yesterday"));
$yesterdaysImages = glob("$localFilePath/sc-$yesterday*.jpg");

If It Fits, It Ships!

The final step in the process is to simply iterate through the resulting array and ship the data to the Team/Shared Drive:

// Now Handle File Upload to Team Drive
foreach ($yesterdaysImages as $yesterdayImage) {
    $content = file_get_contents($yesterdayImage);
    $teamDriveFilename = pathinfo($yesterdayImage)['basename'];
    // Generate Metadata
    $fileMetadata = new Google_Service_Drive_DriveFile(array(
        'name' => $teamDriveFilename,
        'supportsTeamDrives' => true,
        'teamDriveId' => $teamDriveId,
        'parents' => array($teamDriveId)));
    // Create File
    $file = $service->files->create($fileMetadata, array(
        'data' => $content,
        'mimeType' => 'image/jpeg',
        'uploadType' => 'multipart',
        'supportsTeamDrives' => true,
        'fields' => 'id'));
}

You'll likely find much help in troubleshooting by referencing the Drive API documentation. This is especially true with regard to that magic little JSON nugget of 'supportsTeamDrives' => true, which is often the key component (along with 'supportsTeamDrives' => true) to make a regular Google Drive API call work in a Team/Shared drive context.

That's All! Really?

The bits highlighted above and my corresponding GitHub repo for the project are really enough to get you started down this path to automation. It's worked very well for me since I can work with a client to get things configured and started. They can then be fully responsible for handling the output.

Miscellaneous Things to Note

As noted above, this process matches by looking for a date string encoded into the file name, and not the file's datestamps proper (created, modified). Originally I wrote up a super cool way to leverage the file's metadata itself, which was far more powerful at a different scale. However, in testing I discovered a critical error for the specific environments I was using. There was another data transport between the sourcing and transport environments due to a specific DMZ-sort of isolation, and this intermediary transport was also a batch process. The resulting intermediary transport effectively "reset" the file timestamps and caused other issues with that plan.

Since we already had the date/timestamps in the file names themselves, it was just as simple to match against the names. Doing so also didn't add complexity to the processing.

There's also zero error handling in this process, and no clean-up mechanism for the source files. I'd baked in a bit of a log-style debugger originally, but what I saw in our test and production environments was a decidedly smooth and error-free upload process. If you were to run something like this indefinitely or for a longer period of time than a few days or weeks, adding some failsafe mechanisms would be prudent.

Hopefully you find this sort of thing interesting and useful. I've discovered that just poking around at Google's REST APIs has led to a number of other thoughts and ideas. There's a lot of cool stuff you can do!

Headline image via kisspng