Watch-folder - Google

Google Cloud Storage is a RESTful online file storage web service for storing and accessing data on the Google Cloud Platform infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities. Data (objects) stored in GCS buckets belong to a project consisting of a set of users, APIs, billing, authentication, and monitoring settings for those APIs.

When trying to process a larger volume of files through Telestream Cloud one of the first things you will want to do is to automate your workflow. The easiest way to do it is to use watch folders.

The idea behind it is very simple. You specify a storage location and tell Telestream Cloud to monitor it for new files. For example, you may create an S3 store in our web console, then enable it as your input store. Any file uploaded to this location will automatically trigger media processing workflow, e.g. start a transcode using Flip service and delivery to output store once finished.

Create a Project and GCS bucket

If you have an account with Google Cloud, log in to your account and either [create a project for Telestream Cloud](go to https://console.cloud.google.com/projectcreate) or pick one of your existing projects.

Now it time to create a bucket within the project to store your data. From the side menu pick Storage > Browser.

Click "Create Bucket" and follow the instructions.

Give the bucket a name, then choose where to store your data. You have 3 choices here:

Region: data is stored in one selected geographical location, lowest cost
Dual-region: data is stored in 2 selected locations for higher availability and lower latency
Multi-region: data is stored in multiple locations, highest availability and lower latency but higher cost

Pick the option that makes most sense for your use case.

Step 3 is setting storage class for your data. A storage class sets costs for storage, retrieval, and operations depending on how often you access your data. The more frequently the higher the cost. Leaving it at default setting of Standard works best for most use cases.

Click "Continue" to move on to access control configuration. Pick Fine-grained for access control

Click "Create" - you may ignore Advanced settings section for now - to finish creating the bucket. Let's have a look now on how to set permissions for it and obtain Access Key and Secret required to add it to Telestream Cloud.

Granting access in setting permissions

We strongly recommend to create specific Google Service Account that will have access only to this bucket instead of using your main (owner) account. Such account will have its own pair of Access Key and Secret that you can use to grant us access to the bucket.

Creating Service Account

When logged-in to your Google Cloud console and from left side menu select IAM & Admin > Service Accounts and click "Create Service Account".

Give it a meaningful name and copy created Service account ID. You will need it later when assigning this service account with your bucket.

You can ignore next 2 steps, as they are optional. Click "Done" and your new Service Account should be created and visible on the list.

Assigning Service Account to your bucket

Go back to Storage Browser and click on the name of the bucket you would like to use.

You will now see bucket properties grouped in 4 tabs. Go to Permissions tab. This is where all users and service accounts that have access to the bucket are visible.

Click Add members and in config form paste the Service account ID and select role - Storage Admin. This will grant the service account full access to this bucket and its objects. Click Save.

Generating access keys

Now, from left side menu in Google Console go to Storage > Settings and choose Interoperability tab. This is where you will generate Access Key and Secret for this service account that will be needed to add it to Telestream Cloud.

Click "Create a key for service account" and in the modal dialog select service account for which to generate keys and click "Create Key".

A unique pair of two keys - Access key and Secret will be created just for this Service Account. While you can always check Access Key in the web console, Secret will be shown only once in this dialog, so make sure you have it stored securely somewhere.

Now open new tab and log-in to your Telestream Cloud account.

Adding GCS bucket to Telestream Cloud

Go to Manage Stores in Telestream Cloud web console (top right menu) and Click Add Store. Select Google Cloud Storage and paste both keys in their respective fields and enter name of the bucket you wish to use.

Check off "Configure as Input Store"
Click "Add Store" to finish. Your new GCS storage is ready to use.

Configure Watch Rules

Watch rules in general define which Factory or Project in any Telestream Cloud will use the watch-folder as source and allow you to control what types of files are going to be processed. To define a watch rule click "+" button in the Watch Rules column on the list of stores.

This will take you to the Add Watch Rule config view. Pick the service and a Factory/Project that should use this watch-folder and optionally define a path pattern. If you leave it blank, any file that arrives at this watch-folder will trigger a job.

However, if you need more control over what files be processed you can use regular expressions to match specified file types or folders within a bucket:

match only files of a certain type to be processed:
*.(mxf|mp4) will match any file with mxf or mp4 extension
match certain file types in specific folders within the bucket:
folder/${dir1}/${dir2}/.mp4*
match video and captions files from different folders:
folder/${dir1}/${dir2}/${name}.mp4
folder/${dir2}/${dir1}/${name}.scc

You can create multiple patterns for each watch rule.

If you have existing files already in the watch-folder and you'd like to process them, check "Sync Existing Files". Be careful though, as this will automatically trigger jobs. If you have a large number of files to process this can be time-consuming.

Click "Create Rule" to add it to your watch-folder. All you need to do now is to upload the first source file to your GCS watch-folder bucket.