Orchestrating the Cloud

There are probably more cloud-based services providing functional capability for media operations than for any other commercial sector.  These services range from simple format conversion (transcode) to complex AI services which can automatically create temporal logs that ‘describe’ content based on object or other recognition models.  The aggregate of even a small selection of the available services is extremely powerful.  Cloud-provided service functionality has the capability to transform the efficiency of production and content supply chain operations.  Aside from the breadth of functionality available ‘in the cloud’ the structure and fabric of cloud provides direct benefit for media businesses:

• • Scale, infinite scale for material storage and processing.
  • Resilience, at a fractional incremental cost.
  • Security, cloud affords greater security and auditability than on-prem storage.
  • Access, global access to content enabling remote and collaborative operations.
  • Economical, almost universally costs are on a consumption basis.

• Interoperability, through standardised control interfaces*.

*A very welcome development in the broadcast and media industry.

Whilst there has been a significant increase in the use of cloud for production and content preparation workflows, broadly take-up has not been as rapid as it might have been. Aside from the migration issues relating to file size, covered here, the key reasons cited by media companies distil into three headlines: visibility, cost, and control.

These are genuinely held concerns; below are some real-world statements from media operators and details of how BLAM’s functionality answers the issue.

Visibility & Discovery

“we recently put over a petabyte of media into the cloud as a backup / safety copy but now cannot see what we have without restoring it”

This is a simple one for BLAM. The BLAM UI makes all media objects visible and ‘playable’.  For video assets, BLAM creates a full resolution browse / proxy version of all material that is brought into the platform so that it may be played within a standard web-browser.

The browse / proxy file is typically encoded at 2Mb/s (this is configurable) so the file stored on disk is between 0.5-4% of the original / master file size. If you are using AWS S3 storage, this means that only the small browse / proxy file is required to ‘sit’ at Standard Class.  The much larger broadcast or mezzanine format may be archived to the Deep or Glacier class in AWS parlance.

The ability to see media and the relationships between media in this way means that cloud storage and services may be used in an extremely cost-effective way.

Content accessibility is enhanced with discovery tools – principally this means search. BLAM supports accessibility of media through a fast and sophisticated search capability underpinned by advanced metadata management for structured and unstructured data.  BLAM provides:

• Metadata synchronisation with live systems such as traffic or rights management.
• Connectors to a range of legacy platforms for bulk data migration.
• Generation of temporal metadata generation using AI and other services.
• A powerful search capability to support simplified discovery.
• Asset relationship management capability.
• Cloud storage tagging tools to persist data at the object level.

“With all the promise of scalability, our operation still suffers bottlenecks but, in the cloud, ironically, it is more difficult to see where….” 

Another form of visibility that has been designed into BLAM is real-time workflow monitoring. BLAM provides real-time status information for all processes that it is orchestrating – both for internal processes, such as checking technical data on a file, or long-running processes, such as transcoding.  BLAM provides a single interface through which multiple services from multiple vendors (i.e. multi-cloud) can be monitored and managed.  Crucially, BLAM supports the “management by exception” paradigm through alerts in the case of error. Such alerts provide specific detail so corrective action may be enacted quickly.

Cost

“It is extremely difficult to calculate how much as specific asset or job has cost us, particularly as we use the services of multiple providers”

The fundamental of finance control, when using a consumption-based model, is to understand the aggregate cost of performing a specific function.  This calculation is complex when using the services from several vendors – the multi-cloud problem.   BLAM tracks fixed and variable costs based on transactions across all services, regardless of provider, and can compute these to provide meaningful management information.

Finance in BLAM is based on transaction units which are calibrated as time (seconds) or size (bytes) although these can be customised.  Rates are manually set or applied to transactions dynamically through a service API.  Calculating costs is then as simple as multiplying the rate by transaction units (which are tracked within each BLAM Workflows BLidget).  Tracking transaction units at the BLidget level means that BLAM records data at the most granular level possible.  Cost data may then be presented in an entirely customisable way, be it in a graphic form in a BLAM dashboard or in an Excel-based financial report.  A common requirement in the content supply chain context is to see the cost of delivering a media package which includes a breakdown of all the services involved so that this may be recharged.  An equally common request is to see the total cost of ownership information.  BLAM now displays this as standard in the Media view.

“It’s the egress costs, that is where they catch you out”

The fact is that storing large media files in AWS S3 at Standard class is expensive.  It is also a fact that egress costs for large files are not trivial – although this aspect is under increasing commercial pressure and some smaller vendors are now offering zero egress fee models. =Some of the techniques we use to exploit tiered storage and minimise egress include:

• Store objects at S3 Standard class only while they are needed.
• Avoid moving objects out of S3 – bring services (transcode, AI, etc.) to the objects.
• Archive objects as soon as direct access is not required.
• Use slower restore speeds to reduce costs – let the workflow do the planning.
• Following a restore, retain objects at Standard class until all process have completed.
• Use partial restore functions to reduce the data retrieval and therefore cost.

BLAM provides tools to support all these techniques in the workflow builder. The BLAM finance tools provide an early warning – i.e. before the billing statement arrives – if an overall workflow is not as efficient as it should be.

Control

“We are going to have to establish a technology group in order to choreograph the services we need”

Absolutely not. There has been much emphasis within the industry on the fallacy that media companies need to establish a ‘dev ops’ (development operations) capability or become a technology company to access the benefits of the cloud.  In our view, this runs entirely contrary to the accessibility model which the cloud affords.  BLAM has been specifically designed to enhance that accessibility.

BLAM provides a ‘no-code’ development environment for analysts and system administrators to directly build operationally specific workflows using an easy to use drag-drop-connect interface.  BLAM workflows are built by connecting BLidgets (Blue Lucy widgets) together to perform a given set of functions.  The BLidgets ‘talk’ directly to the services through their individual APIs and pass through relevant data (which may be asset specific or set within the simple configuration in the BLidget). The BLidget then monitors the operation of the 3^rd party service and reports this in a monitoring view, in real time.

BLAM currently supports more than 200 functional BLidgets, including connectors to cloud services, such as AI based speech to text or automatic quality control systems, as well as established production tools such as Avid Media Composer and Quantel (GV).

BLAM affords media companies complete control over cloud-based services from multiple providers without the need for any specialist technology resource.

BLAM provides an easy to use toolset to allow media operators to benefit from cloud based services in a controlled way without needing to spend on creating a technology department. This article is based on Managing Media in a Multicloud Environment presented at the Netorium NETS event 2020.  That presentation is available here.