Library Migration made easy with BLAM

When PLAZAMEDIA, a leading provider of smart solutions for media content, sought to migrate their long-term archive away from aging legacy systems they chose Blue Lucy’s BLAM as the platform and netorium as the application integrators for a modern, cost-effective solution.

PLAZAMEDIA is a leading sports TV producer in the German-speaking world and an established provider of digital content solutions for all media platforms. They have provided a range of services designed to support business operations along the entire content value chain, from their base in Munich since 1976. PLAZAMEDIA’s clients include public and private broadcasters, platform operators, international sports federations, and sports rights agencies, with services ranging from OB-based and remote production, staging live events, content management, editing, archiving and content distribution.

Project Scope

The broad requirement for the project was straightforward: to migrate the long-term, deep archive of around 2.5PB of media from an Avid MAM (which utilised an SGL Flashnet Hierarchical Storage Management system and Quantum Scalar i6000 LTO library for media storage) to BLAM, managing a brand-new object-based disk storage from NetApp. Notably, PLAZAMEDIA provide production services to several operators and, as such, there is a need to ensure the content is kept separate but on shared storage resources. PLAZAMEDIA required a multi-tenant media operations management system with sophisticated migration tools.

Multi Tenant 

BLAM enables service providers such as PLAZAMEDIA to offer truly partitioned services to their customers within a single platform which shares resources such as storage, media transformation (transcode), packaging and accelerated transfer.

This is possible because BLAM-3 has been developed from the outset as a multi-tenant platform with an organisational hierarchy model of: Organisation > Groups > Users, with users belonging to one or more groups and an organisation supporting multiple groups.  Logically, organisations are entirely separate meaning that BLAM may be configured to support multiple tenants, in this case PLAZAMEDIA customers.

Integrations

The most significant business gains that may be realised from deploying new technology come from the integration of systems – that is particularly the case in media technology.  The PLAZAMEDIA archive migration required several integration connector BLidgets, for the short-term migration of media as well as to support the ongoing operation.

Avid MAM

The Avid MAM at PLAZAMEDIA is of the Blue Order vintage, developed at a time before open and well documented APIs were common practice.  This presented a potential technical difficulty to overcome. However, given that the Avid MAM was due for retirement on completion of the project, the data only needed to come in one direction – outwards.  As such we developed the BLidgets not as dynamic bi-directional connectors as we would normally but as data readers from a raw dump of the Avid DB.  This made things a little less complex and two BLidgets were developed to manage the data migration.

The first and most complex connector BLidget extracts a list of assets from a dump of the Avid BD file and creates a BLAM Placeholder for each.  The BLidget has a filter in the reader which enables the asset records for each PLAZAMEDIA client to be imported individually to support a controlled migration.  The second BLidget then imports the metadata, including the asset location data for the BLAM Placeholders created in the first step.

Without documentation as to the structure of the Blue Order DB required a ‘reverse engineering’ method but, by taking this approach, the overall data migration may be re-run which means that BLAM now has ‘off the shelf’ BLidgets ready for other customers who may wish to migrate from the legacy Avid / Blue Order MAM platform.

SGL FlashNet

This integration was more straightforward because SGL-FlashNet has an API against which the BLidget could be rapidly developed.  In this case, the BLidget developed by Blue Lucy is a ‘proper’ connector which simply allows a given file, referenced by an SGL GUID, to be restored from LTO and placed on a working cache.   A subsequent BLidget in the workflow is then responsible for archiving the asset to the new NetApp StorageGRID object storage. Interaction with NetApp is also through a true REST API.

Snell Morpheus

As well as providing archive library management capabilities, PLAZAMEDIA’s legacy Avid MAM also managed the transfer of assets to the Snell Morpheus playout system.  In order to replicate this functionality, we developed a BLidget to perform this function. Like all new BLidgets developed to deliver a specific project the Morpheus connector is available ‘off the shelf’ for other customers.

 

NetApp StorageGRID

Since NetApp is a modern platform which supports Amazon’s Simple Storage Service (S3) API, integrating with the NetApp StorageGRID object-based storage was the most straightforward of all the integrations. In development this required only a minor modification to the existing AWS S3 connector BLidget, but this minor tweak provides an operationally powerful integration.

A system administrator may set up a storage bucket of a given capacity for each customer (tenant) within NetApp, and mount this in BLAM. BLAM then monitors the bucket utilisation, which can be viewed in the storage pane of the platform, and the information can be used in workflows, to send notifications or to take other actions.

 

The Migration Process

Working with PLAZAMEDIA engineering and our colleagues at netorium, a method was devised that allows a rapid but careful migration of data – metadata and media.

• The migration begins when the schema and metadata masks for a given organisation in BLAM are approved by the PLAZAMEDIA operations team and end user customer.
• A fresh database dump from Avid is created and the BLAM workflow is run on the DB file to read in the assets and metadata for the given organisation. Each asset has an additional metadata value which records the migrated status, i.e., a Boolean value to record if an asset been migrated from SGL / Quantum to the NetApp object-storage, or not.
• On an ad-hoc basis, typically overnight, a BLAM workflow is run which generates a list of assets which have not been migrated, limits these to a manageable number (around 200) and triggers a subsequent workflow that makes a restore request of SGL FlashNet and then archives the resulting restored file to the NetApp storage.
• Within the migration workflow there are integrity checks to ensure migrated files are complete and error free. In cases where the asset does not have a browse / proxy that has been migrated from Avid, one is created by BLAM.  This process runs until all assets for a given organisation have completed. During this period, which for large collections can be a few weeks, the operational teams continue to use the Avid system.
• On completion of the migration for a given organisation, the Avid metadata import workflow is run again so that any changes which have been made in the Avid MAM during the migration period are copied to BLAM and not lost.
• The process then starts for the next organisation.

Cost Effective Outcomes

Once the migration of assets is complete for all PLAZAMEDIA clients, the Avid MAM, the SGL FlashNet and the Quantum LTO library will be shut down.  In the case of the LTO library, this will yield 7-racks of floorspace, and the new storage will afford faster retrieval times and greater flexibility.  The commercial model for BLAM is SaaS based and is significantly less expensive than the support contracts of the legacy systems which are ended on completion of the migration.

In this case the storage media is an on-prem object-based solution from NetApp but could equally be cloud based storage such as AWS S3.  BLAM instance is obviously ‘on-prem’ which is increasingly rare for BLAM projects but demonstrates that although BLAM-3 has been developed as a ‘cloud-native’ platform it delivers the same scalability and flexibility when deployed on the ground.

Cost-effective Migration

The Avid / SGL / LTO library combination was a common ‘eco-system’ based archive solution in the late 1990’s and early 2000’s.  The delivery of this project demonstrates that a migration to a more modern platform need not be a headache for media companies or service providers.  With a small range of off-the-shelf BLidgets BLAM can release operators from legacy vendor lock-in allowing them to exploit service-based technology models and fully realise the value of archive content.

Get it in touch to find out more.