ProSiebenSat.1 distributes video content to more than 25 linear TV channels including ProSieben, SAT.1, and kabel eins, and more than 200 non-linear channels such as 7TV, maxdome, YouTube (Studio71), and various TV channel websites. As such, ProSiebenSat.1 manages the content of over three million videos for linear TV and the content of nearly half a million additional videos for our non-linear platforms. At the end of last year, our storage volume for video files reached 10 P-bytes and without regular monitoring it would likely grow by at least 2 P-bytes per year, if not more.
Currently, all video content is managed by a variety of systems and platforms. Given the importance of time-to-market and the fact that adapting our existing systems to new business models is usually complicated and time-consuming, additional systems were bought or built as and when needed, then integrated into our existing technical environment – each specialized for different business models or distribution channels. This leaves us with one system for factual content (e.g. stories for magazines and news shows), another system for fiction content (e.g. films, trailer, and ads), three different backend systems to manage the content for online and mobile platforms, a system for ad acquisition, a system for trailer approval, plus a few more. The environment for the management, production, and delivery of video content has become so complex that it cannot be understood and controlled by a select group of experts, let alone one single person. Workflows generated in this IT environment become complex and are often interrupted by workarounds. Operation and further development of the multiple systems are challenging and must consider many dependencies.
Things needed to be changed to stay competitive. Bearing this in mind, we decided to replace considerable parts of our existing environments with a sustainable and innovative Unified Content Platform (UCP). To do so, we began by analyzing business and operational needs, which then led us to new technologies such as docker and Kubernetes, as well as to new ways in which the project could be implemented.
UCP will become ProSiebenSat.1’s major platform for the acquisition, enrichment, management, and distribution of the content for all linear and non-linear distribution channels. We have begun with just 30 users. At a later date, UCP will be utilized by over 2000 internal users and external partners. So, what does the (simplified) content workflow look like, when all the legalities have been addressed?
Acquisition: Licensors or content producers will deliver their audio, video, and image file (essence) content and descriptive metadata to the UCP, primarily via WAN acceleration tools such as Signiant or Aspera.
Quality control (QC): The metadata are then checked and an automated, and if necessary manual, quality control of the video file is performed before the material is made available to other units.
Content production: In the next step, the content is manipulated to create new content (e.g. trailers, magazine stories, versions for a younger audience). To facilitate this, the UCP will provide interfaces to different video editing platforms such as Avid MediaCentral or Adobe Premiere.
Metadata enrichment: Additionally, the content will be enriched with different metadata – such as a description, a list of actors, year of production, and other additional information – that can then be used for electronic program guides (EPG) or on websites. Ad markers can also be set in the UCP to place the advertising breaks.
Scheduling: Content must be scheduled according to the various rights of each distribution channel. This means schedules for linear TV and non-linear video on demand will be generated in separate scheduling systems and the information will be transferred to the UCP. A schedule basically consists of the following information: Video content, timeframe, and distribution channel or platform.
Distribution supply: The schedule information is then used to deliver the video files to the different distribution platforms, e.g. TV playout servers or OTT platforms. This includes transcoding the content to different video codecs and creating specific metadata profiles.
If the platform used to manage the process is flexible enough, this workflow is virtually independent of specific business models and distribution channels. The difference is in the metadata that is required, the video file formats, and the target systems or API for distribution. UCP will not merely replace existing systems, but actually improve our content production and delivery workflows. Managing linear and non-linear workflows in one platform will simplify collaboration between different units. Work that, at the moment, is repeated on a daily basis will become part of the system structure, thus negating the need for daily repetitions. UCP will move most of the information and communication that today is exchanged by e-mail or phone directly to content. This will simplify finding and using content as well. Workflow and task management will increase the amount of automation and sharpen focus on relevant tasks, while also improving transparency throughout the content production process. The ultimate end goal is that our customers will get more relevant content much faster than they currently do.
But how do we ensure scalability, reliability, supportability, and security for such a large amount of content and such a big group of users? How do we achieve rapid deployment of new features and zero downtime during operation? The changes necessary have already been started as part of our requirements and feasibility analysis. For the first time this evaluation included a 10-week proof of concept: Three vendor teams competed to configure and code a real-life simplified workflow with their products. Taking into account each of the products’ features, the various “-ilities”, and in collaboration with the vendor teams, we opted for two lightweight, modern products – Cantemo Portal and Vidispine – and NetApp’s new storage technology, Object Storage. We also decided not to stick with regular system architecture, but to build our own modular platform based on the two software products, different open source tools, and a “Container as a Service” layer. Finally, based on our experience with the implementation and operation of comparable system environments, we decided to manage the whole implementation project ourselves instead of hiring in a general contractor. We believe that taking this approach will help us react to market changes more quickly in the future.
The application layer follows domain-driven design, which means we will have multiple instances of the UCP to cover different parts of the process. Every instance consists of about 12 to 20 modules each capsuled into docker containers. Some of them are standard containers provided by our selected product vendors whereas others contain the custom code built by our development partners Codemill and Dimensional. The custom code interacts with the product via REST API. We consciously decided not to open branches within the products we purchased but to undock custom code from product code as much as possible. The vendors ensure backward compatibility for their APIs so we’ll be able to release upgrades of the core product easier as we do today with other highly customized applications.
Looking at the instances in one stage, you can see that we differentiate between the apps and the Content Hub. The hub is used to manage and store all the content that is ready for broadcast or delivery. It also contains various central services such as quality control or transcoding that are needed in different apps. The different apps are used to import, enrich, modify, or export essence files (audio, video, images, and other) and metadata. They handle only a subset of the metadata and links to the essence files. An app groups the features for the main process steps. Depending on the value chain there will be apps for acquisition, production, distribution, and possibly other processes (enhancement) as well. In some cases, different versions of an app are needed e.g. for the Distribution App. In the past, apps would have been separated by business units; this works as long as the workflows and users don’t change otherwise this causes difficulties and workarounds – or even more versions of this app with all the disadvantages as described at the beginning. In the UCP, different versions of an app will only be built if there are major differences in the workflow such as the scheduling type required for linear or non-linear distribution. Linear distribution needs a sorted playlist per channel and per day which non-linear doesn’t require. However, even in this area we noticed that many elements in the linear app can be used in the non-linear version. Our aim is to have a manageable number of apps. The communication between apps and hub occurs via REST API and uses Kafka for messaging. Kafka also will be used as an interface with other business applications (e.g. for rights management). Users will enjoy a seamless UI integration between the apps.
This concept of hub and apps enables an easy functional scaling without creating another monolithic system. If another business domain needs to be added, it’s relatively easy to integrate it as an app into the UCP and without needing to build a completely new platform as had previously been necessary: A domain such as automated object, speech, and face recognition; an area into which ProSiebenSat.1 is also researching. Within the apps, it’s important not to build too rigid a workflow but to offer a toolbox containing various smaller workflows that can be orchestrated by the users. Each app is handled as a sub-product with its own DevOps team (three teams at the moment). It will also enable performance scaling whereby, based on contemporary requirements, additional instances of an app can be activated on the same on-site infrastructure or in the cloud.
Container as a Service
Over the past two years, we have started implementing automated deployment. Once we had decided to use four or five instances of our chosen products in four or five different stages (from DEV to PROD, a total of at least 300 docker containers), it quickly became clear that just automating deployment is not enough to reach our operational and business goals. We felt it would be too risky to build yet another monolithic platform, as it would inevitably end up being expensive to run and awkward to operate so, in order to make our hardware independent, cloud-ready, highly-available, self-healing, and scalable, the project team built a container platform that enabled the development teams to deliver faster and more reliable software products. These capabilities cover the whole container lifecycle (create, test, deploy, scale, monitor).
The platform is used to orchestrate the whole release process, including a high level of automation, and will enable the DevOps teams to develop and deploy applications quickly. It is treated as an independent product, with its own DevOps team and roadmap, which delivers value to the business by enabling quicker and more efficient new features (or bug fixes). The first goal is to release a sprint to the PROD environment at least once, which means every three weeks instead of three or four times a year. The platform also will reduce the manual effort required for testing (e.g. unit, integration, and regression tests) and reduce or even eliminate maintenance downtime. The link to the ITIL-driven world, where every change needs a ticket and an approval workflow, will occur via an interface to the ticketing tool where every deployment automatically creates an “info-change”. Due to the involvement of all relevant stakeholders within the development process there will no longer be a need for an approval workflow.
The platform is offered as a service to the DevOps application teams who independently create code, build automated tests, and integrate them into docker containers. At the moment, the platform is being used by the three UCP-DevOps teams. The platform team itself is continuously improving the capabilities and services of the platform; these team members act as consultants to the other DevOps teams. At a later date, the intention is to offer the platform to other ProSiebenSat.1 product teams as a central service. To this end, the team is now working on the next level of improvement: The fully automated deployment of the platform in its entirety alongside all hosted services, including multitenancy.
The infrastructure layer has been built entirely of standard IT components to keep it independent of specific vendors. It’s designed to be easily scalable and to guarantee high availability. The ultimate goal for the complete platform is to reach a status of zero downtime. The IT infrastructure layer is based on > 50 nodes with > 750 CPU cores and > 10 TB RAM using CentOS. All essence files and system images are stored on an Object Storage from NetApp that contains > 10 P-bytes net / > 29 P-bytes gross capacity. For the databases, NetApp’s all-flash storage SolidFire has been selected. This will be enhanced by online storage to cover different high performance processes on video files, e.g. content analysis and transformation processes. The infrastructure has been split into two data centers and made ready for the data center migration we are expecting to happen within the next few years, as we build our new campus.
Introducing these new technologies is a challenging but necessary step as we prepare for the future. Steep learning curves and a great deal of effort have been combined as we introduce new technologies and new modes of operation into our ITIL-driven environment. It has also required a culture change within the project team and the organization as a whole, which takes time. Key success factors include considerable creative freedom for the team when deciding how best to build the platform, self-organized teams with a high degree of responsibility, and strong user-participation. As such, most of the crucial decisions regarding the platform can be taken within the team and, if needed, by the key users. Management usually only comes in if prioritization between major stakeholder groups is required or support for big impediments is needed. As we have highly skilled and motivated colleagues in place, we have already taken a good number of steps along this path. The platform is running in production with its first major release, already bringing advantages to users. The feedback from many stakeholders and notable consulting companies shows us that we are on the right track to hit our goals and to get ProSiebenSat.1 content production ready for the future.