​eCP-Production Deployment

About this Guide

This guide is intended for IT infrastructure architects, administrators, and IT professionals who are planning to deploy eMAM system (PAAS) on AWS Cloud.

Overview

eMAM Cloud Platform - Production (eCP-Production) is a PaaS platform which allows editors, artists, and others to collaborate their production workflows worldwide. It gives you web access to AI tagged proxy copies of all your media. You can search, browse, preview and organize media into projects, mark and subclip media and build a sequence/timeline. Editors and artists can use Adobe Creative Suite panels (within Premiere, After Effects, Photoshop, Illustrator, and InDesign) to access projects, bins, sequences and all other media content in the eMAM system. After edit/design, can easily send the completed project back to the eMAM system for sharing.

• Ingest media into your S3 buckets to automatically create lightweight proxy copies for easier use. 
• Organize media into projects, bins, and sequences and categories with restricted permissions. 
• Store the media in your S3 buckets.
• Use the web interface to find clips, mark, sub-clip, and build a timeline.
• Download native resolution or proxy edit versions.
• Send review/approval links.
• Share finished projects checked back into the system.
• Automatically AI tag your contents using Rekognition and Transcribe AI to find faces, keywords, and other metadata.
  eMAM system can be deployed directly from AWS Marketplace or in the AWS cloud using the CloudFormation template. eMAM software installed in EC2 instances (Windows Server 2019 and SQL Server 2019 Web) manages the ingest, transcode, remote production, content delivery, sharing and archive workflows.
  eMAM uses the Cloud Formation Template (CFT) to connect and manage various AWS micro-services like Lambda, S3, Glacier, Elemental Media Convert, Elastic Transcoding, Rekognition and Transcribe. For more details refer AWS services used by eMAM.
  Depending upon the no. of servers , you can either deploy eMAM solution on AWS Cloud using single server CFT or multi server CFT which will only take less than 15 minutes.

Cost & Licensing models

You are responsible for the cost of the AWS services used while running the eMAM system. This is based on flexible pricing model where you only pay for the services you consume, and once you stop using them, there are no additional costs or termination fees. You will pay AWS directly for infrastructure and services and EMAM charges you only for EMAM billing. List of billable services in this CFT deployment are:

• IAM User, Roles & Policies,
• Amazon S3 buckets,
• Elemental MediaConvert and Elastic transcoders,
• SNS Topic
• EC2 instances for DB, Web and App nodes,
• Amazon Transcribe,
• Amazon Rekognition,
• Lambda functions,
• CloudWatch
• SQS
• Dynamodb
• Elastic IP.

eMAM software licensing cost of eCP-Production is $1.50/hr or $11,826/annum. For more details, contact sales@emamcloud.com.

Click Here to get the approximate infrastructure cost of running eCP-Production in a single server environment. Please note that the eCP cost of $1.50 is not included here.

Click Here for our AWS Marketplace pricing model.

Architecture Overview

This is a high-level overview architecture of eMAM Enterprise Configuration deployed in AWS Cloud.

Workflows

eMAM solution Cloud Formation template deploys below workflows:

1. Ingest workflow: Using eMAM solution directly upload asset to S3 buckets from any part of the world using Lambda transcoders (AWS Elemental MediaConvert or AWS Elastic transcoder). Proxy files are created, uploaded and stored in a proxy Amazon S3 buckets.

•      AWS Elastic transcoder

1. 1. 1. Feeder informs eMAM Gateway that a file is getting uploaded to eMAM system and the eMAM dashboard displays status ‘Uploading’. 
      2. Feeder now directly moves the media file to the original S3 bucket. 
      3. A job JSON file is written to the new Ingest bucket. All the values required for the Lambda function are defined in this JSON file.
      4. After the upload is complete (2&3), Feeder will update the eMAM DB status to "Added".
      5. Ingest Bucket is monitored by AWS Lambda function. Therefore, as soon a file comes in the Ingest bucket, it will trigger the Lambda and will get all the necessary info for processing from the JSON file.
      6. AWS Lambda will extract the embedded metadata using Media Info and will now invoke and submit job to the AWS Elastic transcoder.
      7. AWS Elastic transcoder will pick up the file from the Original bucket.
      8. After transcoding, AWS Elastic transcoder will put the output file to the Proxy bucket.
      9. AWS Elastic transcoder will inform the AWS Simple Notification Service (SNS) that the job is complete.
      10. SNS in turn informs AWS Lambda that the job is complete. 
      11. AWS Lambda will now inform the eMAM Gateway system that this job id is complete.

• AWS Elemental MediaConvert workflow works quite similar to Elastic Transcode workflow (as mentioned above) except that the transcoder is Media Convert and instead of SNS, Could Watch communicates with AWS Media Convert and AWS Lambda. Below is a diagram representing the workflow of how files are transcoded in eMAM using Elemental MediaConvert transcoder:
  

2. Remote Editing: eMAM Cloud on AWS Cloud improves workflow collaboration by allowing geographically dispersed users to work together on the same project. Adding eMAM to the mix with its NLE project management, mezzanine and localization features make the remote editing workflows more seamless.
3. Content Delivery: Transcode and Delivery options in eMAM allows the users to push the content to different FTP / Network locations in different formats. eMAM solution uses Amazon CloudFront that securely delivers content with low latency and high transfer speeds.
4. Archive backup and Disaster recovery: eMAM system allows the users to store, track and retrieve the content easily from the cloud storages.

Solution Components

Ingest

When an asset is ingested to eMAM and uploaded to Amazon S3 bucket through the web interface/eFeeder, the ingest process starts. The workflow generates an asset unique identifier, computes and validates an MD5 checksum, and extracts media information such as bitrate, formats, audio channels container format for video, or EXIF information such as GPS location, model, and make for image. The workflow creates a proxy file and thumbnails using AWS Elemental MediaConvert. The proxy files and thumbnail images are stored in the proxy Amazon S3 bucket.
eMAM solution uses Amazon Elastic Compute Cloud (Amazon EC2) as compute engine and requires servers with Microsoft Windows Server operating system and SQL Server database. You can deploy servers with different memory, CPU, storage and network configuration using web based management interface. EC2 instances can be used to setup eMAM Database, Web and Application servers. eMAM uses Lambda functions in Python to manage the ingest/transcoding workflows.
For for details on Amazon EC2 , refer https://www.amazonaws.cn/en/ec2/

Transcoder

eMAM solution is designed to support both server based and serverless transcoding in cloud.
New eMAM Feeder application allows the users to upload the content directly to Amazon S3 from their workstations and manage the transcode queue in cloud using Elastic Transcoder or Elemental Media Convert.
In Server less transcoding, AWS Lambda functions are managing with transcoding process with Elastic Transcoder and Elemental Media Convert. Lambda function can be triggered when certain events occur and can be leveraged to achieve the transcoding functionality without configuring any servers by ourselves. AWS Transcoder grabs input file from the S3 bucket, processes it and based on the trigger set, stores the transcoded file in a target S3 bucket. It supports pre-defined transcoding pre-sets, custom transcoding pre-sets, monitoring and management, notifications, thumbnails, captions, visual watermarks etc.
AWS Lambda acts as a link between the S3 bucket and the Elastic Transcoder. This is done by configuring the events for new object creation in S3 bucket which will trigger the Lambda function. Then the Lambda function will create the Elastic Transcoder Job and add that to the Transcoder pipeline created. Elastic Transcoder then processes the job and produces the transcoded videos in the output S3 bucket. Elastic transcoder uses AWS Simple Notification Service (SNS)- a pub/sub message system from Amazon to communicate with AWS Lambda. Management console and API’s can be used to monitor and check the status of your transcoding jobs.
For more details on Elemental MediaConvert, refer:https://aws.amazon.com/mediaconvert/

In serverless transcoding, eMAM uses AWS Lambda which is a serverless compute service to deploy transcode, archive and delivery management modules.
For more details on AWS Lambda, refer https://aws.amazon.com/lambda/

Storage

eMAM solution supports four types of storages in the cloud supported by eMAM.

1.  Amazon EBS: File based storage that can be mounted as local NTFS drives in the Windows Server. Multiple storage volumes may require for OS installation, Database, Log files, and other file based operations like ingest, transcoding and proxy streaming.
   For more details on Amazon EBS, refer: https://aws.amazon.com/ebs/
   b. Amazon Simple Storage Service (S3): Object based storage where the files are stored inside a bucket with metadata and globally unique identifier (UUID).
   For more details on Amazon S3, refer: https://www.amazonaws.cn/en/s3/ 
   c. Amazon FSx: Network File Storage that can be accessed as CIFS/SMB shares from multiple servers using a UNC path.
   For more details on Amazon FSx, refer https://aws.amazon.com/fsx/
   d. Amazon S3 Glacier: Amazon S3 Glacier and S3 Glacier Deep Archive are a secure, durable, and extremely low-cost Amazon S3 cloud storage classes for data archiving and long-term backup.

     eMAM solution supports below Archive and Retrieval Options:

• Expedited — Expedited retrievals allow you to quickly access your data when occasional urgent requests for a subset of archives are required. For all but the largest archives (250 MB+), data accessed using Expedited retrievals are typically made available within 1–5 minutes.
  • Standard — Standard retrievals allow you to access any of your archives within several hours. Standard retrievals typically complete within 3–5 hours. This is the default option for retrieval requests that do not specify the retrieval option.
  • Bulk — Bulk retrievals are S3 Glacier’s lowest-cost retrieval option, which you can use to retrieve large amounts, even petabytes, of data inexpensively in a day. Bulk retrievals typically complete within 5–12 hours.
  For more details on Amazon S3 Glacier, refer https://aws.amazon.com/s3/glacier/

Error Handling

The eMAM solution uses CloudWatch to monitor the transcode jobs and report the status to eMAM live dashboard. The error handling in Lambda function processes the error by analyzing the execution history and sends the error message to eMAM solution by API call.
eMAM solution applies a catch and retry concept for error handling to the lambda functions to improve the resiliency of the solution by retrying the execution multiple times. When the lambda execution exhausts the retries, it stops the execution and generates an error.
Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real time. It provides detailed insight via comprehensive reports and metrics to ensure seamless and efficient operations. You can send logs and custom metrics to CloudWatch for monitoring. Elemental MediaConvert uses AWS CloudWatch- a monitoring tool to communicate with AWS Lambda. We can create custom rules inside AWS Cloud watch to monitor workflows.
For more details on Amazon CloudWatch, refer https://aws.amazon.com/cloudwatch/

Database

The solution deploys the following Amazon DynamoDB tables:

• A table to store ingest information.
• A table to store delivery information

eMAM uses SQL Server with EC2 as the main database. eMAM also uses DynamoDB as the lambda to handle ingest & delivery operations. Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale.
For more details on Amazon DynamoDB, refer https://aws.amazon.com/dynamodb/

Messaging

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. eMAM uses SQS to manage the archive/restore operations.
For more details on Amazon Simple Queue Service (SQS), please refer https://aws.amazon.com/sqs/

Notifications

This eMAM solution deploys Amazon SNS (Simple Notification Service) to check the progress, completion, warnings or errors in the transcoding jobs in the queue.
Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication. SNS is used to send the object tier change, transcode and AI jobs notifications to eMAM Gateway.
For more details on Amazon Simple Notification Service (Amazon SNS), refer: https://aws.amazon.com/sns/

Machine Learning

eMAM solution uses AI technologies (Amazon Rekognition & Amazon Transcibe) extract insights from the videos to enable smart metadata tagging, automatic transcription and automatic transcribing.
For more details on Amazon Rekognition, refer https://aws.amazon.com/rekognition/
For more details on Amazon Transcribe, refer https://aws.amazon.com/transcribe/

Load Balancing

Load balancer helps to distribute the incoming eMAM web traffic to multiple servers. This will also enable fault tolerance by doing the auto failover to the available nodes if one or more nodes (servers) fails.
Amazon Elastic Load Balancer can be used to set up the load balanced environment with respective cloud vendors.
For more details on Amazon Elastic Load Balancing, refer https://aws.amazon.com/elasticloadbalancing/

Email Server

SMTP (Simple Mail Transfer Protocol) Server is important in the eMAM configurations to send out the workflow and other system notifications to the end users. SMTP services in cloud allows the users to configure the email services in eMAM without going through the hassles of installing, configuring and maintaining a dedicated SMTP server.
For more details on Amazon Simple Email Service (SES), refer https://aws.amazon.com/ses/

Active Directory

AWS Directory Service for Microsoft Active Directory, also known as AWS Managed Microsoft Active Directory (AD), enables your directory-aware workloads and AWS resources to use managed Active Directory (AD) in AWS. eMAM uses AD to connect FSx storage and for SQL Server High Availability configurations.
For more details on AWS Managed Microsoft Active Directory (AD), refer https://docs.aws.amazon.com/directoryservice/latest/admin-guide/directory_microsoft_ad.html

Security

When you build systems on AWS infrastructure, security responsibilities are shared between you and AWS. This shared model can reduce your operational burden as AWS operates, manages, and controls the components from the host operating system and virtualization layer down to the physical security of the facilities in which the services operate. For more information about security on AWS, visit the AWS Security Center.

• Server-Side Encryption: AWS highly recommends that customers encrypt sensitive data in transit and at rest. This solution automatically encrypts media files and metadata at rest with Amazon S3 Server-Side Encryption (SSE).
• Amazon CloudFront: eMAM solution uses Amazon CloudFront that securely delivers content with low latency and high transfer speeds.
• Permissions on Storage buckets: eMAM makes use of AWS S3 Block Public Access feature which provides settings for access points, buckets, and accounts to help you manage public access to Amazon S3 resources. By default, new buckets, access points, and objects don't allow public access. However, users can modify bucket policies, access point policies, or object permissions to allow public access. S3 Block Public Access settings override these policies and permissions so that you can limit public access to these resources.
• IAM User & Roles: It is not recommended to use root user account to deploy the CFT. eMAM creates IAM user with only least privileges with access to limited AWS resources. When creating a stack, AWS CloudFormation makes underlying service calls to AWS to provision and configure your resources. You can use AWS Identity and Access Management (IAM) to manage permissions. IAM roles are used to control access to buckets and objects in cloud storages. This makes it easy to limit users permission without having to modify each bucket or object permission individually. eMAM manages the IAM role which is an IAM identity that you can create in your account that has specific permissions.
• Password Management: eMAM Software uses integrated security option of the Windows server and does not require a separate user id and password to access the database and the application. Customers can change the operating system security level and that does not affect the eMAM application. Click Here for more details about Windows Operating systems security.  We recommend to use AWS tools to rotate the Windows password periodically. Click here for more details. Note: Windows password rotation does not affect eMAM.  

Deployment Options

eMAM uses the Cloud Formation Template (CFT) to connect and manage various AWS micro-services like Lambda, S3, Glacier, Elemental MediaConvert, Elastic Transcoding, Rekognition and Transcribe.
eMAM components (database, web and application) can be installed in one or more windows servers. These servers can be deployed on cloud. We recommend to provision at least two servers in the cloud to run the eMAM database/web and application components. Depending upon the no. of servers , you can either deploy eMAM solution on AWS Cloud using single server CFT or multi server CFT which will only take less than 15 minutes.

eMAM supports all the regions however it is always recommended that you choose a location nearest to you to reduce latency.

Server Specifications

1. eMAM Database Server (Two db servers required for eMAM Enterprise installation)

• • Windows Server 2019 with SQL Server 2019 Standard Edition
  • r5.xlarge instance (4vCPU/32GiB Memory) with Microsoft Windows 2019 Datacenter edition and Microsoft SQL Server 2019 Standard.
  • One General Purpose SSD volume (100 GB) for the operating system (C:)
  • One General Purpose SSD volume (100 GB) to host the SQL Server database files (D:)
  • One General Purpose SSD volume (500 GB) to host the SQL Server log files (E:)
  • One General Purpose SSD volume (500 GB) to host the SQL Server tempdb/backup (F:)
    https://aws.amazon.com/marketplace/pp/prodview-krniomza6bizg 

2. eMAM Web Server (Two Web servers required for eMAM Enterprise installation)

• • m5.xlarge instance (4vCPU/16GiB Memory) with Microsoft Windows 2019 Datacenter edition
  • One General Purpose SSD volume (100 GB) for the operating system (C:)

              https://aws.amazon.com/marketplace/pp/B07QZ4XZ8F

3. eMAM App Servers (Two App servers required for eMAM Enterprise installation)

• • c5.2xlarge instance (8vPU/16GiB Memory) with Microsoft Windows 2019 Datacenter edition
  • One General Purpose SSD volume (100 GB) for the operating system (C:)
  • One General Purpose SSD volume (1000 GB) to host ingest/original/proxy folders.

         https://aws.amazon.com/marketplace/pp/prodview-bd6o47htpbnoe

4. Net X Code Servers (Two servers)

• • c5.4xLarge instance(16vCPU/32GiB Memory)
  • Microsoft Windows 2019 Datacenter edition
  • FSx as the shared storage
  • 1000 GiB SSD storage type with 512 MB/s throughput capacity

Deployment Considerations

Pre-Requisites

Few of the prerequisites to get eMAM system deployed on AWS Cloud are listed below:

• eMAM License: Ensure you have an eMAM product license. To request a trial license, email the eMAM Sales team at sales@emamcloud.com.
• AWS Account: Ensure you have an AWS account (does not require root user) with necessary permissions to provision the IAM resources and deploy the CloudFormation Template. If you don’t have an AWS account, see How do I create and activate a new AWS account?for instructions. If you have a pre-existing AWS account, you can use it for this purpose. Once you have an AWS account, ensure that all technical personnel have been added to the account and that they have been configured with the permissions necessary to properly manage the account. At minimum, there must be one user within the AWS account who has sufficient permissions to run the AWS CloudFormation template, which performs account linking to the eMAM.
• Resource: Deployment should be carried out by an AWS certified engineer or somebody who has moderate knowledge of Amazon Web Services (AWS). If you are new to AWS, visit below links. These will help provide you with the foundational constructs you need. These sites provide materials for learning how to design, deploy, and operate your infrastructure and applications on the AWS Cloud.
  • AWS Identity and Access Management (IAM)
  • Amazon Elastic Compute Cloud (Amazon EC2)
  • AWS CloudFormation

Service Limits

• Maximum media duration is limited to 4 hours : Amazon Transcribe can process files up to 4 hours in length. For more information, see Amazon Transcribe Guidelines and Limits. Concurrent Amazon Rekognition video processes are limited to 20 processes
• Amazon Rekognition supports up to 20 concurrent video processes. For more information, see Limits in Amazon Rekognition.