AWS Deployment Guide: V6

This deployment guide contains all the information that is applicable to successfully deploy Beast Core to commercial platforms. This is possible through the Beast Code Application Suite (BCAS).

BCAS is a collection of light-weight web applications or plugins that can work independently and integrate into Beast Core. Beast Core provides the foundation for these plugins to work seamlessly together as one cohesive user experience.

This is all possible thanks to the plugin architecture that Beast Core and BCAS subscribe to. By following standards maintained by Beast Code, third parties can create plugins of their own to use within Beast Core or with any of the web applications and services that BCAS owns.

1 - Introduction

1.1 - Software Use Cases

• GOAL - Help advance organizational initiatives using modern, open digital technologies.

• One stop shop for full lifecycle support; Break down silos.

  • Development and Source selection.

  • Operational situational awareness through live system data.

  • Maintenance & Logistics.

  • Training.

• Digital twin capability across Program Executive Offices/Special Projects Office, vendors and classifications.
  o Transition support from Legacy to Modern Systems.
  o Systems of Systems - Single pane of glass view of family of twins in one environment.
  • Help shape RFP development to empower improved source selection in common environment... better compare apples to apples.
  • Live sensor integration allows CBM+ capability of life cycle savings & real-time situational awareness.

1.2 - Typical Customer Deployment

Beast Core utilizes a single platform deployment, and it is illustrated in Figure 1 of the Architecture Design section.

1.2.1 - Resource List

1. Proficiency with Kubernetes, Helm, Docker, and AWS.

2. The beast-core-shell Helm chart, which is provided by Beast Code.

3. The images used to by the Helm chart which are provided by Beast Code.

4. The digital twin, which is provided by Beast Code.

5. A cluster on EKS. Beast Core v6 uses Helm charts which require a Kubernetes cluster.

6. K8s objects created during Helm install.

7. Network Load Balancer in EC2.

8. "A Record" in Route 53.

9. Persistent Volume.

10. Persistent Volume Claim.

11. Beast Core images.

1.3 - Deployment Options

Deployment option: Single Platform. This deployment targets one physical asset. When the app is opened, the user will immediately see that asset.

1.3.1 - Resource List

This portion of the task is not applicable as Beast Core doesn’t have options in a list format. The only option in this guide is the choice between using the EFS-CSI controller or local storage.

Here is a list of AWS Standard Resources:

• EFS

• EC2

• ECR

• EKS

• A cluster on EKS (AWS Managed). Beast Core v6 uses Helm charts which require a Kubernetes cluster.

Click this link for the Amazon EFS Container Storage Interface (CSI) driver provides a CSI interface that allows Kubernetes clusters running on AWS to manage the lifecycle of Amazon EFS file systems:
https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html

1.4 - Time Estimate to Deployment Completion

The expected time to deploy Beast Code within AWS is eight hours. 

1.5 - Supported Regions

Beast Code is supporting all 50 United States and no foreign countries.

The supported AWS Regions for Amazon RDS are:

• US East (Ohio)                      –     us-east-2

• US East (N. Virginia)             –     us-east-1

• US West (N. California)        –     us-west-1

• US West (Oregon)                –     us-west-2

2 - Prerequisites and Requirements

2.1 - Technical Prerequisites and Requirements

To deploy Beast Core v6 on AWS you will need:

1.    The beast-core-shell Helm chart, which is provided by Beast Code.

2.    The images used to by the Helm chart which are provided by Beast Code.

3.    The digital twin, which is provided by Beast Code.

4.    Bash or GitBash along with the following commands: aws (along with credentials to access your AWS account), docker, kubectl, eksctl, helm.

These skills are required to successfully deploy Beast Core on AWS:

• Kubernetes

• Helm

• Docker

• Bash

• AWS (IAM, Elastic Kubernetes Service, EC2, EFS, Route 53, Certificate Manager, Elastic Container Registry)

2.3 - Deployment Environment Configuration

To run Beast Core Experience, the following steps need to be implemented.  Each step includes Amazon-provided guides. (Note these settings are what Beast Code uses to test Beast Core Experience.  If you are sufficiently skilled in AWS, Kubernetes, and Helm you may alter the provided Helm chart and K8s manifests to match infrastructure different than that listed in this section.)

1.    Create an EKS cluster with Managed nodes.  (https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html)

2.    Install the AWS Load Balancer Controller (https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html)

3.    Register a domain to be used as part of the URL for the Beast Core Experience site you will create. (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-register.html)

4.    Request a public certificate to match the URL of the Beast Core Experience site you will create.  (https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html)

5.    Install the Amazon EFS CSI driver and create an Amazon EFS file system (https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html)

3 - Architecture Diagrams

3.1 - Deployed Services and Resources

The architecture diagram below displays the relationships between Beast Core, AWS services and resources, with how connections to the Client/Customer are established.

Figure 1. Internal Architecture Diagram

Figure 2. External Architecture Diagram

3.2 - Network Diagrams Demonstrate Virtual Private Clouds (VPCs) and Subnets

This task is Not Applicable in Beast Core. Beast Core uses Elastic Kubernetes Service which handles these networking details for us. The details vary depending on how EFS implements it. 

3.3 - Integration Points

The architecture diagram in Figure 2 includes integration points, including third-party assets/APIs and on-premises/hybrid assets.

4 - Security

4.1 - Does Not Require the Use of AWS Account Root Privileges for Deployment or Operation

Beast Core does not recommend, nor does it require the use of AWS account root privileges for deployment or operation.

4.2 - Provides Prescriptive Guidance on Following the Policy of Least Privilege for All Access Granted as Part of the Deployment

Strategy regarding permissions for access to Beast Core is the responsibility of the customer/license holder.

To access AWS Service Limits Documentation, use this link: https://docs.aws.amazon.com/awssupport/latest/user/service-limits.html

4.3 - Public Resource Documentation

Beast Core makes use of and requires no public resources for successful deployment for any customer.

4.4 - AWS Identity and Access Management (IAM) Role Purpose and IM Policy

While the deployment is being completed, the only Identity required is an administrative ID to complete the deployment. Beast Core operates independently of AWS IAM.

An Administrator Access policy is required as a successful deployment requires establishing and documenting roles and policies that are above the scope of a Power User. An EKS Cluster Administrator is also required to create and administer the cluster for the deployment.

4.5 - Purpose and Location of User Created Keys

This task is not applicable to Beast Core as no keys are required for this deployment.

4.6 - Maintaining Stored Secret Instructions

This task is Not Applicable (N/A) as there are no secret instructions stored within Beast Core.

The customer should create a public cert to add to the ALB in order to enable https. See Deployment Assets section steps 4 and 5 of the deployment guide.

4.7 - Customer Sensitive Data Storage

The entity purchasing beast core is providing the workspaces, which could make it sensitive to customer data. Potentially, the entity would also be providing the other classes of information that may be classified as other classifications depending upon who their customers are.

Potentially sensitive data can be found in the Workspace path that is not applicable within the current constructs of Beast Core.

NOTE

later versions with user login might have more potentially sensitive data where user account information is stored but that does not currently exist. 

4.8 - Data Encryption Configuration

This task is Not Applicable as Beast Core currently has no encryption requirements. Beast Core will link to the AWS encryption requirements.

• Encryption in Transit for EFS CSI driver: https://github.com/kubernetes-sigs/aws-efs-csi-driver#encryption-in-transit

• Encryption at rest for EFS: https://docs.aws.amazon.com/efs/latest/ug/encryption-at-rest.html

• Secret encryption on EKS cluster: https://docs.aws.amazon.com/eks/latest/userguide/enable-kms.html

• SSL/TLS: https://aws.amazon.com/what-is/ssl-certificate/
  

  NOTE

  EBS encryption is not included because this deployment does not use it.

• While EBS encryption may not be required by Beast Core, best practices recommend encrypting volume:https://aws.github.io/aws-eks-best-practices/security/docs/data/
  

• EFS Volume's encryption status cannot be changed after creation so want to call this out and link to documentation for setup process: https://docs.aws.amazon.com/efs/latest/ug/encryption.html

4.9 - Multiple Element Deployments with Network Configuration

One way to let your application be accessible from the internet is to use load balancers. By adding a Load Balancer Controller to your cluster, a load balancer will automatically be created when you make a k8s service of type Load Balancer or an Ingress.

NOTE

See Section 7.1 (Deployment Assets section, Networking subsection) for the instructions on how to create a Load Balancer.

4.10 - Support Customer Ability to Disable IMDSv1

This task is Not Applicable as Beast Core does not make use of IMDS.

Currently "eksctl create" creates three private subnets and three public subnets. They have 13 bits for IP address in each subnet. Beast Code doesn't have requirements for subnets and CIDR range.

Instead, these are configured by eksctl. However, if still answer is not acceptable, artificial requirements are created during the configuration:

• Public subnet: 2

• Private subnet: 0

• Minimum number of IP addresses in a CICD Block: 4

5 - Costs

5.1 - List of Mandatory or Optional Billable Services

The initial pricing strategy is a flat rate of $150 per hour for any services provided. This may evolve and be modified as Beast Code engages with customers and contracts are negotiated.

The estimated cost using this link https://calculator.aws/#/estimate are represented in the table below.

For pricing customized for your specific configuration see the Support Contact Methods in the Different Support Tiers and Service Level Agreement (SLA) Section.

5.2 - Cost Model and Licensing Costs

• The Beast Core software is licensed per user. Licenses are subscription based.

• The monthly subscription provides users with access to software updates.

• The subscription may be paid annually with a 10% discount.

• Site and enterprise licenses may be negotiated and purchased.

• Price is for 2024, subject to change for future years.

 5.2.1 - Commercial Plugin License Costs

• Bundles may be purchased for a subset of the organization’s users.

• The bundle may be paid annually with a 10% discount.

• Site and enterprise licenses may be negotiated and purchased.

• Price is for 2024, subject to change for future years.

 5.2.2 - Service Level Agreement (SLA)

• Government and commercial licenses include Tier 1 support.

• Tier 1 support is basic troubleshooting via email during business hours (9AM - 5PM Central).

• Tier 1 inquiries will be responded to within 24 hours.

• Tier 2, Tier 3, and After-Hours support may be purchased separately.

  • Tier 2 includes remote login support by technical staff.

  • Tier 3 includes dedicated developers to support installation, data migration, and plugin development.

  • After-Hours support is any support outside of normal business hours.

• User Training may be purchased separately.

6 - Sizing

6.1 - Scripts to Provision Required Resources or Guidance for Type and Size

The “Create the PV and PVC” section that is part of the Deployment Assets section provisions the needed storage for the workspaces. Currently the storage setting for both K8s objects is set to 5 Gi. If you use workspaces that require more storage, adjust this amount as needed.

Creating the cluster on EKS is created by default two m5 large nodes, which can be scaled up depending on customer usage needs.

7 - Deployment Assets

7.1 - Instructions for Deploying the Workload on AWS

Prerequisites:

For the list of prerequisites, see the Prerequisites and Requirements section.

Networking

1.    In values.yaml, please change the value of alb.ingress.kubernetes.io/certificate-arn to match your certificate.  This ARN is created in step 4 of “Deployment Environment Configuration”

Workspaces

Create the PV and PVC

2.    cd into the k8s-objects folder in the deployment package. In pv.yaml change volumeHandle to your file system's "File system ID".  (See step 5 in "Deployment Environment Configuration".) Then run kubectl apply -f  pv.yaml

Recommend performing a health check: run kubectl get pv to make sure your PV exists.

3.    Create the beast-core namespace: kubectl create namespace beast-core

Recommend performing a health check: run kubectl get namespaces to verify the beast-core namespace is included on the list.

4.    cd into the k8s-objects folder in the deployment package.  Run kubectl apply -f pvc.yaml -n beast-core

Recommend performing a health check: run kubectl get pvc -n beast-core to verify the PVC is bound to your PV.

Next, Copy the Workspace(s) to the PV

5.    cd into the k8s-objects folder in the deployment package and run kubectl apply -f pv-transfer.yaml -n beast-core

This pod is used to copy the workspaces folder.

Recommend performing a health check: run kubectl get pods -n beast-core to see if the pv-transfer pod has been created and appears in the directory.

6.    cd into the directory on your machine containing the workspaces folder.

Recommend performing a health check: run ls and verify workspaces folder is one of the listed items.

7.    Run kubectl cp workspaces pv-transfer:/my-pv -n beast-core 

This may take a few minutes.

Recommend performing a health check: run kubectl exec -it pv-transfer -n beast-core -- sh, run cd /my-pv, run ls to make sure the workspaces folder is there.

Note

For subsequent copies one must consider kubectl cp doesn't remove files already in destination but not in source.To prevent extra files being in the workspace, before running kubectl cp please exec into the pv-transfer pod and delete the workspaces folder. If you know which files are in the destination but not the source, you can instead just delete those to save time.Add Images to Your Container Registry

Create the following repositories on "Elastic Container Registry" (Choose the setting of "Private" in "Visibility Settings"):

• aggregator

• atlas-service

• bcore-shell-host

• bcore-shell-plugin

• busybox

• details-pane-plugin

• document-details-sub-plugin

• document-viewer

• dynamic-workspace-geometry-service

• geometry-selector

• input-controller

• search-plugin

• search-service

• system-viewer-babylon

• visualizer-atlas

• visualizer-system-designation

• workspace-service

Recommend performing a health check: In AWS console, go to "Elastic Container Registry" and verify all the registries exist.

9.    In values.yaml, replace 123456789012.dkr.ecr.us-east-2.amazonaws.com with your registry.

10.  cd into the folder called "image tar files".

Recommend performing a health check: run ls and verify *.tar files are listed.

11.  Authenticate with the repositories by clicking on one of the repositories, click

"View push commands", Do step one for "macOS/Linux". If you are using Windows, use Git Bash.

Recommend performing a health check: Do the next two steps and verify there are no errors during the "docker push" command.

12. Change the registry variable in pushImages.sh to match your registry.

13.  run ./pushImages.sh in bash.

Recommend performing a health check: All the repositories should now contain images.

Run the Helm Chart

14. cd into the beast-core-experience Helm chart

Recommend performing a health check: run ls, you should see several folders and files including charts, templates, and Chart.lock, etc

15. Run helm upgrade --install beast-core-experience -n beast-core .

Recommend performing a health check: run kubectl get pods -n beastcore and verify all pods created in this step are STATUS == Running.

Networking (Part 2)

16. Find the Application Load Balancer created during the last step.  In the AWS console, go to "EC2", click "Load balancers".  The Application Load Balancer you are looking for is the one created since performing the previous step.  You will need this for the next step.

17. Create an A record. In the AWS console:

• Go to "Route 53."

• Click on your hosted zone.

• Click "Create record." Fill in the subdomain.

• Record type is "A."

• Enable "Alias."

• "Choose endpoint" is "Alias to Application and Classic Load Balancer".

• Choose your region.

• "Choose load balancer" is the alb you found in the previous step.

• Routing policy is "Simple routing."

Recommend performing a health check: In Chrome enter https://<subdomain>.

<Hosted zone name> to verify the page loads.

7.2 - Prescriptive Guidance for Testing and Troubleshooting

Testing

Enter the URL you created into the browser and make sure the site comes up. Try loading the system to see if it works.

Troubleshooting

The Site Doesn’t Load

Try using the URL of the load balancer. If this doesn't work, you can "kubectl port-forward" into the host pod. (e.g. kubectl port-forward beast-core-shell-host-799775dfcd-pjm76 -n beast-core 80:9001) This will help you pinpoint where the problem is.

For more information about "kubectl port-forward" see https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/#forward-a-local-port-to-a-port-on-the-pod

Unable to delete a Persistent Volume Claim

If you try to delete a PVC but it automatically reappears, it might have a finalizer preventing deletion. To remove the finalizer, make the following patch.yaml

metadata:
  finalizers:

Then do:

kubectl patch pvc <your-claim> -n <your-namespace> --patch-file patch.yaml

For more information on kubectl patch see https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/
For more information about PVCs and PVCs see https://kubernetes.io/docs/concepts/storage/persistent-volumes/

At this point, you should have all of the following roles under AWS > IAM > Roles:
AmazonEKSVPCCNIRole 
AmazonEKSLoadBalancerControllerRole
AmazonEKSEFSCSIDriverRole
AmazonEKSClusterRole
AmazonEKSNodeRole

The PVC Remains in "Pending" State

This may be the result of the PV already referencing a PVC.  If the PV no longer needs to reference a PVC, you can free it up by removing the reference.

To do that make the following patch.yaml.

spec
  claimRef

A K8s Object Not Found

When doing kubectl or helm commands and you get an error about a K8s object not found, it might be because they are not on the same namespace. The PVC and the transfer pod that uses it need to be in the same namespace. Likewise, the Helm install and PVC it uses need to be on the same namespace.

For more information about namespaces see https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

My File System on EFS is Showing Only 6 KiB even Though I Already Transferred a Whole Digital Twin

There is lag time between uploading to EFS and it is showing up on "Metered size". To verify the digital twin is really on the PV, you can recreate the transfer pod and shell into it to see if the data is there.

8 - Health Check

8.1 - How to Assess and Monitor the Health and Proper Function of the Application

Follow the steps below in order. They show you how to assess and monitor the health and proper function of the application:

1. Set a liveness probe in the values.yaml file (The liveness probe will cause a pod to be restarted if the probe fails).

2. With the probe(s) set, run kubectl get events -n <namespace> to see if any pods are frequently restarting.

3. Use the site to ensure functionality is working.

This link will direct you to monitoring an EKS cluster: https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/amazon-eks-logging-monitoring.html

9 - Backup and Recovery

9.1 - Data Stores and Configuration Backup and Recovery

The data store for the workspaces is a file system. There are no configurations to back up, just the files themselves. The file system isn't proprietary, so Beast Code is not required to provide instructions for backup and recovery.

Use this link to backup EFS for AWS: https://docs.aws.amazon.com/efs/latest/ug/awsbackup.html

10 - Routine Maintenance

10.1 - Rotating Programmatic System Credentials and Cryptographic

System credentials created in AWS do not interact with the Beast Core program itself. There is no need to rotate any credentials within Beast Core when AWS credentials change.

This page describes how to change the IAM password: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_user-change-own.html#ManagingUserPwdSelf-Console?icmpid=docs_iam_help_panel

10.2 - Prescriptive Guidance for Software Patches and Upgrades

Copy the workspace(s) to the PV.

At this point, you should have all of the following roles under AWS > IAM > Roles:
AmazonEKSVPCCNIRole 
AmazonEKSLoadBalancerControllerRole
AmazonEKSEFSCSIDriverRole
AmazonEKSClusterRole
AmazonEKSNodeRole

Once you have a PV and PVC, you'll need to copy the workspaces into the PV. If you wish, you can "kubectl apply" the following object to facilitate copying:

apiVersion: v1
kind: Pod
metadata:
name: pv-transfer
spec:
containers:
- name: pv-transfer
image: alpine:latest
imagePullPolicy: Always
command: ["sh"]
stdin: truet
ty: true
volumeMounts:
- name: my-pv
mountPath: 
volumes:
- name: my-pv
persistentVolumeClaim:
claimName: <name of your pvc>

Once pod exists, cd into the directory on your machine containing the workspaces folder.

Do:

kubectl cp workspaces pv-transfer:/my-pv -n <namespace>

/my-pv in the pod should now contain the workspaces folder.

To verify the transfer happened, exec into the pod that has the PV mounted:

kubectl exec -it pv-transfer -n <namespace> -- sh

When you cd into /my-pv, you'll be at root of your Persistant Volume, which in this example should contain the workspaces folder.

10.3 - Prescriptive Guidance on Managing Licenses

The number of allotted licenses v/s licenses used will be managed by the license key creation and management module of Beast Core 6. There will be floating licenses which means the number of used licenses cannot exceed the number of allotted licenses. The license users can switch/change, but the number of users logged in using the license count as used license.

10.4 - Prescriptive Guidance on Managing AWS Service Limits

AWS service limits management is In Accordance With (IAW) standard AWS documentation. Beast Core has no special requirements.

https://docs.aws.amazon.com/awssupport/latest/user/service-limits.html

11 - Emergency Maintenance

11.1 - Handling Fault Conditions

In its current version (6.0.0), the software provides no communication if things are not working. Currently, it requires the technician to review the deployment health checks and potentially review any logs that relate to containers failing to provide functionality to Beast Core 6.

NOTE

See Section 12: Support in this guide for the procedure to recover the software

This page talks about how to get support from AWS:
https://repost.aws/knowledge-center/get-aws-technical-support
Because the cluster is spread across two availability zones, if one AZ goes down the cluster is expected to still run.
If a node becomes unhealthy the maintainer of the cluster is expected to find out what the problem is and troubleshoot.
Here is an example of troubleshooting an unhealthy node:
https://repost.aws/knowledge-center/eks-worker-node-not-ready

11.2 How to Recover the Software

If the Helm chart or any of the images got deleted or unwanted changes were made to them, follow these steps to recover them:

1. Look for any copies of the deployment package sent by Beast Code.

2. If you can't find the deployment package, contact Beast Code to obtain a current copy.

3. Once you have the Helm chart and images, use them to redeploy Beast Core v6 according to the Deployment Assets section in the deployment guide.

12 - Support

12.1 - How to Receive Support

There are three ways to contact Beast Code Support

1. Email: The user Point of Contact (POC) can send an email to support@beast-code.com with the issues and Beast Code will create a ticket and address it based on agreed Service Level Agreement (SLA). This will be part of Standard Tier1 support.
   

2. Phone: The user POC can also contact support by calling the Beast Code Support line phone number during normal workday business hours (Mon to Fri 8AM-4PM Central). This will be part of Extended Support (Tier2 & Tier3) agreement.
   

3. Help Center: The user POC will also have access to Beast Code Help Center. The POC can open a new ticket as well as view and/or update an existing ticket. This will be part of Extended Support (Tier2 & Tier3) agreement.

For more information, see the How to Receive Support article.

12.2 - Technical Support Tiers

Regardless of the channel used to report an issue to Beast Code, user, the POC will receive email updates on the progress of the ticket based on the SLA.

Beast Code will also provide an escalation path if the SLA is not followed, or the issue resolution is not satisfactory.

The Beast Code SLA support team will be available during regular business hours (Mon to Fri 8AM – 4PM Central) except Federal Holidays.

12.3 - Different Support Tiers and Service Level Agreements (SLAs)

The following content is the complete Beast Code Service Level Agreement 1.0. 

12.3.1 - Application Owners

Beast Code will interface directly to the Customer administrator Point of Contact (POC) for all matters related to the Beast Code Application Support. All activities and requests must follow the Beast Code change control process and be approved by the Customer admin. Beast Code will NOT interface with Customer users without Customer admin approval. 

12.3.2 - Support Contact Methods

Beast Code Help Desk - support.beast-code.com

Email - support@beast-code.com

Phone - 850-702-3600 ext. 5

12.3.3 - Severity Levels

Beast Code Level 1 and 2 technical support operates standard “SLA Initial Response Time” hours from 8:00AM to 4:00PM Central Time, excluding Saturdays, Sundays, Federal Holidays, and Beast Code Holidays.

The customer may submit a ticket and access the help desk online / knowledge base twenty-four hours a day.

NOTE

Access to Technical Support is limited to designated administrator(s) for Customer. 

Technical Support requests received by the Beast Code help desk will be given a Severity Category from 1 to 4 based on how important responding to the request is to the primary business of the Customer as a whole, as well as the availability of resources. Severity categories are as follows:

• Severity #1 – a problem has made a critical application function unusable or unavailable and no workaround exists.
  

• Severity #2 – a problem has made a critical application function unusable or unavailable, but a workaround exists; or a problem has made an important application function unusable or unavailable and no workaround exists.
  

• Severity #3 – a problem has diminished critical or important application functionality or performance, but the functionality still performs as designed.
  

• Severity #4 – a problem has diminished supportive application functionality or performance.

Severity codes are used to determine appropriate response and resolution times.  Response and resolution times are measured from when the incident is opened by the help desk.  If the problem is not resolved within the defined timeframe, continuous effort will be applied until the problem is resolved. 

12.3.4 - Response Time Definitions

• Initial Response: is when an open ticket is replied to by Beast Code. Circumstances may make it necessary for the Customer to notify Beast Code about a problem by email or phone, but for the purposes of establishing the correct response time, the clock starts request when a ticket is opened by the Customer, or by Beast Code on behalf of the Customer.
  

• Subsequent Response: is the frequency with which the Customer is updated on the resolution status.
  

• Resolution: is the point at which the request is resolved.

The table below represents standard response times. Beast Code will work with the Customer to determine actual severity.

12.3.5 - Enhancements

Enhancements are site features that are either missing or not operating as desired (though operating as they were designed).  This includes the following types of activities:

• Application Changes: Modifications to any application hosted on the site. This includes adding features, capabilities, etc.

• Application Features: This entails adding features or capabilities to the application.

• Database Modifications: – Changes to the data or database.

Enhancements will be defined, estimated, prioritized, and managed by Beast Code. Enhancements that require additional resources to complete due to Customer deadlines, schedule requirements, complexity, or scope will be executed through a formal statement of work at Beast Code.

Beast Code will work with the Customer to define the requirements, acceptance criteria. 

Beast Code will periodically enhance the Beast Code Application system at its own cost and extend new capabilities to our customers. There is no additional cost for access to these new capabilities, and Beast Code will define the release schedule for the internal enhancement efforts. 

12.3.6 - Approval and Acceptance

Acceptance of the Services and each Deliverable shall occur only when:

• Beast Code has corrected necessary defects identified during testing.

• Beast Code has provided to Customer all Services and Deliverables required to be provided to Customer pursuant to the applicable SOW.

• Customer notifies Beast Code in writing that all testing for the Services and Deliverables, as applicable, has been met to Customer’s satisfaction.

Nothing else, including the Customer’s use of the Services or a Deliverable, or any portion thereof, in a live, operational environment, shall constitute Acceptance.

12.3.7 - Request Ticketing System

Beast Code will maintain a Ticketing System (TS) for all work, errors, bugs, and enhancements requested by Customer to be fulfilled by Beast Code.  

The appropriate Customer personnel will make a request via a ticket to Beast Code. If the request is accepted, the ticket will be added to Beast Code’s backlog. Beast Code will determine prioritization of the ticket. When work is started on the request, the activity will be marked as in progress. Upon completion of the task, the activity will be marked as ready for review. Upon approval, the changes will be migrated to the UAT environment for testing and final approval. Last, the changes will be migrated to the production systems and the task will be marked as complete. The originator of the request will receive an email indicating that the activity is complete.

If a problem arises after the change has been launched, a new ticket MUST be submitted for that problem. The TS provides a reporting mechanism for all requests. Beast Code will provide a detailed invoice to Customer for any activities requiring additional resources via a formal SOW. 

12.3.8 - Customer Response Expectations

At times, Beast Code will need feedback or decisions from the Customer to complete tasks. To assure effective use of resources is provided, Customer administrators are expected to respond to normal requests via email or phone within two business days. Delays in or lack of response will impact deadlines and delivery schedules. 

12.3.9 - Reporting Requirements

Beast Code will maintain an activity tracking tool that will enable the customer to pull real-time and historical work reports. 

Where needed, Beast Code and the customer may agree on an alternate reporting format on a case-by-case basis.