PowerProtect Cyber Recovery - Content (494 pages)

o PowerProtect Cyber Recovery Concepts .................................................................... 1

o PowerProtect Cyber Recovery Features ................................................................... 19
o PowerProtect Cyber Recovery Administration .......................................................... 32
o PowerProtect Cyber Recovery Implementation ......................................................... 86
o PowerProtect Cyber Recovery Integration .............................................................. 118
o PowerProtect Cyber Recovery Design .................................................................... 122
 o PowerProtect Cyber Recovery Concepts
➢ Challenges with Security

Data is the currency of the internet economy and a critical asset that must be protected, kept
confidential, and made available at a moment’s notice. The global marketplace of today
relies on the constant flow of data across interconnected networks, and digital
transformation efforts put more sensitive data at risk.
The importance of the data of the data of an organization makes it an attractive and lucrative
target for cyber criminals. Cybercrime has been called the greatest transfer of wealth in
history, and it is all about the data. Accenture (a global consulting firm with a service line
that focuses on technology) estimates that US $5.2 trillion of global value is at risk by
cybercrime in the next five years.
Regardless of the industry or size of the organization, cyber-attacks continually expose
business and governments to compromised data. Cyber-attacks result in revenue loss due
to downtime, reputational damage, and costly regulatory fines. The average annual cost of
cybercrime per company is increased to US$18 million in 2018, a surge of 72% in the last
five years.
Having a cyber recovery strategy has become a mandate for business and government
leaders. According to a 2019 Marsh and Microsoft study, 79% of global executives rank
cyber-attacks as one of their organization’s highest risk management priorities.
The US government sent a memo to corporate executives, and business leaders on June
2, 2021, stating strengthening resilience from cyber-attacks is a top priority of the president.
The concern is based on an increase in number and size of ransomware incidents.
➢ Types of Attacks

A cyberattack is an attempt by hackers to damage, destroy, or control a network or system.

It includes any type of offensive action and can also target information systems,
infrastructures, networks, or personal computers. The purpose of the attacks include
stealing, altering , hijacking, or destroying data or information systems. Listed here are the
most common cyberattacks.
• Denial of Service

Denial of service attacks attempt to bring systems to a halt. These attacks overwhelm the
resources of the system with excessive requests that consume all the resources. Distributed
Denial of Service launches the attack from many other host machines. The purpose of denial
of service attacks is to bring down a system to initiate another attack or affect the system by
a business competitor.

1
 • Unauthorized Currency Mining

Digital currency relies on blockchain, which requires distributed computing power to mine
and process operations. The systems involved in mining receive a commission for facilitating
the transaction. While digital mining is a legitimate operation, hackers can use compute
resources of many victims to mine for cryptocurrencies without their authorization. This
attack is known as crypto jacking.
• Spam

Unsolicited bulk messages sent through email, instant messaging, or other digital
communication assets are known as spam. While spam might be a common practice for
marketing, it can be used to trick victims into providing sensitive information that can be sued
later to perpetrate crime.
• Adware

Adware is part of greyware, potentially unwanted programs that are not a virus or malicious
software but have problematic code or hidden intensions. Adware collects information about
a user with the purpose of advertisement. These programs on a computer are usually
referred to as adware, while programs on a mobile device are referred to as madware.
Adware has the potential of slowing down a system and can work with spyware.
• Malicious Web Scripts

Malicious web scripts can be in existing legitimate websites or in websites that are redirected
from legitimate websites. Malicious web scripts are scripts that when run can detect and
exploit the vulnerabilities of a system of visitors to the website. Whether they are a redirect
or embedded in the legitimate website, customers feel safe because they are visiting a
known source.
• Business Email Compromise

Business email comprise is a phishing attempt that relies on deception. There are several
forms of this scam, but the common trait is that scammers target employees. If their interests
are financial, attackers trick employees into transferring to bank accounts. Employees
believe that these bank accounts belong to the trusted partners. Attackers can be interested
in proprietary information or trade secrets. After gaining their victim’s trust, they can obtain
private company information that should not be public. These attacks can be perpetrated
through email spoofing, social engineering, identity theft, and malware, among others.
• Banking Trojan

A banking trojan tricks users into downloading a “harmless” file that becomes malware that
identifies a user’s banking information. This attack is very profitable because it gains access
to bank accounts and can transfer funds from it. This malware can target businesses or
individuals and is also perpetrated through social engineering, phishing, and spam emails,
exploit kits, and so on.

2
 • Ransomware

Ransomware is also a form of malware, different from adware; it is malicious software that
encrypts the entire hard drive of the computer, locking a user out of the system. Alternatively,
it can be crypto ransomware, which encrypts specific files, most commonly documents and
images in the systems. When a system is infected with ransomware malware, it asks the
user to pay a fee to unlock and reclaim the data, or else the data is lost or made public.
Ransomware is normally distributed through phishing emails or exploit kits. It is more
common than the different categories of cybercrime because it provides significantly less
effort for a greater gain.
➢ Size and Impact of the Problem

Global crime damage is predicted to cost US $6 trillion annually by 2021. This number is an
increase from US $3 trillion in 2015.
The 2013 Yahoo breach is now known to have affected the entire 3 billion accounts
subscribed. This breach alone almost ruined Yahoo’s 2015 acquisition by Verizon. Also, this
information is believed to have been sold to spammers and other hackers for US $900,000.
Cyber-attacks are becoming more common every day. They can be categorized into
different types depending on their method and intention.
➢ Technology Limitations

Traditionally, prevention from these attacks results in using multiple layers of protection.
Backing up data is the most important and effective way of combating ransomware. Often
the backed-up data is stored online or even on the cloud and is accessible by the hacker.
This method makes the backup copy vulnerable to a cyberattack. An approach is to keep
the backup copies offline, where cyberattacks cannot access the secure copies. While
keeping secure copies offline is safest, backups can be time consuming due to the amount
of data an organization might manage. Strict RTOs and RPOs are difficult to meet.
Other protection best practices include keeping security software up to date on latest
definitions of virus and malware. Other best practices are to keep the operating systems and
software updated with the security patches. Because email is the main infection method,
educate employees to be aware of links or attachments in suspicious email messages.
The PowerProtect Cyber Recovery (PPCR) solution mitigates ransomware and other
attacks in a simple and secure way. The solution provides the security of offline backups
with the flexibility of performing them online.

3
 ➢ How to Access the Cyber Recovery Documentation?

CR information can be found in the Dell Technologies Cyber Recovery micro site and the
support site.
1. To reach the Cyber Recovery micro site, go to the Dell website.
2. From the main menu, go to Products and select Data Protection.
3. Look for Cyber Recovery in the Data Protection Portfolio and click it.
4. Explore the Cyber Recovery micro site.

After watching the Cyber Recovery Overview video:

1. Which type of strategy should every organization have?

Cyber Resilience Strategy

2. Which three attributes are presented as key features of the Cyber Recovery
solution?

Protection, integrity, and confidentiality.

• Cyber Recovery Support Documentation

Documentation for Cyber Recovery can also be found in the Dell support website. Under
Search Support, search for PowerProtect Cyber Recovery. On the Cyber Recovery section
an overview, drives and other files, documentation and advisories can be found.
The documentation pages of the support website have three different sections Top
Solutions, Knowledge Base Articles, and Manuals and Documentation. All the content can
be filtered based on the model version. To filter, select the drop-down menu. Once a version
is selected, the filters are presented on the left.
Release notes describe features, known issues, and workarounds for the current version of
Cyber Recovery.
The Product Guide provides information about how to use Cyber Recovery to configure and
manage users, policies, jobs, assets, and alerts and events. The document also describes
the Cyber Recovery UI and CLI and provides a link to the Cyber Recovery REST API
documentation.
The Security Configuration Guide provides an overview of PPCR settings for access control,
log files, communication, and data security. This guide also includes useful information
about PPCR licensing and code integrity, security patches, malware protection, and manual
vault security.
The Installation Guide provides instructions on how to install, implement, and deploy PPCR.
It also provides a methodology to verify it is working properly.
The AWS Deployment Guide describes how to deploy the PPCR solution to an AWS virtual
public cloud (VPC).

4
The Azure Deployment Guide describes how to deploy the PPCR solution to the Microsoft
Azure public cloud.
The GCP Deployment Guide describes how to deploy the PPCR solution to the Google
Cloud.
The CLI Reference Guide describes how to use the PPCR command-line interface (CLI).
➢ Solution Overview

The solution maintains mission-critical business data and technology configurations in a

secure, air-gapped Vault environment that can be used for recovery or analysis. The Cyber
Recovery Vault is physically isolated.
The Cyber Recovery solution enables access to the Cyber Recovery Vault only long enough
to replicate data from the production system. At all other times, the Cyber Recovery Vault is
secured and off the network. Deduplication expedites the replication process so that
connection time to the Cyber Recovery Vault is as short as possible.
The Cyber Recovery software creates point-in-time (PIT) retention-locked copies. The
copies can be validated and then used for recovery of the production system. Policies and
retention locks are part of the Cyber Recovery solution.

5
 ➢ Benefits
• Introduction

Modernize and automate recovery and business continuity strategies and leverage the latest
intelligent tools to detect and defend against cyber threats. The result is reduced business
risk that is caused by cyber-attacks and a more cyber resilient approach to data protection.
PPCR provides proven, modern, and intelligent protection to isolate critical data, identify
suspicious activity, and accelerate data recovery. The solution allows the users to quickly
resume normal business operations. PPCR also identifies suspicious activity and
accelerates recovery to allow you to quickly resume normal business operations.
• Reduced Risk of Cyber Attacks

The impact of being unable to recover critical data and resume business operations after an
attack can be devastating. PPCR provides automated air gap with data isolation and
governance, CyberSense analytics and machine learning to monitor data integrity, and
forensic tools to discover, diagnose, and remediate ongoing attacks.
• Data Isolation and Governance

The isolated data center environment that is disconnected from corporate and backup
networks are restricted from users other than those with proper clearance.
Policy-driven automated workflows allows users to securely move business critical data into
an isolated environment. An intuitive yet powerful dashboard gives users the power to create
protection policies in fewer than five steps while monitoring potential threats in real time.
• Automated Data Copy and Air Gap

Create unchangeable data copies in a secure digital vault and processes that create an
operational air gap between the production or backup environment and the vault.
• Intelligence Analytics and Tools

Machine learning and full-content indexing with powerful analytics are used in the solution
by keeping the vault safe. Automated integrity checks are performed to determine whether
the data is impacted by malware and tools to support remediation.
• Recovery and Remediation

Workflows and tools to perform recovery after an incident using dynamic restore processes
and existing DR procedures of a customer. PPCR protects and isolates critical data from
ransomware and other sophisticated threats. Machine learning identifies suspicious activity
and allows recovery of known good data.
PPCR provides automated restore and recovery procedures to bring business critical
systems back online. Recovery is integrated with the company’s incident response process.
After an event occurs, the incident response team analyzes the production environment to
determine the root cause of the event.

6
CyberSense also provides post-attack forensic reports to understand the depth and breadth
of the attack and provides a listing of the last good backup sets before corruption. Then,
when the production is ready for recovery Cyber Recovery, CyberSense provides
management tools and technology that performs the actual data recovery.
• Sheltered Harbor

The Sheltered Harbor standard incorporates cyber resilience and data protection best
practices and safeguards for protecting U.S. financial data. Cyber threats, including
ransomware, data destruction, or theft targeting production and backup systems, put
consumer and corporate financial data at risk.
Dell Technologies is the first solution provider in the Sheltered Harbor Alliance Partner
Program that has developed a Sheltered Harbor turnkey data vaulting solution for U.S.
financial institutions. Sheltered Harbor endorsed PPCR on-premises turnkey data vaulting
solution. It meets all technical product requirements for Participants implementing the
Sheltered Harbor standard.
• Solution Planning and Design

Optional Dell Advisory Services help you determine which business critical systems to
protect and can create dependency maps for associated applications and services, as well
as the infrastructure needed to recover them. The service also generates recovery
requirements and design alternatives. It identifies the technologies to analyze, host and
protect your data, along with a business case and implementation timeline.
➢ Architecture

As shown in the following diagram, the CR solution uses PPDD systems to replicate data
from the production system to the CRV (Cyber Recovery Vault). Replication is done through
a dedicated replication data link. The CRV is disconnected from the production network
through an automated air gap. The vault stores all critical data off-network to isolate it from
attack. Cyber Recovery automates data synchronization between production systems and
the vault by creating immutable copies with locked retention policies.

7
 ➢ Components
• Components

PPCR provides proven, modern, and intelligent protection to isolate critical data. CR
identifies suspicious activity and accelerates data recovery allowing you to quickly resume
normal business operations.
• Production DD System

The source PPDD contains the production data that the CR solution protects.
• Cyber Recovery Vault (CRV)

The PP CRV offers multiple layers of protection to provide resilience against cyber-attacks
even from an insider threat. It moves critical data away from the attack surface, physically
isolating it within a protected part of the data center and requires separate security
credentials and multi-factor authentication for access. The PPDD system in the CRV is the
replication target for the source PPDD.
• CyberSense

PPCR is the first solution to fully integrate CyberSense, which adds an intelligent layer of
protection to help find data corruption when an attack penetrates the data center. This
innovative approach provides full content indexing and uses machine learning to analyze
over 100 content-based statistics and detect signs of corruption due to ransomware.
• Cyber Recovery Software

The CR software orchestrates synchronization, manages, locks the multiple data copies that
are stored on the PPDD in the CRV, and orchestrates recovery. The software also governs
the optional process of performing analytics on data that is stored on the PPDD in the CRV
using the CyberSense feature.
CR software is installed on the management host. This server is installed in the vault
environment.
• Policies

A policy, which can be scheduled, orchestrates the workflow between the production
environment and the CRV. A policy is a combination of objects (such as PPDD storage and
applications) and jobs (such as synchronization, copy, and lock).

8
The CR solution uses policies to perform replications, create point-in-time (PIT) copies, set
retention locks, and create sandboxes. Note the following details about Cyber Recovery
policies:
- A CR policy can govern one or more PPDD MTrees. Only a PPDM policy type can
govern more than one MTree.
- A user can create, modify, and delete policies.
- A single action or carrying multiple actions in sequence can be performed when you
run a policy. For example, run a policy so that it only performs a replication. Or, run
the same policy so that it performs a replication, creates a PIT copy, and then
retention locks the copy.
- Concurrent Sync or Lock actions for a policy cannot be run.
- PPCR 19.12 supports up to 32 CR policies.
• Retention Lock

DD Retention Lock software provides data immutability for a specified time. Retention Lock
functionality is enabled on a per-MTree basis, and the retention time is set on a per-file
basis. Retention Lock is not required for CR but is strongly recommended as an additional
cyber-resiliency measure.
PPDD systems support both Governance mode and Compliance mode retention locking.
Compliance mode is a stricter type of retention locking, which enables you to apply retention
policies at an individual file level. You cannot delete or overwrite locked files under any
circumstances until the retention period expires. Retention Lock Compliance mode is not
supported on:
- The CR solution on AWS, Microsoft Azure, and Google Cloud Platform.
- Dell PP DP4400 Integrated Data Protection Appliance.
- Dell PP DD3300 appliances and PP DDVE storage appliances running versions of
DD OS earlier than DD OS 7.10.
• Recovery Hosts

The backup application recovery server is a designated server to which the backup
application (NetWorker, Avamar, PPDM, or other applications or combination of
applications) and backup application catalog are recovered. Multiple servers can be
deployed, depending on the recovery requirements of the solution. The backup application
recovery server is sized so that all backup applications that are being protected by the CR
solution can be recovered.
If the CR solution is protecting a physical, single-node Avamar system in a production
environment, a single-node Avamar system must also reside in the vault for recovery
purposes.

9
 ➢ Operations

Recovery managers can perform continuous and iterative operations that maintain recovery
data in the CRV if they are needed for restoration. These operations can be performed
separately or in combinations. Except for a recovery, operations can also be scheduled or
triggered manually as needed.
• Replication

PPDD MTree replications are performed from the PPDD production system to the PPDD
system in the Cyber Recovery Vault. Each replication uses PPDD deduplication technology
to match the data in the vault incrementally. A replication operation is referred to as a Sync
in Cyber Recovery.
• Copy

A PIT fast copy is made of the most recent replication. If data recovery is required, the copy
serves as a PIT restore point. You can maintain multiple PIT copies to ensure an optimal
number of restore points. You can mount each copy in a sandbox. The sandbox is a
read/write PPDD fast copy inside the CRV. A fast copy is a clone of files and directory trees
of a PIT copy from the cr-policy-<policy-id>-repo MTree. Data can be scanned for a malware
or analyzed as needed in the sandbox.
• Lock

All files can be secure in a PIT copy from modification by retention locking for a specify
duration.
• Analyze

Locked or unlocked copies can be analyzed with various tools that search for indicators of
compromise, suspicious files, or potential malware. These anomalies might identity a copy
as an invalid source for recovery.
• Recovery

The data in a PIT copy can be used to perform a recovery operation.

• Recovery Check

Run a scheduled or on-demand recovery check on a PPDM recovery to ensure that after a
successful recovery a copy can be recovered.

10
 ➢ PowerProtect Cyber Recovery Workflow

1. Overview

A typical PPCR environment is shown here. The production site includes a DD system with
Avamar, PPDM, or NetWorker installed. The Vault side has a similar build with Cyber
Recovery and CyberSense.
2. PPDD MTree

When the data is in the PPDD MTree, we are going to write an AV Checkpoint. This
checkpoint includes the data and metadata of Avamar in the PPDD.
3. Cyber Recovery Air-Gap

When the data is stored in the production PPDD, a link is established between the
Production site and the Vault, and an initial replication takes place. Data is replicated from
the production MTree to the vault MTree. After the initial replication concludes, the link will
be disabled.
4. Cyber Recovery Management

CR software exists in the vault. Storage is added by providing the vault PPDD credentials
to the software.
Next, a policy is created. It dictates which PPDD is used, which replication context is
managed, when to synchronize, and how long to keep the copies. The policy specifies the
Ethernet interface to use for replication.
5. Vault MTree

CR opens communication between the Production and the Vault PPDD. PPDD performs a
sync-as-of-time which pulls the replication into the vault. PPDD compares the production
and vault data to make sure it is consistent. After the data is copied over, the context is
disabled, and the link is air-gapped.

11
 6. Second MTree

Once data is copied over, an MTree is created where retention lock is applied. A Fast Copy
(pointer-based replication) is performed between the original replication MTree and the
retention lock MTree. Manual or scheduled copies can be easily created with CR. When the
retention lock date expires, copies can be deleted manually or through a policy.
7. Sandbox

To do sandbox analysis, create it from CR. CR can create the MTree as it was in the
production site and make a Fast Copy into the MTree. This is then rehydrated to create an
identical copy as it was in production.
8. CyberSense Analysis

When the Sandbox copy is created, CR through its APIs, communicates with the
IndexEngines CyberSense server. Cyber Recovery indicates an MTree is available for
analytics. The MTree is mounted on the PPDD system, via an NFS mount, and initializes an
analysis on the backup.
The result of the analysis is sent from the CyberSense host to Cyber Recovery to indicate
the status of the analysis. A green light means there were no changes to the data, a red light
means the data changed and could have been tampered with.
The analysis begins with an index job, where more than 100 analytics and statistics are
created for each job. An index jobs results in the creation of one or more segments. Large
jobs have multiple segments.
After the index job creates a segment, the post-processing phase begins. This phase
optimizes the segment for analysis and searching.

12
 ➢ Cyber Recovery in the Public Cloud
• The Case for Multicloud Data Services

Hybrid and multicloud environments offer operational flexibility, the ability to scale up quickly,
and access to innovative services and hardware. However, the approach of scattering and
duplicating data across multiple clouds can lead to new security and compliance risks,
potential synchronization issues, and increased resource costs. This approach can also
reduce visibility across various environments, leading to insufficient protection from
constantly evolving cyber threats of today.
A better way is needed to make data simultaneously accessible to public cloud providers
without compromising security. A better way is needed to retain freedom to choose any
cloud provider and avoid vendor lock-in. As more workloads and data are moved to the
cloud, it is imperative to invest in a cyber protection solution for critical data. The solution
should exist wherever the data lies. Dell Technologies delivers a secure data vault and
intelligent analytics that safeguards your critical data from cyber-attacks, ransomware, and
insider threats.
When combined with Multicloud Data Services for Dell PP, clients achieve sovereign data
protection across all clouds (AWS, Google Cloud, Oracle, and Azure). Clients are able to
protect their critical data within a secure CRV. Multicloud Data Services for Dell PP can be
used as a multipurpose system: a backup target for cloud-native application data or a
replication target for existing PP systems. The CRV is an additional option that can be added
to provide isolation of critical data from cyber-attacks and validation of data integrity.
Customers can replicate data from an on-premises PPDD to a CRV in one of the data
centers of Faction (leading multi-cloud service provider). If a cyber-attack occurs, users can
quickly identify the most current clean copy of data within the remote CRV. Users can then
recover their critical system back on-premises or choose to recover into the cloud if their
service has been architected with this recovery motion.
For cloud-native applications already using PP DDVE, the CRV service is an optional
service. This service enables customers to replicate critical data to a secure vault.

13
 • Cyber Recovery in Azure

Dell PPCR for Azure provides proven, modern, and intelligent protection to isolate critical
data. The solution also accelerates data recovery, allowing users to resume normal business
operations quickly.
PPCR for Azure offers multiple layers of protection to provide resilience against cyberattacks
and insider threats. It moves critical data away from the attack surface, physically and
logically isolating it from access within Azure with a secure, automated operational air gap.
Networking controls lock-down access to management interfaces unlike standard cloud-
based backup solutions. Management interfaces can require separate security credentials
and multifactor authentication for access.
Automated workflows securely move business critical data to an isolated environment within
Azure. The vault components are never accessible from production. Access to the vault
storage, when the air gap is unlocked, is limited, and is protected within a secure Azure
Virtual Network.
Cyber Recovery for Azure enables recovery of critical data from the vault after a cyberattack
or for recovery testing procedures. These process recovers the data back to the corporate
data center, or an alternate, or to a new VNET or clean environment within Azure.

14
 • Cyber Recovery in AWS

The Cyber Recovery solution can be deployed on AWS. The solution works with the PP
DDVE storage appliance in an AWS VPC. The components store replicated data from a
production DD system in a secure vault environment. This data can then be recovered to
the production DD system.
The production environment can be on premises or also deployed on AWS or another cloud
provider.
The software enables and disables access to both a private subnet and DDVE in CRV. Flow
of data is allowed by enabling both the replication link and the replication port of the DD
system. When a policy finishes synchronizing data in to the CRV using the replication link,
the CR software disables the replication link. When all policies no longer use a specific DD
port to synchronize data into the CRV, the CR software disables the port by bringing down
the interface.
CR for AWS is the latest data protection solution available as a transactable offer through
AWS Marketplace enabling users to leverage their existing AWS subscription.

➢ Cyber Recovery in GCP

The CR software manages a virtual air gap between a production environment and the CRV.
It disables replication links and replication ports con the DD system in the CRV when CR
policies are idle. The software enables and disables access to both a private subnet and
DDVE in the CRV, which are installed during the solution deployment, through GCP firewall
rules.
When a policy runs, the CR software enables the flow of data into the CRV by enabling both
the replication link and the replication port of the DD system. When a policy finishes
synchronizing data into the CRV using the replication link, the CR software disables the
replication link. Also, when all policies no longer use a specific DD port to synchronize data
into the CRV, the CR software disables the port by bringing down the interface.
GCP firewall rules provides virtual private cloud (VPC) security that provides additional
security measures for the CRV. The CR software enables and disables access to a private
subnet and enables and disables access to an instance through firewall rules.

15
 ➢ CyberSense Solution Overview

Real-time cybersecurity solutions are designed to protect against an attack. However, these
solutions are not 100% effective, and corporate data is still corrupted daily.
CyberSense adds a layer of protection to the real-time solutions, finding corruption that
occurs when an attack has successfully entered the data center. It enables quick recovery
after the cyberattacks so that you can avoid business interruption.
CyberSense uses a unique approach in uncovering cyberattacks. It observes how data
changes over time and uses analytics to detect signs of corruption due to ransomware. The
approach uses machine learning to analyze over 100 content-based statistics and finds
corruption with up to 99.5% confidence. It also helps to protect business-critical
infrastructure and content.
It detects mass deletions, encryption, and other types of changes in files and databases that
result from common attacks. If CyberSense detects signs of corruption, an alert is generated
with the attack vector and listing of files affected.
CyberSense provides forensic reports to diagnose the cyberattack. With CyberSense,
organizations can proactively audit the files and databases to determine when an attack
begins. It also helps in quickly recovering with the last good version of the data before there
is any interruption to the business.

• Features and Benefits of CyberSense

CyberSense delivers a unique approach: auditing data content to determine if it has been
compromised. Here are some key features and benefits of CyberSense:
• Fully integrated with PP CRV by directly scanning all common backup software
images.
• More than 100 statistics are generated to look inside the data for any unusual
behavior.
• A machine learning algorithm generates a Yes or No indicator to identify an attack.
• Forensic tools are used to find any corrupted files and diagnose the attack vector.
• CyberSense allows users to restore the last good file to minimize any business
interruptions.

16
 • CyberSense Terminology

Term Definition

CyberSense scans critical data sources in the PP CRV. This includes

Scan
unstructured files and databases to create an observation.

More than 100 statistics generated from each observation. Statistics include
Analytics analysis of file, entropy, similarity, corruption, mass deletion, or creations,
and much more.

Machine learning algorithms are used to analyze the statistics to indicate if

Analysis
an attack on the data has occurred.

The process repeats as CR backs up data incrementally to the vault and a

Repeat new observation is created. New observations are compared to previous
observations to see how data changes.

Forensic reporting and analysis tools are available after an attack to find
Investigate
corrupted files and diagnose the types of ransomware.

• CyberSense Architecture

The graphic shows the architecture of CyberSense in PPCR solution. CyberSense

integrated with PPCR provides a secure and powerful solution to combat ransomware and
other cyber-attacks.
When an attack gets past real-time defenses and corrupts the files or databases, you have
confidence that clean data is isolated in the CRV and CyberSense is analyzing the data.
• CyberSense Operations Overview

Here is the workflow when CyberSense is integrated with PPCR solution: After data is
replicated to the CRV and retention lock is applied:
1. CyberSense scans the backup data creating point-in-time observations of files and
databases. The scanning occurs directly on the backup data within the backup image
without the need for the original backup software.
2. The analytics are generated including the file type mismatch, corruption, known
ransomware extensions, deletions, entropy, similarity, and more.
3. The machine learning algorithms use the analytics to make a deterministic decision on
data corruption which is the indication of a cyberattack.
a. With CyberSense, organizations can proactively audit the files and databases to
determine when an attack begins. It also helps in quickly recovering with the last
good version of the data before there is any interruption to the business.
b. The observations of data allow CyberSense to track how contents of file change
over time.
4. A critical alert is displayed in the CR dashboard when an attack occurs.

The forensic reports and reporting tools are available after the attack to diagnose the
corrupted files and recover from the ransomware attack.

17
 • Supported Data Types

CyberSense generates analytics from a comprehensive range of datatypes. The datatypes

includes core infrastructure such as Domain Name Server (DNS), Lightweight Directory
Access Protocol (LDAP), Active Directory (AD); unstructured files such as documents,
contracts, and agreements; intellectual property and databases such as Oracle, DB2, SQL,
Epic Cache, and others.
• CyberSense Documentation

The knowledgebase articles, manuals, release notes, and documents on CyberSense can
be accessed from the Dell Support portal
All CyberSense documentations and downloads can be downloaded from the Index Engine
Support Portal.

18
 o PowerProtect Cyber Recovery Features
➢ PowerProtect Cyber Recovery Workflow

PPCR protects the data that drives business, the same data that cyber-attacks and
ransomware target. Automation and intelligent security isolates data away from the attack
surface with an operational air gap. Stored immutable within a dedicated cyber vault, users
can respond, recover, and resume normal business operations with confidence that the data
and business are protected with PPCR.

➢ Cyber Recovery UI

The CR solution provides a web-based UI, API, and CLI. The web-based CR UI is the
primary management and monitoring tool. It enables users to define and run policies,
monitor operations, troubleshoot problems, and verify outcomes.

To access the CR UI, go to https://<hostname>:14777, where the hostname represents the

CR management host.

➢ CRCLI
The CRCLI is a command-line alternative to the CR UI. The commands represent a subset
of the functionality that is available in the CR UI. If the CR software is installed using the
default locations, the CRCLI is in the /opt/dellemc/cr/bin directory. Use the crcli help
command to view the help system.
➢ Cyber Recovery REST API
The CR REST API provides a predefined set of operations that administer and manage
tasks over HTTPS. REST API is used to create a custom client application or to integrate
CR functionality into an existing application.
To access the CR REST API documentation, go to https://<hostname>:14780, where
<hostname> is the hostname of the management host.
➢ CyberSense UI
Together with PPCR 19.12, CyberSense 8.0 was released. This new version of CyberSense
introduces a new UI, different Linux support, and multiple performance improvements.
CyberSense 8.0 is supported in AWS. Installation parameters are configured. A new
Analyze Dashboard is introduced.

19
 ➢ Cyber Recovery in the Public Cloud
◼ The Case for Multicloud Data Services
Hybrid and multicloud environments offer operational flexibility, the ability to scale up quickly,
and access to innovative services and hardware. However, the approach of scattering and
duplicating data across multiple clouds can lead to new security and compliance risks,
potential synchronization issues, and increased resource costs. This approach can also
reduce visibility across various environments, leading to insufficient protection from
constantly evolving cyber threats of today.
A better way is required to make data simultaneously accessible to public cloud providers.
Data can be accessible without compromising security, retain freedom to choose any cloud
provider and avoid vendor lock-in. As more workloads and data are moved to the cloud, it is
imperative to invest in a cyber protection solution for critical data. The solution should protect
wherever the data lies. Dell delivers a secure data vault and intelligent analytics that
safeguards your critical data from cyber-attacks, ransomware, and insider threats.
◼ Multi-cloud Data Services
When combined with Multi-cloud Data Services for Dell PowerProtect, clients achieve
sovereign data protection across all clouds (AWS, Google Cloud, Oracle, and Azure).
Clients are able to protect their critical data within a secure CRV. Multi-cloud Data Services
for Dell PP can be used as a multipurpose system: a backup target for cloud-native
application data or a replication target for existing PP systems. The CRV is an additional
option that can be added to provide isolation of critical data from cyber-attacks and validation
of data integrity.
Customers can replicate data from an on-premises PPDD to a CRV in one of Faction’s
(leading multi-cloud service provider) data centers. This approach gives organizations the
best possible chance for recovery when their production or primary backups have been
compromised or their DR location has been breached or infected. If a cyber-attack occurs,
they can quickly identify the most current clean copy of data within the remote CRV. The
customer can recover their critical systems back on-premises or choose to recover into the
cloud if their service has been architected with this recovery motion.
For cloud-native applications already using PP DDVE, the CRV service is an optional service
that enables customers to replicate critical data to a secure vault.

20
 ◼ Cyber Recovery in an Azure Environment
Dell PPCR for Azure provides proven, modern, and intelligent protection to isolate critical
data and accelerate data recovery, allowing you to resume normal business operations
quickly.
PPCR for Azure offers multiple layers of protection to provide resilience against cyberattacks
and insider threats. It moves critical data away from the attack surface, physically and
logically isolating it from access within Azure with a secure, automated operational air gap.
Unlike standard cloud-based backup solutions, access to management interfaces is locked
down by networking controls and can require separate security credentials and multifactor
authentication for access.
Automated workflows securely move business critical data to an isolated environment within
Azure. The vault components are never accessible from production, and access to the vault
storage, when the air gap is unlocked, is limited, and is protected within a secure Azure
Virtual Network.
CR for Azure enables critical data from the vault after a cyberattack or for recovery testing
procedures. These process recovers the data back to the corporate data center, or an
alternate, or to a new VNET or clean environment within Azure.
PPCR for Azure is available as transactable offer through Azure Marketplace. Users can
leverage their existing Azure subscription.
◼ Cyber Recovery in an AWS Environment
The CR solution can be deployed on AWS. The solution works with the PP DDVE storage
appliance in an AWS VPC to store replicated data from a production DD system in a secure
vault environment. This data can be recovered to the production DD system. The production
environment can be on premises or also deployed on AWS or another cloud provider.
The software enables and disables access to both a private subnet and DDVE in the CRV.
Private subnets are installed during the solution deployment, through security groups and
ACLs.
When a policy runs, the CR software enables the flow of data into the CRV by enabling both
the replication link and the replication port of the DD system. When a policy finishes
synchronizing data into the CRV using the replication link, the CR software disables the
replication link. Also, when all policies no longer use a specific DD port to synchronize data
into the CRV, the CR software disables the port by bringing down the interface.

21
 ◼ Introduction to Cyber Recovery in Google Cloud Platform
Customers can deploy the PPCR solution on the GCP. The solution works with the PP DDVE
storage appliance in a GCP VPC to store replicated data from a production PPDD system
in a secure vault environment. This data can be recovered to the production PPDD system.
The production environment can be on premises, or deployed on GCP, or another cloud
provider. GCP firewall rules provide VPC security that provides additional security measures
for the CRV. The CR software enables and disables access to a private subnet and enables
and disables access to an instance through firewall rules.
◼ PowerProtect Cyber Recovery for GCP Components
The CR software is made available as VM image. To deploy the CR software in GCP, use
a Terraform template. The Terraform template also deploys a CR jump host. The Windows-
based jump host is available in the VPC to access the CR and DDVE instances. The
management path is through the jump host.
Terraform templates provide capabilities for packing infrastructure and configuration
components in the same place. Virtual machines, network components, databases, and
configuration files are stored in an easy to use format.
The Terraform template creates:

• Two CR VPCs - The VPCs includes all the components required for the CR solution.
• Three subnets - The three private subnets include a subnet with the CR jump host,
a subnet with the CR management host and DDVE, and a subnet with a second
DDVE network interface that is used for replication.
• A Google storage bucket for DDVE storage.
• Firewall rules.
The CR deployment using Terraform does not include a VPN. Dell Technologies strongly
recommends setting up a VPN when deploying CR. Use a VPN gateway or Google Cloud
Interconnect to access the jump host.
◼ PowerProtect Cyber Recovery on GCP Architecture
The basic CR solution on GCP architecture includes a single region, two VPCs, and a single
availability zone (AZ).

22
 ➢ Assets Overview
Assets in the CRV are represented as storage, application, and vCenter server objects.
These assets are in the Vault. When the assets are discovered, CR operations can be
performed on them. Power on all assets before you add them to your CR deployment.
◼ Storage Objects
Storage objects represent storage systems, such as PPDD systems. Define a storage object
for each PPDD system that is running in the CRV. The CR software uses the PPDD system
to perform replications, store PIT copies, and apply retention locking. Storage objects are
required to protect data through MTree replication.
◼ Application Objects
Application objects represent applications, such as Avamar, NetWorker, or PPDM, or the
CyberSense feature. The CyberSense feature is only supported as a component of the CR
solution in the CRV. The CyberSense feature is not supported on the production system.
The CR software integrates with the CyberSense feature application, which analyzes
backup data for the presence of malware or other anomalies. After the CyberSense feature
is installed on a separate host in the CRV, define an application object to it. Then, CR
policies can call the CyberSense feature to analyze PIT copies of supported datasets.
Avamar, NetWorker, and PPDM backup applications are in the CRV when the PPDD system
is integrated with those applications in the production systems. The CRV does not require
these applications to protect the data because MTree replications copy all the data to the
CRV. Running the applications in the CRV enables you to recover and restore your data so
that it can be used to rehydrate production backup applications, if necessary.
◼ vCenter Server Objects
If PPDM is used to perform a recovery in the CRV, add a vCenter server asset. Otherwise,
a PPDM recovery fails.
➢ Manage Storage Objects
◼ Add
When adding PPDD, a replication context must be configured between production and vault
systems. The following steps are required to add storage objects.
1. From the Main Menu, select Infrastructure > Assets.
2. Click VAULT STORAGE at the top of the Assets content pane.
3. To add a storage object, click ADD.
4. Complete the storage configuration.
5. Click SAVE. The VAULT STORAGE table lists the storage object.
6. Click in the row for the storage objects to view more detailed information that is
retrieved from the PPDD system. This information includes replication contexts and
the Ethernet interface.

23
 ◼ Edit
1. To edit vault storage, go to infrastructure, and then assets.
2. Under assets, the first tab shows vault storage. The vault storage shows the PPDD
systems added.
3. Select the checkbox next to the storage to edit. Then select Edit from the top menu.
4. The edit vault storage window is displayed. Users can change all the parameters for
the storage.
5. If the FQDN or IP of the storage changes, select the Reset Host Fingerprint
checkbox.
6. Select Save to complete the Edit Process.
◼ Delete
1. To delete vault storage, select the checkbox next to the storage to delete. Select
delete.
2. A confirmation message is displayed. Select delete to confirm.
The storage is deleted and no longer present in the vault storage view. If active policies are
configured in this PPDD system, it cannot be deleted.
➢ Manage Applications
Applications that are installed in the CRV must be represented to the CR software.
Applications include the Avamar, NetWorker, and PPDM applications, and the CyberSense
feature. The application must be installed and running at the CRV location before they can
be defined in the CR UI.
◼ Add
The process of adding an application to the CRV varies depending on the type of application
to be added. Host OS credentials and sometimes, application credentials are required. The
host credentials that are required must be the root user. For PPDM, the system requires the
vCenter where it resides to be added. This way the vCenter is selected from a drop-down
list.
1. From the Main Menu, select Infrastructure > Assets.
2. Click APPLICATIONS at the top of the Assets content pane.
3. To add an application, click ADD.
4. Complete the application configuration parameters.
5. Click SAVE. The APPLICATION table lists the storage object.
6. Click in the row for the application to view more detailed information.

24
 ◼ Edit
To edit applications in CR, follow these steps:
1. From the Infrastructure, assets section, select the applications tab.
2. A list of added applications is shown. Select the checkbox next to the application and
then select Edit.
3. The edit vault application window is shown. Password fields are blank. However, the
passwords are not necessary when editing an application.
4. If the FQDN or IP address of the application changes, ensure to select Reset Host
Fingerprint checkbox.
5. Select save to complete the changes to the application.
◼ Delete
To remove an application, follow these steps:
1. Under Infrastructure, select Assets and select the Applications tab.
2. Select the checkbox next to the application and select the remove button.
3. A confirmation message is displayed. Select Delete to complete the application
deletion process.
Applications that have active policies cannot be removed.
➢ Manage vCenter Servers
◼ Add
When a vCenter system is installed in the CRV, users must present the vCenter to the CR
software.
1. To add vCenter servers, select Infrastructure, and then Assets. Select the vCenters,
and then select Add.
2. Specify a nickname and FQDN or IP address for the vCenter Server.
3. Then, specify the vCenter host administrator username.
4. Tags are optionally set to provide useful information about the application. If a tag
exceeds 24 characters, only 21 characters are displayed.
◼ Edit
To edit a vCenter Server in CR follow these steps:
1. Check the checkbox next to the vCenter. Select Edit.
2. The nickname and FQDN or IP address can be updated. A new username and
password can be specified too.
3. Notice there is no option to Reset Host Fingerprint. It is not necessary to create a
new host fingertip.
4. Tags are not edited; however a tag can be cleared, and more tags can be created
from this view.
5. To complete the edit, select Save.

25
 ◼ Delete
The process for deleting a vCenter Server is the same as storage objects and applications.
1. Select the checkbox next to the vCenter server to delete.
2. Then, select Delete.
3. A confirmation message is displayed. Select delete to complete the process.
➢ Policies and Copies
The CR solution uses policies to perform replications, create PIT copies, set retention locks,
and create sandboxes. Users can create, modify, and delete policies. When a policy is
executed, a single action or multiple actions in sequence can be performed. For example, a
policy can be ran so that it only performs a replication. Or, the policy performs a replication,
creates a PIT copy, and then retention locks the copy.
A CR policy can govern one or more PPDD MTrees. Only a PPDM policy type can govern
more than one MTree.
◼ Policies Actions
The CR UI support the Secure Copy Analyze, Secure Copy, Sync Copy, Copy Lock, Sync,
and Copy policy actions.

• Copy
A Copy action makes a PIT copy of an MTree’s most recent replication in the CRV and
stores it in the replication archive.

• Copy Lock
A Copy Lock action retention locks all files in the PIT copy.

• Sync
A Sync action (or replication) replicates an MTree from the production system to the CRV,
synchronizing with the previous replication of that MTree. From the CRCLI, you can perform
a Sync action to a system other than the CRV DD system. Replicate an MTree from the
CRV DD system to the production DD system or an alternate DD system.

• Sync Copy
A Sync Copy action combines the Sync and Copy actions into one request. It first performs
the replication and then creates a PIT copy.

• Secure Copy
A Secure Copy action performs a replicate, creates a PIT copy, and then retention locks all
files in the PIT copy. Retention locking an existing PIT copy is also allowed.

• Secure Copy Analyze

A Secure Copy Analyze action performs a replication, creates a PIT copy, retention locks all
files in the PIT copy, and then runs an analysis on the resulting PIT copy.

26
 ◼ Migrate Replication Contexts
CR software detects the context when a policy is created with a Retention Lock compliance
replication context. Likewise the context is detected when an existing policy is modified to
add a Retention Lock Compliance replication context. If the deployment is running DDOS
7.8, the CR software modifies a setting on the DD system in the CRV. This one-time
modification enables the CR software to support Retention Lock Compliance contexts.
When a policy is created that uses a Retention Lock Compliance replication context, the CR
UI and CRCLI prompt you for the Security Office (SO) credentials. By default, the security
authorization for disabling replications is set to enabled. This setting means that the PPDD
system continues to prompt for the SO credentials when the CR software attempts to disable
a replication at the end of any Sync action. So that the workflow is not impeded, when a
policy is created that uses a Retention Lock Compliance replication context, the CR software
changes the setting to disabled. This change ensures that for subsequent workflow actions
that disable replications and require SO credentials, the CR software is not required to
provide these SO credentials.
If a replication context that is configured in a CR policy is migrated to a Retention Lock
Compliance replication context using the same name, the CR software cannot detect this
change. The replication context is migrated to a Retention Lock Compliance replication
context, but the CR software does not modify the setting on the DD system.
◼ Authorization for Replication
Unlike a policy creation, the CR software does not change the authorization for replication
disable setting to disabled on the PPDD system if it is in the enabled state (the default
setting). The user must change the setting manually on the DD system.
The following command is run on the PPDD system to verify the current authorization for
replication disable setting on the PPDD system:
System replication security-auth repl-disable status
If the status is enabled, run the following command on the PPDD system to set the
authorization for replication disable setting to disabled:
System replication security-auth repl-disable disable
This command requires SO credentials. It provides a one-time modification on the PPDD
and enables future Retention Lock Compliance migrations to work properly.

27
 ◼ Copies
Copies are the PIT MTree copies that serve as restore points that can be used to perform
recovery operations. In the CR UI, users can retention lock a copy or analyze its data to
detect the presence of malware or other anomalies. Also, unlocked copies can be deleted.
Secure a PIT copy for a specific retention period during which the data in the PIT copy can
be viewed, but not modified. If a copy is already retention locked, you can extend (but not
decrease) the current retention period. Analysis are performed for a PIT copy by using the
CyberSense feature in the CRV.
➢ Sandboxes
A sandbox is a unique location in the CRV in which you can perform read/write operations
on a PIT copy. This copy is a read/write copy of the locked data in the CRV. The CR software
supports two types of sandboxes:
System Sandboxes
The CR software enables users to create custom sandboxes manually to perform operations
by using applications that are not in the CR default list. A sandbox contain only one PIT
copy; however, the user can create multiple sandboxes for one PIT copy. Sandboxes are
created as needed for data analysis or validation operations. The CyberSense feature
software automatically creates a system sandbox when an analyze operation is created on
a PIT copy.
Recovery Sandboxes
The CR software automatically creates recovery sandboxes when a NetWorker, Avamar, or
PPDM recovery is initiated.
➢ CyberSense Solution
Real-time cybersecurity solutions are designed to protect against an attack. However, these
solutions are not 100% effective, and corporate data is still corrupted daily.
CyberSense adds a layer of protection to the real-time solutions, finding corruption that
occurs when an attack has successfully entered the data center. It enables quick recovery
after the cyberattacks so that you can avoid business interruption.
Uses a unique approach in uncovering cyberattacks. It observes how data changes over
time and uses analytics to detect signs of corruption due to ransomware. The approach uses
machine learning to analyze over 100 content-based statistics and finds corruption with up
to 99.5% confidence. It also helps to protect business-critical infrastructure and content.
Detects mass deletions, encryption, and other types of changes in files and databases that
result from common attacks. If CyberSense detects signs of corruption, an alert is generated
with the attack vector and listing of files affected.
It provides forensic reports to diagnose the cyberattack. With CyberSense, organizations
can proactively audit the files and databases to determine when an attack begins. It also
helps in quickly recovering with the last good version of the data before there is any
interruption to the business.

28
 ➢ Features and Benefits of CyberSense
CyberSense delivers a unique approach: auditing data content to determine if it has been
compromised. Here are some of the key features and benefits of CyberSense:

• CyberSense is fully integrated with Dell PPCRV by directly scanning all common
backup software images (Avamar and NetWorker).
• More than 100 statistics are generated to look inside the data for any unusual
behavior.
• A machine learning algorithm generates a Yes or No indicator to identify an attack.
• Forensic tools are sued to find any corrupted files and diagnose the attack vector.
• CyberSense allows you to restore the last good file to minimize any business
interruptions.
➢ Analyze a Copy Using CyberSense
The CyberSense feature in the CRV allows users to analyze a PIT copy that is created by
the policy. The CyberSense feature is only supported as a component of the CR solution in
the CRV. It is not supported on the production system. The copies can be analyzed using
the Analyze option.
To analyze a PIT copy from the PPCR interface:
1. Select Policies from the main menu and click Copies to display the list of existing
copies.
2. Select the copy and click Analyze.
If you do not have a valid license for the CyberSense feature, and then the Analyze option
is disabled.
3. From the Application Host list box, select the CyberSense feature.
4. Optionally, you can choose the content format (either Filesystem, Databases, or
Backup) from the drop-down menu, and click Apply.
You can retrieve detailed analysis reports for completed jobs using the Analysis Report
Actions option.
As result, the policy starts a job that can be viewed on the Jobs page. If the analysis indicates
possible malware or other anomalies, the CR software generates an alert, and the job status
is listed as Critical. Otherwise, the job status is listed as Success.
The Analysis Report Actions option in the Copies tab is disabled when you request a report
for partially completed, failed, or canceled analysis jobs; and multiple copies: you can
request a report for only one analyzed copy at a time.

29
Analysis report is available in two ways:

• Download Analysis Report

Downloads an analysis report for a specified copy to the local system.

• Email Analysis Report

Sends an analysis report for a specified copy in an email message. You can send the
analysis report at to one or multiple valid email addresses.
An analysis report is available only for a successful completed job for a single copy.
➢ User and Credential Management
◼ User Roles
CR users are assigned roles that determine the tasks that they can perform in the CRV
environment. The CR installation creates the default crso user and assigns the Security
Officer (crso) role to this user. The SO user must perform the initial CR login and then create
users.
With the release of CR 19.12, multiple SO users are now supported. The crso superuser
can create SOs. SOs other than the crso cannot create additional SOs.
There are three CR user roles:

• Dashboard
This role enables the user to view the CR dashboard but not perform tasks.

• Admin
Create, manage, and run policies and associated objects. Acknowledge and add notes to
alerts. Change administrative settings. Modify own user account. Change own password.
Manually secure and release (unsecure) the CRV.

• Security Officer (SO)

All Admin permissions. Create, modify, and disable users. Change and reset user
passwords. Change the SO password. Set the duration after which passwords expire for all
users. Disable multifactor authentication for an Admin user. Configure the daily activity
report. Configure the number of login sessions.
◼ Default Credentials
A CR virtual appliance deployment requires you to set a password for the root user and
admin user during the CR installation. The CR virtual appliance system uses the following
default user accounts and default passwords.

User Account Default Password Description

root changeme Linux operating system root account.
Admin changeme Linux operating system administrative account.

30
31
 o PowerProtect Cyber Recovery Administration
➢ Solution Overview

PPCR protects and isolates critical data from ransomware and other sophisticated threats.
Machine learning identifies suspicious activity and allows you to recover known good data
and resume normal business operations with confidence.
The solution maintains mission-critical business data and technology configurations in a
secure, air-gapped ‘vault’ environment that can be used for recovery or analysis. The CRV
is physically isolated from an unsecure system or network.
The CR solution enables access to the CRV only long enough to replicate data from the
production system. At all other times, the CRV is secured and off the network. A
deduplication process is performed in the production environment. Deduplication expedites
the replication process so that connection time to the CRV is as short as possible.
Within the CRV, the CR software creates point-in-time (PIT) retention-locked copies. The
copies can be validated and then used for recovery of the production system. Policies and
retention locks are part of the CR solution.
➢ CyberSense Solution Overview

Real-time cybersecurity solutions are designed to protect against an attack. However, these
solutions are not 100% effective, and corporate data is still corrupted daily.
CyberSense adds a layer of protection to the real-time solutions, finding corruption that
occurs when an attack has successfully entered the data. CyberSense enables quick
recovery after the cyberattacks so that you can avoid business interruption.
It uses a unique approach in uncovering cyberattacks. It observes how data changes over
time and uses analytics to detect signs of corruption due to ransomware. The approach uses
machine learning to analyze over 100 content-based statistics and finds corruption with up
to 99.5% confidence. It also helps to protect business-critical infrastructure and content.
CyberSense detects mass deletions, encryption, and other types of changes in files and
databases that result from common attacks. If CyberSense detects signs of corruption, an
alert is generated with the attack vector and listing of files affected.
CyberSense provides reports to diagnose the cyberattack. With it, organizations can
proactively audit the files and databases to determine when an attack begins. It also helps
in quickly recovering with the last good version of the data before there is any interruption
to the business.

32
 ➢ CyberSense Operations Overview

After the data is replicated to the CRV and retention lock is applied:
1. CyberSense scans the backup data creating PIT observations of files and
databases.
a. The scanning occurs directly on the backup data within the backup image
without the need for the original backup software.
2. The analytics are generated including the file type mismatch, corruption, known
ransomware extensions, deletions, entropy, similarity, and more.
3. The machine learning algorithms use the analytics to make a deterministic decision
on data corruption which is the indication of a cyberattack.
a. With CyberSense, organizations can proactively audit the files and databases
to determine when an attack begins. It also helps in quickly recovering with
the last good version of the data before there is any interruption to the
business.
b. The observations of data allow CyberSense to track how content of files
change over time.
4. A critical alert is displayed in the CR dashboard when an attack occurs.
5. The forensic reports and reporting tools are available after the attack to diagnose the
corrupted files and recover from the ransomware attack.

The workflow phases go as follow: Scan, Analytics, Analysis, Repeat, and Investigate.
• Supported Data Types

CyberSense generates analytics from a comprehensive range of datatypes. The datatype

includes core infrastructure such as Domain Name Server (DNS), Lightweight Directory
Access Protocol (LDAP), Active Directory (AD); unstructured files such as documents,
contracts, and agreements; intellectual property and databases such as Oracle, DB2, SQL,
Epic Cache, and others.
➢ Differences Between Recovery Types

CR is not operational recovery and or disaster recovery. The first type of recovery is
operational recovery. This recovery is commonly secured through traditional backups and
restores. These can be secured depending on the level of complexity within the same
storage unit or host, across different hosts, or even across remote sites. When a system
becomes inoperable, the latest backup is restored to make it operable again. Metrics such
as recovery point objective (RPO), or recovery time objective (RTO) are important measures
of an operational recovery strategy.
DR relates to larger and more complex causes of inoperability. It is an organization’s method
of regaining access and functionality to its IT infrastructure after events like a natural
disaster, cyber-attack, or even business disruptions related to the COVID-19 pandemic. A
variety of DR methods can be part of a DR plan. When a disaster strikes a local site is
assumed to be lost. Total recovery of the environment must occur from recovery copies.

33
While CR coexists with operational and DR, it serves a very different purpose. The primary
purpose of CR is to secure and restore data that has been compromised by cyber criminals.
The amount of data that is compromised and could be lost is not estimated, as every attack
is different. Recovery is not total or absolute, it is selective. Only data that is needed is
recovered. The recovery takes place from one or many checkpoints depending on the level
of corruption.
• Operational Recovery
o Loss Assumption
▪ Limited loss of data.
o Recovery Plan
▪ Selective recovery.
o Recovery Technique
▪ Restore from production backups.
• Disaster Recovery
o Loss Assumption
▪ Assumes site loss.
o Recovery Plan
▪ Top to bottom recovery from disaster recovery plan.
o Recovery Technique
▪ Recover from DR copies.
• Cyber Recovery
o Loss Assumption
▪ Unknown amount of loss.
o Recovery Plan
▪ Selective recovery. Recover only what is needed.
o Recovery Technique
▪ Recover from one of many checkpoints.

➢ Cyber Recovery Vault Metrics

➔ Protection Objectives
- DDO - Destruction Detection Objective

Expected amount of time from the point of incursion required to detect the intrusion.
- DAO - Destruction Assessment Objective

Once an incursion has been detected, amount of time that is allowed to assess the extent
of the damage and decide of whether to repair or restore a checkpoint copy.

34
 ➔ RPO/RTO Equivalents
- CRP - Cyber Recovery Point

RPO equivalent of data in the vault.

- CRT - Cyber Recovery Time

RTO equivalent of the amount of time that is expected to recover from a cyber-destruction
event.
➔ Checkpoints
- Checkpoint Interval

A measure of time of how frequently to make Checkpoint Copies. Typically 24 hours or

weekly.
- Checkpoint Count

Number of checkpoints to maintain in the vault. Typically 1 to 2 weeks.

35
 ➢ Recovery Process
• Separate Copy Streams

Separate copy streams provide better recovery capabilities. Malware in nearly all cases is
an attack against Production hosts. It is recommended to create binary and executable
distros in a cleanroom. As operating system and application distributions are created, run
them through Change Management Governance. Then make a copy with a backup on an
MTree copied into the CRV. Make the copy into the CRV and perform normal backups on
the production hosts.
Dormant malware can be present on the production host. This is not necessarily bad.
Bringing malware into CR provides an opportunity to use CyberSense to detect the malware
at-rest. The malware will not cause harm in the CRV.
At Recovery, the decision can be made whether to recover from a gold copy OS distro or a
backup.

➢ Typical Recovery Process

The essential steps of a formal Cyber Incident Response plan include the following:
1. Invoke Cyber Incident Response Plan.

The scene of the crime is secured, and the CRV protected by the Air Gap.
2. Perform Forensics

Understand what cause the attack, identify if there are any patches to fix.
3. Perform Damage Assessment

Identify what is working, what can be repaired, and what was destroyed.
4. Prepare for Recovery

Define which recovery technique should be used (restore, repair, or rebuild). Identify which
data checkpoint is the best to use for recovery. Prioritize the applications that need to be
recovered.
5. Recover the Production Environment.

36
 ➔ Restore from Backup

The first recovery technique is to restore from the last backup. This option is viable if the
backup has not been tampered with. This is the fastest possible recovery technique. A
difficulty with this recovery is the attack might re-occur.
1. Start by cleansing the production environment.
2. Continue restoring the binaries. To do so start by restoring backups. Choose
between clean room binaries and patch the exploit if possible. Alternatively, use a
backup image and look for and optionally cleanse the malware. Finally, restore the
binaries to the host.
3. Once the binaries are restores, restore the data. Restore config files from a
cleanroom copy. Restore backups and optionally find clean checkpoints. Then stores
the data and roll-forward available database logs.

➔ Repair Binaries

The second recovery technique is to repair the binaries. This option is necessary if the
binaries have been corrupted.
1. Start by cleansing the production environment.
2. Then repair the binaries. To do so, restore backups and look for clean binary copies.
If no clean copy exists, remove the malware. Optionally fix the exploit. Restore the
binaries to the host.
3. Finally restore the data. Restore the backups and find a clean checkpoint. Restore
the data and optionally roll-forward the available database logs.

37
 ➔ Rebuild from the Cyber Recovery Vault

Rebuilding from the CRV is the recovery method with the highest confidence. This method
provides patches for an exploit and reduces the concerns about dormant malware. The
downside of the process is it takes time to execute. A combined alternative is to restore the
best copy of data and rebuild the computer platforms in the background. Then connect the
data to the new servers when it is more convenient to do so.
1. Start by formatting the data stores. To do so format the internal and external storage.
2. Rebuild the binaries. To do so, restore the binaries and apply exploit patches. Then
distribute to the different hosts.
3. Recover the application by restoring config files and data. Roll forward clean logs
and perform application recovery.

➢ Assets

Assets in the CRV are represented as storage, application, and vCenter server objects.
- Storage Objects

Storage objects represent storage systems, such as PPDD systems. The CR software uses
the PPDD system to perform replications, store point-in-time (PIT) copies and apply
retention locking.
- Application Objects

Application objects represent applications, such as PPDM, Avamar, NetWorker, or the

CyberSense feature.
- vCenter Server Objects

If you plan to use PPDM to perform a recovery in the CRV a vCenter server asset is required.
Otherwise, a PPDM recovery fails.

38
 ➢ Add, Edit, and Delete Storage Objects
➔ Add

When adding PPDD, a replication context must be configured between production and vault
systems. The following steps are required to add storage objects.
1. From the Main Menu, select Infrastructure > Assets.
2. Click VAULT STORAGE at the top of the Assets content pane.
3. To add a storage object, click ADD.
4. Complete the storage configuration.

This includes the Vault Application Fields:

- Nickname
o Enter a name for the storage object.
- FQDN or IP Address
o Specify the PPDD host by using FQDN or IP Address.
- Storage Username
o Specify a dedicated CR PPDD administration account (for example,
cradmin), which the CR software uses to perform operations with the PPDD
system. This PPDD account must be an admin role and on the DD boost
users list. The sysadmin user cannot be used for this.
- Storage Password
o Enter the password for the PPDD administrator.
- SSH Port Number
o Enter a storage SSH port number.
- Reset Host Fingerprint
o (Security Officer only) If the FQDN or IP address of the PPDD host is
changed, select to reset the fingerprint. The CR software then sends an alert
message.
- Tags
o Optionally, add a tag that provides useful information about the storage
object. The tag is displayed in the details description for the vault storage in
the Assets content pane in the CR UI.
5. Click SAVE. The VAULT STORAGE table lists the storage object.
6. Click in the row for the storage object to view more detailed information that is
retrieved form the PPDD system. This information includes replication contexts and
the Ethernet interface.

39
 ➔ Edit
1. To edit vault storage, go to infrastructure, then assets.
2. Under assets, the first tab shows vault storage. The vault storage shows the PPDD
systems added.
3. Select the checkbox next to the storage to edit. Then select Edit from the top menus.
4. The edit vault storage window is displayed. Users can change all the parameters for
the storage.
5. If the FQDN or IP address of the storage changes, select the Reset Host Fingerprint
checkbox.
6. Select Save to complete the Edit process.
➔ Delete
1. To delete vault storage, select the checkbox next to the storage to delete. Select
delete.
2. A confirmation message is displayed. Select delete to confirm.

The storage is deleted and no longer present in the vault storage view. Notice if active
policies are configured in this PPDD system, it cannot be deleted.
➢ Add, Edit, and Delete Applications Overview

Applications that are installed in the CRV must be represented to the CR software.
Applications can include the Avamar, NetWorker, and PPDM applications, the CyberSense
feature, or other applications.
The application must be installed and running at the CRV location before they can be defined
in the CR UI.
• Avamar and NetWorker Pre-Configuration

To configure the Avamar or NetWorker for PPCR perform the following steps:
1. Connect through SSH to the system.
2. Log in with the admin credentials.
3. Change to the root user.
a. admin@ave01:~/> su -
b. Password: <specify password>
4. Edit the SSH configuration file.
a. admin@ave01:~/# vi /etc/ssh/sshd_config
5. Change the following parameters in the file:
a. PasswordAuthentication Yes
b. PermitRootLogin Yes
6. Save the configuration file.
a. :wq!
7. Restart the SSH service.
a. admin@ave01:~/> service sshd restart
8. Close the connection
a. admin@ave01:~/# exit | Both |→ admin@ave01:~/> exit

40
 • PowerProtect Data Manager Pre-Configuration

To configure the PPDM for PPCR perform the following seps:

1. Connect through SSH to the system.
2. Log in with the admin credentials.
3. Change to the root user.
a. admin@ppdm01:~/> su -
b. Password: <specify password>
4. Edit the SSH configuration file.
a. ppdm01:~ # vi /etc/ssh/sshd_config
5. Change the following parameter in the file:
a. PasswordAuthentication Yes
6. Save the configuration file.
a. :wq!
7. Restart the SSH service.
a. ppdm01:~ # service sshd restart
8. Close the connection.
a. ppdm01:~ # exit
b. admin@ppdm01:~/> exit
➔ Add

The process of adding an application to the vault varies depending on the type of application
to be added. Host operating system credentials and sometimes, applications credentials are
required. The host credentials that are required must be the root user. For PPDM, the system
requires the vCenter where it resides to be added before PPDM. This way the vCenter is
selected from the drop-down list.
1. From the Main Menu, select Infrastructure > Assets.
2. Click APPLICATIONS at the top of the Assets content pane.
3. To add an application, click ADD.
4. Complete the application configuration parameters (Vault Application Fields).
5. Click SAVE. The APPLICATION table lists the storage object.
6. Click in the row for the application to view more detailed information.
➔ Edit

To edit applications in Cyber Recovery follow these steps:

1. From infrastructure, asset section, select the applications tab.
2. A list of added applications is shown. Select the checkbox next to the application and
then select Edit.
3. The edit vault application window is shown. Password fields are blank. However, the
passwords are not necessary when editing an application.
4. If the FQDN or IP address of the application changes, ensure to select Reset Host
Fingerprint checkbox.
5. Select save to complete the changes to the application.

41
 ➔ Delete

To remove an application, follow these steps:

1. Under infrastructure, select Assets and select the Applications tab.
2. Select the checkbox next to the application and select the remove button.
3. A confirmation message is displayed. Select Delete to complete the application
deletion process.

Applications that have active policies cannot be removed.

➢ Add, Edit, and Delete vCenter Servers
➔ Add

When a vCenter system is installed in the CRV, you must represent it to the CR software.
1. To add vCenter servers, select Infrastructure, then Assets. Select the vCenters, then
select Add.
2. Specify a nickname and FQDN or IP address for the vCenter Server.
3. Then, specify the vCenter host administrator username.
4. Tags are optionally set to provide useful information about the application. If a tag
exceeds 24 characters, only 21 characters are displayed.
➔ Edit

To edit a vCenter Server in CR follow these steps:

1. Check the checkbox next to the vCenter. Select Edit.
2. The nickname and FQDN or IP address can be updated. A new username and
password can be specified too.
3. Notice there is no option to Reset Host Fingerprint. It is not necessary to create a
new host fingertip.
4. Tags are not edited however a tag can be cleared and more tags can be created
from this view.
5. To complete the edit, select Save.
➔ Delete

The process for deleting a vCenter Server is the same as storage objects and applications.
1. Select the checkbox next to the vCenter server to delete.
2. Then, select Delete.
3. A confirmation message is displayed. Select delete to complete the process.

42
 ➢ Policies and Copies
- Policies

The CR solution uses policies to perform replications, create point-in-time (PIT) copies, set
retention locks, and create sandboxes. Note the following details about CR policies:
• A CR policy can govern one or more PPDD MTrees.
o Only a PPDM policy type can govern more than one MTree.
• A user can create, modify, and delete policies.
• When a policy is run, a single action or multiple actions in sequence can be
performed.
o A policy can be run so that it only performs a replication. Or the same policy
can also be run so that it performs a replication, creates a PIT copy, and then
retention locks the copy.
• Concurrent Sync or Lock actions for a policy cannot be run.
- Copies

Copies are the PIT MTree copies that serve as restore points that can be used to perform
recovery operations. In the CR UI, you can perform a retention lock on a copy or analyze its
data to detect the presence of malware or other anomalies. Also, unlocked copies can be
deleted.
The CR UI supports the following policy options: Copy, Copy Lock, Sync, Sync Copy, Secure
Copy.
➔ Copy

A Copy action makes a point-in-time (PIT) copy of an MTree’s most recent replication in the
CRV and stores it in the replication archive.
➔ Copy Lock

A Copy Lock action retention locks all files in the PIT copy.
➔ Sync

A Sync action (or replication) replicates an MTree from the production system to the CRV,
synchronizing with the previous replication of that MTree.
➔ Sync Copy

A Sync Copy action combines the Sync and Copy actions into one request. It performs the
replication and then creates a PIT copy.
➔ Secure Copy

A Secure Copy action performs a replication, creates a PIT copy, and then retention locks
all files in the PIT copy.

43
 ➢ Manage Policies

Create policies to perform replications, make point-in-time (PIT) copies, set retention locks,
and perform other CR operations within the CRV. Policies can also be modified and deleted.
Before a policy is created, ensure that a storage object is available to reference in the policy.
Also, ensure that it has an unprotected replication context. Only one policy can protect a
replication context. Policies that perform recovery or analysis operations require an
application.
Up to 25 policies can be created for a maximum of five PPDD systems in the CRV.
The CR software supports PPDM policies that govern multiple MTrees.
A copy from a disabled policy can be used to perform a recovery operation manually or from
the Recovery window.
• Add and Edit a Policy

To add a policy, select policies from the main menu on the left. In the policies tab, select
Add.
- Policy Parameters

Name Specify a policy name. Password Password for security officer.

Policy Select Standard or PPDM. Storage Required when you select
Type Standard denotes NetWorker, Security Compliance. Enter the
Avamar, Filesystem, and Other Officer username and password of the
policy types. (SO) storage instance Security
Username Officer. This username was
created on the PPDD system.
Storage Select the storage object Retention Specify the minimum retention
containing the replication Lock duration that this policy can
context that the policy protects. Minimum apply to PIT copies. This value
cannot be less than 12 hours.
Context Under Context, select the Retention None, if retention locking is not
MTree replication context to Lock Type supported. The retention fields
protect and the interface on the are then removed from the
storage instance that is dialog box.
configured for replications. Governance, if it is enabled on
Under Ethernet Port, select the the storage instance.
interface on the storage Compliance, if it is enabled on
instance that is configured for the storage instance.
replications. Do not select the
data or management Ethernet
interfaces.
Replication Set timeout value in hours for Retention Specify the default retention
Window how long a job for a Sync Lock duration that this policy applies
action runs before CR issues a Duration to PIT copies
warning. The default value is 0.

44
Retention Specify the maximum retention Enforce If the default value in the
Lock duration that this policy can Replication Replication Window field is
Maximum apply to PIT copies. This value Window changed, the Enforce
cannot be greater than 1,827 Replication Window checkbox
days. For minimum and is displayed. Enable the
maximum retention lock, if the checkbox to stop a Sync
lock type is set to Compliance operation that continues to run
and the value is edited, enter beyond the replication window
the Storage SO username and limit for that policy. When the
password. replication window limit is
exceeded, the operation
completes the current PPDD
snapshot replication and does
not replicate queued
snapshots.
Enable For DDOS version 6.2 or later, Tags Optionally, add a tag that
Auto if the retention lock type is provides useful information
Retention Governance or Compliance, about the policy. The tag is
Lock click the checkbox to enable displayed in the details
the automatic retention lock description for the policy in the
feature. There is a five minute Policies content pane in the
delay before the lock is Cyber Recovery UI.
applied. The feature cannot be
disabled after you enable it.

To edit a policy, select the checkbox next to the policy, then select Edit. The edit view has
the same parameters as the add policy view. The storage cannot be changed when editing.
On an edit policy view, governance can be disabled.
If retention lock is set to compliance, security officer username and password must be
specified again. Select Save to complete adding or editing a policy.

45
 • Disable and Delete Policies

Policies can be disabled or deleted. A policy can be disabled so that the replication contexts
of that disabled policy can be used to create a policy. If the contexts of that disabled policy
can be used to create a policy. If the contexts of a disabled policy are used, the policy cannot
be enabled. A disabled copy of a policy can be used to perform a recovery operation
manually or from the Recovery window.
When policies are created, they are enabled by default. The policy is displayed in the list of
disabled policies, and the Status column indicates that the policy is disabled. To disable a
policy, select the policy checkbox from the list, and select Disable.
To delete a policy, select the checkbox next to the policy and select Delete. A disabled policy
cannot be deleted until the retention lock has expired, and all copies are deleted. A policy
cannot be deleted if there are any active copies that are associated with the policy. Delete
the copies before the policy.
CR software does not remove the MTree from the PPDD system. The software does not
delete unlocked PIT copies. Remove them manually.
• Run Policies

Run a policy manually at any time so that it performs a specified action or actions. To run a
policy, select Policies from the Main Menu. Select the policy to run. Click ACTIONS and
select one.
The policy starts a job that you can monitor on the Jobs page. Concurrent sync or lock
actions for a policy cannot be run. When a policy is run, and the same policy with an action
that performs either a sync or lock operation is run, CR displays an informal message and
does not create a job. When the initial job is completed, the policy is run.
• Schedule Policies

A policy can be run manually or can be scheduled. Schedules cannot be configured if CR is

not licensed. The policy action that is scheduled might require a previous operation. For
example, a PIT copy must exist if you want to perform the Lock action.
Multiple schedules can be created for the same policy. However, multiple schedules cannot
be created for a policy that run simultaneously. Each schedule specifies the action that the
policy performs. From the policies section of the main menu, select the schedules tab. Select
Add to add a schedule.
Schedules can be disabled. With schedules disabled the operations will not be performed.
Schedules can also be deleted, or permanently removed.

46
 - Schedule Parameters

Field Description

Schedule Name Specify a schedule name.

Policy Select the policy that you are scheduling.

Action Select the action that the policy performs when it runs under this
schedule.

Retention Lock Only if Secure Copy or Copy Lock was selected as the action, enter
Duration the duration of the retention lock that this policy applies to PIT copies.

Application Host Only if Analyze was selected as the action, select the host for
CyberSense feature.

Frequency Enter the frequency in days and hours.

Next Run Date Select the date to start running the policy under this schedule.

➢ Manage Copies

The Policies page, in the copies tab enables users to view, secure, analyze, and delete PIT
copies. From the main menu on the left, select Policies. Select the copies tab. Each row
shows the copy and policy names, the copy creation date, the retention lock expiration date,
an analysis assessment, and the recovery status. Child copies are not displayed.
To view details about a copy, click in the row for the copy. The Details window displays the
information and provides links to the policy and sandboxes. Copies can be retention locked,
analyzed, and deleted. If the Expiration Date column for a copy displays a date, the copy is
retention locked and cannot be deleted.
If a PPDM copy that has associated child copies is deleted, those child copies are also
deleted.
• Secure Copies

Secure a PIT copy for a specific retention period during which the data in the PIT copy can
be viewed, but not modified. If a copy is already retention locked, the current retention period
can be extended but not decreased.
When the retention period of a copy expires, the data is no longer protected from deletion.
Under copies, select the copy that you want to secure and click LOCK. In the LOCK dialog
box, specify the retention period and click SAVE.
The Policy Retention Range field displays the minimum and maximum retention value of the
policy. Specify a duration within this range. Once the retention lock is set the Expiration Date
column changes from No lock set and displays the expiration date and a locked icon. When
the retention lock expires, the Expiration Date column displays the expiration date and an
unlocked icon.

47
 • Analyze a PIT Copy

Analyze a PIT copy by using the CyberSense feature in the CRV. An Analyze operation can
only be run on a Standard policy copy. Prerequisites to analyzing a copy are:
• To use the CyberSense feature, a valid license is required.
• A policy must create the PIT copy to analyze.

A CyberSense feature license is based on TB capacity. If the license capacity is exceeded,

the analysis is completed, and the CR software provides an alert. Until the licensed capacity
is updated, the alert is shown every time an Analyze operation is run. There is a 90-day
grace period to increase the licensed capacity.
If the licensed capacity is not increased after 90 days, the Analyze operation status is Partial
Success and the CR software indicates that security analytics were not generated because
the license is invalid. If the license expires, the Analyze operation fails. The CR software
indicates that there is a missing or invalid license.
- Analysis Procedure
1. Select Policies from the Main Menu.
2. On the Policies content pane, click COPIES to display the list of existing copies.
3. Select the copy to analyze and click ANALYZE. If there is no valid license for the
CyberSense feature, the ANALYZE button is disabled.
a. An analysis cannot be run concurrently on a copy of the same policy.
Otherwise, the CR software displays an information message and does not
create a job. When the initial job is completed, run the analysis on the copy.
4. From the Application Host list box, select the application the nickname for
CyberSense feature.
5. A content format can be optionally selected from the drop-down menu.
6. Select Apply.
a. The policy starts a job that you can view on the Jobs page. If the analysis
includes indicators of possible malware or other anomalies, the job status is
listed as Critical. Otherwise, the jobs tatus is listed as Success.
7. An analysis can be canceled at this point. An analysis job can be reviewed by
selecting the running Analyze job under jobs.
8. When the analysis is complete, return to the list of copies and click in the row of the
copy. The Last Analysis column shows the results as Suspicious, Good, or Failed.
a. If an analysis job that is in progress is canceled or the analysis skips any files,
the Last Analysis column shows the result as Partial, and the job status is
Canceled. An email message and the logs indicate that the analysis job was
partially successful.
b. If the analysis detects an anomaly, the Last Analysis column shows the result
as Suspicious, and the job status is Critical. An alert notifies you about the
anomalies.
c. If an Analyze job fails, the CR software generates an error.

48
 • Analysis Report

To retrieve a detailed report about a complete Analyze job, select the analyzed copy, click
Analysis Report Actions, and then select either Download or Email Analysis report. If the
user choose to download the analysis report a CSV file is generated. The copy-name.csv
file is downloaded to the location configured for download in the browser.
To receive an email message with the analysis report, which is a CSV file, which is attached,
select Email Analysis Report. In the Email Analysis Report window, enter an email address
for the specified recipient: enter at least one email address.
A report is only available for a successful Analyze job. If an Analyze job fails, the CR software
generates an error. If you select a copy on which there has been no analysis or on multiple
copies, the Analysis Report Actions button is disabled.
➢ Manage Sandboxes

A sandbox is a unique location in the CRV in which you can perform read/write operations
on a PIT copy. This copy is a read/write copy of the locked data in the CRV. The CR software
supports two types of sandboxes:
• System Sandboxes: The CR software enables the creation of custom sandboxes
manually to perform operations by using applications that are not in the CR default
list. A sandbox can contain only one PIT copy; however, multiple sandboxes can be
created for one PIT copy. Sandboxes are created as needed for data analysis or
validation operations.
• Recovery Sandboxes: The CR software automatically creates recovery sandboxes
when a Networker, Avamar, or PPDM recovery is initiated.
• Manage Sandboxes

Create sandboxes as needed for data analysis or validation operations. The CyberSense
feature, which analyzes backup data for the presence of malware or other anomalies,
requires a sandbox. Follow these steps to manage sandboxes:
1. Select Recovery in the main menu.
2. Select COPIES and then select a PIT copy from the list.
3. Click Sandbox.
4. Select an application host that is configured in the CRV. Enter a unique sandbox
name (notice that the CR prefix is appended to the sandbox name). Indicate if the
file system will be mounted and where to mount it. Notice CR supports mounting
filesystem operations for UNIX operating systems only. Click Apply.
5. From the Recovery content pane, select SANDBOXES. View the list of sandboxes.
The details window displays additional information. To delete the sandbox, select the
sandbox, then select Delete.

49
 • Manage Recovery Sandboxes

The CR software creates a recovery sandbox during a recovery operation and populates it
with the selected copy. The sandbox is available to the application host. After a recovery
operation is run, follow the steps to manage recovery sandboxes:
1. From the Main Menu, select Recovery.
2. On the Recovery content pane, click RECOVERY SANDBOXES.
3. To view the recovery details select the recovery name. To validate if it is successful,
click Launch App. This accesses the NetWorker or PPDM UI in the CRV. The Launch
App button is not available unless a recovery has completed successfully.

To delete the sandbox, click Cleanup.

➢ PowerProtect Data Manager with Cyber Recovery
• PowerProtect Data Manager

Backup applications are transforming to provide more than just access to backups and
restore capabilities including:
• Analysis and reuse for dev/test.
• Leverage the cloud to extend data center capabilities.
• Protect cloud native applications.
• Enable self-service backup and restore from native applications.
• Maintain centralized governance and control.
• Increase business resiliency to rapidly recovery from cyber incidents.

In order to address this wide range of requirements, PPDM is at the forefront of this
transformation to modern data protection.
• Recovering PowerProtect Data Manager Data

When the recovery is initiated, complete the recovery form the PPDM application in the CRV.
When a recovery is initiated, the CR software prepares the environment so that a PPDM VM
recovery can be run from the application console. As part of this process, the software
creates a production DD Boost username and password and reboots the PPDM appliance.
It also takes a VM snapshot of the PPDM appliance that you use to revert the PPDM
software after you complete the Recovery.
Only one recovery job can be run per application at a time.

50
 • Prerequisites

The following prerequisites should be met before you initiate a PPDM recovery:

• The CRV PPDD system must be running DD OS Version 6.2 or later.

• The CR virtual appliance is deployed in the CRV. The PPDM application must be
installed as the admin user.
• The UIDs that are associated with the production PPDM DD Boost users are
configured in the CRV PPDD system. These UIDs must be available in the PPDD
system in the CRV.
• The PPDM application in the CRV must be configured with the credentials of the
PPDM application on the production system. Remember after 90 days root and
admin passwords expire.
• PPDM server hostname and IP address within the CRV does not have to match
production PPDM hostname and IP address.
• The PPDM application is defined as an application asset in the CR software. Use
either the CR UI or the CRCLI to add the application.
• Ensure that there are no snapshots of the PPDM VM that is deployed in the vCenter
Server.
• Run application and server backups in the PPDM production environment. Then,
perform a Secure Copy operation to copy data to the CRV environment.
• A policy is created for the VM data and a policy for the server backup.
• Initiate Recovery

A recovery can be initiated in the CR UI. The CR software completes the recovery operation
automatically.
The CR software prepares the environment so that a VM recovery can be run from the
PPDM application console. As part of this process, the software creates a production DD
Boost username and password and reboots the PPDM appliance. To initiate a recovery,
follow the steps:
1. Select Recovery from the Main Menu.
2. On the Recovery content pane, select the copy, and then click APPLICATION.
3. In the Application dialog box, select a PPDM application host and then click APPLY.

The CR UI software runs a job to create a recovery sandbox. Once the sandbox is created
it populates it with the selected copy, and then makes the sandbox available to the
application host.
4. Optionally, cancel the recovery. To do so, select Jobs, then select the running
recovery job and click Cancel Job.
5. Wait for the recovery application job to complete. A recovery sandbox is created for
the PPDM application.
6. Click RECOVERY SANDBOXES from the top of the Recovery pan. To view the
recovery details, select the recoverapp_<ID> name and view the status detail.

To validate success, click Launch App to access the PPDM UI in the CRV. The button is
only active when recovery is completed successfully. To delete the sandbox, click Cleanup.

51
 • Run Recovery Check

Run a scheduled or on-demand PPDM recovery check to ensure that a successful recovery
a copy can be recovered.
When the CR software completes a recovery check action, the status of the copy is marked
as recoverable or nonrecoverable. The CR software reverts PPDM back to its initial state
from which you can run a recovery. However, you can run a recovery manually to determine
if the copy is recoverable and manually perform the cleanup.
Scheduled projects are added from Policies, then Schedules. When a new schedule is
created, under actions, select recovery check. This perform a recurring recovery check.
For on-demand recovery checks, select Recovery from the main menu on the left. Under
copies, select the copy checkbox, then select Recovery Check.
• Postrecovery

After the PPDM recovery is completed, perform required post recovery steps. Other post
recovery steps include deleting the sandbox. Delete the sandbox that was created when you
initiated the PPDM recovery. To do so, select Recovery, then Sandboxes from the main
menu.
Select the Sandbox, then delete. The sandbox is deleted, and the CR software reverts the
PPDM software to the snapshot that the copy was created when the recovery initiated.
To validate success, log in to the PPDM application in the CRV. The Welcome to PPDM
window opens.
Optionally, on the PPDD system, run the filesys clean command. This step deletes the DD
Boost storage unit. If you choose not to perform this step, the DD Boost storage unit is
deleted during the next scheduled cleaning operation.
• PPDM Recovery Enhancements

Lockbox passphrase and root password are no longer needed while adding Vault PPDM
application through CLI or UI. The Application UI password is set to the production
credentials after the recovery and root, or admin operating system passwords are set to
whatever the user-defined in the CRV PPDM application.

52
 ➢ NetWorker with Cyber Recovery
• NetWorker

Dell EMC NetWorker software for unified backup and recovery of your enterprise
applications and databases. With centralized administration, NetWorker helps you take
advantage of the data protection that fits your needs best: deduplication, backup to disk and
tape, snapshots, replication, and NAS.
NetWorker protects both physical and virtual environments including VMware and Microsoft
Hyper-V. Whether you are protecting applications and data residing within your data center
or the public cloud, NetWorker provides the same enterprise-level user experience.
With cloud capabilities, NetWorker offers a solution for cloud data protection with
optimizations to secure your data everywhere. NetWorker is available as virtual edition or
as a component of the Dell EMC Data Protection Suite, which offers you a complete suite
of data protection software options.
• NetWorker with Cyber Recovery

NetWorker recovery can be performed from PPCR. To do so, use a PIT copy to rehydrate
NetWorker data in the CRV. The NetWorker application must be installed as the root user
in the CRV.
Before a recovery operation, run application and server backups in the production
environment. Then, perform a Secure Copy policy operation to copy data to the CRV
environment. Finally, use the CR UI to initiate the recovery. The CR software creates a
sandbox so that you can run the recovery from the NetWorker application.
Only one recovery job can be run per application at a time.
• NetWorker Recovery Enhancement

The NetWorker Automated Recovery process has a new option to choose folder containing
bootstrap backup or folder that is used as bootstrap device in the UI or CLI. Selecting the
folder saves recovery time as the system does not have to scan all the device looking for
bootstrap backup.
Also, two unnecessary background steps were removed to save on the recovery time.
The automated NetWorker process also instructs the recovery process to keep the original
NetWorker Authentication Service database file. The original file will not be replaced with
the recovered file. Since the file is not replaced, recovery will not fail from mismatch of
authentication between the production and vault NetWorker.

53
 • User and UID

Before performing a NetWorker recovery, create the DD Boost account that is associated
with the copy in the CRV.
1. To determine the UID required for recovery, run the following CRCLI command on
the management host:

# crcli policy show -n <policy_name>

# crcli policy list-copy—policyname <policy_name> -c <copy_name>

2. Note the output from this command, as shown in the following example:

# Source Storage UID: 503

3. To determine if the account exists for this UID, log in to the PPDD system in the CRV
and run the following command:

# user show list

4. To determine the UID required for recovery, run the following CRCLI command on
the management host:

# crcli policy show -n <policy_name>

If the output lists the UID, you can proceed with the recovery procedure. If the output does
not show that the UID exists, go to the next steps.
5. Create the UID. When adding the application asset, if you defined a tag, reference
the tag go determine the production system DD Boost user name. If you are running
DDOS 6.1.2.10 or later, create the username and account by running the following
command:

# user add <NetWorker_ddboostname> uid <UID_from_user_show_list_output>

For earlier versions, run the user add command until you get the UID required for recovery.
For example, if you have a UID 510, you might have to create up to nine temp accounts.
Note that users add on the PPDD system starts at UID 500.

54
 • Initiate NetWorker Recovery

Initiate a recovery in the CR UI. After you initiate a recovery, the CR software uses the lates
system device to complete the recovery operation automatically.
Ensure that the credentials for the CRV hosts on which the NetWorker application is installed
and for the NetWorker application are secured. The NetWorker server host within the vault
has the same IP address and hostname as the NetWorker production host. The NetWorker
application is installed in the CRV and defined as an application asset in CR. The DD Boost
user within the vault has the same UID as the production DD Boost user. A policy has
created a PIT copy to use for the recovery. The UID associated with this copy has been
created in the CRV PPDD system.
To initiate the recovery, perform the following steps:
1. Select Recovery from the Main Menu.
2. On the Recovery content pane, select the copy, and then click APPLICATION.
3. In the Application dialog box, select an application host, enter DD Boost username
and password, and then click APPLY. Optionally, enter the name of the folder that
includes the last bootstrap backups.
a. The CR software runs a job to create a recovery sandbox, populates it with
the selected copy, and then makes the sandbox available to the application
host.
4. Wait for the recovery application job to complete creating the sandbox. The recovery
sandbox is created for the NetWorker application.
5. Click the job and view the status detail. The Status Detail provides the name of the
newly created sandbox.
6. Click RECOVERY SANDBOXES from the top of the Recovery pane and do the
following:
a. To view the recovery details, select the recover_app<ID> name.
b. To validate success, click Launch App to access the NetWorker UI in the
CRV. The Launch App button is active only when the recovery is completed
successfully.
c. To delete the sandbox, click Cleanup.

55
 ➢ Avamar with Cyber Recovery
• Avamar

Companies are redefining their backup and recovery solutions to meet challenges that are
brought on by accelerated virtualization and movement to the cloud. These conditions
require that companies design data protection from edge to core to cloud.
Comprehensive backup and restore Part of Data Protection Suite, Dell EMC Avamar
provides flexible deployment options for fast, daily full backups supporting:
• Virtualized and physical environments.
• Multicloud backup and disaster recovery.
• Enterprise applications.
• NAS systems.
• Remote offices.
• Desktops or laptops.
• Avamar Recovery with Cyber Recovery

To recovery with Avamar, use a PIT copy to rehydrate Avamar data in the CRV. The Avamar
application must be installed as the root user in the CRV.
Before a recovery operation, run application and server backups in the production
environment. Then, perform a Secure Copy policy operation to copy data to the CRV
environment.
A recovery operation is a two-step process:
1. From the CR UI, copy the PIT copy into a read-writable sandbox.
2. Perform manual recovery steps on the application host.
a. Only one recovery job can be run per application at a time.

56
 • Prepare Production Avamar

Perform the following procedure if you want to create a checkpoint before performing a
Secure Copy policy operation.
1. Log in to the production Avamar server as root user and run a checkpoint operation.
This step might take some time.
o Run Checkpoint Operation
• Type su admin -c “mcserver.sh—flush”:

root@ave-03:~/#: su admin -c “mcserver.sh—flush”

=== BEGIN === check.mcs (preflush)
check.mcs passed
=== PASS === check.mcs PASSED OVERALL (preflush)
Flushing Administrator Server...
Administrator Server flushed.

• Type mccli checkpoint create:

root@ave-03:~/#: mccli checkpoint create

0,22624,Starting to create a server checkpoint.
root@ave-03:~/#: mccli checkpoint show
0,23000,CLI command completed successfully.
Tag Time Validated Deletable

----------------- ----------------------- --------- --

cp.20180316130025 2020-08-16 09:00:25 EDT Validated No

cp.20180316130301 2020-08-16 09:03:01 EDT No

cp.20180316151143 2020-08-16 11:11:43 EDT No

• Verify the checkpoint information.

root@ave-03:~/#: mccli checkpoint show
0,23000,CLI command completed successfully.
Tag Time Validated Deletable

----------------- ----------------------- --------- --

cp.20180316130301 2020-08-16 09:03:01 EDT No
cp.20180316151143 2020-08-16 11:11:43 EDT Validated No
2. On the CR host, run a Secure Copy policy action for the PPDD MTree.

57
 3. Validate the size of the production PPDD system MTree that was replicated is the
same as the replicated MTree on the destination PPDD system and the CR MTree.
o List the MTree
• Type mtree list, as shown in the following code example:

sysadmin@crmgmthost# mtree list

Name
Pre-Comp (GiB) Status

-------------------------------------------------- ---
----------- -------
/data/col1/avamar-1560177494-repl 4.2
RO/RD
/data/col1/backup 0.0
RW
/data/col1/cr-policy-5d5ad66394422f0001ced229-repo 0.0
RW/RLGE
/data/col1/cr-policy-5d5ad69994422f0001ced22a-repo 4.2
RW/RLGE
/data/col1/nw02-repl 0.0
RO/RD

-------------------------------------------------- ---
----------- -------

D : Deleted
Q : Quota Defined
RO : Read Only
RW : Read Write
RD : Replication Destination
RLGE : Retention-Lock Governance Enabled
RLGD : Retention-Lock Governance Disabled
RLCE : Retention-Lock Compliance Enabled

• Verify that the production-, target-, and policy-replicated MTrees are the same.

58
 • Checklist

Perform the following tasks for the Avamar system in the CRV.
• Add the Avamar application as the root user.
• Obtain the credentials for the host on which the Avamar application is installed.
• Ensure that the Avamar version and build are identical to the production system.
• Ensure that the Avamar FQDN is identical to the production system. A different IP
address can be used in the CRV. The FQDN must be identical.
• Ensure that all Avamar credentials such as MCUser/GSAN accounts have the same
passwords. For Avamar services to start properly, the Avamar credentials must be
the same.
• Ensure that the DD Boost username and UID in the CRV match the credentials of
the production system.
• Obtain Avamar licenses, if necessary.
• Establish Avamar applications in the CRV. This task enables rehydrating
applications in the CRV.
• Ensure that DDOS version in the CRV is compatible with the Avamar applications.
• Configure the PPDD hostname in the Avamar application.
• Create Avamar Account

Before performing an Avamar recovery, create the DD Boost account that is associated with
the copy in the CRV. To do so, perform the following steps:
1. To determine the UID required for recovery, log in to the CRCLI and run the following
command on the management host:

# crcli login -u <CR_user>

# crcli policy show -n <policy_name>

2. Note the output from this command, as shown in the following code example. 505 is
the UID that you associated with this policy.

Source Storage UID: 505

3. To determine if the account exists for this UID, log in to the Data Domain system in
the CRV and run the following command:

# user show list

If the output lists the UID, you can proceed with the recovery procedure. If the output does
not show that the UID exists, got to the next step.
4. Create the UID. When adding the application asset, if you defined a tag, reference
the tag to determine the production system DD Boost user name. If you are running
DDOS 6.1.2.10 or later, create the username and account by running the following
command:

# user add uid role admin

59
 5. For earlier versions, run the user add command until you get the UID required for
recovery. For example, if you have a UID 510, you might have to create up to nine
temp accounts.

The user add command on the PPDD system starts at UID 500.
• Initiate Recovery

Initiate a recovery in the CR UI and then complete the recovery by performing manual steps
on the application server in the CRV.
This procedure assumes that the Avamar application is installed in the CRV and defined as
an application asset in CR. A policy has created a PIT copy to use for the recovery. The UID
associated with this copy has been created in the CRV PPDD system. To initiate a recovery,
perform the following steps:
1. Select Recovery from the Main Menu.
2. On the Recovery content pane, select the copy and click APPLICATION.
3. In the Recovery dialog box, select the Avamar application host and click APPLY. The
CR software runs a job to create a recovery sandbox. It then populates the sandbox
with the selected copy, and then makes the sandbox available to the application host.
4. Wait for the recovery application job to complete creating the sandbox. The recovery
sandbox is created for the Avamar application.
5. Click the avamar-<GUID> name. The Status Detail provides the name of the newly
created sandbox. Use this name for the following recovery steps.
• Manual Steps for Recovery

After initiating an Avamar recovery in the CR UI, perform the following manual steps on the
Avamar server host in the CRV.
1. In the CRV, log in to the Avamar server as root.
2. Edit the /etc/hosts file to alias the PPDD data IP on the Vault as the PPDD name.

This change ensures that the restore operation uses the required production PPDD name.
In the following example, ddve-05 is the name of the production PPDD system:
/#: cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 localhost.localdomain localhost
192.168.2.83 ave-03.vcorp.local ave-03
192.168.2.106 ddve-05.vcorp.local ddve-05

60
 3. Verify that the PPDD hostname resolves correctly:

# ping ddve05.vcorp.local
4. CR creates the recovery sandbox with the same name that Avamar uses in
production. The HFS creation time (hfsctime) value is after the Avamar_prefix. For
example, the recovery sandbox is created as avamar_1491947551 and the hfsctime
is 1491947551.
5. Run a checkpoint restore operation from the recovery sandbox by using the HFS
Time of the Avamar DD Boost storage unit. Use the DD Boost user that is associated
with that storage unit.

→ Checkpoint Restore
Before proceeding with this command, ensure that the ddr-user name matches the name on
the production system, including the UID.
# cprestore—hfsctime=1491947551 --ddr-server=ddve-0.5.vcorp.local—ddr-user=ddboost
a. When prompted, enter the DD Boost password. The script displays a list of
restorable checkpoints and asks which one to restore.

Mount NFS path ‘ddve-05.vcorp.local:/data/col1/avamar-1491935387/GSAN’ to

‘ddnfs_gsan’ Mount path
‘ddnfs_gsan’ already mounted… skipping.
There are 4 available checkpoints.

cp.20180315171722
cp.20180316130025
cp.20180316151143
cp.20180316151143_1521213451

Checkpoint to restore or ‘quit’ to stop?

b. Enter the checkpoint that must be restore and, when prompted, type yes to
confirm the entry. The restore procedure is performed from the recovery
sandbox, and the script terminates with messages that confirm the operation.
6. On the CRV PPDD system, perform the steps that are listed here:
a. Create the checkpoint snapshot by using the same checkpoint name that you
selected in the previous step.

# snapshot create <checkpoint_name> mtree <name_of_avamar_tree/sandbox>

b. An example is found here:

# snapshot create cp.20190826122838 mtree

/data/col1/avamar-1560177494

61
 7. Log back in to the Avamar system as root and stop the Avamar services. To do so,
perform the steps that are listed here:

→ Stop Avamar Services

a. Stop the Avamar services on the Avamar server. This step can take some
time.

# dpnctl stop
b. When asked if you want to shut down the instance, enter y:

Do you wish to shut down the local instance of EM Tomcat?

Answering y(es) will shut down the local instance of EM Tomcat

n(o) will leave up the local instance of EM Tomcat

q(uit) exits without shutting down

y(es), n(o), q(uit/exit): y

c. When the process is completed, use the dpnctl status command to verify the
status.
d. Stop the Avamar Agent service.

# /etc/init.d/avagent stop
e. Clear out the Avamar client ID (CID).

# rm -f /usr/local/avamar/var/client/cid.bin
f. Start a rollback recovery of the checkpoint. This step might take a long time.

# dpnctl start—force_rollback
g. When asked if you want to continue, enter y:

Have you contacted Avamar Technical Support to ensure that this is the right thing to do?
Answering y(es) proceeds with starting all.
n(o) or q(uit) exits
y(es), n(o), q(uit/exit): y

h. When a message indicates the following, enter 3 to select a specific

checkpoint. The script displays a list of available checkpoints.

The choices are as follows:

1 roll back to the most recent checkpoint, whether or not validated
2 roll back to the most recent validated checkpoint
3 select a specific checkpoint to which to roll back
4 do not restart
q quit/exit

62
 i. The script displays a list of available checkpoints. Enter the number that
corresponds to the exact checkpoint name that you selected in the previous
steps and on which you created the snapshot. Then enter y when prompted
to confirm the recovery.
j. If the system asks a user to choose to restore the local EMS data, enter y.

Do you wish to do a restore of the local EMS data?

Answering y(es) will restore the local EMS data
n(o) will leave the existing EMS data alone
q (quit) exists with no further action.
Please consult with Avamar Technical Support before answering y(es).

Answer n(o) here unless you have a special need to restore the EMS data, e.g., you are
restoring this node from scratch, or you know that you are having EMS database problems
that require restoring the database.
y(es), n(o), q(uit/exit): yes
dnnctl: INFO: Restoring EMS data...
dpnctl: INFO: EMS data restored.

k. Wait for the rollback recovery to complete and the Avamar Services to start
up.
8. Validate that all required services are up and running:

# dpnctl status
9. Add the SSH key for the CRV PPDD system to the newly restored Avamar server.

# echo \”Username: ddboost@ddve-05.vcorp.local\”; cat ~admin/.ssh/ddr_key.pub | ssh

ddboost@ddve-05.vcorp.local adminaccess add ssh-key
10. Update the security configuration on the newly restored Avamar server by performing
the steps listed here.

→ Update Security Configuration

a. Regenerate the security certificates:

# enable_secure_config.sh—certs
b. View the session security settings:

# enable_secure_config.sh—showconfig
c. Run the avsetup mccli command and accept all the defaults except for the
MCUser password. Do not use the default value for MCUser.

# avsetup_mccli

63
 d. Restart the Avamar MCS services.

# su admin -c ‘mcserver.sh—restart—force’
e. Run the avsetup mccli command. Press Enter if you do not want to change
the hostname or the port number.

# avsetup_mccli
f. Edit the PPDD system configuration.

# mccli dd edit—name=ddve-05.vcorp.local
g. Confirm the PPDD system properties.

# mccli dd show-prop—name=ddve-05.vcorp.local
h. From the PPDD system, revoke token access for DD Boost.

# ssh cradmin@ddve-05.vcorp.local “ddboost user revoke token-access ddboost”

i. Stop the Avamar Agent service.

# /etc/init.d/avagent stop
j. Edit the client properties.

# mccli client edit—domain=/MC_SYSTEM—name=ave-03.vcorp.local—activated=false

k. Start the Avamar Agent service.

# /etc/init.d/avagent start
11. Log in to the Avamar UI on the host server. Verify that the PPDD system is displayed
in the main window. Verify that the data that is represented on the PPDD system
matches that of the Avamar PPDD system. Verify that all the policies, clients, and
other configuration items match the policies of the production system.

64
 ➢ CyberSense Feature

CyberSense is an optional feature. CyberSense is a third-party tool that validates and

analyzes PIT copies for the presence of malware or other anomalies.
• Features and Benefits
- Direct indexing of content in backups. There is no need to rehydrate, making it a
more secure solution.
- The original backup software does not have to restore the last good file or database.
- An initial scan detects attacks with 95% accuracy, subsequent passes increases
accuracy to 99%.
- CyberSense is fully integrated with Cyber Recovery.
- Reporting tools detect indicators of compromise and attacks.
- The solution detects the most common attacks (encryption, ransomware,
destruction, and slow corruption).
- Forensic analysis tools help detect and recovery from cyber-attack and diagnose the
attack vector.

A report provides indication of compromise.

• CyberSense Workflow
1. Synchronization

This diagram provides a view of the production and CRV environments. Production and CRV
have PPDD systems. The CRV has the CR Server which is in charge of the CR operations.
The CR software controls the Index Engines CyberSense software.
In the synchronization stage, CR opens the ports and allows communication between the
Production and the CRV PPDD systems.
Backups are replicated from the production to the CRV PPDD system.
2. Immutable Copies

When synchronization is complete, CR closes the ports and crates an air gap between the
two sides. Immutable copies are created in the CRV PPDD system. These copies cannot
be changed or tampered.
3. Sandbox

A Sandbox copy of the backup is created on the PPDD system. As opposed to immutable
copies, the sandbox copy is a read/write copy which can be changed.
4. CyberSense Notification

When the Sandbox copy is created, CR through its APIs communicate with the IndexEgines
CyberSense server. CR indicates an MTree is available for analytics.

65
 • Terminology

These terms are common regarding CyberSense.

- Index: An index job defines the source of the backup file (MTree/location mounted
ia NFS) to be indexed and the indexing options.
- Segment: Segments hold the indexing results for one or more jobs that were run
simultaneously.
- Post Processing: Post Processing enables and optimizes the segments for analysis
and searching.
- Analysis: Passes the job to the Machine Learning engine and compares the content
changes from the previous backup.
5. Analysis

The MTree is mounted on the PPDD system, over an NFS mount, and initializes an analysis
on the backup. The result of the analysis is sent from the CyberSense host to CR to indicate
the status of the analysis. A green light means that there were no changes to the data, a red
light means that the data changed and could have been tampered.
The analysis begins with an index job, where more than 100 analytics and statistics are
created for each job. An index jobs results in the creation of one or more segments. Large
jobs have multiple segments.
After the index job creates a segment, the postprocessing phase begins. This phase
optimizes the segment for analysis and searching.
An analysis uses the CyberSense Machine Learning (ML) engine to compare the content
with the previous backup.
Additional information is available when a ransomware infection is detected. Forensic search
capabilities are provided, including the identification of the corrupt files.
• Licensing

A valid license is required for the CyberSense feature. Dell administers CyberSense
licenses. Each license has two components:
• System Licenses - There is a license for every server.
• Capacity Licenses - Based on the analyzed capacity, these licenses can be split
across multiple servers.

The capacity license is measured in TBs of data before deduplication. The license is based
on the total amount of unique active data to be analyzed. The license can be split across
multiple engines. This split is done in the Index Engine support website.
A subscription model is available in 1, 3, 5-year terms.

66
If the capacity exceeds the license capability a message is sent to CR. However,
CyberSense uses the 90-day grace period to continue to analyze backups. If the grace
period is exceeded and a new license is not applied, analyses are ceased. The same
condition happens if the license expires.
Requirements include:
- CyberSense feature must be installed at the same location as the CRV.
- A valid feature license for CR is required.
- CyberSense Version 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, or 7.7 are required.
- A dedicated host running CentOS or Red Hat Enterprise Linux, on which the
CyberSense feature is installed, that acts as the validation host.

The validation host provides direct integration between the Cyber Recovery software and
the CyberSense feature.
• CyberSense 7.7 Integration Enhancements

CyberSense 7.7 now supports analyze of a PPDM copy. The following PPDM workloads are
supported: Filesystem (Linux and Windows), Oracle Database, VMware.
Analysis report can now be sent to additional email addresses other than the logged in user
who is used to start or schedule analysis job. Select the copy from Policies→Copies, tab,
choose “Send Analysis Report” under “Analysis Report Actions” and mention the email
addresses.
• Syslog Integration and FIPS Support
➔ Syslog Integration

All events that appear in the Message Center are now outputted to syslog in Common Event
Format (CEF). Customers can log aggregation tools like Splunk to track CyberSense jobs
and other activities.
➔ FIPS 140 Support

The Index Engines application has been modified to run on an operating system configured
for FIPS 140 compliance. This support is for non-docker installs.
• Using Local Repos without Internet in the Vault

Index Engines now provide repositories customers can use for both docker and nondocker
installs. Perform a yum install of the Index Engines RPM after the repositories are
configured.
Install the repos using the below command:
# sh <repo_bundle_filename> install

67
The command disables any repositories that were enabled before installing and enabling
the Index Engines provided repositories. The above steps need not be repeated for
subsequent updates of Index Engines software unless you are instructed to do so.
• Include or Exclude option and analysis datatype

Users can include or exclude files from the CyberSense analysis jobs. While starting the
analyze operation from the CR CLI and REST API, users can choose to include or exclude
the files. Included or Exclude files can be passed in an array using REST API. Comma-
separated values can be given, which contains a list of file path or directory to be included
or excluded.
An Analysis datatype can also be passed while starting the analysis which improves the
measurement of the license capacity usage of CyberSense.
Analyze Options:
• -p, --excludefilepath string (optional for analyze)

• excludefilepath “/opt/dellemc/tmp/exclude.txt”

• -e, --excludefiles string (optional for analyze)

• excludefiles “testinfo/logs”

• -f, --includefilepath string (optional for analyze)

• includefilepath “/opt/dellemc/tmp/include.txt”

• -d, --includefiles string (optional for analyze)

• includefiles “backup/dr.swap”

• -t, --mtreeContentFormat string (optional for analyze)

• mtreeContentFormat “file system” or databases or backup.

• Re-Analyze Suspicious Copy

Starting from this release, if a user analyzes the already found Suspicious copy again, the
status of the copy will not change to Good. The copy status will remain as Suspicious.
• CyberSense Configuration

The CyberSense in a PPCR system provides two distinct capabilities:

• Audits the data managed by CR to detect signs of corruption due to trojans and
ransomware.
• Provides postattack forensic reports for diagnosis and recovery from the attack.

Prerequisite: The application must be installed and running at the CRV location before
defining it in the CR UI.

68
The first step to add CyberSense to the PPCR system:
1. From the main menu, select Infrastructure > Asset.
2. Select Applications tab from the Assets content pane and click Add.
3. Enter the following information in the Add Vault Application page, and click Save:
a. Application Object Name
b. FQDN or IP Address
c. Host Username
d. Host Password
e. SSH (SSH) Port Number
In CyberSense configuration, you must select CyberSense application for analysis
capabilities. You can also add Avamar, NetWorker, and PPDM as applications.

f. Application Type
It is optional to add a tag that provides useful information about the application. The tag is
displayed in the Assets content pane in the PPCR UI.

g. Tags
4. Notice that the CyberSense application is added in the list.
• Defining Schedules for Scanning

Once the CyberSense application is added to the PPCR UI, you must define a schedule for
scanning the data source. The following tasks must be performed to define a schedule:

1. Select Policies from the main menu.

2. Select Schedules tab from the Policies content pane and click Add.
3. Enter the following information in the Add Schedule page, and click Save:
a. Schedule Name
b. Policy
c. Action
For a CyberSense schedule, select Analyze as the action type that the policy performs when
it runs under the schedule.

d. Retention Lock Duration

If you select Secure Copy or Copy Lock as the action type, then you must set the retention
lock duration.

e. Application Host
If you select Analyze as the action type, then the application host is enabled, and you can
select the CyberSense application.

f. Frequency
g. Next Run Date
h. Next Run Time
4. Notice that the schedule is added in the list.

69
 • Monitoring CyberSense Job and Alerts
➔ Monitor Jobs

The CR software creates a job when you run a policy schedule or recovery operation. The
Jobs content pane shows the job status, which indicates the progress of the job. Lists the
jobs that are running, successfully completed, canceling, or canceled.

When a job is completed, the status is either Success, Warning, or Critical. If the status of a
job is Critical, then a critical alert is also associated with the job.

➔ CyberSense Alerts

The CyberSense alerts appear as Critical alerts in the PPCR dashboard. The alert indicates
that an event occurred and might require some action.

➢ Administration

Administrative tasks can be performed from either the CR UI or on the management host by
using the CRCLI. Common administrative tasks in PPCR include:

• Manually secure and release the CRV.

• Managing users and roles.
• Email server configuration.
• Change lockbox passphrase or database password.
• Backup CR config.
• Reset IP addresses and passwords.
• Collect logs.
• Perform Garbage collection.
• Disaster recovery.
• Cyber Recovery Vault (CRV)

If a security breach occurs, the Security Officer or an Admin user can manually secure the
CRV. During this time, the CR software performs no replication operations.

To secure or release (unsecure) the CRV, log in to CR and access the dashboard. Under
Status, do one of the following:

- To secure the CRV if you suspect a security breach, click SECURE VAULT so that
the CRV status changes from Locked to Secured. All Sync policy operations stop
immediately, and no new Syn policy operations can be initiated. The CR software
also issues an alert that the CRV is secured.
- To unsecure the vault when you are confident that there is no longer a security threat,
click RELEASE VAULT. The CRV status returns to Locked. Syn policy operations
can now be initiated.

70
 • Backup Configuration

Data can be migrated data from a CR software deployment to a CR virtual appliance

deployment. This data backup the configuration and restores it in the target.

To perform the procedure, follow these steps:

- Deploy the CR virtual appliance in the CRV.

- For a migration, upgrade the current CR software deployment to the same version
as the CR virtual appliance.
- On the host that is running the CR software, run the command to create a backup
copy.

crsetup.sh --save

- Copy the newly created backup file onto the CR virtual appliance.
- On the CR virtual appliance, run the command to perform a recovery and restore the
data.

crsetup.sh --recover

• Email Server

If the configuration allows email to leave the CRV, specify which users receive email
notifications about alerts and connect to an SMTP email server. By default a CR deployment
uses Postfix to rout and deliver CR email notifications to CR users. Postfix is an open-
sourced mail transfer agent that is included with most non-Windows systems. Optionally,
enable and configure the option to use an external email service.

• Select Administration, then Alert Notifications from the Main Menu. The table lists
CR users, their email addresses, and roles.
• For each user that you want to receive email messages, select either or both the
Receive Critical Alerts and Receive Warning Alerts check boxes. If Receive Warning
Alerts is selected, by default, the user also receives critical alerts.
• To send a test email to the user, click SEND TEST EMAIL. Contact the intended
user to verify if the email was received.

71
 • Delete Unneeded Objects

Delete alerts, events, expired and unlocked copies, and jobs when they are no longer
needed. By setting a CR cleaning schedule, you can avoid system slowdown. The CR
software provides a default cleaning schedule, which you can modify with:

• From the Masthead Navigation, click the gear icon to access the System Settings
list.
• Select Maintenance.
• To modify the default cleaning schedule, click Cleaning Schedule.

Specify the frequency for when the schedule runs, the time that the schedule runs next, and
the age of the objects to delete. The cleaning operation runs, using the values that you
defined in the cleaning schedule.

• To run the cleaning schedule on demand click Clean Now, then click RUN NOW.

The cleaning operation runs immediately, using the values that you defined in the cleaning
schedule.

• Reports

With the release of CR 19.7 and 19.8, new reports are available.

➔ Cyber Recovery Daily Reporting

CR now sends daily reports to all the configured admin users. The report is sent once a day
and can be scheduled using the CRCLI. The report has information about all the jobs for the
last 24 hours.

➔ Cyber Recovery Telemetry

Telemetry is used to gather data on the use and performance of applications and application
components. These components include how often certain features are used,
measurements of start-up and processing times, hardware, application crashes, and general
usage statistics and user behavior.

Sometimes, detailed data is reported like individual window metrics, counts of used features,
and individual function timings. This kind of telemetry can be essential to software
developers to receive data from a wide variety of endpoints. These endpoints cannot all be
tested in-house. Data on the popularity of certain features and whether they should be given
priority or be considered for removal is also useful.

Due to concerns about privacy since software telemetry can easily be used to profile users,
telemetry in user software is often the choice of the user. They are commonly presented as
an opt-in feature (requiring explicit user action to enable it) or user choice during the software
installation process.

72
 ➢ Disaster Recovery

On-demand DR backups of the CR configuration can be scheduled or run on a predefined

CRV PPDD MTree. Users now have the option of viewing DR backups that they run for the
CR software in the CR UI. If there is any issues with the software, users can recover the CR
configurations from the backup.

To launch the DR Backup configuration, select the gear icon on the top-right menu. Select
DR Backups.

• DR Backup Configuration

The DR Backups window has two sections. The first section is configuration. The DR Backup
requires an MTree that will be used for the backup. Enable backups as the first step.

Once DR backups are enabled, select the PPDD system in the CRV. Specify the MTree for
replication. Finally, set the frequency of the backups in days and hours, and select Save.

• DR Backup Information

The second section relates to managing backups. Once the DR backups are configured,
system backups can be run on demand by selecting backup now.

Also, backup file names and creation dates are also listed here. While backups are being
run, they can be seen under Jobs.

• Recovery

The CR configuration recovery process must be completed manually by exporting the PPDD
share and mounting it on a new CR host. Then run recover: # crsetup.sh -recover

Two additional NFS exports are seen on Vault PPDD when DR backup is configured. One
NFS export is used to list the backups and other one to run the backup. Bot the operations
can be run simultaneously.

➢ Monitoring
• Vault Status

The vault status indicates if the vault connection to the production system is open or closed.
The vault status is in the Locked state unless CR is replicating. If necessary, the Security
Officer can manually disconnect the vault. The vault connection states are listed here:

73
 Status Description

The connection is closed because no replication is being performed. If a

Locked replication policy is run, the CR opens the connection and changes the
vault status to Unlocked.

The connection is open because a replication is being performed. The

Unlocked
status returns to Locked when the replication completes.

All replication network connections are secured because the Security

Officer or an Admin user manually locked the connection due to a security
Secured
breach. You cannot initiate any replication policy actions. When the CRV is
released and returns to the Locked state, you can run replication policies.

There are cases when there are multiple PPDD systems in the CRV. If one
Degraded PPDD system is unable to communicate with the CR software, the vault
status is Degraded.

There are cases when there are multiple PPDD systems in the CRV. If all
Unknown the PPDD systems are unable to communicate with the CR software, the
vault status is Unknown.

• Alerts and Events

CR generates notifications about alerts and events. An alert indicates that an event occurred
and might require you to fix the condition. Alert categories include:

• System

Indicates a system issue that might compromise the CR system such as a failed component.

• Storage

Indicates storage issues such as insufficient disk space.

• Security

Indicates that a user cannot log in or malware might have been detected.

74
 • Jobs

When you run a policy or recovery operation, a job is created. The job status indicates the
progress of the job.

The Jobs content pane shows the job status, which indicates the progress of the job. It lists
jobs that are running, successfully completed, or canceled. When a job is completed, its
status is either Success, Warning, or Critical. If a status is Critical, a critical alert is also
associated with the job.

When a policy is created or edited, set an optional job window timeout value in hours for
how long a job for a Sync action runs. If the duration of the job reaches the timeout limit, CR
issues a warning alert. Cancel the job, if necessary.

➢ User and Credential Management

Preloaded accounts and default credentials are presented here. Select Default Credentials
to learn more about them. Direct root login is disabled for security reasons.

• User Roles

CR users are assigned roles that determine the tasks that they can perform in the CRV
environment. The CR installation creates the default crso user and assigns the Security
Officer (crso) role to this user. The Security Officer user must perform the initial CR login
and then create users. There is only one Security Officer per CR installation; you cannot
create another Security Officer.

User Role Account Description Permissions

Name

Dashboard dashboard The dashboard user View dashboard.

enables the user to view
the CR dashboard but
not perform tasks.

Admin Cyber- The CR installation Create, modify, and disable dashboard users.
recovery- procedure creates the
admin Security Officer user Create, manage, and run policies and associated objects.
account. This user must Acknowledge and add notes to alerts.
perform the initial CR
login and configuration. Change administrative settings.
Admin users can be
created by the crso. Modify own user account and change own password.
Manually secure and release CRV.

Security crso The CR installation All Admin permissions.

Officer procedure creates a
system user on the Create, modify, and disable users.
management host Change and reset user passwords.
system.
Change the crso password.
The management host
system user owns Set the duration after which passwords expire for all users.
certain CR files, folders,
and processes within the Disable multifactor authentication for an Admin user.
application.

75
 • Default Credentials

A CR virtual appliance deployment requires you to set a password for the root user and
admin user during the CR installation. The CR virtual appliance system uses the following
default user accounts and default passwords:

- root (changeme) - Linux operating system root account.

- Admin (changeme) - Linux operating system administrative account.

The cyber-recovery-admin user is assigned User ID 14999 and Group ID 14999. User ID
14999 and Group ID 14999 can be allocated for another system user. If the ID is assigned
the installation procedure prompts whether to continue the installation with that other system
user or cancel the installation.

➢ Authentication

Previous versions of CR did not employ Multi Factor Authentication (MFA). If an attacker
reaches the vault network or if the vault is exposed to an attacker, a security breach could
occur. There was no further protection to getting into the account of the customer with the
admin or crso user.

• MFA

After initial login to the CR UI, users can optionally enable MFA. Authentication can be done
over the UI or CLI to provide added protection for CR software and its resources such as
copies and sandboxes that are stored on PPDD. Any authentication application can be used
to generate security codes.

• MFA Configuration

With any of the supported virtual MFA applications, scan the QR code. Enter the two
consecutive security codes and select Save. If nonconsecutive security codes are entered,
MFA is not enabled. Wait for the authenticator to generate a new security code and ensure
that you enter the next consecutive security code.

• MFA Login

Once MFA is enabled and configured, when a user attempts to log in to the UI a new step
is added. The security code is requested. This code is generated by the virtual MFA app.

• Disable MFA Access

Each user must enable MFA for their own accounts. Only the crso user can disable MFA for
any user who is created in the CR UI. To do so, go to Administration and select Users. Select
the user to disable MFA and select disable MFA.

76
 ➢ Login Count Settings

To improve CR protection, the simultaneous login count maximum value is reduced from 10
attempts to 3. If a customer has it set to a value higher than 3 only for the security user, the
upgrade to 19.7 will set the maximum value for the security-officer login count to 3. The
count only applies to the security officer. Admin and Dashboard roles remain at a maximum
value of 10 attempts.

➢ Multi-Factor Authentication Disable Alerting

An email is sent to the CRSO and the Admin user whose MFA was disabled in CR when
multifactor authentication is disabled. If MFA for the crso is disabled, then only the crso
receives the email. The email is irrespective of which user has disabled MFA and who has
opted to receive an email alert. This security enhancement reduces the risk of intrusion to
the system. At the UI, the alert is generic, and a tag is added on who disabled which MFA
of the user.

If an email is modified, the crso and the user receive an email message that indicates the
change. The old email address, which has since been modified, receives the email
message.

➢ Custom Certificate Support

• Self-Signed Certificate

In cryptography and system security, a self-signed certificate is a security certificate that is

not signed by a certificate authority (CA). These certificates are easy to make and do not
cost money. However, when a website owner uses a self-signed certificate to provide
HTTPS services, people who go to that website sees a warning in their browser.

CR generates a self-signed certificate for each internal service (nonedged service). In 19.8
release, CR encrypts all the internal key files.

• Certificate Signed Request (CSR)

In public key infrastructure (PKI) systems, a certificate signing request (also CSR or
certification request) is a message that is sent from an applicant to a registration authority
of the PKI in order to apply for a digital identity certificate. It usually contains the public key
for which the certificate should be issued. The key identifies information (such as a domain
name) and integrity protection (for example, a digital signature).

CR customer can use crsetup script to generate a CSR on the same server they installed
the CR on. The customer submits the CSR to their CA and apply for a CA Signed Certificate.

77
 • Certificate Authority and Root Certificates

In cryptography, a certificate authority or certification authority (CA) is an entity that issues

digital certificates. A digital certificate certifies the ownership of a public key by the named
subject of the certificate.

• CA Signed Certificate

A CA signed certificate is a certificate that has been issued and signed by a publicly trusted
CA. A CA signed certificate is trusted automatically and authenticated by all popular
operating systems (Windows, Android, iOS, so on) and web browsers (Chrome, Firefox,
Edge, Safari, so on). This authentication ensures that your customers can access your
website without experiencing any security errors.

• How to Add a Custom Certificate

Customer must run crsetup.sh --gencertrequest and crsetup.sh --addcustcert commands on

the CR host. This is the host where CR was installed.

• To generate a CSR, run the following command:

crsetup.sh --gencertrequest

• Submit the CSR to the CA to apply for a CA signed certificate. This results in a digital
identity certificate.
• Add the signed certificate into the CR system.

crsetup.sh --addcustcert

• Considerations

During the upgrade from 19.7 or older release to 19.8, the customers must follow the process
to generate a CSR. The CSR needs to be signed and added to the CR system.

If customers upgrade from 19.8 to any later build, signed certificates are automatically
imported.

If a hostname is changed for CR, customers are asked if they want to replace their certificate
with a new self-signed certificate. If CA signed certificate is being used, customer should
answer no to skip this step and go through the steps to add a custom certificate. Customer
should NOT reuse the previous generated CSR file to apply for a new CA certificate.

78
 ➢ PowerProtect Cyber Recovery for Sheltered Harbor

The Sheltered Harbor standard was created in 2015 by the financial industry. It incorporates
a set of cyber resilience and data protection best practices and safeguards for protecting
U.S. financial data.

Cyber threats, including ransomware, data destruction, or theft targeting production and
backup systems, put consumer and corporate financial data at risk. A successful cyberattack
on a U.S. bank, credit union, or brokerage firm would damage that financial institution’s
reputation. Additionally, it could undermine consumer confidence in the U.S. financial
system, and possibly trigger a global financial crisis.

Sheltered Harbor enhances U.S. financial stability and institutions’ cyber resilience by
isolating critical customer account records and other data immutably within a digital vault. In
the event an institution’s primary or backup systems are compromised by cyberattack or
other event, rapid recovery of this critical data is enabled, facilitating the continuity of critical
customer-facing banking services, ensuring public confidence is maintained.

The Sheltered Harbor initiative was launched by the industry in 2015 to ensure that in a
worst-case scenario:

• Public confidence in the financial sector is maintained.

• Critical data sets are protected across the industry.
• Critical services can continue even when systems and backups are down.
• An impacted financial institution has a lifeline to survival.
• All of the above must be achievable independent of the event’s origin.

Dell is the leader in Cyber Resilience solutions and brings its experience and the Cyber
Recovery Solution to meet the strict requirements of the Sheltered Harbor specification.

Its commitment to being the 1st provider to have an endorsed on-premises Sheltered Harbor
data vaulting solution is further proof of their commitment to help their customers protect the
integrity, confidentiality and availability of their data and their data-driven business.

Dell will continue to build on Dell Technologies’ and Dell EMC’s mission to help their
customers transform their data protection and security strategy to address the modern
challenges of cyber threats across the spectrum from ransomware to insider attacks.

• First Sheltered Harbor-endorsed turnkey vaulting solution.

• Built upon a proven solution with 5 years in the market, hundreds of customers.
• Fast deployment with automated vaulting operations and monitoring.
• First Solution Provider member in the Sheltered Harbor Alliance program.
• Simple vault expansion to protect critical data unrelated to Sheltered Harbor.
• Single vendor support, help desk and regular product updates from Dell.

79
 • Sheltered Harbor Data Protection Process

The Sheltered Harbor data protection process consists of three basic steps:

1. Data that has been identified by Sheltered Harbor as critical to consumer confidence
is extracted from the institution’s system and written format. Participant performs
nightly extraction of critical customer account data in standard Sheltered Harbor
format.
2. The data is packaged and encrypted, and then sent to a “data vault” that meets
certain requirements specified by Sheltered Harbor. The data vault is encrypted,
unchangeable and completely segregated from the institution’s other infrastructures
including backups.
3. If there is a cyber-attack and the Sheltered Harbor Resiliency Plan is activated, that
data is removed from the vault and transmitted to a restoration platform, where
access for customers can be restored. Secure recovery and restoration is
independent of external systems to quickly resume business operations.

• Cyber Recovery and Sheltered Harbor

Dell Technologies is the first Sheltered Harbor Alliance partner that developed a turnkey
data vaulting solution for U.S. financial institutions. PPCR for Sheltered Harbor is the first
on-premises turnkey data vaulting solution designed to meet all technical requirements for
Participants implementing the Sheltered Harbor standard.

➔ Data Vault

Nightly backups of critical data in the Sheltered Harbor standard format are created by the
participating institution or service provider. The data vault is encrypted, unchangeable and
isolated from the institution’s infrastructure.

➔ Isolation & Governance

An isolated, secure environment disconnected from corporate networks restricts users other
than those with proper clearance. Automated data copy and air gap management assure
preservation of data integrity, security, and confidentiality.

➔ Recovery & Remediation

If a Sheltered Harbor Resilience Plan is activated the participating institution can quickly
recover data from the vault to enable the fastest restoration and resumption of banking
operations.

80
 • Vaulting Process

PPCR for Sheltered Harbor operates similarly to the “standard” CR Solution. There are a
few extra steps to meet the specific requirements of the Sheltered Harbor Specification.

1. Extract: Critical data is extracted from the institution’s systems and written in the
Sheltered Harbor-designated format. It is written by the customer to a PPDD MTree.
2. Sync: The data set in the MTree is synchronized across the air gap and securely
copied into the vault.
3. Copy: Data is copied from the replication folder (MTree) into a retention folder.
4. Process: The data is processed (packaging, encryption, etc.) according to the strict
requirements of the Sheltered Harbor Specification.
5. Lock: The finished retention set is locked per the Sheltered Harbor specification. It
cannot be changed or deleted before the retention period expires.
6. Restoration Platform: If an event is declared, the data can be accessed in the vault
and transmitted, per the Sheltered Harbor specification, to a restoration platform.
➢ Sheltered Harbor Implementation

To comply with the Sheltered Harbor Specification, the CR vault architecture is being
extended to perform the Archive Generation and Secure Repository processes. Extracted
Sheltered Harbor data is saved in production, then securely replicated via a logical, air-
gapped, dedicated connection to the vaulted environment. The remaining steps, such as
retention locking, are performed.

By creating a dedicated, isolated environment, physically separated from corporate

networks and backup systems, critical data sets are available in standardized format.
Sheltered Harbor participants are required to protect these data sets. Basic banking services
can therefore be quickly resumed for customers. Deployment can be measured in a matter
of weeks instead of months, and with a certainty of compliance with the Sheltered Harbor
Specification.

• Enable Sheltered Harbor

Sheltered Harbor will not be enabled and available as an option by default. Once customer
register them with Sheltered Harbor, only Dell EMC Service Representative is allowed to
enable it on CR through CLI. Once enabled, Sheltered Harbor option will be visible under
Infrastructure Tab.

The Sheltered Harbor option provides the possibility of adding, editing, or deleting one or
multiple financial institutions.

81
 • Sheltered Harbor Policies and Copies

When Sheltered Harbor is enabled in the infrastructure a new type of Policy is permitted.
Notice the policy type of Sheltered Harbor is available. It allows users to choose the financial
institution for which the policy is created. Unlike other types of policies, retention locking is
mandatory for a Sheltered Harbor policy.

For Sheltered Harbor Policy types, only one option “Sheltered Harbor Copy” is available.
This option performs the following actions:

• Sync: Synchronizes the data from the production DD to the Vault DD.
• Verify: Verifies the input data against the manifest file.
• Copy: Creates a fastcopy of the replication destination and places it into the CR
repo.
• Certify: Zips and encrypts the data in the CR repo copy per the Sheltered Harbor
specification.
• Lock: Retention lock the copy.
• Report: On success, sends the Sheltered Harbor attestation message out of the
vault to the email configured in the Financial Institution
➢ Recovery Process from the Sheltered Harbor Copy

The recovery process of a Sheltered Harbor copy is manual. It is necessary to setup

separate recovery host with SH recovery scripts to recover the data from the Sheltered
Harbor Copy.

Sheltered Harbor manages the monitoring log utility, which accepts and records the daily
attestation messages. Sheltered Harbor monitors these attestation messages, notes
noncompliance, and escalates with noncompliant participants, and provides compliance
statistics to the financial industry.

➢ Stealth Agents

Unisys Stealth Agents are installed on endpoints running supported Operating Systems to
enforce Stealth security policies. Stealth agents can be installed in the non-OVA machines,
which are the Recovery Host and the CyberSense server.

82
 ➢ Secure Virtual Gateways

The Secure Virtual Gateways uses Virtual Stealth Endpoints (VSEs) as network devices to
translate and route traffic between the Stealth-enabled network. Devices running in the clear
text network that you configure to communicate with Stealth enabled endpoints are known
as clear text endpoints.

Clear text endpoints include servers, workstations, or other devices (for example, printers)
that are unable to run the Stealth endpoint software because they are running unsupported
operating systems. By using the Gateway, clear text endpoints are enabled to participate in
Stealth Communities of Interests (COIs).

Secure Virtual Gateway are also used for policy enforcement in situations where Stealth
agents are not supported, such as legacy systems, appliance-based devices and IoT
devices. Can be deployed in Layer 3 (route mode - SVG deployed as a router) or Layer 2
(cleartext proxy mode - SVG deployed as a bump in the wire). Layer 2 mode allows the SVG
to be deployed in existing environments within incurring changes to the network.

➢ Terminology
➔ Index Job

An index job defines the source of the backup file (MTree/location mounted via NFS) to be
indexed and the indexing options.

➔ Segment

Segments hold the indexing results for one or more jobs that were run simultaneously.

➔ Post Processing

Post Processing enables and optimizes the segments for analysis and searching.

➔ Analysis

Passes the job to the Machine Learning engine and compares the content changes from the
previous backup.

➢ Unisys Stealth

Unisys Stealth is a software-based solution that provides zero trust security through identity-
driven encrypted microsegmentations. In the context of PPCR, Stealth microsegmentations
provides the option to be implemented with or without changes to the network architecture.

83
 • Stealth Principles

Stealth principles are simple; trust no user or device and grant as little access as possible.
User or device trust is inside or outside the private network. Access is granted based upon
reliable identification.

Stealth implements identity-based microsegmentations, leveraging identity management

systems such as Active Directory or LDAP to define and manage security policies.

Stealth creates dynamic, identity-based microsegments called communities of interests

(COI) to reduce risk and treats all network traffic as untrusted until COI membership is
confirmed.

Microsegmentation of Stealth cryptographically hides and protects devices from all

unauthorized devise.

• Community of Interest (COI)

COI is the key concept in Stealth Network Security Policies. COIs are secure virtual network
enclaves on existing IP networks. COIs are cryptographically isolated from each other.
Stealth-enabled systems are “dark” to unauthorized traffic. Traffic in each COI is encrypted
with ephemeral keys.

• Benefits of Stealth in Cyber Recovery

There are multiple benefits of Stealth in PPCR. The PPCR solution with Unisys Stealth
protects systems of record data in their original form and secures all traffic with (and within)
the vault with trusted encrypted access.

• Stealth adds an additional level of protection to the PPCR solutions to increase

deployment flexibility and lower your risk profile.
• Stealth provides a cryptographic wrapper around a PPCR vault, so it is undetectable
and inaccessible from other systems in the production network.
• The PPCR solution with Unisys Stealth secures the replication link as well as traffic
within the vault and enables swift cyber recovery.

84
 • Components

There are several Unisys Stealth components that can be used to secure a PPCR
environment.

The Stealth Management Server consists of an Enterprise Manager and Authorization

Services. The Enterprise Manager is a web-based UI for managing and configuring Stealth
objects and security controls. It also manages license distribution to Stealth enforcement
points. The Authorization Services verify the identity of the enforcement points and
provisions the security policies to the enforcement points. The services can be deployed on
a separate server for better resilience and performance.

Stealth Enforcement points enforce the security policies that are provisioned by the
Authorization Services. Security policies include COI membership and filters to control
communication access between Stealth-enabled endpoints.

85
 o PowerProtect Cyber Recovery Implementation
➢ Production System Installation Requirements

The production environment must have at least one PPDD system. This PPDD system must
have at least one MTree replication context. The replication context is configured for
replication for the PPDD system in the CRV.

When multiple PPDD systems are deployed in the production environment, they can be
configured to replicate to as many as five PPDD systems in the CRV.

➢ Storage Systems
• PowerProtect DD

PPCR supports PPDD systems running DDOS 6.0.2.20 and later. Ensure that the CR PPDD
system has more space than the production PPDD system. The following features are not
supported in PPCR implementation:

• PPDD with Cloud DR and Cloud Tier in the CRV.

• PPDD with Cloud Tier in the CRV.
• DP4400

The replication target can be a supported PPDD system or a DP4400 Integrated Data
Protection Appliance.

If the replication target in the CRV is a DP4400 Integrated Data Protection Appliance, the
production-side system must also be a DP4400 Integrated Data Protection Appliance.

Other than DDOS and AVE, the CR software does not support other features on the
Integrated Data Protection Appliance in the CRV. It is recommended that you disable them.

• DP5300/5800

DP5300 and 5800 Integrated Data Protection Appliances are not supported as a replication
target in the CRV. These IDPA models have been qualified for production environment
replication to a supported PPDD system target in the CRV.

• DP8300/8800

The DP8300 and DP8800 IDPA are not supported in the production or the CRV environment
due to Avamar Grid support limitations. However, replication through a single node or Virtual
Edition supported.

86
 ➢ Production Backup and Recovery Applications
• Avamar

PPCR supports Avamar versions 18.1 and later. Single-node physical appliances and AVE-
only servers are supported. Avamar grids are not supported. Validated Avamar checkpoints
are stored on the PPDD system.

• NetWorker

PPCR supports NetWorker versions 18.1 and later. The NetWorker server database and
data devices are stored on the PPDD system.

• PPDM

PPCR supports PPDM versions 19.3 or later. If you plan to use PPDM for backup and
recovery with CR version 19.8 or later, upgrade to PPDM version 19.3 or later. Otherwise,
you cannot use PPDM with the CR software.

The PPDM server backups and policy data are stored on the PPDD system. DDOS must be
version 6.2 or later.

➢ CR Vault System Requirements

The CRV storage environment includes a minimum of one and a maximum of five physical
or virtual PPDD systems. These systems are on the same network as the CR software. Each
PPDD system has the following requirements:

• Version

PPDD systems running DDOS 6.0.2.20 or later. Deployments that use the PPDM application
for recoveries must run DDOS version 6.2 or higher.

• Ethernet Interfaces

Two Ethernet interfaces:

• A primary interface is for the PPDD hostname.

• CR software manages a second dedicated interface for replication.
• Account

A PPDD account with the admin role for use by the CR software to manage PPDD
operations. The account name recommendation is cradmin, however, another name can be
provided. The sysadmin account cannot be used for the CR PPDD system.

• Licenses

Valid licenses for DD Boost, Replication, Retention Lock Governance, and Retention Lock
Compliance.

87
 • Retention Lock

PPDD Retention Lock software provides data immutability for a specified time. Retention
Lock functionality is enabled on a per-MTree basis, and the retention time is set on a per-
file basis.

Retention Lock is not required for CR but is recommended as an additional cyberresiliency

measure. CR does not support Indefinite Retention Hold capability of Retention Lock
Governance, or Compliance modes.

Retention Lock Compliance mode is not supported on the following platforms:

• PPDD3300 appliance: Only Retention Lock Governance Mode is supported.

• DP4400: Only Retention Lock Governance Mode is supported.
• DDVE: Only Retention Lock Governance Mode is supported.
• Policies

For each CR policy in the vault, capacity for at least three MTrees to protect one production
MTree.

For a high availability (HA) deployment, two floating IP addresses: one for the CR
management host and one for replication. HA must be enabled on the PPDD system.

If a failure occurs during a Sync operation, the job fails and the CRV might remain unlocked.
The CR software provides an alert every hour until you lock it manually.

It is recommended that you perform an initial replication between the production and vault
systems for each replication context before you define CR policies.

➢ Cyber Recovery Virtual Appliance

The CR virtual appliance is a preconfigured virtual machine that can be readily deployed
onto a VMware hypervisor. The CR virtual appliance has the following requirements:

• VMware vCenter/ESXi, version 6.5, 6.7, and 7.0.

• Approximately two G for deploying the OVA file.
• Approximately 195 GB for three disks that are partitioned as follows:
➔ Disk 1 and 2, 48 GB
➔ Disk 3, 97 GB
➔ A thin provisioned environment does not use all the space.
• 4 CPUs, single core per socket.
• 8 GB memory.

The CR virtual appliance deployment is configured with one interface by default. Optionally,
after the deployment, extra virtual Ethernet adapters can be added.

88
 ➢ Cyber Recovery Management Host

The management host is a physical or VM host with the following requirements:

One of the following operating systems with the latest updates, patches, and security
patches:

• CentOS Linux Version 7.6, 7.7, 7.8, and 7.9.

• Red Hat Enterprise Linux Version 7.6, 7.7, 7.8, and 7.9.
• SUSE Linux Enterprise Server Version 12 SP5.

The following system requirements:

• 4 GB RAM
• 200 GB disk space
• 1.5 GB free space to extract the CR software
• 10 GB or more free space for installation of CR software

The following table lists the required and optional network ports that CR functions require:

Port Required Service Direction Description

Provides web browsers with HTTPS access
14777 Yes Nginx Inbound
to the CR UI.

Provides the HTTPS connection for the

14778 Yes REST API Inbound
user and UI REST interface.

Cyber Used to upload or download the Docker

Recovery container images. The installation and
14779 Yes Inbound
Docker upgrade scripts retrieve the images from
Registration the registry, if needed.

Provides access to the CR REST API

14780 No Swagger Inbound
documentation.

Provides bi-directional communication

22 Yes SSH Outbound between the SSH client and the remote
systems in the CRV.

Used for SMTP email notifications about

25 No Notifications Outbound
alerts and events.

Used to perform NFS mounts between the

Inbound and
111 Yes NFS Client PPDD system and the CR management
Outbound
host.

Controls the time synchronization of CR to

123 No NTP Inbound
another reference time source.

89
 Port Required Service Direction Description
Used to perform NFS mounts between the
Inbound and
2049 Yes NFS Client PPDD system and the CR management
Outbound
host.

2052 Yes NFS Client Outbound Used to mount to the PPDD system.

Provides access to the database that

27017 Yes MongoDB Inbound holds CR configurations. The installation
process configures these ports.

Used to upload or download the Docker

CR Docker
5000 Yes Inbound container images. The installation process
Registration
configures these ports.
<

➢ Cyber Recovery Vault Backup and Recovery Applications

• Avamar

Applications can be deployed optionally in the CRV environment. Avamar versions 18.1 and
later are supported. The following configurations are required:

• The same Avamar version that is deployed on the production system.

• A single-node or AVE server (Avamar grids are not supported).
• An uninitialized and correctly sized Avamar instance that is equivalent to the size of
the Avamar instance on the production system.
• A hostname that matches the production hostname.
• The PPDD system has the same Avamar DD Boost account name and UID.
• NetWorker

NetWorker versions 18.1 and later are supported. The following configurations are required:

• The same NetWorker version that is deployed on the production system.

• An uninitialized and correctly sized NetWorker instance that is used to perform a
nsrdr operation. The NetWorker uses data that is replicated from the production
PPDD system to the CRV PPDD system.
• PPDM

PPDM version 19.3 and later, with the following requirements:

• DDOS version 6.2 and later.

• Credentials for the PPDM host and the PPDM application that match the production
system.
• A UID that matches the production user UID.

The CR software enables you to perform a VM recovery or a file system recovery for a
PPDM deployment.

90
 • Other Considerations

Follow the documented Avamar, NetWorker, and PPDM procedures for deployment in the
CRV environment. Follow the CR documentation to run the recovery procedures.

The CR software can protect third-party application data. Information about third-party
applications is beyond the scope of Dell Technologies documentation. See the DR
protection and recovery procedure guidelines for the application vendor for more
information.

Data formats such as vDisk and VTL are unsupported for protection in the PPDD system in
the CRV.

➢ Firewall Configuration

Before the CR software and the Docker components are installed, ensure that the firewall
settings are configured appropriately for the environment. Determine if the CRV must be a
firewall-enabled environment or a firewall-disabled environment.

• Enable the Firewall

Enable the firewall on the CR host, in the CRV. This procedure takes place before Docker
is configured and the PPCR software is deployed.

1. From the CLI, verify the status of the firewall.

systemctl status firewalld

2. Enable the firewall

systemctl enable firewalld

3. Start the firewall.

systemctl start firewalld

• Configure SELinux

When the firewall is enabled, configure SELinux.

1. Edit the SELinux configuration file to use the firewall.

vi /etc/selinux/config

2. In the SELinux option, replace the state of SELinux to enforcing. This change
enforces the security policy.

SELinux=enforcing

3. Save the configuration file. (wq!)

4. Reboot the system. (reboot).

91
 • Disable the Firewall

Disable the firewall on the CR host, in the CRV. This procedure takes place before Docker
is configured on the PPCR software is deployed.

1. From the CLI, verify the status of the firewall.

systemctl status firewalld

2. Stop the firewall.

systemctl stop firewalld

3. Disable the firewall.

systemctl disable firewalld

• Configure SELinux

When the firewall is disabled, configure SELinux.

1. Edit the SELinux configuration file to use the firewall.

vi /etc/selinux/config

2. In the SELinux option, replace the state of SELinux to disabled. This change disables
the security policy.

SELinux=disabled

3. Save the configuration file. (wq!)

4. Reboot the system. (reboot)
➢ Docker Containers

Docker is a platform for developing, shipping, and running applications. With Docker, it is
easy to decouple the applications from the infrastructure. The applications can be deployed
quicker and more flexibly when they are decoupled.

Applications are packaged and run in containers. Docker Compose is a tool for defining and
running multicontainer Docker applications. Docker Engine is a client/server application with
these major components:

92
 • Docker Requirements

The following Docker components are required to install CR software:

• Docker Version 17.06.0, 18.09.7, 19.03.5, 19.03.8, 19.03.12, 19.03.13, and 20.10.2.

RedHat Linux and SUSE Linux Enterprise Server only support Docker Enterprise Edition
(EE). CentOS Linux also supports Docker Community Edition (CE). Ensure that you install
a proper version of Docker.

• Docker Compose Version 1.21, 1.24, 1.25.3, 1.25.4, 1.26.2, and 1.27.4.

Set up the firewall before installing Docker. At installation, ensure that you enable Docker to
restart and to configure firewall settings automatically when the management host reboots.

➢ Install the Cyber Recovery Software

- Part 1

The CR software is installed using the crsetup.sh setup script. The installation takes
approximately 5 minutes.

1. Log in to the CR management host as root.

2. Download the CR installation package from the PPCR support site. This file requires
approximately 1.5 GB of free space.
- Part 2
3. Untar the installation package.

# tar -xzvf cr-release-bundle-19.8.0.1-148.tar.gz

4. Go to staging directory and make the crsetup.sh setup script an executable script:

# cd staging

# chmod +x ./crsetup.sh

5. Verify that the prerequisite software is installed:

# ./crsetup.sh --check

- Part 3
6. Use the hostname -i command to determine if there are multiple IP addresses that
are associated with the management host. If the command returns multiple IP
addresses, use the following command to specify the IP address for the CR software.
This is used to communicate with the PPDD storage in the CRV.

# export dockerHost=10.127.25.1333

7. Begin the installation.

# ./crsetup.sh --install

93
 8. When prompted, press Enter to view the EULA. Enter q to exit the EULA at any time,
and then enter y to accept the EULA. If you decline the EULA, the installation stops.
Otherwise, the installation continues.
• UID:GID

The installation procedure attempts to create a Linux user (cyber-recovery-admin) on the

management host in the CRV. It assigns a reserved UID:GID of 14999 to the cyberrecovery-
admin user. This user owns specific installation directories.

If the reserved UID:GID 14999 is assigned to another user or the cyber-recovery- admin
user exists but is not assigned the reserved UID:GID 14999, the installation procedure
issues a warning message. Otherwise, the installation procedure continues.

9. If the installation procedure displays a warning about creating the cyber-recovery-

admin user, indicate if you want to continue or cancel the installation.
If you complete the installation, the CR software operates correctly, however, a non-
cyber-recovery-admin user might own some installation directories.
- Part 4

10. When prompted, specify the directory where you want to install the CR software, or
press Enter to accept the default location.
11. When prompted, specify the directory where you want to install the database or press
Enter to accept the default location.
Output is displayed about creating directories, loading Docker containers, and
starting the Docker registry and MongoDB database.

The installation procedure also creates internal IP addresses that enable communication
between the Docker containers.

12. At the prompts that follow, enter, and confirm a lockbox passphrase, database
password, and Security Officer (crso) account password of your choosing.

Remember the lockbox passphrase. It is required to perform upgrades and reset the
Security Officer’s password. If you forget the lockbox passphrase, you must reinstall the CR
software.

Enter a unique passphrase or password for the lockbox, the database, and the crso account.

The passphrase and password requirements are:

• Between 9-64 characters.

• At least one uppercase character.
• At least one lowercase character.
• At least one number.
• At least one special character: ~!#$^&*()+={}|:”;?[]-_,’.

94
 • Results

The installation procedure starts CR services and then exits. The installation procedure
loads the cyber-recovery.service file. If the CR management host restarts after a shutdown,
this file directs the management host to start the CR services automatically.

In your browser, go to the URL shown at the end of the installation script. Then, log in to the
CR UI using the default Security Officer (crso) account and the password that you created.

If your system has an active firewall, ensure that the ports that are listed at the end of the
installation script are open on the firewall.

➢ Install the Cyber Recovery Virtual Appliance

This topic demonstrates how to deploy the CR virtual appliance file to a VMware ESXi host
in the CRV.

1. Download the OVA file from the PPCR support site.

2. Ensure that the configuration parameters are determined. These parameters include
DNS, default gateway, FQDN, and IP address of the VM.
- Part 1
1. From the vSphere Client in the CRV, use the Deploy OVF Template wizard to deploy
the Cr virtual appliance file.
a. Select the Deploy OVF template and select the CR vApp OVA file.
b. Select the name of the vApp and location to store it.
c. Select the ESXi where the vApp will be deployed.
d. Review the template configuration.
e. Review and accept the license agreements.
f. Select the virtual disk format and datastore to save the vApp.
g. Select the destination network, IP allocation, and IP protocol.
h. Customize the deployment with network and NTP configuration.
i. Review the configuration and start the deployment process.
2. When the CR virtual appliance deployment is completed, open the vCenter console
for the newly deployed appliance.
3. Log in as the root user using the default password changeme.
4. Run the crsetup.sh script with the deploy option to begin the installation.

# crsetup.sh --deploy

95
 - Part 2
5. At the prompts, change the admin password of the CR VM. Then, change the root
password for the CR VM.
6. At the prompts, enter a unique passphrase or password for the CR Security Officer
(crso), the CR lockbox, and MongoDB.

Remember the lockbox passphrase. It is required to perform upgrades and reset the
Security Officer’s password. If you forget the lockbox passphrase, you must reinstall the CR
software.

• Results

The installation procedure starts CR services and then exits.

The installation procedure loads the cyber-recovery.service file. If the CR management host
restarts after a shutdown, this file directs the management host to start the CR services
automatically.

In a browser, go to the URL shown at the end of the installation script. Then, log in to the
CR UI using the default Security Officer (crso) account and the password created.

➢ Installation Login

The CR installation procedure adds the crso user to the database. This user has the Security
Officer orle and must perform the initial login and then create one or more admin users.

In a supported browser, go to https://:localhost:14777.

In the Username field, enter crso and the password set for the user.

➢ Pre-Installation Tasks

Before a CyberSense software installation, the following preinstallation tasks must be

verified:

For a successful installation of CyberSense, it is recommended that you perform a site

survey before the installation. Consider the following:

• Resources

Review the server requirements and confirm that the environment can adequately
accommodate the system. If the environment cannot meet the minimum requirements, then
you can expect unpredictable behavior.

96
 • Support Portal Access

Ensure that you have an account that is created in the Index Engines Support portal prior
installation. The license owner must sign into the portal and sign the EULA. You must accept
the EULA. Otherwise, you cannot register the new system and activate or install the license.
The deployment engineer is given a project key or added by Index Engines as a project
member.

Important Note: The CheckEngine.sh script is available on the Index Engines Support
Portal. Install the files in /usr/local/bin and make executable. The script should be run to
assure the system is ready for a CyberSense software installation and then to confirm the
final configuration.

• Server Requirements

The table below shows the server requirements for deploying Index Engines software:

MEDIUM: Amount of LARGE: Amount of front-

Component front-end backup data to end backup data to be
be indexed ≤ 15 TB indexed ≥ 15 TB

CPU (cores) 20 32

Memory 192 GB 384 GB

CentOS (7.9); Red Hat (Version 7.x where x is ≥ 5; support for

OS
version 8.0 is pending).

SELinux Disable SELinux before installing the Index Engines software.

Architecture X86_64

Filesystem XFS is preferred. EXT4 is also acceptable.

/partition: 120 GB
/boot partition: 500 MB
/opt/ie partition: The Index Engines Sales Engineer provides
Host Operating System guidance on the size of the partition that is based on the details
Partitioning Schema: of your data.
MINIMUM Requirements and
Storage (Direct attached, *Swap = 2 (Memory) + 64
local, or SAN) • Memory: 128 GB, SWAP: 320 GB
• Memory: 192 GB, SWAP: 448 GB
• Memory: 256 GB, SWAP: 576 GB
• Memory: 384 GB, SWAP: 832 GB

At least one 10 GB network interface is required for proper

performance. The Index Engines Sales Engineer can provide
Connectivity to PPDD
guidance on the number and type of network connection that
are required for the server.

97
 ◼ Firewall Port Configuration

Ideally, the firewall should be disabled on the CyberSense server. Disable and turn off the
firewall and iptables unless the Linux firewall must be enabled.

Firewall Port Configuration

Open Port To Allow When

To access the Index Engines

80 or 443 Inbound connection
software

All appropriate
To access the corresponding
NDMP and NFS Outbound
sources
PORTS

Inbound and outbound

22 For SSH communications
connection on all engines

Inbound connection on To participate in a federation. A

Federation Manager; outbound federation consists of two, or
5432
connection on each Federation more Index Engines systems
Member joined to share work

22, 7776, 7779,

Inbound and outbound
7781, 7795, To participate in a federation
connections on all engines
7799, and 8476

111, 1110, Inbound and outbound To allow archiving in a federation

2049, and 4045 connection on all engines when using local caching

◼ Disable SELinux

Ensure that SELinux is disabled. The sestatus command displays the status of SELinux. If
the status is anything other than “SELinux status: disabled”, then SELinux is not fully
disabled.

To disable SELinux, you must edit /etc/selinux/config file by setting the value of SELinux to
disabled as shown in the screenshot.

Once the changes are saved, a reboot is required before the installation of ie-docker RPM.

◼ Set Hostname

Set the hostname before installing Index Engines application. Failure to set hostname
provides unpredictable results which require support intervention. Reboot is required after
setting the hostname before installing ie-docker rpm. Consistently use the hostname of the
engine (preferred to use FQDN). Check if the hostname is same inside and outside of the
ie-container after the application is installed.

98
 ◼ Map Local Host to Loopback Address

The local host must be mapped to the loopback address. If the local host is not mapped,
then edit the /edit/hosts file as shown in the example.

➢ Pre-Requisite Software Installation

With the release of CyberSense 7.6.0, Index Engines provide local repositories from which
you can install the nbd-kmod kernel module and its dependencies. The repositories also
include Docker Community Edition (CE) and its dependencies.

◼ Install nbd-kmod without Local Repositories

You can install the nbd-kmod kernel module without any local repositories. If the nbd-kmod
without local repository option is chosen, then you must resolve all the dependency issues
that may be identified at the time of installation.

The nbd-kmod RPM file can be downloaded from Index Engines repository. The RPM file
can be installed on your host using the RPM command or another utility. To install using the
RPM command, log in as root user and type: rpm -ivh <rpmfile>

◼ Install Local Repositories

The local repository files are available on the Index Engines support portal.

The file names are of the form: repobundle-ie-docker-OS-YYMMDD. Where:

• Operating System can be centos-7, rhel-7.5, rhel-7.6, rhel-7.7, rhel-7.8, or rhel-7.9.

• YYMMDD is the year, month, and date.

Download the repository that matches the operating system to a temporary folder (/tmp/ie-
repos) on the server.

Once the file is downloaded, go to the directory, and run the following command:

sh <repo bundle name> install

The command disables any repositories that were enabled before installation and from
enabling Index Engines provided repositories.

99
 ◼ Install Docker Community Edition

The Docker CE is in the local repositories.

Run the following command to install the Docker CE:

yum install docker-ce

After installing Docker CE, run the following commands to enable and start the docker
services:

• systemctl enable docker

• systemctl start docker
◼ Configure Docker Storage Driver

Once the docker service is started, confirm that Docker uses the overlay2 storage driver by
running the following command:

Docker info | grep “Storage Driver”

The output of the command should be Storage Driver: overlay2.

If the storage driver is not overlay2, then edit /etc/docker/daemon.json file to add the
following information: “storage-driver” : “overaly2”K.

Restart the docker service by running the following command after the configuration change:

sudo systemctl restart docker

◼ Install and Configure Atop

It is highly recommended that you install atop outside of the container for diagnostic
purposes. atop is already packaged in the repository which must be installed.

Run the following commands to install, enable, and start atop:

• yum install atop

• systemctl enable atop
• systemctl start atop

100
 ◼ Download ie-docker RPM

After installing and starting atop, you must download the ie-docker RPM package from the
Index Engines website.

To download the software:

1. Log in to the Index Engines support portal and click the Releases tab.
2. Select the specific release that you want to download.
3. Select the specific build version from the list.
4. Review the availability date, features, and fixes and click Download.
5. From the Download Build dialog box, select the following from the drop-down list:
a. Type: Docker
b. OS: Select supported OS.
6. Finally, click OK to download the file.
a. Move the downloaded file to the Index Engines system and into a temporary
directory.

➢ Install ie-docker RPM File

Once the RPM file is available, you must install the ie-docker file and start the container and
the Index Engines services.

The following steps must be performed to install the RPM file:

1. Log in to the host system using the command-line interface as a root user.
2. Go to the directory containing the RPM file.
3. Run the install command:

yum install ie-docker-<version>.rpm

The command unpacks, verifies checksum, imports, and loads the software into the Docker
container. Click Next on the upper right corner of the page to go to the next page. The
installation takes ten minutes or more, depending on the speed of the system.

101
 ◼ Verify Installation and Start Services

Once the RPM file is available, you must install the ie-docker file and start the container and
the Index Engines services.

1. Verify Successful Image Upload

After the RPM file is installed, you can verify if the image is successfully loaded by running:

docker images

2. Start Container and Services

Before running the first job after the new installation, see the CyberSense Installation
checklist to be sure that all pre-requisites are performed successfully. After verifying the
checklist, you must start the container, which also starts the Index Engines services. Run
the following command to enable and start the services:

systemctl enable ie.services

systemctl start ie.services

The ierestart command with the -r option ensures that the Docker container is removed and
is then created. Use this option from outside the container if there are any changes to the
hostname. Also, the changes must be propagated into the container after the container has
already been created.

➢ Post-Deployment Tasks

Before using the installed software, you must make a note of the Engine ID, register the
engine, and upload and activate the license.

◼ Log into the Index Engines Graphical User Interface (GUI)

Once the software is installed, you must log in the Index Engines UI with the admin
credentials. Result: The Upload License page appears.

Some important notes to keep in mind:

• Ensure that the Index Engines cookies are enabled in the browser.
• If you are unable to access the Sign In page, verify that ports 80 and 443 are open
on the firewall.

◼ Locate the Engine ID

From the Upload License page, notice that the Engine ID appears in the title bar. Important
Note: Record the Engine ID that is required while registering the engine.

102
 ◼ Register the Engine

Once logged into the Index Engines support portal, you must register the engine with the
Engine ID.

To register the engine:

1. Click the New button from the Engines tab. Resulting in the Edit Engine dialog box
opening.
2. Enter the following details in the Edit Engine dialog box:
a. Project - The project key must be added.
b. Hostname - The hostname that is configured must be added.
c. IP Address - The IP address of the host must be added. The IP address
should not be the network IP address of the Docker container.
d. Engine ID - The recorded Engine ID must be added.
3. Once the Engine details are added, click Save.
◼ Activate the License

Once the engine is registered, click the Licenses tab from the support portal. The license is
listed with a Pending status. The pending status changes once the license is activated
completely or partially. Dell EMC or Index Engines support assists to activate the license.
The support helps you decide whether to activate the license all at once or split the license
and activate only a portion of it to start using the system.

The split feature is only visible on pending licenses. This feature creates a child license. For
example, if you are splitting a system license, then it allows you to create a system license
that can be applied to a different engine.

Index Engines sends an email with the license attached as a .txt file. To apply and activate
the license:

1. From the Upload License page, click Browse to locate the license file that Index
Engines sent over email.
a. The file name of the license includes the Engine ID to which the license
applies. Be sure that it matches the Engine ID shown on the Upload License
page of the system. If the Engine ID is not matching, then contact the Index
Engines support team.
2. Click Open to select the license file.
3. On the Upload License page, click Upload File option which uploads the file, and the
license details are displayed.

103
 ➢ AWS Requirements

The following chart lists the general AWS requirements and where to find the information to
set up AWS for CR.

- Create an AWS account

To deploy CR to AWS, you must have an AWS account. To set up an account, go to Getting
Started with AWS. Note: Ensure that the AWS account includes the AWS user with privileges
to create resources.

- Identify and access management

AWS recommends that you create an identity and access management (IAM) user or role
for authenticating with AWS. Never use root credentials to deploy a Cloud Formation
template. The IAM user must be allowed to perform AWS Cloud Formation actions. The
following links provide more information about AWS best practices:

- Security and operational best practices

Amazon recommends that you enable AWS CloudTrail logs to enable governance,
compliance, and operational and risk auditing of your AWS account. AWS CloudTrail
enables you to do the following:

• View the event history of your AWS account activity, including AWS Management
Console actions, AWS SDKs, CLI, and other AWS services.
• Identify the initiator of actions, resources that are involved, and event timing. This
event history helps to simplify security analysis, resource change tracking, and
troubleshooting.

➢ Cloud Formation Template and AMI

◼ Cloud Formation Template

The Cloud Formation template declares the AWS resources that make up a stack. The
template is a text file that you can edit in any text editor.

Send a request for access to the CR Cloud Formation template to

CyberRecoveryCloudRequest@Dell.com and provide the following information:

• Customer name
• Sales order number
• AWS region in which you want to deploy the CR solution. This information is required
to ensure access to the correct AMI and Cloud Formation template.
• Your AWS account ID, which is required for access to the CR AMI.
• Your AWS User Canonical ID, which is used for access to the Cloud Formation
template.

104
 ◼ AMI - Amazon Machine Image

The AMI is a packaged environment that contains the CR configuration and other
components that are required to set up an instance. As part of the stack deployment, the
following AMIs are deployed:

• CR management host (SUSE Linux Enterprise Server 12)

• PP DDVE (DDVE DDOS 7.4.0.5)
• Jump host CIS Microsoft Windows Server 2019 Benchmark - Level 2.
➢ Preparing Your Environment

Before you deploy the CR solution on AWS, ensure that you meet the prerequisites and
prepare your environment. Ensure that you have reviewed General requirements for
deploying the CR solution on AWS.

◼ Steps for Preparing Your Environment

1. Accept DDVE and CIS jump host terms and complete the subscriptions.
a. Go to AWS Marketplace: Dell EMC PP DDVE and accept DDVE terms.
b. Go to AWS Marketplace: CIS Windows Server 2019 Benchmark - Level 2
and accept CIS jump host terms.
2. Before you deploy the CR solution on AWS, create Amazon EC2 key pairs. The key
pairs provide secure logins in the AWS VPC to access the:
a. Jump host
b. CR management host
c. DDVE

Note: Dell Technologies recommends that you create three key pairs, one for each instance.
When the EC2 instances are deployed, use the key pairs to access the instances. Ensure
that you have access to the key pairs.

3. Next, create Amazon EC2 key pairs.

a. VPC in Classless Inter-Domain Routing (CIDR) format

The minimum CIDR block range for a CR AWS VPC is /27, which provides 32 total IP
addresses. This minimum allows each of the subnets that are created to meet the AWS
minimum CIDR range of /28, which provides 16 IP addresses per subnet.

b. Subnets in CIDR format

Subnets must be within the VPC range. The minimum subnet is a /28 subnet, which
provides 16 addresses. The maximum is /16, which provides 65,534 addresses.

c. IP address of the production PPDD systems

d. IP address of the production workstation that connects to the jump host in
AWS.
4. If there are five VPCs in the region in which you want to deploy the CR software,
remove a VPC or request an increase from AWS. The limit is 5 per region.
5. Ensure that you have access to the CR and AMI.

105
 ➢ Deploy Cyber Recovery to AWS

Use the AWS Cloud Formation template to deploy the CR solution to an EC2 instance in an
AWS VPC. The Cloud Formation template also deploys the DDVE appliance, a Window-
based jump host. The jump host enables connection to the CR management host and to the
DDVE appliance.

◼ Deploying the CR Software to AWS

Use the Cloud Formation template to create a stack, which then silently installs the CR
software. Prerequisites:

• All the prerequisites are met for General AWS requirements.

• The required key pairs are created.
• The DDVE and CIS jump host terms at the following AWS sites are accepted and
the subscriptions are completed:
➔ AWS Marketplace: Dell EMC PowerProtect DD Virtual Edition (DDVE) at:
https://aws.amazon.com/marketplace/pp/ prodview-2x2p43yvgswtm

➔ AWS Marketplace: CIS Windows Server 2019 Benchmark - Level 2 at:

https://aws.amazon.com/ marketplace/pp/B07XY9KLJJ

The following procedure lists steps for deploying CR to AWS.

1. Log in to the AWS management console and go to the Cloud Formation service.
2. Click Stacks > Create stack. Then click With new resources (standard). The Create
stack pane opens.
3. Under Prepare template, enable the Template is ready option.
4. Under Specify template > Template source, do the following:
a. If the Cloud Formation template has been shared with you over S3, enable
Amazon S3 URL. Then, enter the S3 URL for Cloud Formation template and
click Next.
b. If the Cloud Formation template has not been shared with you over S3,
enable Upload a template file. Then, click Choose file and click Next.
The Specify stack details pane is displayed.
5. Enter a stack name, complete the parameter fields that are described in the following
table, and then click Next. The table below shows the parameters fields for creating
a stack.

106
 6. Provide tags. Tags typically include key value pairs such as Name: CR-AWS.

The private key is the key pair that you specified during the Cloud Formation deployment.

7. Leave the Permissions and Advanced Options at the default values and then click
Next.
8. Review your input and click the checkbox to provide an acknowledgement.
9. When you are satisfied with your input, click Create a Stack.
a. You can monitor the progress of the stack creation, which takes a few
minutes.
b. After the stack is created, view details in Cloud Formation by clicking the
Resources, Events, and Output tabs.
10. Go to the created resources in the AWS UI to validate that they were created
properly.
11. Connect your site-to-site VPN gateway to the VPC private subnets.
12. When the VPN is in place, connect to the jump host by using the Remote Desktop
Connection.
13. Obtain the Windows password from the AWS UI:
a. Select the jump host instance.
b. Click Actions > Security > Get Windows Password.
c. Follow the prompt to upload the key provided during the Cloud Formation
deployment. This sub step decrypts the password.
d. Copy the decrypted password to the jump host.
14. Connect to the private IP address that is assigned to the jump host instance using
the username administrator and the password that you copied in the previous step.
a. You can find the IP address in the Output tab in the Cloud Formation UI or in
the EC2 UI for that instance.
15. Copy the private key for the CR host to the jump host:
a. In a text editor, open the key pair on your production workstation.
b. Copy the entire contents in the text editor.
c. On the jump host, create a file and paste the contents from the text editor.
d. Save the file using the same file name and extension as the key pair on the
production host.

The private key is the key pair that you specified during the Cloud Formation deployment.

107
 ➢ Deploy Additional DDVE Appliances in the Vault

One production PPDD system can replicate data to multiple PP DDVE appliances in the CR
vault. After the CR solution is deployed on AWS, you can add additional DDVE appliances
to the CR vault.

• Follow instructions for deploying a DDVE on AWS in the PP DDVE on AWS

Installation and Administration Guide at Dell Online Support.
• Configure the newly added DDVE so that it is on the same subnetwork as the existing
DDVE.
• Configure the newly added DDVE so that it uses the same security group as the
existing DDVE.
• Create an S3 bucket as described in the Guide.
◼ Perform the Following Steps to Ensure Connectivity Between the Jump Host
and an Additional DDVE in the CRV.
1. In the AWS Console, go to Services > VPC.
2. Under SECURITY on the left side menu, click Network ACLs.
3. Under Network ACLs in the main window, click _PPCR Jump Host Subnet ACL.

The default value for the Allow/Deny field is Allow. This field indicates that the port range
from the source IP address, which is the newly added DDVE, is allowed.

4. Edit the jump host ACL:

a. Click the Inbound rules tab, and then click Edit Inbound Rules.
b. In the Edit Inbound Rules window, click Add new rule.
c. Add a rule that includes the ephemeral range 1024-65535 for the destination
IP address of the newly added DDVE, and then click Save changes.
d. Click the Outbound rules tab and then click Edit Outbound Rules.
e. In the Edit Outbound Rules window, click Add new rule.
f. Add a rule that includes the ephemeral range 1024-65535 for the destination
IP address of the newly added DDVE.
g. Click Add new rule again.
h. Add a rule that includes HTTPS port 443 for the destination IP address of the
newly added DDVE, and then click Save changes.
5. Under SECURITY on the left side menu, click Security Groups.
6. Under Security Groups in the main window, click _PPCR Mgmt Host SG.
7. Edit the management host security group:
a. Click the Outbound rules tab, and then click Edit Outbound Rules.
b. In the Edit Outbound Rules window, click Add rule.
c. Add the following four rules for the destination IP address of the new DDVE:
i. Add SSH port 22.
ii. Add Custom TCP port 2052.
iii. Add Custom TCP port 2049.
iv. Add Custom TCP port 111.
d. Click Save rules.

108
 8. Under VIRTUAL PRIVATE CLOUD on the left side menu, click Endpoints.
9. Select the endpoint that corresponds to the S3 Gateway endpoint that was created
during the initial Cloud Formation deployment.

The endpoint type is displayed as Gateway.

10. Click the Policy tab, and then click Edit Policy.
11. In the Edit Policy window, under the Resource section of the policy, add the Amazon
Resource Name (ARN) for the S3 bucket that was created for the newly added
DDVE.
a. “arn:aws:s3:::secondary-dds3bucket”,
“arn:aws:s3:::secondary-dds3bucket/*”
12. Click Save.
a. You can now access the CR jump host on AWS and connect to the newly
added DDVE.
13. Return to the PP DDVE on AWS Installation and Administration Guide instructions
to create a file system on S3 object storage on the newly added DDVE.
➢ Reset Passwords

After the CR software is deployed on AWS, reset the CR passwords. Use SSH to log in to
the jump host to access the CR management host.

Before you log in to the CR management host, wait approximately 10 minutes to ensure that
the installation is completed.

The CR solution for AWS does not support password-based SSH access to the CR
management host. Instead, use the private key from the key pair that is assigned to the
instance when the CR software was deployed.

◼ Perform the Following Steps to Reset the Cyber Recovery Passwords

1. From the jump host, open a command prompt with admin privileges.
2. Use SSH to access the CR host.
a. ssh -i <path-to-key-file-above> ec2-user@<ip-of-cr-instance>
3. Reset the CR passwords and follow the prompts:
a. sudo /opt/dellemc/cr/bin/crsetup.sh --reset
4. Follow the displayed directions to access the CR URL from a supported browser.
The login page for the CR UI is displayed.
5. Reset the CR management host root user password:
a. sudo passwd root

109
 ➢ Install Browser on Vault Jump Host

The jump host installs on AWS as part of the CR deployment. The jump host comes with the
Internet Explorer browser installed. Optionally, you can manually install a different browser,
such as Google Chrome, Microsoft Edge, or Firefox, for the CR deployment on AWS.

Prerequisites:

• The CR solution is successfully installed on AWS.

• A stand-alone or offline browser installer executable file is downloaded to your
workstation.

Do not download the default installer. During installation, most web browsers access the
latest packages over the Internet. You cannot perform this action from the CRV jump host.

◼ Perform the following steps to install the browser on the jump host
1. Use Remote Desktop Protocol (RDP) to log in to the jump host on AWS.
2. Open a PowerShell session with admin privileges.
3. If necessary, copy the CR key pair to the jump host. If you did not delete the key pair
that was used for the CR solution deployment, ignore this step. Otherwise, copy the
key pair to the jump host again.
4. Use SCP to copy the aws-cis-regedit registry edit program to a local directory:

scp -i ec2-user@: /home/ec2-user/aws_cr/aws-cis-regedit.exe C:/Users/Administrator/Desktop

5. From the PowerShell session, enable file transfers over RDP:

a. Run .\aws-cis-regedit.exe enableFileTransfer
b. Press Enter to confirm the command and acknowledge the reboot. The jump
host reboots automatically.
6. After the reboot, use RDP to log in to the jump host.
7. Copy and paste the stand-alone browser executable form the local workstation to
the jump host.
8. Install the browser executable.
9. Verify that the browser is operational.
10. Open a PowerShell session with admin privileges.
11. From the PowerShell session, disable file transfers over RDP:
a. Run .\aws-cis-regedit.exe disableFileTransfer
b. Press Enter to confirm the command and acknowledge the reboot. The jump
host reboots automatically.
12. After the reboot, use RDP to log in to the jump host.
13. Delete the aws-cis-regedit registry edit program.
14. Optionally, delete the key pair if they are no longer required. As best practice, it is
recommended that you delete key pairs when you no longer need them.
15. Empty the Recycle Bin.

110
 ➢ Log in to Cyber Recovery

The CR installation procedure adds the crso user to the database. This user has the Security
Officer role and must perform the initial login and then create one or more admin users.

Use the following steps for logging into CR:

• Open a supported browser and go to https://<CR>:14777, where <CR> is the

hostname of the management host where the CR software is installed.
• In the Username field, enter crso.
• In the Password field, enter the Security Officer (crso) password that was created in
the installation procedure and click Log In.
➢ Configure DDVE and Storage

To configure PP DDVE and storage, use the bucket that was created during the deployment.

You can find the bucker in the AWS UI. Under Services, go to Cloud Formation > Stacks >
<your_stack>. Click the Outputs tab to list the outputs, which include the S3 bucket.

➢ Configure Disaster Recovery

Configure the CR software to back up critical server data on a periodic basis automatically.
This procedure protects your CR instance from catastrophic data loss.

➢ Assets

Assets in the CRV are represented as storage, application, and vCenter server objects.

- Storage Objects

Storage objects represent storage systems, such as PPDD systems. The CR software uses
the PPDD system to perform replications, store PIT copies, and apply retention locking.

A storage object is defined for each PPDD system that is running in the CRV. A PPDD
system in the CRV serves as the repository for the data that is replicated from the production
system and protected by the CR solution.

- Application Objects

Application objects represent applications such as PPDM, Avamar, NetWorker, or the

CyberSense feature. When an application is installed in the CRV, represent the application
to the CR software. PPDM is included when the PPDD system is integrated with those
applications in your production systems.

The CRV does not require these applications to protect the data because MTree replications
copy all the data to the CRV. However, running the applications in the CRV enables you to
analyze, recovery, and restore your data so that it can be used to rehydrate production
backup applications, if necessary.

111
 - vCenter Server Objects

If you plan to use PPDM to perform a recovery in the CRV a vCenter server asset is required.
Otherwise, a PPDM recovery fails. When you install a vCenter system in the CRV, you must
represent it to the CR software.

➢ Add Assets
- Add Storage Objects

From the Main Menu, select Infrastructure > Assets.

1. Click VAULT STORAGE at the top of the Assets content pane.

2. To add a storage object, click ADD.
3. Complete the storage configuration.
4. Click SAVE. The VAULT STORAGE table lists the storage object.
5. Click in the row for the storage object to view more detailed information that is
retrieved from the PPDD system, such as the replication contexts and the Ethernet
interface.
- Add Applications

From the Main Menu, select Infrastructure > Assets.

1. Click APPLICATIONS at the top of the Asset in the context pane.

2. To add an application, click ADD.
3. Complete the application configuration parameters.
4. Click Save. The APPLICATION table lists the storage object.
5. Click in the row for the application to view more detailed information.
- Add vCenter Servers

From the Main Menu, select Infrastructure > Assets.

1. Click VCENTERS at the top of the Assets content pane.

2. To add an vCenter, click ADD.
3. Complete the vCenter configuration parameters
4. Click SAVE. The VCENTERS table lists the application.
5. Click in the row for the vCenter to view more detailed information.

112
◼ Parameters mentioned in the third points of the Adding Assets.

113
 ➢ Prepare to Upgrade Cyber Recovery

Listed are the CR upgrade prerequisites:

• Run the crsetup.sh --save command to back up data. Save the backup copy outside
of the CR server to an external network location.
• Ensure that all CR users are logged out.
• Ensure that there are no jobs running.
• Ensure that there are no scheduled jobs about to start.
• As an extra level of protection, take a VMware level snapshot.
• To use PPDM for backup and recovery with CR Version 19.3 or later, upgrade to
PPDM Version 19.3 and 19.4. Otherwise, you cannot use PPDM with the CR
software.

Upgrades have no effect on existing assets, policies, and other CR objects. If the CR
software is installed using the CR virtual appliance file, follow the upgrade procedure that
uses the crsetup.sh setup script to upgrade the CR software.

➢ Upgrade Pre-Check

A precheck option is added for upgrading CR software. The command to perform the
precheck is as follows:

# crsetup.sh --upgcheck

The precheck includes three checks. The first is compatibility for upgrades from the current
version. The CR registry service must have a running status. The CR operating system size
requirements are checked. A minimum of 2 GB are required for upgrade.

➢ Upgrade Paths

Follow these paths when upgrading the software:

- CR Version 18.1.0-529 or 18.1.0-532

Upgrade first to 18.1.14, then to 19.1.0.9, and finally to the latest CR version.

- A version earlier than 18.1.1.7 (other than 18.1.0-529)

Upgrade to 18.1.1.7, then to 19.1.0.9, and finally to the latest CR version.

- CR Version 18.1.1.7

Upgrade to 19.1.0.9 and then to the latest CR version.

- CR Version 19.x or later

Upgrade directly to CR Version 19.8.

If the current environment includes a pre-version 19.8 CR virtual appliance deployment, it is

recommended that the CR virtual appliance security patches are applied. Apply the patches
before or after upgrading to CR Version 19.8 or later.

114
 ➢ Upgrade Cyber Recovery
◼ Before you begin
1. Ensure that you satisfy all system requirements.
2. Verify that the prerequisite software is installed.

# ./crsetup.sh --check

3. Run a preupgrade check to ensure readiness for the software upgrade.

# ./crsetup.sh --upgcheck

4. Ensure that you have saved a data backup copy outside of the CR server.
◼ Part 1

Upgrades have no effect on existing assets, policies, and other CR objects.

1. Log in to the management host as root.

2. Download the CR upgrade package to a directory with approximately 1.5 GB of free
space.
3. Untar the file. The file is untarred to the staging directory (within the current directory).
The extraction includes the crsetup.sh setup script.

# tar -xzvf <filename>

4. Go to the staging directory and make the crsetup.sh setup script an executable file.

# cd staging
# chmod +x ./crsetup.sh

5. Begin the upgrade

# ./crsetup.sh --upgrade

◼ Part 2
6. At the prompt, indicate that you want to continue the upgrade.
7. For a CR software upgrade only, if you are upgrading from Version 19.1 to Version
19.3, and the upgrade procedure displays a warning about creating the cyber-
recovery-admin user, indicate if you want to continue or cancel the upgrade.
a. If you complete the upgrade, the CR software operates correctly, however, a
non-cyber-recovery-admin user might own some installation directories.
8. When prompted, enter the MongoDB password.
9. When prompted, enter the lockbox passphrase.
a. The upgrade proceeds and starts the CR system.
10. If you are upgrading a pre-version 19.8 CR virtual appliance, upgrade the security
patches and base operating system components.

115
 ◼ Migrate Data

Migrate data from a CR software deployment to a CR virtual appliance deployment.

1. After an upgrade operation, on the host that is running the CR software, run the
crsetup.sh --save command to create a backup copy.
2. Copy the newly created backup file on to the CR virtual appliance.
3. On the CR virtual appliance, run the crsetup.sh --recover command to perform a
recovery and restore the data.
➢ Use Cyber Recovery to Apply Patch in CR Vault Environments

With CR, it is unnecessary to take a laptop or external storage into the physical CRV. To
upgrade vault components, move patch software from your production system into the CRV
securely.

Software patches can be applied to upgrade the CR management host and PP systems.
Applications such as the NetWorker, Avamar, PPDM, CyberSense, among others can also
be patched.

On the production PPDD system, create a dedicated MTree. On the production and CRV
PPDD systems, create and initialize a PPDD replication. On the CR system, create a CR
policy and select the replication context that is associated with the patch software.

◼ Procedure
1. Place the patch software on the host.
2. On the production PPDD system, export the dedicated MTree to a host.
3. NFS mount the production MTree to the host.
4. Download the patch software to the NFS location from the host.
5. Perform a checksum and run a scanner to ensure that the downloaded patch
software is uncorrupted.
6. Optionally, test the software upgrade on a test system.
7. On the CR system, perform a Sync Copy operation to replicate the MTree on which
the patch software resides.
8. After the Sync Copy job completes, create a CR sandbox of the copy. Export the
copy to the host on which you want to access the patch software.
9. Optionally, run a scanner to ensure that the downloaded copy of the software patch
is uncorrupted. Perform an analysis by using the CyberSense feature.
10. Apply the patch software.

116
 ➢ Apply Cyber Recovery Virtual Appliance Patches

CR virtual appliance can also be patched. Before you apply the security patches, back up
data and take a VM snapshot of the CR virtual appliance. Then, save the backup data and
snapshot outside of the CR virtual appliance.

This procedure requires a reboot of the virtual appliance.

1. Go to Dell Technologies Online Support to obtain the cyber-recoveryosupdate-

<current-release>.bin file.
2. Take a VM snapshot of the CR virtual appliance.
3. Run the crsetup.sh --save command to create a backup copy.
4. Save the backup data and snapshot outside of the CR virtual appliance.
5. Run the following command:

# ./cyber-recovery-osupdate-<current-release>.bin

6. Reboot the CR virtual appliance.

117
 o PowerProtect Cyber Recovery Integration
➢ Solution Overview

PPCR protects and isolates critical data from ransomware and other sophisticated threats.
Machine learning identifies suspicious activity and allows you to recovery known good data
and resume normal business operations with confidence.

The solution maintains mission-critical business data and technology configurations in a

secure, air-gapped ‘vault’ environment that can be used for recovery or analysis. The CRV
is physically isolated from an unsecure system or network.

The CR solution enables access to the CRV only long enough to replicate data from the
production system. At all other times, the CRV is secured and off the network. A
deduplication process is performed in the production environment. Deduplication expedites
the replication process so that connection time to the CRV is as short as possible.

Within the CRV, the CR software creates PIT retention-locked copies. The copies can be
validated and then used for recovery of the production system. Policies and retention locks
make part of the CR solution.

➢ Configure External Email Server

After an SMTP email server is configure din the CR UI, enable the option to use an external
email service to route and deliver CR email notifications to CR users. If this option is not
enabled, by default, the CR software uses Postfix as the default email service.

◼ Procedure
1. From the Masthead Navigation, click the gear icon to access the System Settings
list.
2. Click Mail Server Settings.
3. In the Mail Server Settings dialog box, enable this option. The dialog box displays
configuration fields.
4. Enter or modify the values in the following fields:

Mail Server: Specify the CR email server.

Port: Specify a port number. The default port number is 25.

Sender’s Email Address: Specify the email address that delivers CR alert messages. The
default value is noreply@cyberrecovery.

Authentication: Specify the password for the email address.

Username: Optionally, specify the username that is associated with the CR email server.

Password: Optionally, specify the password that is associated with the CR email server.

5. Click Save.

118
 ➢ Authentication to External Systems
◼ Configure Remote Connection

Configure the CR deployment with remote systems to manage and audit the data flow in the
CRV. With the CR UI, CLI, and REST API, Admin users can define these supported CRV
assets. Assets are defined so that they are represented in the CR environment.

The Storage asset is used to define the storage systems, which are PPDD systems. The
Application asset is used to define the following:

• Applications that are installed in the CRV environment, such as the Avamar,
NetWorker, and PPDM applications.
• Recovery host to which the backup and application and data re recovered by using
applications that are installed in the CRV environment.
• Validation host on which scanning, and validation is performed by using software
that is installed in the CRV environment, such as CyberSense feature software.
◼ Add a Supported Component

During CR remote configuration with other systems in the CRV, information such as
credentials for the remote hosts is required to perform successful authentication and
configuration. By using the CR UI, CLI, or REST API, you can make changes.

1. Log in to the CR UI.

2. From the Main Menu, click Assets.
3. On the Assets page, click Applications.
4. Click Add.
5. In the Add Vault Application window, enter information about the system.
◼ Remove a Supported Component

Remove an online and available asset from the Assets page in the CR UI. To remove an
offline and unavailable asset, use the CR CLI. Follow this procedure to remove an added
component in the CRV by using the CLI:

1. Log in to the CR CLI:

crcli login --username <AdminUser>

2. To view the list of assets, type the following commands.

crcli dd
crcli apps list

3. Obtain the asset’s nickname from the second column in the output.
4. Type either one of the following commands:

crcli dd delete --nickname

crcli apps delete --nickname

119
 ➢ Cyber Recovery REST API

An application programming interface (API) provides a means of communicating with an

application without understanding its underlying architecture. APIs allow programmers to
create channels of communication between applications and hardware, as well as pool
resources from multiple sources such as databases and news feeds. APIs can be
precompiled code to be leveraged in programming languages and can also be web-based.

◼ Cyber Recovery API

The CR REST API provides a predefined set of operations that administer and manage
tasks over HTTPS. Use the REST API to create a custom client application or to integrate
CR functionality into an existing application.

◼ Cyber Recovery Ports

The following ports are used for CR communication through UI, API, and documentation.

- Port 14777 for CR UI

- Port 14778 for CR REST API
- Port 14779 for CR Registry
- Port 14780 for CR API Documentation
◼ API Capabilities

The CR API provides functionality for all the UI operations.

- Authentication

Log in with username and password to perform all the operations.

- Users

Retrieve, create, find, update, and change user settings.

- Storage

Get storage endpoints or create storage endpoints. Get storage by id, update the
parameters, replace the source, or delete. Get and configure storage configuration and
settings.

- Applications

Get, create, update, and delete application and application settings.

- Policies

Get, create, update, and delete policies.

- Vault

Get lock state, secure, and release the lock, and get and update vault settings.

120
 - Actions

Perform sync, sync-copy, securecopy, copy, copy-lock, lock, analyze, recover, recover
check, and create sandboxes.

- Schedules

Get schedules, create schedules, update, and delete schedules, and get and update
settings.

- Notifications

Perform email tests, retrieve alerts, acknowledge alerts, get events. Get notification settings,
email settings, and patch the email server.

- Systems

Get the dashboard data, license information, set licenses, get, and perform operations,
create log bundles, and get system settings.

- vCenters

Get, update, and delete vCenters. Get and update vCenter settings.

- Versions

Get REST API and CR version information.

121
 o PowerProtect Cyber Recovery Design
➢ Business
◼ State of Cyber Security

Across industries and among organizations of every size, cyberattacks are on the rise. Cyber
Security Ventures estimates that every 11 seconds a cyber or ransomware attack occurs.
Attacks are nonstop. The cost per attack continues to increase, with Accenture estimating
that $13 million is the average cost to organizations resulting from cybercrime. Organizations
become increasingly aware of the cybersecurity risks that threaten their mission-critical
operations and their reputation. IT security has become an essential part of enterprise digital
strategy.

Protecting an organization starts with protecting the data against ransomware and other
sophisticated cyber threats. Yet, cyber threats are becoming more sophisticated. These
threats present many opportunities for criminals using modern tools and tactics to use critical
data for various purposes. Among them to destroy and ransom it for some benefit.
Furthermore, 64 percent of organizations are concerned that they experience a disruptive
event in the next twelve months.

◼ White House Brief

In June 2021, the White House of the United States sent a memorandum to corporate
executives and business leaders. The memo stated strengthening the resilience from cyber-
attacks was a top priority to the president. Cyber-attacks became a high priority to the
government due to the significant increase in ransomware incidents.

The memo invites business leaders and executives to take ransomware crime seriously and
ensure the corporate cyber defenses match the threat. The memo reinforces that companies
should view ransomware as a thread to their core business operations. The US Government
recommends the following measures to protect from cyberattacks. Implement the five best
practices from the Executive Order of the President.

• Back up data, system images, and configurations, regularly test them, and keep the
backups offline.
• Update and patch systems promptly.
• Test the incident response plan.
• Check the work of the security team.
• Segment the networks.
◼ Cyberattack Prevention

The modern threat of cyberattacks and the importance of maintaining the confidentiality,
availability, and integrity of data require modern solutions and strategies to protect vital data
and systems. Having a cyber resiliency strategy is becoming a mandate for all organizations
and government leaders. This strategy can be seen as a competitive advantage in the data-
driven world of today.

122
 ◼ Data Recovery

PPCR solutions and services from Dell Technologies provide the highest levels of
protection, integrity, and confidentiality for the most valuable data and critical business
systems. The solutions and services are a critical component of a comprehensive Cyber
Resiliency strategy. This assurance that you can quickly recover your most critical data and
systems after a cyber or other disruptive event is a critical step in resuming normal business
operations. A modern and powerful cyber resiliency strategy and Dell EMC Data Protection
are key to enabling customers to increase business agility, accelerate time-to-market,
improve their cloud economics, and reduce business risk.

➢ Architecture

As shown in the following diagram, the CR solution uses PPDD systems to replicate data
from the production system to the CRV. Replication is done through a dedicated replication
data link.

1 : Production Environment

In the production environment, applications such as Avamar, NetWorker, and PPDM

applications manage backup operations. The backup operations store the backup data in
MTrees on PPDD systems. The production PPDD system is configured to replicate data to
a corresponding PPDD system in the CRV.

2 : Vault Environment

The CRV is a customer-provided secure location of the PPDD MTree replication destination.
It requires dedicated resources including a network, and though not required but
recommended, a name server such as DNS and a clock source. The CRV can be at another
location.

The CRV environment includes the CR management host, which runs the CR software and
a PPDD system. If required for application recoveries, the CRV can also include NetWorker,
Avamar, PPDM, and other applications.

If the CyberSense feature is installed and licensed, you can validate and analyze your data.

123
The CR software enables and disables the replication Ethernet interface and the replication
context on the PPDD system in the CRV. This operation controls the flow of data from the
production environment to the vault environment. For short, periods of time, the CRV is
connected to the production system over this dedicated interface to perform replications.
Because the management interface is always enabled, other CR operations are performed
while the CRV is secured.

➢ AWS Cloud Formation Template

The AWS Cloud Formation Template creates the CR Virtual Private cloud. The image and
description below gives an explanation of the various components that make the AWS
architecture for PP CRV on AWS.

1. The VPC includes all the components that are required for the CR solution.
2. The CR management host and DDVE are on subnet 2.
3. AWS jump host - The Windows-based jump host is available in the VPC to access
the CR and DDVE instances. The management path is through the jump host.
4. Network access control lists (ACLs). The ACLs provide a layer of security for the
VPC that act as a virtual firewall for controlling traffic in and out of the subnets.
5. A security group for each instance - The security group protects the instance by
acting as a virtual firewall to control inbound and outbound traffic.
6. Amazon Simple Email Service (SES) is used for one-way email from the CR
management host.

124
 ➢ Terminology

The following definitions are some of the terms that are used with PPCR.

- Air-Gapped

Physically isolated from an unsecure system or network.

- PPCR Policy

Combination of objects (such as PPDD systems and applications) and jobs (such as
synchronization, copy, and lock). A policy, which can be scheduled, orchestrates the
workflow between the production environment and the CRV.

- PP CRV

Secure location at the site of the customer, which is the target for PPDD MTree replication.
The CRV requires at least one PPDD system and a dedicated network.

- Logical Air Gap

Physical connection but logical isolation from the network.

- Sandbox

Read/write fast copy (clone) of files and directories that are in the CRV.

- Synchronization

PPDD MTree replication between at least one PPDD system on the production network and
one PPDD system in the CRV.

➢ Dell Technologies Consulting Services

Dell Technologies offers the following Consulting Services for customers who want to deploy
this solution:

◼ Cyber Recovery Workshop

A 1-day business workshop that assists customers in understanding CR best practices and
providing recommendations for designing a customized Dell EMC PPCR solution.
Deliverables include:

• A list of cyber threats vectors, real-world examples of emerging cyberattacks, and

strategies for recovery.
• A summary of priorities and CR recommendations.

125
 ◼ Cyber Recovery Advisory

A 1-week detailed exploration of the most critical data assets of the customer. Deliverables
include:

• A list of cyber threat vectors, with real world examples of emerging cyberattacks, and
strategies for recovery.
• Recovery strategic considerations, best practices, and potential solutions.
• Prioritized recommendations for CR preparedness.
◼ Cyber Recovery Advisory and Roadmap

A 4-week engagement to design a customer-tailored strategy and solution. Deliverables

include:

◼ Cyber threat vectors, real-world examples of emerging cyberattacks, and strategies

for recovery.
◼ A rating of maturity.
◼ A tailored strategy and solution.
◼ An actionable road map for CR preparedness.
➢ Design Overview

CR supports several designs variants that are based on the required cyber resiliency. The
design that is implemented depends greatly on the environment and requirements. A base
design and its options are described.

While not required, Dell CR Advisory Services can increase confidence that the CR solution
meets your business objectives.

◼ Environments

As part of a base-level design, the following environments are configured:

• Production environment: Production data to be protected by the CR solution must

be stored on a PPDD MTree in the production environment.
• CRV environment: The CRV environment contains a PPDD system and the CR
management host that runs the CR software. Data from the production environment
enters the CRV environment through PPDD MTree replication. This environment can
also contain various recovery and analytics and indexing physical or virtual hosts
that integrate with the solution.
◼ Network Connectivity

The production and vault environment networks are not directly connected to each other,
except for a replication data link between the DD systems in the two environments. The
solution also provides for an optional dedicated link from the CR management host in the
vault environment to the production network operations center or security operations center
for events reporting.

126
 ◼ Other CR Vault Components

The CR solution frequently includes the following CRV components:

• Analytics and indexing hosts (physical or virtual) that the CR software can use to
perform data analysis. One example is an analytics host that is installed with the
CyberSense software and integrated with the CR software.
• Recovery hosts (physical or virtual) that the CR software can use to perform a
recovery. The CR software can expose sandbox data copies to any host to perform
in-vault recoveries of data. Some examples include protected data by Dell EMC
NetWorker, Avamar, DP4400 IPDA, or Dell EMC PPDM software, third-party backup
data, and file system data. After recovering a backup application within the vault
recover application data that is stored by the backup application to additional
recovery hosts in the vault.
• A Rsyslog server or Splunk Server that is installed in the vault that is used to
centralize log files for archiving and troubleshooting. The Rsyslog server can be
configured on SUSE Linux Enterprise Server, CentOS, and Red Hat Linux Enterprise
distributions.
◼ SMTP Server

In addition to these CRV components, consider including an SMTP server in the production
environment for receiving CR alerts. CR can transmit alert details through the SMTP to a
mailbox. This functionality requires one-way SMTP connectivity from the CR management
server to the SMTP server. Alerts can be received by using one-way Data Diode device.

◼ Customization

The base-level CR solution architecture consists of a pair of PPDD systems and the CR
management host. In this base-level configuration, the CR software runs on the
management host. The software enables and disables the replication Ethernet interface
along with replication contexts on the PPDD system in the CRV. This action controls the
flow of data from the production environment to the vault environment.

There are several ways of customizing the base level solution. Use a data diode from OWL
Cyber Defense Solutions for secure one-way communication. This communication occurs
from within the vault environment to the production environment for UDP Protocols such as
SMTP and SNMP alerts. Set up a Zero Trust Network in the Vault Environment using Unisys
Stealth. Install a firewall on the replication data path to ensure that only expected data traffic
can traverse the secure link into the vault. The link must connect directly to the CRV PPDD
system and not go through the CRV switch.

127
 ➢ Server Design Considerations

Server infrastructure is installed in the vault environment and is not shared with or connected
to the production environment. Keeping vault server equipment separate from the production
environment helps ensure that any ongoing issues do not propagate into the vault
environment.

The solution requirements help determine the infrastructure type to be deployed. For
example, a VMware-based hyperconverged appliance such as VxRail simplifies server
infrastructure management in the CRV. It also makes the solution more scalable whenever
you must add storage or compute for larger restores or additional analytics.

◼ Server Types

The following server types are part of the CR solution:

• CR management server.
• Application analytics server.
• Backup application recovery server.
• Application recovery server.

Additional server types can be implemented depending on the solution requirements.

◼ Server Infrastructure

The server infrastructure in the CRV can be deployed in multiple ways. CR is also available
as an application that customer can install on a VM with CentOS / RHEL. We can support
discrete physical servers, VMware ESXi (with or without VSAN), Dell EMC VxRail appliance,
and Hyper-V.

◼ Cyber Recovery Management Server

The CR management server is where the CR software is installed and from where the CR
solution is managed. The software is available as OVA or as an application.

◼ Analytics Server

The analytics server is a designated server used to check that the protected data by the CR
solution on the PPDD system in the CRV is recoverable and intact. The type of analytics
tools that are used depends on the analysis requirements of the solution. CR and the
CyberSense feature provide direct, end-to-end analytics of certain datasets using the data
stored on the PPDD system in the CRV.

CyberSense only reads the client backup data blocks that have changed since the previous
client backup copy. Other analytics techniques might require that the data is rehydrated off
the PPDD system. The data must be restored to an application recovery server before
performing analytics operations against the data.

128
 ◼ Backup Application Recovery Server

The backup application recovery server is a designated server to which the backup
application and backup application catalog are recovered. Backup applications include
NetWorker, Avamar, DP4400, or PPDM, or other applications or combination of
applications).

Backup applications other than the ones listed are supported for data protection. Recovery
is the responsibility of the customer. Other backup applications require a BRDC qualifier.
Multiple servers can be deployed, depending on the recovery requirements of the solution.
The backup application recovery server is sized so that you can recover all backup
applications that the CR solution is protecting.

If the CR solution is protecting a physical, single-node Avamar system in a production

environment, a single-node Avamar system also resides in the vault for recovery purposes.
CR does not support grids and IDPA grid models (DP8000 series appliances).

DP5300 and 5800 PPDP Series Appliances are not supported as a replication target in the
CRV. These components have been qualified for production environment replication to a
supported DD system target.

◼ Application Recovery Server

The application recovery server is a designated server to which applications are recovered.
Some applications might require that other dependent applications are recovered first. The
infrastructure within the CRV is sized to support the recovery of the largest production
application that the CR solution protects.

If an incident occurs, more than one application might have to be recovered. Choosing a
balance between available capacity (compute, memory, storage) and cost would then be
required.

➢ Network Design Considerations

The air-gapped CRV environment has both a physical and logical separation from the
production environment. The separation further reduces the attack surface of the CRV. The
base-level design for the vault network starts with the vault having its own network switching
infrastructure. No inter-vault communication is routable to any other environment. The only
connectivity between the vault and another environment is as follows:

• Replication data link between the vault-environment and production-environment

PPDD systems.
• An optional dedicated link from the CR management host in the CRV to the
production network operations center. Alternatively, the link can connect to the
security operations center for events reporting.

129
 ➢ Replication Link

The CR software manages the replication link. The replication link on the PPDD system in
the CRV uses its own unique Ethernet interface. For the replication link that connects the
production PPDD system to the PPDD system in the CRV, it is recommended to use the
fastest link speed possible, preferably 10GB/s Ethernet (GbE). The amount of data to be
stored in the CRV and the change rate of the data determine how long the replication
connection stays open.

➢ Vault Network Security Guidance

◼ Secure Network Links

To secure the network links that connect the vault environment, or any other network, it is
recommended that a firewall is installed on both the PPDD replication link and the SMTP
link. If a hyper-converged VMware appliance is installed in the CRV, the VMware NSX
Distributed Firewall is a good firewall option for reducing complexity in the vault environment
and protecting VMware-based infrastructure.

Additionally, the VMware NSX Edge firewall is a potential software-defined option for
protecting the PPDD replication link between production and vault PPDD systems at near
wire speed.

◼ Traffic Specification

The replication link between the production PPDD system and the PPDD system in the CRV
should transfer only PPDD replication traffic. For the events reporting link from the CR
management server to the production network operations center or security operations
center, only trusted outbound traffic should be permitted.

◼ VPN Tunnel

As an additional layer of security, a one-way VPN tunnel can be enabled for the events
reporting connection. This tunnel allows only secure communications to be transmitted from
the vault environment to the production environment.

The VPN can be set up to allow access by specify users only. VMware NSX Edge VPN is a
good option and supports IPsec and SSL. Dell Technologies ProDeploy Plus services can
install and implement these tools.

In lieu of using a VPN tunnel for transmitting event details to the production network or
security operations center, you can use a data diode to provide secure one-way
communications from the vault. A data diode ensures that only one-way communication is
possible, reducing the possibility of the vault environment becoming compromised.

If other network links are required between the production and vault environments, secure
those network links to the greatest extent possible by using a firewall, VPN, or data diode.

130
 ◼ Zero Trust Network

Set up a Zero Trust Network in the CRV using Unisys Stealth. Stealth is a “defense grade”
solution that uses identity-based segmentation. Network segments can be defined and
managed by using an identity management system that has high business alignment, such
as AD or LDAP.

The Stealth principle is to trust no user or device and grant as little access as possible
(always based on reliable identification). This is true both inside and outside the private
network.

➢ Other Network Design Considerations

◼ NTP

When designing the network, also consider how to keep the vault environment time
synchronized. If a reliable NTP time source is not available for the PPDD system in the CRV,
the PPDD Retention Lock functionality might not function correctly.

Time-of-day clocks on Intel- and AMD-based systems are not reliable; we have observed
time skews of 24 hours or more. An NTP source should exist within the CRV. If one is not
available, with the appropriate security and access controls in place, an NTP source that is
external to the vault should be allowed access to vault components. The solution design
requirements determine the better option. More options such as GPS-based systems might
be available, depending on your environment.

◼ DNS and Active Directory

DNS and Active Directory are commonly used critical components in any data center. In this
solution, it is not recommended that production DNS or Active Directory instances extend
into the vault environment. Such extension would require connectivity between vault and
production components, which is not recommended. Instead, within the vault, separate DNS
and Active Directory instances can be instantiated for only the vault components.

Optionally, you can periodically copy production Active Directory and other foundation
services into the CRV along with the business-critical data to enable recovery of those
components.

For DNS, using host files is another, more secure option. Regardless of whether an Active
Directory instance is implemented in the vault environment or local logins are sued for vault
components, passwords must be unique.

◼ Bandwidth

In addition, consider the bandwidth that will be required to support data recovery after a
cyber-attack. Ensure that the bandwidth between the CRV and the recovery environment is
sufficient to meet the solution’s recovery time objectives. If feasible, 10 GbE links, one for
replication and two for recovery, should be available. The CR software has its own design
considerations that must be understood before a CR solution is implemented. CR supports
up to five PPDD systems in the CRV and a total of up to 25 policies over the five DD systems.

131
 ➢ Network Segmentation

The figure shows how network segmentation can be configured within the vault. The
production side of the diagram is for illustrative purposes only. The only links that connect
the CRV to the production environment are the replication network link and, optionally, the
events-reporting network link.

All connections that span both the production and vault environment should be secured by
using a firewall, VPN, or data diode. Inter-vault communication is segmented based on the
needs that are specified in the CR solution design. As an example, Unisys Stealth can
provide network segmentation in the CRV.

➢ Cloud Formation Template and AMI

Acquiring the Cloud Formation template and AMI:

◼ Cloud Formation Template

The Cloud Formation Template declares the AWS that make up a stack. The template is a
text file that you can edit in any text editor. Send a request for access to the CR Cloud
Formation template to CyberRecoveryCloudRequest@Dell.com and provide the following:

• Customer name
• Sales order number
• AWS region in which you want to deploy the CR solution. This information is required
to ensure access to the correct AMI and Cloud Formation Template.
• Your AWS account ID, which is required for access to the CR AMI.
• Your AWS User Canonical ID, which is used for access to the Cloud Formation
template.
◼ AMI

The AMI is a packaged environment that contains the CR configuration and other
components that are required to set up an instance. As part of the stack deployment, the
following AMIs are deployed:

• CR management host (SUSE Linux Enterprise Server 12)

• PP DDVE (DDVE DDOS 7.4.0.5)
• Jump host CIS Microsoft Windows Server 2019 Benchmark - Level 2

132
 ➢ Storage Design Considerations

Each CR solution implementation requires its own storage design review. The review
determines the amount of data that is to be protected in the CRV and the growth rate of the
data.

Follow the standard PPDD sizing process to determine the optimal PPDD model and
capacity point. If PPDD Retention Lock is used for storage of vault-environment data copies,
account during data-copy retention when determining the size of the PPDD system in the
CRV. The longer that unique data must be retained on the vault PPDD system, the more
capacity the system requires.

➢ Physical Environment Design

A CR solution design must provide for sufficient physical security. The vault environment of
the solution is a secure enclave, and physical security is as important as logical
segmentation. An internal bad actor can take advantage of weak physical security.

Install the CRV equipment in a dedicated room or cage with physical access controls. This
secured room should have a limited access list with key sign-out or two-person key access.
Video surveillance of entry points into the cage or room and of the equipment should be in
place.

For the utmost security, the CR software must be accessible only by physical access to the
CR management server. The server has an associated keyboard and mouse. If this option
is not feasible, monitoring the CR console through the cage (if the monitor is up and showing
messages) is another possibility. Also, with implementing a VPN, firewall, or other security
tools, you can configure a jump server within the vault environment. The server allows a
client in the production environment to securely access the CR management server.

➢ Limitations and Considerations

Each production PPDD MTree that is protected by using a CR policy requires three or more
MTrees on the CRV PPDD for the following purposes:

• One as the replication destination.

• One or more for Retention Locked copies.
• One or more for read/write sandboxes.

The CyberSense feature requires its own sandbox MTree in addition to any other sandbox
MTrees.

133
 ➢ Mechanisms for Data Protection

The CR solution uses the following additional mechanisms to further protect the data being
stored in the CRV:

• Replication traffic in and out of the vault is encrypted using PPDD encryption.
• Other data being sent to the production environment, such as CR alerts, can be
encrypted using other tools.
• The PPDD system in the CRV is disconnected (air gapped) from the production
network most of the time.
• The CR vault is set up as a separate security zone by using a VPN tunnel and a
DMZ.
• Access to the CRV is limited using least-access-privilege concept.
• Temporarily access for recovering testing is set up before testing and brought down
immediately after testing.
• The CRV functions as an enclave and can operate without production IT services.
• Power and HVAC can be common to the rest of the environment.
• The data and binaries that are stored in the CRV can be analyzed forensically and
in a nonexecutable format.
• Two-factor authentication can be implemented for access to critical vault
components.

◼ Data Protection Mechanism

The CR software controls data synchronization from the production environment to the vault
environment by using PPDD MTree replication. After the datasets and their associated
MTrees to be protected by the CR solution are determined, replication contexts are set up
between the production and vault PPDD systems. MTree replication is designed so that all
data within an MTree is replicated securely between two PPDD systems. The initial
synchronization is completed, and all data is copied to the vault PPDD. After the initial
synchronization, each subsequent synchronization operation copies only new and changed
data segments. There is no limit to the number of MTree replication contexts that the solution
supports. However, there are limits to the number of MTrees that each PPDD model
supports.

◼ Data Protection Synchronization

The CR software manages the synchronization of MTree replication contexts and the
number of data copies it creates for each replication context on the PPDD system in the
CRV. If an incident occurs, the data copies that the CR software creates are possible
recovery points. The CR software can apply a Retention Lock for all files in the MTree based
on the CR policy specifications. This condition happens when PPDD system in the CRV is
licensed with either Retention Lock governance or compliance mode.

134
Retention Lock provides data immutability and is key to the CR software operations on the
PPDD system in the CRV. Enabling Retention Lock on data copies within the vault ensures
that data copies can be trusted for recovery. The duration of the Retention Lock and the
amount of data to which the Retention Lock is applied must be carefully understood. If you
disregard the values that are used during sizing, the PPDD system in the CRV might reach
capacity more quickly than planned. The two types of Retention Locks (governance and
compliance) should be weighed against the requirements of each. Compliance is stricter
and more secure; it should be implemented.

➢ Planning and Sizing the Environment

Proper sizing of a CR solution requires gathering many details about the current
environment and determining the business-level solution requirements. Although not all-
inclusive, this section addresses some of these considerations. Dell Consulting Services
can help sizing and implementing a CR solution:

◼ Protection Objectives

Each organization implementing CR must determine the CR metrics and goals to regulate
recovery. Different metrics are used for CR than are used for traditional business continuity
and disaster recovery. Organizations must set time and recovery objectives to ensure a
predictable recovery from an event. The data protection metrics that are key to ensure that
the CR solution is properly sized are found below.

➔ Destruction Deletion Objective (DDO)

The amount of time between the point of incursion and when the incursion is detected. CR
mechanisms (including analytics) must operate within the DDO rolling window.

➔ Destruction Assessment Objective (DAO)

The amount of time that is allotted to the cybersecurity team to assess damage after an
incursion is discovered. The purpose of the assessment is to determine the amount of
destruction and if the data can be cleansed or if a fallback to a previous data copy is required.

➔ Cyber Recovery Point (CRP)

The point in time to which you can return after a destructive cyberattack. This metric is
analogous to a recovery point objective in a disaster recovery scenario. The CRP most
commonly spans from days to months, depending on the dataset that is being protected by
CR.

➔ Cyber Recovery Time (CRT)

The amount of time it takes to recover from an incident.

➔ Cyber Recovery Synchronization Interval

The frequency at which data is copied from the production environment to the CRV. This
interval is based on the established recovery point objective (RPO) for the CR solution.

135
 ➔ Cyber Recovery Data Copy Count

The number of data copies held in the CRV. The data copy count, coupled with the CR
synchronization interval, roughly translates to how far back in time data can be recovered.

◼ What to Protect

It is important to characterize the data to be protected. The CR solution can protect any data
that can be stored on a PP MTree. If a CR is to protect an entire backup application and its
backup data, the backup software must be able to store both its backup catalog (metadata)
and backup data on one or more PPDD MTrees.

For CR to support Avamar data protection, the Avamar system must store its checkpoint on
a PPDD MTree, which is an option for all Avamar VE and Avamar single-node
implementations. If the Avamar system is not configured in such a way, you cannot
reconstitute and restore Avamar protected data within the vault.

➔ Dell Consulting Services

To identify and characterize the data to be protected and to ensure that a thorough analysis
is performed, you can use optional Dell Consulting Services. Details to be determined
include:

• Mission-critical and business-critical applications that must be protected.

• Characteristics and dependencies of each application, including host platform,
location and amount of data, and CR objectives and metrics; in addition, any
dependencies on core infrastructure services (such as DNS, LDAP, and Active
Directory) that must be protected to ensure a successful recovery.
• Data, such as application binaries, boot images, and backup catalog, that must be
protected.

These details, along with the previously defined objectives, help determine the ideal size of
the PPDD system in the CRV and an estimate of the time that will be required for data
replication on an operational basis.

➔ Backup Streams

Recovery requirements and the type of data to be protected help determine the data
synchronization frequency and data retention time. For the greatest recovery flexibility,
categorize data to be protected in one of the following backup streams:

• Full-application and file-system backups, including image-level (if possible) and

application-specific data.
• Binary and executable backups, including base-level operating system distribution
and application builds.

136
 ➔ Synchronization Frequency and Retention
In the production environment, backups of applications and their data, including image level
backups, are typically performed daily. Backups are made to one or more MTrees on the
production PPDD system.
During solution sizing, the production MTrees to be protected are identified based on which
applications and critical data must be protected in the CRV.
If an MTree contains a large amount of data and the CR solution must protect only a subset
of the data, we recommend copying the desired subset to separate MTree. Dell EMC backup
software can perform this operation with limited overhead. The CR software enables you
specify on an MTree basis the data synchronization frequency and retention time.
➔ Binaries and Executables
In addition to protecting application data, we recommend that you also protect binaries and
executables to enable full reconstruction of an application if needed.
If the production environment is subject to a destructive cyber-attack that infects base-level
operating system and application components, a complete re-creation of application hosts,
beginning at the operating system level, might be necessary.
Because cyber attackers can remain dormant within operating system binaries for a long
time, the retention period for such data is typically measured in years.
◼ Data Analysis Techniques
A plan to confirm that the validity of the vault data should be crafted. This is done after the
CR objectives and metrics are determined and the data to be protected is defined. The list
of techniques in this section is not all-inclusive, but it provides an overview of the types of
analytics options that are available.
Some of the analytics techniques require third-party software and associated infrastructure
to run the software. The CR software and the CyberSense feature provide automated
analysis of backup data in native format directly off the vault PPDD system.
➔ System-level Analytics
System-level analytics focuses on analyzing that data copies are successfully created on
the PPDD system in the CRV. The goal is to ensure that the steps involved in synchronizing
the data and creating immutable (Retention Locked) copies were completed successfully.
System-level analytics provide assurance that the restore point is recoverable. This type of
analytics also identifies health issues that are related to the overall CRV infrastructure and
the CR software. System-level tools perform the required level of analysis and issue alerts
when needed.

137
 ➔ Full Content Analytics
Cyber threats are increasingly becoming more sophisticated by how they penetrate the data
center. Even with the most advanced security products deployed, organizations are still at
risk of having data that is attacked and corrupted by bad actors. CyberSense adds a last
line of defense to your existing security solutions, finding corruption that occurs when an
attack has successfully breached the data center.
➔ CyberSense
CyberSense uses data backups to observe how data changes over time and then uses
analytics to detect signs of corruption indicative of a ransomware attack. Machine learning
then examines over a 100 content-based statistics to find corruption with up to 99.5 percent
confidence. Machine learning users protect their business-critical infrastructure and content.
CyberSense detects mass deletions, encryption, and other suspicious changes in core
infrastructure
➔ Forensic Reports
When suspicious behavior occurs, CyberSense provides postattack forensic reports to
diagnose the cyberattack further. The report provides details about the statistics used with
the analytics and the attack vector that is used for the attack. With CyberSense, when data
corruption is detected, a list of the last known good backup datasets is available to support
rapid recovery and minimize business interruption .
➔ Content-based Analytics
CyberSense is the only product on the market that delivers full-content-based analytics on
all the protected data. This capability sets CyberSense apart from other solutions that take
a high-level view of the data. These solutions use analytics that look for obvious sings of
corruption that is based on metadata. Metadata-level corruption is not difficult to detect; for
instance, changing a file extension to .encrypted or radically changing the file size. These
types of attacks do not represent sophisticated attacks that cybercriminals are using today.
➔ Data Corruption Detection
CyberSense goes beyond metadata-only solutions because it is based on full-content
analytics that provide up to 99.5 percent confidence in detecting data corruption. It audits
files and databases for attacks that include content-only based corruption of the file structure
or partial encryption inside a document or page of a database. These attacks cannot be
found using analytics that do not scan inside the file to compare how it changes over time.
Without full-content-based analytics, the number of false negatives is significant, providing
a false sense of confidence in your data integrity and security.

138
 ◼ Recovery Techniques
If a destructive cyber-attack requires a recovery, a plan must be formulated that specifies
how data is recovered and what infrastructure must exist in the vault to support the recovery
operation.
➔ Restore Data and Application Requirements in the CR Vault
Data and application binaries in the CRV can be restored as follows:
1. Identify the restore points that were created before the attack occurred.
2. Using the forensic findings, identify the malware and where it has been persisted. If
binaries or operating system images have been compromised, decide whether to
cleanse the malware from the backup image. Then restore the binaries from the vault
PPDD system.
3. Apply security patches if possible.
4. Restore the data to a recovery host that is located within the CRV using the disaster
recovery runbook for the associated application. Segment the application from the
rest of the CRV infrastructure and then launch the application. Determine if the
recovery process has eliminated the effects of the offending malware.
5. Test-run production applications using the CRV compute.
6. Cleanse or reimage the production environment and connect the recovery host to
production (either logically or through physical shipment). Then copy the application
and data back to the original production servers.
➔ Restore Process Diagram
The following figure illustrates the restore process.

139
 ➔ Completely Rebuild from the CR Vault
Completely rebuilding from the CRV is more comprehensive and conservative, but it is a
slower recovery method. This method also minimizes concerns around dormant malware.
The high-level steps for a complete rebuild are as follows:
1. Reformat the production system based on the damage and forensics assessment
that was done as part of the incident response.
2. Rebuild the binaries by restoring the appropriate CRV data copies. This recovery
process is consistent with the previous scenario. Apply security patches if possible
and distribute them to freshly formatted hosts.
3. Recover the application and data to the original production environment. To do so,
locate and restore the appropriate copy, configuration files, data, and perform
application recovery using the disaster recovery runbook for the application.
➔ Rebuild Process Diagram

◼ Test the Recovery

In addition to perform the recovery, create a plan to enable authorized individuals to test the
application recovery. How those individuals carry out the test can vary. If a jump box is
configured within the vault environment, a user can log in to that server. From the server,
the user can access the vault infrastructure to recover data. If a jump box is not configured,
the user must be physically present within the vault environment to have access to the
necessary equipment.
➢ CyberSense Sizing Calculator
The CyberSense sizer allows users to view the server requirements based on their
environment. The inputs include backup analysis intervals, including the first backup.
Workload information is added, together with total number of VM clients, their average size,
and maximum size. The number of files in large VMs and or file servers are estimated too.
The output is the server requirements, and per-server requirements when there are more
than one server.

140
 ➢ Hardening the Solution
During solution implementation, ensure that all components in the vault are secured as best
as they can be. Dedicated security guidelines might be available for some products that are
installed in the vault environment. If they are available, follow the guidelines and lock down
the products as best as possible. For example, disable unused ports and nonessential
protocols, and use unique and limited-access credentials. Dell Services provides an offering
that ensures that the PPDD system is secured in accordance with best practices.
➢ Review Settings
For hardening the target DD system, be familiar with the following:

• System passphrase
• Access control settings
• Log settings
• Communication security settings
• Data security settings
• Secure serviceability settings
• Dell Secure Remote Services
• Security alert system settings
• System hardening (to comply with the DISA STIG standards)
➢ Best Practices
Follow these best practices and precautions when hardening the DDOS:

• When configuring a client list, do not use a wildcard character that enables access
for any user. Type individual IP addresses or client names.
• The PPDD system must use a FIPS 140-2 approved cryptographic hashing algorithm
for generating account password hashes.
• Enable HTTPS and disable HTTP.
• Do not enable Telnet.
• Use strong passwords.
• If the SSH client does not comply with the ciphers that are supported by default, use
the CLI to add the additional ciphers. They are added so that the SSH client can
connect to the system.
• Change the default SSH port.

141
 ◼ SSH Port Recommendations
The Admin interface enables only port 22 and port 443. If the ports change, there is no way
to change the ports of the Admin interface and they are left exposed. To avoid this exposure,
assign the Admin’s interface to a temporary interface. Then, bring that interface down so
that there are no processes listening to the old port numbers.
The default filter function for SSH is 22 and can be disabled. In SE mode, add a port number
by using these net filters add operation. Identify specific addresses that can access the new
port number and a specific interface. When you add the filter function, the common
operations, such as enable, disable, add, and delete, apply.
If the SSH port changes, the net filter blocks the port unless the auto detection option is
enabled. While this option is enabled by default, it is recommended that you disable it in a
secure environment. To do so, use the net filter autolist delete ports all command. The only
way to enable the new port without auto is through SE mode.
◼ System Passphrase
The passphrase is used to encrypt the encryption keys, cloud access, secure keys, imported
host certificate private keys, and DD Boost token keys. It enables a system to be transported
with encryption keys on the system but without the passphrase being stored on it. The
system uses the passphrase to encrypt imported host private keys and DD Boost token
keys. If the system is stolen in transit, an attacker cannot easily recover the data.
At most, they can recover the encrypted user data and the encrypted keys. Data at rest
encryption keys require this passphrase, and therefore, the use of a stronger passphrase is
mandatory. A valid passphrase must contain:

• A minimum of nine characters

• A minimum of one lowercase character
• A minimum of one uppercase character
• A minimum of one numeral
• A minimum of one special character
• No spaces

◼ Passphrase Security
DDOS supports a passphrase of up to 1024 characters. The passphrase is encrypted and
stored in a file on the head unit of the DD or PP system. The encryption key that is used to
encrypt the passphrase is hard coded. Use the following hidden sysadmin command to
choose to not store the passphrase on disk:
system passphrase option set store-on-disk no
Then, change the passphrase after running the command. A side-effect of not storing the
passphrase is that you must unlock the file system every time that you reboot the system.
Until the file system is unlocked, all backup jobs and replication are impacted. If there is no
concern that an attacker can gain physical access to the appliance in the environment, then
store the passphrase on disk.

142
 ◼ Multifactor Authentication (MFA)
The system requires additional authorization for certain commands to promote better
security and protection. Sysadmin or security-officer credentials are required to run these
commands.
When MFA is enabled, the system prompts for the MFA passcode in addition to sysadmin
or security-officer credentials for certain commands to promote better security and
protection. An MFA passcode is usually a time-based one-time password (TOTP) that
changes every 30 to 60 seconds.
Different MFA providers support different ways to generating TOTP. Common MFA
providers include RSA SecurID, Google Authenticator/Microsoft Authenticator, and Authy.
DD supports RSA SecurID as the MFA provider.
◼ Access Control Security
Use the following table to verify and record the access control security settings for the PPDD
system:
Configuration YES NO Notes
Are session timeouts configured for
SSH?

Are session timeouts configured for

HTTPS?

Have host-based access lists been

created?

Are ACLs set up for file shares?

Has the default sysadmin account

password changed?

Are accounts of the users defined Create separate Admin and Security Officer accounts to
for them? manage the DD system in the CRV. Do not share the same
accounts for production or other environments. Do not use the
sysadmin, SE, or root account to manage any DD
environment.

Is there a syslog server? Forward the syslog to the management host. Use VPN or
another secured mechanism such as a data diode or Unisys
Stealth. Push the logs from the target DD system to the
secured external management host with this mechanism.

Are logs being sent to the syslog

server?

Is there strong password policy in

place?

How is the system passphrase Who knows it? Is it stored securely? Who knows how to access
protected? it?

Is an encryption key manager in the If the target DD system has several MTrees and uses
CRV design? encryption consider using RSA Data Protection Manager
(DPM) or a similar product. These products are used in the
CRV to manage encryption keys.

143
 ➢ Perform Hardening Procedures
Perform hardening procedures on the target DD system.
◼ Administrator Access
The following connection settings are required for the administrator:

• FTP - Disabled
• FTPS - Disabled
• HTTP - Disabled
• HTTPS - Disabled (This forces CLI only access)
• SCP - Disabled
• SSH - Enabled (Allow only CR management host access)
• Telnet - Disable

◼ Password Policy
Use AD to maintain all users other than sysadmin, ddboost, and security officer. The
following recommended rule settings apply to the sysadmin, ddboost, and security officer
login IDs. The following settings are recommended to harden the password policy.

• Minimum Days between change - 0

• Maximum Days between change - 30 (or as appropriate)
• Warn Days before Expire - 7
• Disable Days after Expire - Never (or as appropriate)
• Minimum Length of Password - 16
• Minimum Number of Character Classes - 3
• Lowercase character Requirement - Enable
• Uppercase character Requirement - Enable
• One Digit Requirement - Enable
• Special Character Requirement - Enabled (recommended)
• Max Consecutive Character Requirement - Disable
• Number of Previous Passwords to Block - 12
• Maximum login attempt - 5
• Unlock timeout (seconds) - 300

144
 ◼ Account Configurations
The following configurations are recommended on user created accounts.
Username - sysadmin

• Management role - admin

• Notes - Update the password to be 20 characters long. Passwords must be 20
characters in length, containing a combination of numbers, letters (upper and lower
case), and symbols. Keep the password in a secure safe and ensure that it is
accessible only to designated personnel. Do not use this account for day-to-day
operations so you can identify who accesses the DD system.
Username - ddboost

• Management role - none

• Notes - The DD Boost account is only used for authentication.
Username - security

• Management role - security

• Notes - Required for Retention Lock compliance.
Username - Other users

• Management role - admin

• Notes - Users managing the target DD system must have their own accounts. This
account is used to add the target DD system to the CRV.

◼ Authentication Settings
The following authentication configuration is recommended.

• Activity Directory / Kerberos Authentication - Disabled (Default)

• Workgroup Authentication - Enabled (Default)
• LDAP Authentication - Disabled (Default)
• Single Sign-On - Disabled (Default)
• NIS Authentication - Disabled (Default)

◼ Mail Server Settings

The following configurations are recommended on the mail server.

• Mail Server - Complete this setting because email notification is key for detecting
issues. Configure the local mail server in the CRV. Alternatively use data diodes or
Unisys Stealth software to send email notifications outside the CRV to a secured
external management host.
• Time and Date - Provide a reliable NTP server for time synchronization.
• System Properties - Set the appropriate values.
• SNMP - Disable if not required in the CR environment.

145