Disaster Recovery

Overview

The DR features are also referred to as resilient secrets.

The Disaster Recovery (DR) feature is a tool for emergency access to critical systems in the event of an emergency, such as a network outage. DR generates updates at regular intervals from one Secret Server instance (the data source) to another, or many other, instances (the Replicas). A select set of folders and the secrets they contain will have any changes sent securely to the configured replicas so that, in the event the data source becomes inaccessible, the replicas can be quickly used for emergency access to vital systems

Resilient Secrets Coverage

Overview

The Resilient Secrets (RS) feature, which is or DR solution, is tailored to replicate data from a primary data source to a secondary data replica. The features and specific data replicated do not constitute the entirety of an instance of Secret Server. Instead, replicates the prioritized vital data and functionality needed so that, in the event of an outage of the primary data source, the most important information and functionality can be accessed on the secondary data replica to aid restoring the primary data source while keeping minimal operations running.

The following features are what RS replicates, denoting what data is and is not replicated:

Excluded Features

The following features not replicated at all:

  • Advanced LDAP Connection Settings
  • Advanced Session Recording
  • Audits
  • Auto Export
  • Backup Configuration
  • Custom Reports
  • Dev Ops Secret Vault
  • Disaster Recovery Configuration
  • Discovery
  • DoubleLock
  • Dual Controls
  • Event Pipelines
  • Event Subscriptions
  • HSM Configuration
  • Inbox Information and Rules
  • IP Address Restrictions
  • Jumbox Routes
  • Licenses
  • Node Information
  • RDP Proxy Settings
  • Remote Password Changing
  • SAML Configuration
  • Scripts
  • SDK Client Configuration
  • Secret Policies
  • Session Recordings
  • SSH Commands
  • Ticket Systems
  • Workflows

Conditionally Replicated Features

Configuration

Secret Server
Covered Features

The only global configuration settings replicated are:

  • Enable directory service integration.
  • Allow authentication against directory services.
  • Application hardening state
  • Allow quantum state encryption

The SSH cipher suite configuration is replicated.

Exceptions

Any configuration settings not listed above are not replicated.

Delinea Platform

The Platform configuration and permissions are replicated to allow the replica to communicate with Platform when needed, so Platform users can authenticate.

Folders and Secrets

Covered Features

Folders are replicated, along with extended values associated with their function:

  • Subfolders and the entire tree structure, as specified by the RS configuration.
  • Any permissions associated with the folder, including whether or not to inherit them from its parent folder.
  • For personal folders, the associated user.
  • Secret template restrictions placed on folders.

Secrets are replicated along with essential information relating to them:

  • The template, active status, name, and the folder the secret is in.

  • Several secret settings, including:

    • Multi factor authentication required.
    • Check out enabled or RPC interval.
    • Require comment.
    • SSH proxy enabled.
    • Web launcher requires incognito mode.
    • Hide launcher password
  • The secret field, item value, and whether or not it is a file attachment. If it is a file attachment, its contents are replicated as well
  • Any permissions associated with the secret, including whether to inherit them from the folder.
  • Audits are not replicated, but, for data integrity, secret item history, including the history of file attachments, is replicated.
Exceptions

These are not covered:

  • Password change, heartbeat, and custom expiration information is not replicated because we do not allow RPC to happen from a replica.
  • Secret policy information is not replicated.
  • Secret dependencies, checkout hooks, and one-time-password settings are not replicated.

Launchers and Mappings

Covered Features

All launchers are replicated, along with:

  • Their fields.
  • Mappings to secret templates.
  • Any default associated and privileged secrets.
Exceptions

Custom icons for launchers are not replicated.

Lists

All list information is replicated, including:

  • Categories.
  • Item values.
  • Team restrictions.
  • Secret item mappings.

Metadata

All metadata on secrets and folders that are configured for replication are replicated. All Metadata on users and groups will be replicated. This includes:

  • Metadata field sections.
  • Metadata fields.
  • Metadata item data.

Roles and Permissions

Roles and permissions are replicated, as are their assignments to users and groups.

Secret Templates

Covered Features

These are covered:

  • Secret templates, their settings, permissions, and secret fields are replicated.
  • Password requirements, associated rules, and their character sets are replicated.
Exceptions

These are not covered:

  • Remote password changing and expiration-related settings are not replicated. Replicas are not intended to run RPC.
  • One-time-password settings are not replicated.

Sites

Covered Features

All sites are replicated.

Exceptions

Site connector information is not replicated, and engine information is not replicated. This is because the encryption information related to the site is not replicated.

Teams

All teams are replicated, along with their group and site mappings.

Users and Groups

Covered Features

All users and groups in the system are replicated and, for licensing purposes, we recommend the RS configuration be set to having users inactive by default and only enable the ones needed for vital access. User accessibility is maintained between source and replica. Group memberships are replicated and any associated roles and permissions mapped to them will come across as well.

Any directory services on the source is replicated to the data replica, and all users and groups mapped to these services retain system accessibility, provided the data replica is on a network that can reach the service endpoint.

Exceptions

These are not covered:

  • Two-factor-authentication (2FA) configuration or details are not replicated. Users are not automatically configured for 2FA on the replica.
  • User passwords are not replicated. Local users need to have their passwords manually reset by an administrator on the replica.

Setup

Secret Server Database Preparation

Overview

We provide several SQL scripts to help you identify any data duplication you may have that could cause issues with DR. We recommend that you run all of them.

These searches will be built into future DR installations. We provide these as an interim measure till that happens.

They are:

  • Secret Template Names: Searches for duplicates
  • Secret Field Slug Names: Searches for duplicates
  • Character Set Names: Searches for duplicates
  • Password Requirements Names: Searches for duplicates
  • Domain Names: Searches for duplicates of formal domain names (those in URLs)
  • Domain Friendly Names: Searches for duplicates of human-readable domain name equivalents
  • Folder Paths: Searches for duplicates of secret folders
  • Group Names: Searches for duplicates of user group names
  • Role Names: Searches for duplicates of user role names

Procedure

Copy and paste each query into a Creating and Editing Reports and run it. Alternatively, database admins can run the scripts directly on the Secret Server On-Premises database. After running the queries, if you receive any results, that indicates duplicates to address before enabling the DR feature. Most issues can be resolved by simply renaming or removing the duplicate items.

SQL Scripts

Secret Template Names
Copy 
-- Secret Template Names
SELECT
    SecretTypeName
FROM
    tbSecretType
GROUP BY
    SecretTypeName
HAVING
    COUNT(*) > 1
Secret Field Slug Names
Copy 
-- Secret Field Slug Names
SELECT
    st.SecretTypeName,
    t.FieldSlugName
FROM
(
    SELECT
        sf.SecretTypeId,
        sf.FieldSlugName
    FROM
        tbSecretField AS sf
    JOIN
        tbSecretType AS st2
    ON 
        st2.SecretTypeID = sf.SecretTypeID
    GROUP BY
        sf.FieldSlugName, sf.SecretTypeId
    HAVING
        COUNT(*) > 1
) AS t
JOIN
    tbSecretType AS st
ON 
    st.SecretTypeID = t.SecretTypeID
Character Set Names
Copy 
-- Character Set Names
SELECT
    cs.Name
FROM
    tbCharacterSet AS cs
GROUP BY
    cs.Name
HAVING
    COUNT(*) > 1
Password Requirement Names
Copy 
-- Password Requirement Names
SELECT
    pr.Name
FROM
    tbPasswordRequirement AS pr
GROUP BY
    pr.Name
HAVING
    COUNT(*) > 1
Domain Names
Copy 
-- Domain Names
SELECT
    Domain
FROM
    tbDomain
GROUP BY
    Domain
HAVING
    COUNT(*) > 1
Domain Friendly Names
Copy 
-- Domain Friendly Names
SELECT
    FriendlyName
FROM
    tbDomain
GROUP BY
    FriendlyName
HAVING
    COUNT(*) > 1
Folder Paths
Copy 
-- Folder Paths
SELECT
    FolderPath
FROM
    tbFolder
WHERE
    UserID IS NULL
GROUP BY
    FolderPath
HAVING
    COUNT(*) > 1
Group Names
Copy 
-- Group Names
SELECT
    GroupName
FROM
    tbGroup
WHERE
    IsPersonal = 0 
    AND IsPlatform = 0 
    AND SystemGroup = 0 
    AND DomainId IS NULL
GROUP BY
    GroupName, DomainId
HAVING
    COUNT(*) > 1
Role Names
Copy 
-- Role Names
SELECT
    r.Name
FROM
    tbRole AS r
WHERE
    r.RoleType = 1 
    AND r.IsSystem = 0
GROUP BY
    r.Name
HAVING
    COUNT(*) > 1

Role Assignment

Assign the "administer disaster recovery" role permission:

Do this on both the data source and replica instances.

  1. Click Administration in the main menu. The Secrets Administration page appears.

  2. Click the Roles link in the Users, Roles, Access Management section. The Roles page appears.

  3. Select or create the role you want to configure this feature with.

  4. Click the Permissions tab.

    image-20230516151656748

  5. Click the Add or Edit button.

  6. If you do not see the role in the Add Permissions list, type Administer Disaster Recovery in the search box.

  7. Drag the Administer Disaster Recovery role from the Add Permissions list and drop it in the unlabeled permissions box. The role now has the permission.

Data Source DR Configuration

Configure the data source DR configuration:

  1. Click Administration in the main menu. The Secrets Administration page appears.

  2. Search for and click Disaster Recovery. The Disaster Recovery page appears.

  3. Click the Outgoing Configuration tab.

  4. Click the Outgoing Setup Steps button. A popup appears.

  5. Click the copy button to copy the URL from the Data Source URL and save it where you can easily access it.

  6. Click the Copy Data Source Key button to copy the key to the clipboard. The popup disappears.

  7. Copy the key and save it to the same location you used for the URL. It should look something like this:

    https://mydomain.com/Playground BgIAAACkAABSU0ExABAAAAEAAQAPeEYJLZ3u1F26EF+bBiRwokrGusSAICUpRXlIJFdm

    2/lo3Ad7dpTEl7j3rvG9+T+j8DKTsmi6Xfj

    c/J0hkoX00b6LOq4feo6mvnf6Lp8agopN2XWjLy4KU7ICG2iAoL4wgIdpgWVCHdZUcHH

    Vhe6RSkahRDOC4ctpRxb/KOI8Bbk6ftporjwZAfEfCVEOotnnm4Z8qjI0XKKbhLl2eTH

    2lDTJk53dt8z7g/Aj4nz7aYUMJX1vDOKKARpd3GiVfu5fcb0hpIqw64pjOs8trqOVxuy

    oyfSLwYPC9rtK4/JZ7Xodq93p9IIEJfSY7cup9kCHhFzU8d3RNgWj5pKKVLsPOVUreXV

    6vceGmRA3OeKY/a8/I6aEAkmkWjJAP5bmMosYVbcRR2PxI0bxXGUSKbMdKdC2Akgm3PW

    jWqezONsv0QFmJPYEGrLIxeJc2/9X+TUnx5ON/Cgeb2bHSf5CLXizjxlB67gzDZpH3q0

    mmJzm0aSDmqIc20UxFYk2YZeEzICI5lnAVPiAbaCoKuhQdhaxxaD+wdMQDCCEooKiyhW

    Yo3NFYY79k6scJh2+sn62xZNPdpxi0m6rP/F+mpELKMyElwJdFxVm+l14ksaKhIA6bDn

    xRLJ/f/SMdxfmIcTqxLr/r9k6RAiJo8DiBoWMkjk4VUCFIL4rYpUIJb/aKmx9npg==

Replica DR Configuration

  1. Click Administration in the main menu. The Secrets Administration page appears.

  2. Search for and click Disaster Recovery. The Disaster Recovery page appears.

  3. If necessary, click the Incoming Configuration tab.

  4. Click the Edit button. The page becomes editable.

    image-20230516153255099

  5. Click to select the Enabled check box.

  6. Type or paste the data source URL you saved before.

  7. Type or paste the data source key you saved before.

    image-20230516152204301

  8. Click the Save button.

  9. Refresh the Disaster Recovery page showing the Outgoing Configuration tab on the Data Source Disaster Recovery page. A new block appears for the new replica connection:

    image-20230516154003855

  10. Click the v on the block to expand it.

    image-20230516154132794

  11. Click the Approve button.

  12. Returning to the Ingoing Configuration tab, click the Test Connection button to verify everything is configured correctly.

Finishing up

  1. Once configuration is complete, return to and refresh the Outgoing Configuration tab.

  2. We recommend expanding the block and clicking the Manage button for folders.

    image-20230516154822012

    This enables you to select which folders, subfolders, and secrets get replicated. This reduces the scope of the replicated data, enabling faster replication and a smaller duplicated data footprint.

This will enable the regular interval synchronization

Replication

By design, the first time a replication from the data source to the replica occurs, all data is replicated. This is likely the slowest DR operates and can take up to 20 minutes to replicate a data source containing 100,000 secrets.

All subsequent replications are significantly smaller. Only information that has been changed since the previous replication is replicated again.

Replica Mode

When you configure a Secret Server instance as a replica, it is automatically placed in replica mode. While in replica mode, the following features are disabled

  • ConnectWise synchronization
  • Discovery
  • DoubleLock
  • Heartbeat
  • Pipeline and bulk operations on secrets
  • Remote password changing
  • Secret import
  • Secret policy changes to secrets

Replicated User Status

The "replicated user status" setting determines how the "enabled" status for users is handled during replication. The setting is used primarily when the source and replica user counts do not match—that is, new users are on the source that are not on the replica. Delinea strongly recommends both source and replica user counts match so all users and their access is replicated to the DR instance.

When these settings are changed, they impact any new users from that point forward. Existing user statuses do not change.

The possible settings are:

  • User status mirrors source (automatic): This is the default setting. Any new users in the source are replicated as enabled.
  • New users from source are disabled by default (manual): This setting is for admin special cases and must be manually set. New users are replicated as disabled.

Recommendations

We recommend placing a Secret Server replica into read-only mode. This further reduces the possibility of data changes to the instance.