Engine Management

The Delinea Platform manages and protects endpoints using small software packages called Delinea Platform Engines that handle the orchestration of Delinea services called workloads.

The platform’s Engine Management feature provides administrators with a single interface for managing these engines and workloads, which are automatically updated and maintained after installation — removing the need for separate installers and management processes traditionally necessary on individual machines.

Engine Management Components

The components of the Engine Management feature are as follows:

The Delinea Platform Engine and its workloads run on a server endpoint, but they exchange data with the Delinea Platform.

Engine Management Architecture

The diagram below illustrates customer options for running one workload per engine, or running multiple workloads on a single engine.

Server Hardware and System Requirements

The table below provides details about the hardware and other system requirements for the server where the Platform Engine is installed, as well as the related workloads for Audit Collector, Command Relay, and Privileged Access Management.

Queue - Fully Qualified Domain Names (CloudAMQP)

The following Fully Qualified Domain Names are deployed by CloudAMQP using public IP ranges of Amazon, Azure, DigitalOcean, and Google Cloud, and are used by the Platform Engine to facilitate communication with the platform by means of encrypted messages over the CloudAMQP messaging service.

Outbound firewall rules should include the following Fully Qualified Domain Names (selected by databoundary), rather than static IP ranges of these URLs, as these IP ranges can change.

Engine Security

Platform Engines retrieve workloads from the Delinea Platform, which supplies securely signed packages for the engine to download.

The Platform Engine only runs workload deployment binaries that are both signed and trusted.

During deployment execution, Platform Engines maintain a file integrity check for both the working and binary directories. Any unauthorized modifications to these directories will render the deployment invalid and trigger its recycling, which may require downloading the deployment packages again.

Platform Engines send heartbeats to the platform to fetch configuration updates using a stamp. This stamp verifies whether the engine configuration matches the machine's configuration. These heartbeats are dispatched every five minutes, ensuring prompt detection of any new updates during this interval.

Engine States

Account Permissions and Roles

The table below describes each permission available with an Engine Management domain admin account.

Network Communication

Upstream and Downstream

The Engine Management service can manage Platform Engines on millions of endpoints per tenant. To achieve high availability, the request from the engine to the server is sent only once during the engine registration. The rest of the time, communication is carried over message queues.

Upstream communication includes every message from the engine to the server. Downstream communication is from the server to the engines. Downstream doesn’t have a gRPC option.

• The Platform Engine can get a new configuration from the server passively. This means that engines that aren't active (no current workloads) can get the up-to-date configuration from the server. This reduces the load on the system.

• Platform Engines always send their heartbeats upstream to the server.

• If the server determines the engine is out of sync, it sends a single message to the groups the engine belongs to. The engine that forced that message, and all others that have the wrong group stamp, will update. This strategy reduces the load on the bus, and engines eventually coalesce. If an engine stamp itself is outdated, the server sends a message on the engine topic for the engine to update.

• The ultimate goal is to minimize the engine-server communication: outbound messages upstream from the engine to the server and inbound messages downstream from the server to the engine.

Types of messages sent:

• Engine registration (upstream, gRPC)

• Workload registration (upstream, gRPC)

• Engine heartbeats (upstream, engine to server)

• Group changes (downstream, server to applicable engines)

• Workload changes (downstream, server to applicable engines)

• Engine changes (downstream, server to specific engine)

• Engine upgrade (downstream, server to applicable engines)

• Engine uninstall (downstream, server to specific engine)

Protocols

gRPC (upstream). The gRPC protocol is used for the engine registration. After registration:

• The server sends its new configuration downstream to the engine.

• The IT Admin can request a new configuration using the gRPC call.

MQTT (upstream / downstream). Message Queuing Telemetry Transport is an OASIS standard messaging protocol for the Internet of Things (IoT). MQTT is the preferred protocol for sending upstream and downstream fire-and-forget messages. MQTT is used when the message to be sent is under the maximum payload length of 64KB. MQTT uses TLS port 443.

AMQP (upstream / downstream). Advanced Message Queuing Protocol is an open standard for passing business messages between applications or organizations. When the message payload exceeds 64KB, AMQP protocol is used as the message transfer protocol. AMQP uses TLS port 5672.

About Delinea Platform Engine Sites

A site functions as a logical divider for engines, closely resembling network demarcations.

A site doesn't restrict the workloads an engine can run, but it does influence the engine’s communication scope. For instance, a site could correspond to a data center or a main office.

• Similar term: Zone

• Definition: A logical grouping of engines based on location, most likely a network boundary.

• Examples:

  • Data center 1

  • Remote Office

  • DMZ

About the Delinea Platform Engine

The Platform Engine functions as a system daemon on an endpoint (server or workstation), to facilitate data exchange with the Delinea Platform. The Platform Engine transmits data to the platform about the endpoint where it is installed, and it receives instructions from the platform about the workloads it should execute.

• Similar terms: Node

• Function: When installed within an environment (on either a physical or virtual machine), the Windows service/Linux daemon enables the execution of authorized and validated packages from the platform’s various services.

• Installation Process: Click Add Engine from your chosen engine site, and an installation script is displayed. To install the engine, copy and run this script as an administrator. After installation, the engine is registered with the Engine Management service.

• Data Transmitted:

  • The engine assumes the responsibilities of downloading, executing, and monitoring package processes.

  • Communication between the engine and the Engine Management service includes registration, authentication, receipt of communication configuration, and relevant manifests.

• Organization:

  • Engines are grouped into sites.

  • An engine can only be assigned to a single site.

  • An engine cannot be moved to a new site.

Managing Engine Sites

Creating a Site

1. Click Settings from the left navigation menu, then click Engine Management.

2. Click Create Site.

3. Enter a Site name and Description.

4. Click Save.

Editing a Site

1. Click Settings from the left navigation menu, then click Engine Management.

2. Click the name of the site. The site page opens to the Overview tab.

3. Click Edit to update the Site name or Description.

Deleting a Site

1. Click Settings from the left navigation menu, then click Engine Management.

2. Select a site.

3. Click Delete Site.

   

Managing Platform Engines

Adding a Platform Engine

1. Click Settings from the left navigation menu, then click Engine Management.

2. Click the name of a site where you want to add an engine.

3. Click the Engines tab.

4. Click Add Engine. The Add a new engine page is displayed.

   

5. Select the OS (Windows or Linux).

6. Select the desired Capabilities.

7. Click Generate script. A device code is generated for the new engine installation script. This device code expires in 24 hours. If the engine is not installed by then, you must regenerate the script.

8. Once the script is generated, click Copy script to clipboard.

9. Access the target system.

10. Run thescript from your clipboard on the target system with elevated privileges.

The installation script for Windows automates the process of downloading and installing the engine.

The installation script for Linux automates the process of downloading and installing the engine.

After the engine is successfully installed and has established communication with the platform, the new engine appears in the engines table of your site on the platform.

Copying a Platform Engine

An engine’s capabilities can be copied from an existing engine.

1. Click Settings from the left navigation menu, then click Engine Management.

2. Click the name of a site where you want to add an engine.

3. Click the Engines tab.

4. Click Add engine.

5. Click Copy existing engine. The following dialog is displayed: Choose an existing Windows engine to copy.

   

6. Select the engine you wish to copy. The capabilities of the engine are automatically selected.

7. Click Generate script.

8. Once the script is generated, click Copy script to clipboard.

Adding Capabilities to a Platform Engine

You can add PCS and PRA capabilities to the Platform Engine using the workloads below.

• Privilege Control for Servers:

  • Audit Collector workload (Windows)

  • Command Relay workload (Windows)

• Privileged Remote Access:

  • PRA workload (Windows) (currently in Private Preview)

  • PRA workload (Linux) (currently in Private Preview)

For details, see the following:

• Engine Workloads

• Privilege Control for Servers

• Privileged Remote Access

To add capabilities to an existing engine, follow the steps below.

1. Click Settings from the left navigation menu, then click Engine Management.

2. Click the name of a site where you want to add an engine.

3. Click the Engines tab.

4. Select the engine you wish to add capabilities to.

5. Click the Capabilities tab.

   

6. Click Add Capabilities. The Add capabilities dialog appears.

   

7. Select the capability you wish to add to the engine.

8. Click Add. The new capability is displayed in the capability table.

   

Automatic Platform Engine Update

The engines consistently transmit their state to the platform, including information about active workloads and configurations. If the platform finds that the engine's state has become outdated, it sends upgrade instructions or other maintenance updates to the engine. These instructions may upgrade the engine itself or update settings or workload versions. The engine automatically upgrades itself, then restarts. The log folder should also contain a SelfUpgrade log.

Manual Platform Engine Update

Beginning in Platform Engine version 1.5.8, automatic upgrades are supported. You should run a manual upgrade only if you are facing unknown exceptions.

To manually update the engine, follow the steps below.

1. Run the PowerShell script as described in Manually Uninstall an Engine from Host Machine.

2. Navigate to Engine Management.

3. Open the existing site.

4. Wait for the Engine to disappear from the engine list.

5. Once the old version of the engine disappears, click Add Engine to install the latest version of the engine.

6. Follow the steps in Adding a Platform Engine to add the latest version of the engine.

Uninstalling an Engine from the Platform

Uninstalling the engine from the platform severs its association with the site and removes it from the site list.

1. Click Settings from the left navigation menu, then click Engine Management.

2. Select the site where you want to delete an engine.

3. Click the Engines tab.

4. Hover your cursor to the right of the engine name you want to delete

5. Click the ellipses that appears (...)

6. Click Delete engine.

7. Confirm your selection.

The platform then sends uninstall instructions to the chosen engine, which then automatically shuts itself down and uninstalls itself.

You can also delete engines using these procedures:

• Click anywhere in the row of the engine you want to delete, then click Delete engine from the preview panel that opens to the right.

• To delete engines in bulk, select the box to the left of the name of each engine you want to delete, then click Delete engine.

Manually Uninstall an Engine from Host Machine

To completely remove the engine from the Windows host machine, including all program files and installed ProgramData, execute the command below as an administrator.

cd "C:\Program Files\Delinea Engine\scripts";.\uninstall-engine-win.ps1;

Engine uninstall output:

To completely remove the engine from the Linux host machine, including all program files and installed ProgramData, execute the command below as an administrator.

(InstallDir=`mktemp -d /tmp/delinea-engine-installer-XXXXXX`; trap "sudo rm -rf $InstallDir" 0 1 2 3 15; cd $InstallDir; echo "Downloading packages. This may take a moment..."; URL='https://enginepool-downloads-dev.azureedge.net/shell-installer/latest/linux-x64.tgz'; curl -fLO $URL; gunzip -c *.tgz | tar -xf -; sudo ./Delinea.EnginePool.Engine.Installer uninstall; sudo systemctl restart delinea-engine.service;) 

Successful execution should yield a confirmation message indicating proper completion.

Windows Platform Engine and Log Directories

The Windows Platform Engine is installed at the following location:

• C:\Program Files\Delinea Engine\[version number]\

Windows Platform Engine logs and deployment files can be found on the engine endpoint in these folders:

• C:\ProgramData\Delinea Engine\log

• C:\ProgramData\Delinea Engine\[version number]\

The log folders contain bootstrap (startup) logs, registration (with platform) logs, default (engine runtime) logs, and individual workload logs.

Linux Platform Engine and Log Directories

The Linux Platform Engine is installed at the following location:

• /opt/delinea-engine/[version number]/

Linux Platform Engine deployment files can be found in versioned folders at the location below. The versioned deployment folders are used for temporary processes, such as downloading and extracting deployment installations.

• /var/delinea-engine/[version number]/

Linux Platform Engine logs:

• /var/delinea-engine/log

The default log contains the following engine runtime information:

• Process logging

• Starting and stopping deployments

• Sending heartbeats to the platform

Platform Engine Log Levels

The current log levels for various components are found under LogLevel and MinimumLevel to the right of each component. These values can be changed to other values from the table below. Each level includes the levels below it. For example, if the level is set to Warning, messages at the Error level are also recorded.

The default logging levels provide a record of normal engine functions. If a different amount of logging is desired, all log levels should be changed to reflect the new desired level.

Adjusting Windows Platform Engine Log Levels

To change the engine log level, open the file at the following location in a text editor with administrator privileges:

• C:\Program Files\Delinea Platform Engine\[version number]\appsettings.json

After setting the desired log levels, save this file:

• appsettings.json

If the save is not successful, make sure you are running your text editor with administrative privileges.

Restart the Windows Platform Engine service using the Windows Services Manager.

Adjusting Linux Platform Engine Log Levels

To change the engine log level, open the file at the following location in a text editor with administrator privileges:

• /opt/delinea-engine/[version number]/appsettings.json

After setting the desired log levels, save this file:

• appsettings.json

If the save is not successful, make sure you are running your text editor with administrative privileges.

Restart the Linux Platform Engine service using sudo systemctl restart delinea-engine.service

Checking Logs in the Engine Management Interface

The logging page provides a comprehensive view of system logs, helping users to monitor and troubleshoot various components.

Logs are retained for seven days, then they are automatically deleted from the database.

To check logs follow the steps below:

1. Click Settings from the left navigation menu, then click Engine Management from the secondary menu.

2. Click the name of a site where you want to add an engine.

3. Click the Engines tab.

4. Click the name of the engine with the log you want to check.

5. Click the Logs tab.

You can check logs of engines or logs of specific workloads by changing the value in the Source field.

The logs table displays data in four columns: Date, Source, Level, and Message.

• Date: The date and time when the log entry was generated. Format: MM/DD/YYYY, HH:MM:SS. (US Eastern Time Zone)

• Source: Users can filter logs based on the origin of the log entry (Engine or a specific workload).

• Level: The severity or importance of the log entry. Different levels help users to prioritize and address issues accordingly.

  • Trace: Detailed information for tracing the execution of the system

  • Debug: Information useful for debugging purposes

  • Information: General information about system operations

  • Warning: Indicates a potential issue that may need attention

  • Error: Indicates that an error has occurred, requiring investigation

  • Critical: Represents a serious problem that requires immediate attention

• Message: Detailed information about the log entry, including context and specifics related to the log level.

Engine Workloads

Engine Troubleshooting