Troubleshooting

The following are Hyper-scalable Privileged Access Service frequently asked questions and information about specific features and functionality as follows:

  • Scripts won't run.
  • Unknown or non-existent node listed in NodeList.
  • Web node is installed but site does not appear.
  • What is the Logging Relay?
  • How to retrieve Node Logs
  • How to retrieve Connector Logs without a Logging Relay
  • How to provide a Support Report

Scripts Won't Run

If you receive an error such as:

Message: File <file name> cannot be loaded. The file <file> is not digitally signed. You cannot run this script on the current system. For more information about running scripts and setting execution policy, see about_Execution_Policies at http://go.microsoft.com/fwlink/?LinkID=135170.

+ CategoryInfo : NotSpecified: ( [Write-Error], WriteErrorException

+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Centrify-Pas-Deploy.ps1

Review PowerShell Execution Policy for more information.

Unknown or Non-existent Node Listed in NodeList

If you see nodes that no longer exist listed when you run Centrify-PAS-NodeList.

Common Cause

The Node was destroyed, lost, or it was unable to connect to the database when it was deprovisioned using Centrify-PAS-Deploy -RemoveNode on the node itself.

Solution

Centrify-PAS-RemoveNode from the Management node will remove the node from the database.

Web Node is Installed But Site Doesn’t Appear

After you have deployed a web node using Centrify-PAS-Deploy -WebNode, set it active, browsing to the host name doesn’t work.

Common Causes

There are several possibilities:

The name is not registered

To browse to the Web node, the host name must be registered with the appropriate name server. To verify this, from your client system, enter:

nslookup <hostname>

Example:

nslookup pas.corpnet.com

The return IP address should match the public IP address of the node or the node’s load balancer.

For example:

Copy 
PS C: \> nslookup pas.corpnet.com

Server: dns.google

Address: 8.8.4.4

Non-authoritative answer:

Name: corpnet.com

Address: 108.167.88.99

Aliases: pas.corpnet.com

This tells us that:

  1. Name Servers (in Windows Control Panel) are set to Google’s DNS (8.8.4.4).

  2. Pas.corpnet.com is listed and has a public IP address (meaning: not

    192.168.*.* or 10.0.*.*).

If, instead, we got:

PS C:\ > nslookup pas.corpnet.com

Server: dns.google

Address: 8.8.4.4

*** dns.google can't find pas.keybounce.com: Non-existent domain

This indicates that the name could not be resolved. Ensure it is plugged into the correct authoritative name server, such as AWS’ Route53, or GoDaddy, and so on.

This address is not the internal address of the Web node(s), but rather the public internet-facing port for the Load Balancer or Firewall.

Inaccessible IP Address

If the listed address from the above step comes back as a Private IP address or in any of the following ranges...

  • 10.0.0.0 – 10.255.255.255
  • 172.16.0.0 – 172.31.255.255
  • 192.168.0.0 – 192.168.255.255

...the IP Address is not accessible from the outside world. It needs an external public (generally static) IP Address. The IP address is not for the Web node, unless there is only one Web node (not recommended), but rather for the Load Balancer.

Load Balancer Health Check Fails

Once you have verified that the name resolves to the Load Balancer, ensure the Load Balancer can see healthy web nodes.

  • The Health Check point is /health/check. You should see all web nodes listed and at least those on the current deployment (Centrify-PAS-SetActiveDeployment) displaying “healthy”.

  • If you do not see any Web nodes, check your load balancer configuration.

  • If you see the correct Web nodes, but they display as “unhealthy,” verify that they are on the correct deployment. Navigate to the Web node by namefrom the node (this will generally work as the deployment process adds thename to the local hosts file at c:\Windows\System32\Drivers\Etc\hosts) or IP Address, adding the “/health/check” path.

    alt

In this case, we see that the Role is active, with the Instance Name of “WR_Second.” If the Web nodes list as offline, ensure they are powered up and booted.

  • From the Management node, ensure the Web node is listed as online and active from Centrify-PAS-NodeList.

    • If it is offline, it is not accessing the database and may not be running.
    • If it is online but inactive, it has the wrong deployment ID. You need to either change the active deployment with Centrify-PAS-SetActiveDeployment or you will need to deploy a node of the correct deployment.

  • RDP into the Web node and verify that IIS is running and that there is a c:\CentrifyNode directory.

    If the above are not the case, it may be necessary to re-image and re-deploy this Web node.

What is the Logging Relay?

The Logging Relay provides several features including the following:

  • Aggregates logs from all deployed Web and Background nodes, providing a

    single place to retrieve them.

  • Enables the Management Node to watch the logs, using LogWatcher

    (Centrify-PAS-WatchLogs).

In addition to being essential for trouble-shooting, the output provided by a Logging Relay plus LogWatcher can be fed into a custom or Splunk-like parser to generate real-time analytics and alerts.

How to Retrieve Node Logs

On the Logging Node, you can find the logs at c:\Centrify\Logs. Their names contain the date ranges and log type.

For example, for an installation with a hostname (URL) of pas.corpnet.com, generated from the hours of 9:00pm - 11:59pm on May 14, 2020, the log names will look similar to the following:

  • 2020-05-14-21-pas.corpnet.com-navel.log
  • 2020-05-14-21-pas.corpnet.com.log
  • 2020-05-14-22-pas.corpnet.com-navel.log
  • 2020-05-14-22-pas.corpnet.com.log
  • 2020-05-14-23-pas.corpnet.com-navel.log
  • 2020-05-14-23-pas.corpnet.com.log

The plain .log files have standard log data in them, while the -navel.log files are not human-readable, and contain timing data about internal operations that help Delinea determine where a task might be taking longer than expected.

For convenience, you can use Centrify-PAS-GetDiags.ps1 on the Logging Node to specify a start date, start hour, and duration (hours) for the run. This will package the logs from all nodes and the connector logs.

How to Retrieve Connector Logs without a Logging Relay

The documented process is to install a Logging Relay prior to installing any other nodes.

Delinea cannot guarantee support of an installation that did not follow the documented process.

If your Logging Relay is not available for some reason, Centrify-PAS-GetDiags can also be run from the Management Node. You can only retrieve connector logs using this method since the Management node can't reach the Web or Background Node logs.

How to Provide a Support Report

In addition to logs, basic information about the installation and environment can help Delinea quickly find the cause of most reported issues.

The Support Report includes information about all deployed nodes, the versions of the database and binaries installed, and various run-time data including:

  • Delinea connectors, including current status and latency.

  • CurrentDeploymentId

  • DatabaseConnections. This is for debugging database issues. There is no PII in this.

  • DeploymentHistory and SchemaHistory, including binary (cloud) versions.

  • Running and Queued Jobs. In a healthy system, this is usually empty or nearly empty.

  • Nodes including type, name, and the basic environment.

  • StatSnap. These are scale statistics. For example, the count of (but not enumeration of) devices, entitlements, systems, etc.

    None of this information expose any confidential data, but you may still want to scan over the information prior to submitting.

Delinea cannot retrieve this information directly, unless you provide explicit remote access and permission. The information can only be generated using one of the following methods:

  • In the Admin Portal, using the Support menu located in the upper right

    area of the screen.

  • By calling the /health/SupportInfo endpoint. For example, with CCLI.
  • By running Centrify-PAS-NodeList.ps1 -Support on the Management Node.