PCS Troubleshooting Guide

Issue: I don't know where to find all my log files.

Resolutions:

Delinea Connector

C:\Program Files\Delinea\Delinea Connector\log.txt

Delinea Platform Engine

C:\ProgramData\Delinea Platform Engine\<engine_version>\log

Command Relay

Command Relay stores logs in 2 places:

Abridged Log - C:\ProgramDataC:\Program Files\Delinea Engine\<engine_version>\delinea\command-relay\<version>\log
Detailed Log - C:\ProgramData\Delinea\CommandRelay\Logs

Privilege Control Agent

Linux - /var/log/centrifydc.log
Windows (default location): - C:\Program Files\Common Files\Centrify Shared\Logs\

You can change where the Windows agent log files are stored using Privilege Elevation Service Settings:

Open “Delinea Agent Configuration”
Click “Settings” button under “Privilege Elevation Service”
Go to “Troubleshooting” tab in “Privilege Elevation Service Settings” and click “Options” button
Change Log folder path as you want. You also could change the trace level in this “Options” dialog.

Issue: Cannot connect to Delinea Platform

Connection issues can be caused by improperly configured Integrated Windows Authentication (IWA).

Resolution: Use the command https://<connector_host_name>:<https_port>/iwa/sitecheck to verify whether IWA is working on your Delinea Platform host. For details, see Verifying IWA Over HTTPS.

The command /iwa/ping can also be used, but /iwa/sitecheck gives more information, as shown in the section linked above.

Issue: Windows Diagnostics Error for MFA

Error: One or more validations have a problem. The environment might not be configured appropriately and some features might not be functioning properly.

Resolution: This message indicates erroneously that MFA is not working for PCS. You can ignore this message.

Issue: Zero pass-through is not honored for MFA

When configuring multi-factor authentication, you can set the pass-through duration to zero in an authentication profile. This pass-through setting should prevent any time from elapsing before a user is prompted again for MFA authentication. However, when privilege elevation is done with an MFA authentication profile set, the "no pass-through" setting is not honored.

Cause: The command dzdo is used to perform commands with privilege elevation. The issue occurs because, by default, dzdo has an authentication timeout interval of 5 minutes. This means that once dzdo has been authenticated, it does not have to authenticate again for 5 minutes. This 5-minute interval overrides the intended effect of setting the pass-through interval to zero.

Resolution: To solve the issue, you must force dzdo to use an authentication timeout of 0, to match the pass-through interval. In the configuration file /etc/centrifydc/centrifydc.conf on the Linux agents, uncomment the parameter dzdo.timestamp_timeout and set it to 0.

Policies

Issue: When searching for a known user to add as a subject for a PCS policy, the user's name does not appear, or no user names appear.

Resolution:

Open the Delinea Connector Configuration UI.
On the Status tab, look at the Last connection result.
If the message reads, "Connector is not available," select the Connector tab and click Start.
If the message reads, "Successful" but the Last connection time was a long time ago, select the Connector tab and click Stop.
When the connector stops, click Start. It might take several seconds for the connector to stop and start.
If your known user or all users are still not showing up in your search to a policy target, check the Connector logs and contact Delinea support if necessary.

Issue: My Policy is stuck on “Activating” (or “Deactivating”) status?

Resolution: Normally a Policy stuck on Status “Activating” or “Deactivating” is a indication of problem with the Command Relay. Please check if you have a Command Relay running. For more Command Relay debug options, please see Command Relay / Engine Pool section below.

Issue: My Policy status is “Active” but it's not being enforced.

Resolution: Policy changes may take up to 30 minutes to be enforced after its status becomes Active or Inactive, due to the agent internal caching. If Policy is not being enforced after that time, please notify Delinea support.

Issue: My Login (or Elevation) Policy status is “Inactive” but I can still perform Login (or Elevation) on the machine. Why?

Resolution: Policy changes may take up to 30 minutes to be enforced after its status becomes Active or Inactive, due to the agent internal caching. If Policy is not being enforced after that time, please notify Delinea support.

Issue: The Policy’s Target list is not showing the machine I want to select.

Resolution: The Targets you can select come from Inventory. If you are looking for a machine and it is not showing in the Targets list, please check if that same machine appears in the Inventory list.

If the machine not listed under Targets is listed under Inventory, please notify Delinea support.

If the machine not listed under Targets is also not listed under Inventory, then you need to “Discover” that machine by doing a Secret Server Discovery process.

Command Relay / Delinea Platform Engine

Command Relay is one of the workloads deployed by the Delinea Platform Engine and heavily depends on the Engine to run. Therefore, when troubleshooting Command Relay it is also important to investigate potential problems with the Engine. Currently the main tool for Command Relay troubleshooting is looking at the logs that are saved by Command Relay.

Issue: How do I turn on debugging for my Engine?

Resolution: The default setting for Engine Pool logs ensures the logging of critical errors only without much detail. If you want detailed information, increase the default ‘verbose’ level to “Debug” in the Engine Pool’s appsettings.json file (Logging Level)

C:\Program Files\Delinea Engine\<engine_version>\appsettings.json

Issue: Is Command Relay setting in Engine Pool for all engines under the same site?

Resolution: Yes. You could create another site if you want to user a different domain.

Issue: Why does Command Relay need the AD domain admin credentials?

Resolution: Command Relay uses the credentials to communicate with AD to store the Policies. By default, AD users in “Domain Admins“ group have all the required permissions.

Issue: What happens if I provide the wrong Active Directory domain admin credentials or if they expire?

Resolution: Command Relay will stop working (and therefore no other Policy change would be applied) and you would see in the Command Relay log the following:

Issue: My selected secret for Command Relay stopped working (service account)

Resolutions:

This could happen if the selected secret is changed, e.g. if you moved the secret to a personal folder in Secret Server, it will remove the EngineWorkload shared permissions on the secret which will cause permission failure in Command Relay.
This could also happen if the underlying service account associated with this secret is changed, e.g. password expired/not synced, account locked, AD permissions removed, etc. There will be a log on failure log with error details about those.

Issue: Command Relay can't log in using the secret that works for Secret Server Discovery service

Resolution: In the Command Relay log, you see that the Command Relay cannot login:

2024-01-29 14:17:11,592 [10] INFO CommandRelay [(null)] - RunAsProcess info: domain=eric-sp-1.eric user=svc-ssd

2024-01-29 14:17:11,592 [10] INFO CommandRelay [(null)] - Normalized RunAs Info: user=svc-ssd domain=eric-sp-1.eric

2024-01-29 14:17:11,616 [10] ERROR CommandRelay [(null)] - Invalid Domain Credentials. Logon user failed: svc-ssd, errorCode=1385

2024-01-29 14:17:11,628 [10] ERROR CommandRelay [(null)] - Invalid domain creds detected, Exception=Delinea.CommandRelay.Common.LogonException: Invalid Domain Credentials. Logon user failed: svc-ssd, errorCode=1385

---> System.ComponentModel.Win32Exception (1385): Logon failure: the user has not been granted the requested log on type at this computer.

Issue: IWA not working properly when installing Connector

Symptom: When trying to deploy Delinea Connector onto a host, communication issues occur between the host and Integrated Windows Authentication (IWA). The host name also appears truncated wherever it appears in the Platform UI; for example, in the inventory and the list of engines.

Cause: The host computer has a host name longer than the maximum Windows NetBIOS name length of 15 characters. The Powershell script supplied in Generating a Self-Signed Delinea Connector IWA Host Certificate for generating a certificate uses the truncated name, and therefore gets the wrong DNS name for the machine.

Resolution:

Rename the host computer with a name that is no more than 15 characters long.
Generate a new certificate for the host.
Remove the host from enrollment with IWA identity services.
Force removal of all data.
Re-enroll the host with the identity services provider using the new host name.

Secret Server

Issue: My Secret Server Distributed Engine is not working.

Resolutions:

Check if the Engine has been Activated.
Check if the Windows clock of the machine where the agent is running is correct.

Privilege Control for Servers Agent

Issue: How do I turn on/off debugging for Linux agents?

Resolution:

To turn on debugging, run the following commands as the root user:

/usr/share/centrifydc/bin/addebug set cloud.object TRACE
/usr/share/centrifydc/bin/addebug on

Logs can be found in /var/log/centrifydc.log.

To turn off debugging, run the following commands as the root user:

/usr/share/centrifydc/bin/addebug off

Issue: How do I turn on debugging for the sshd server?

Resolution:

Run ps -ef | grep sshd to check if you are using CentrifyDC-openssh or system stock sshd.

If you are using CentrifyDC-openssh

add LogLevel DEBUG3 into /etc/centrifydc/ssh/sshd_config
restart the server by running this command as the root user: systemctl restart centrify-sshd

If you are using system stock sshd

add LogLevel DEBUG3 into /etc/ssh/sshd_config
restart the server by running this command as the root user: systemctl restart sshd (or systemctl restart ssh on Ubuntu/Debian)

Issue: How do I collect debug info for the Delinea team to investigate an issue?

Resolution:

Turn on debugging for Linux agent and sshd
Reproduce the issue
Run adinfo -t command as the root user

the /var/centrify/tmp/adinfo_support.tar.gz file is what the Delinea support team will need for investigation.

Issue: My AD forest has multiple domains, so will each domain have a DelineaZone created?

Resolution: No, there will be only one DelineaZone created in the forest when you deploy the very first Engine Pool in the forest.

Issue: I can not log in to my Linux agent after enabling Session Recording

I can not log in to my Linux agent after enabling Session Recording in a Granular Privilege Elevation policy for Linux (see Policy Details).

Resolution: The Linux agent requires Direct Audit to be enabled on the Agent when policies have session recording enabled.

Direct audit can be enabled on the Linux agent by following the steps in Step 11: Set up Audit and Session Recording.

Issue: My AD user cannot log in to the domain-joined Linux machine

Resolution: You will need a root shell for the following steps.

Suppose your AD user name is “tom@acme.com”

Please note it may take up to 30 minutes before the Linux agent refreshes the latest authentication and authorization info from AD after the policy deployment. You can also manually run adflush -f to force a refresh at any time.
Verify whether the AD user is visible on the Linux machine
- adquery user tom@acme.com
  - If you get “tom@acme.com is not a zone user”, please verify whether the Command Relay has successfully deployed the policy.
Verify whether the AD user has login permissions
- dzinfo --role tom@acme.com
  - an example output would look like this:

User: tom

Forced into restricted environment: No

MFA Service authentication: Supported

Privileged commands:

Name Avail Command Source Roles

--------------- ----- -------------------- --------------------

__pe_sys_6240d3 No * Mansion-Grove-Elevat

33-6256-4221-9a ion/DelineaZone

23-39bfc381202c

/DelineaZone

__pe_6240d333-6 No * Mansion-Grove-Elevat

256-4221-9a23-3 ion/DelineaZone

9bfc381202c/Del

ineaZone

...

If you don’t see “Password login” and “Non password login” in the “Effective rights”, please verify whether the Command Relay has successfully deployed the policy.

If both adquery and dzinfo commands show the expected result, please collect debug info and contact Delinea support.

Issue: My AD user can't run DZDO commands in the domain-joined Linux machine

Resolution: You will need a root shell for the following steps.

Suppose your AD user name is “tom@acme.com”

Please note it may take up to 30 minutes before the Linux agent refreshes the latest authentication and authorization info from AD after the policy deployment. You can also manually run adflush -f to force a refresh at any time.
Verify whether the AD user has privileged command rights
- dzinfo --commands tom@acme.com
  - an example output would look like this

User: tom

Forced into restricted environment: No

MFA Service authentication: Supported

Privileged commands:

Name             Avail Command               Source Roles

---------------  ----- --------------------  --------------------

__pe_sys_6240d3  No    *                     Mansion-Grove-Elevat

33-6256-4221-9a                              ion/DelineaZone

23-39bfc381202c

/DelineaZone

__pe_6240d333-6  No    *                     Mansion-Grove-Elevat

256-4221-9a23-3                              ion/DelineaZone

9bfc381202c/Del

ineaZone

...

...

If you don’t see anything in the “Privileged commands”, please verify whether the Command Relay has successfully deployed the policy.

If dzinfo command shows the expected result, please collect debug info and contact Delinea support.

If the Connector appears Active at Settings > Connectors but you see the error message, Unable to communicate with the Delinea Platform, you can ignore the message.

Issue: I need Some Useful Commands and Tips for AD Client on *.nix

AD-bridging commands ("ad" commands)

adinfo - provides information about the status of the agent

Looking-up Basic Information

To check the general status of the client: $ adinfo

To see the current domain controller the client is using: $ adinfo --server

To see the current domain the agent is joined to: $ adinfo --domain

To see the status (mode) of the agent (connected to ad or in offline mode): $ adinfo --mode

To see the version of the installed client: $ adinfo --version

To see the corresponding Delinea PCS Version: $ adinfo --suite-version

To view Active Directory connectivity to the current domain: $ adinfo --test

To view the current Active Directory site: $ adinfo --site

To see the current joined Delinea zone: $ adinfo --zone

$ adinfo --zonedn (in distinguishedName format)

Advanced/Troubleshooting Information

DNS

To check for the "joined-as" name (local host name and joined as name may be different): $ adinfo --name

To check the status of the DNS cache and stats: $ adinfo --diag dns

Connectivity

To check connectivity with an AD domain: $ adinfo --test [domain.name]

To check network connectivity statistics: $ adinfo --sysinfo neststate

To test connectivity against a specific domain controller: $ adinfo --T --servername [domain.controller.name]

Active Directory

To see the current AD Global Catalog: $ adinfo --gc

To see the domain/forest map: $ adinfo --sysinfo domain

To see the status of the AD computer trust relationship: $ adinfo --sysinfo adagent

Configuration

To parse the contents of the centrify.conf file: $ adinfo --config

To show the client's in memory configuration parameters: $ adinfo --sysinfo config

Microsoft Kerberos

To view Kerberos information like supported encryption types, key version and registered SPNs: $ adinfo --computer

PKI: adcert - delinea Microsoft PKI client

To perform Auto-enrollment of Computer PKI Certificates (requires eligible template and communications)

Using the computer object to authenticate: $ dzdo /usr/share/centrifydc/sbin/adcert --enroll --machine

Using a user to authenticate: $ dzo /usr/share/centrifydc/sbin/adcert --enroll --user [ADusername]

Testing a user's password: $ adinfo -A --user [username] #
This will prompt you for a password, the output is: Password for user "username" is correct/incorrect

Dynamic DNS

addns - a dynamic DNS client for AD DNS or RFC 2136-compliant servers

To renew DNS using machine credentials: $ sudo addns --update --machine

To renew DNS using user credentials: $ sudo addns --update --user [ADusername]

To renew DNS only on a specific interface (e.g. eth0): $ sudo addns --update --machine --interface eth0

Querying Delinea-enabled AD Users and Groups:

adquery - provides information about Active Directory users and groups that are UNIX-enabled by Delinea

To view all Delinea UNIX-enabled users: $ adquery user will show all AD users in Express mode / Only authorized in Zone mode

To view all Delinea UNIX-enabled groups: $ adquery group will show all AD groups in Express mode / Only unix-enabled in Zone mode

To view a user's entry (UNIX passwd file style): $ adquery user [username]

To view a group entry (UNIX group filestyle): $ adquery group [groupname]

To view only the user or group's AD group memberships: $ adquery user [user] --adgroup

To view all information about a user or group (including AD object attributes): $ adquery user|group [user or group] -A

To view the distinguishedName a user or group: $ adquery user|group [user or group] --dn

To view all information and include password expiration, account lockout/enabled state: $ sudo adquery user [user] -A

To view information about a computer: $ adquery user [computername]$ -A

To get results from cache (instead of fetching from AD): $ adquery user|group [options] --cache-first

Delinea Cache Commands

adflush - clears the Delinea cache in the local computer (dc, gc, credential & dns)

To flush the authorization cache: $ dzdo adflush --auth

To rebind and force a new DC selection: $ dzdo adflush --bindings

To flush the DNS cache: $ dzdo adflush --dns

To expire the information from domain controllers and global catalogs: $ dzdo adflush --expire

To force complete removal/expiration even when disconnected (use carefully): $ dzdo adflush --force

To refresh the krb5.conf file: $ dzdo adflush --trusts

To clear the health history: $ dzdo adflush --health

To clear the cloud connectors (in MFA scenarios): $ dzdo adflush --connectors

Group Policy-related Commands

adgpupdate - triggers the group policy refresh interval

To refresh the GPOs in the system: $ adgpupdate

To refresh only computer GPOs: $ adgpupdate --target Computer

To refresh only user GPOs: $ adgpupdate --target User

adgpresult - to view a RSOP (resultant set of policy) to the local system or user

To view the report for computer and user: $ adgpresult

To view the report for the computer: $ adgpresult --computer

To view the report for the current: $ adgpresult --user

To view the report for a particular user: $ dzdo adgpresult --user [user.name]

Joining Active Directory

adjoin - joins an Active Directory domain

To run adjoin successfully, you need the following:

to be root or sudo
to have the credentials (or the keytab) of an AD user that can join computers to a container (NOT Domain Admin)
to know the Distinguished Name (e.g. "ou=servers,ou=unix") of the container that you will place the system in AD
to know the domain name you're joining
to have a clear network path to the DC or DCs you're using (dns, global catalog, kerberos, ldap, cifs, ntp).

To join AD in workstation/express mode (AD user must be able to add computers to "ou=workstations,ou=unix"):
$ sudo adjoin --workstation --container "ou=workstations,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To join AD in Self-Service mode (AD/Delinea admin pre-created the machine ahead of time using Access Manager or Delinea Poweshell cmdlets): $ sudo adjoin --selfserve [domain.name]

To join AD in zone mode (e.g. Global zone): $ sudo adjoin --zone Global --container "ou=servers,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To join AD in zone mode and don't initialize (precache): $ sudo adjoin --noinit --zone Global --container "ou=servers,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To join AD and trust the Computer for Delegation (must know what you're doing - security implications): $ sudo adjoin --trust Global --container "ou=servers,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To join AD in workstation mode and specify a workstation license: $ suod adjoin --licensetype "workstation"--workstation --container "ou=workstations,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To use an specific domain controller to join (e.g. dc1.hq.fabrikam.com): $ sudo adjoin --server dc1.hq.fabrikam.com Global --container "ou=servers,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To join a Mac in Workstation mode and instruct Delinea to use the Apple algorighm to generate UID/GID scheme: $ sudo adjoin --enableAppleIDGenScheme --container "ou=macs,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To join AD and provide a different "AD name" than the local system name (e.g. adserver vs. localhost): $ sudo adjoin --name adserver --container "ou=servers,ou=unix" --user [AuthorizedADUser] --verbose [domain.name]

To join AD using keytab (kinit Authorized AD user keytab first, then run adjoin without the --user option): $ env KRB5_CONFIG=[/path/to/krb5.conf] /usr/share/centrifydc/kerberos/bin/kinit -kt /path/to/keytab [principal]: $ sudo adjoin --zone Global --container "ou=servers,ou=unix" --verbose [domain.name]

Leaving Active Directory

adleave - leaves an Active Directory domain

To run adleave successfully, you need:

> sudo or root
> for online leave, authorized AD user credentials

Leave the domain and disable the computer object (orphan object left behind): $ dzdo adleave --user [Authorized ADUsername]

Leave the domain and remove computer object (frees license): $ dzdo adleave --user [Authorized ADUsername] --remove

Offline/forced leave (no AD connectivity required, must clean-up in AD): $ dzdo adleave --force

Privilege Elevation ("dz" commands):

dzinfo - displays information of the user's access controls

To view self access (all): $ dzinfo

To view the properties of the role(s), including effectiveness: $ dzinfo --roles

To view how you can access the system (PAM rights): $ dzinfo --pam

To view the commands you can run: $ dzinfo --commands

To view the computer roles that apply to the system (requires elevation): $ dzinfo --computer-role

To view authorization information about another user (requires elevation): $ dzdo dzinfo [user.name]

To test a command against the role: $ dzinfo --test [path/to/binary] [options]

Delinea -enhanced sudo

dzdo - delinea-enhanced sudo. Uses Delinea zone data in AD for commands, otherwise identical to sudo.

To view version information (as of 2015, based on sudo 1.8.10p3): $ dzdo -V

DirectAudit Commands ("da" commands)

dainfo - shows information about the status of the audit agent

To view the audit agent status: $ dainfo

To view status with verbose output: $ dainfo --diag (or dadiag)

To view contents of the configuration file: $ dainfo --config

To view audited status of another user (must elevate): $ dzdo dainfo --username lisa.simpson

dacontrol - controls the status/configuration of the directaudit client (requires elevation)

To set the installation (if not set by Group Policy): $ dzdo dacontrol --installation [installation-name]

To check if the audit agent is enabled: $ dzdo dacontrol --query

To enable direct audit: $ dzdo dacontrol --enable

To disable direct audit: $ dzdo dacontrol --disable

What Happens When adjoin is Run Succesfully?

This activates the DirectControl agent (adclient/ DelineaDC service):

Creates a computer object in AD and sets SPNs for http, host, nfs, cifs, afpserver
Establishes a secure communication channel between the system and Active Directory
A forest/domain/site map is created to locate the nearest DCs
The Kerberos environment (krb5.conf, krb5.keytab) are maintained by Delinea (configurable). A backup is created.
Network time is synchronized with AD DCs (configurable)
The PAM (Pluggable Authentication Modules) are modified to include Delinea auth, account, password, session modules. A back-up of the previous configuration is made.
The NSS (Name Service Switch) providers for users and groups defaults to AD first, then other methods (e.g. files, ldap, etc). A backup of the previous configuration is made. Note: in the OS X platform, the PAM/NSS functions are channeled via the Directory Services Plugin API.
An Access Control Model is enforced depending on the zone mode:
- In zone mode: Authorization (RBAC) follows zone rules (defaults to closed, only authorized users can access and enabled groups are visible)
- In express/workstation mode: Only Authentication is facilitated. The system is open for all AD users and all groups are visible.
Privilege Elevation: Delinea -enhanced sudo (dzdo) becomes active based on the roles/rights defined.
User/Group identity (RFC2307) data in AD is stored within the Delinea zone, NOT with the user/group object.
The virtual registry is initialized and group policies are enforced.

What Happens When adleave is Run Successfully?

1. Online the --remove object: The object in AD is removed from the container and from the zone (frees license)

2. Online the without --remove object: The object in AD is marked as disabled. Must be overwritten to rejoin.

2. Offline: The object in AD is left orphaned. Cleanup must happen via any Delinea API (AM, PowerShell, adedit)

3. The UNIX environment is reset and rolled back (Kerberos, PAM, NSS)

4. The Delinea adclient (DelineaDC) service is disabled.

Important Files and Folders

/usr/share/centrifydc/

bin > contains user binaries, including Delinea-enhanced openldap tools like ldapsearch

sbin > contains system binaries, including adcert and Delinea-enhanced OpenSSH

samples > sample files for hadoop, adedit and local account management

Note: on OS X El Capitan, things changed to /usr/local/share/centrifydc

/etc/centrifydc

centrifydc > config files for the DirectControl agent

centrifyda > config files for the DirectAudit agent

centrifycc > config files for the Privilege Service CLI Toolkit for AAPM

openldap > config files for Delinea-enhanced OpenLDAP proxy if installed

ssh > config files for Delinea-enhanced OpenSSHls

/var/centrifydc

kset* files > dynamic information about the environment

reg > virtual registry, contains the computer and user hives (user GPO disabled on Servers)

/var/centrify

net/certs > location of any Microsoft Certificate Authority auto-enrolled certs, keys and trust chain

Issue: Trouble using DirectControl Authentication on *NIX systems

When the directory /var is NFS mounted, DirectControl may not work properly.

Resolution: Directory /var should not be NFS mounted or else DirectControl may not work properly.