ArcSight CEF format
The Common Event Format (CEF) standard format, developed by ArcSight, enables vendors and their customers to quickly integrate their product information into ArcSight ESM.
CEF defines a syntax for log records comprised of a standard header and a variable extension, formatted as key-value pairs.
When syslog is used as a transport mechanism, CEF uses the following format, comprised of a syslog prefix, a header, and an extension:
Jan 18 11:07:53 host CEF:Version|Device Vendor|Device Product|Device
Version|Device Event Class ID|Name|Severity|[Extension]
The following example illustrates a general CEF message using syslog transport:
Sep 19 08:26:10 host
CEF:0\|Centrify\|Centrify_Cloud\|1.0\|Cloud.core\|Cloud.core.MfaSummary\|5\|src=10.0.0.1
dst=2.1.2.2 spt=1232
Using CEF Without Wyslog
Syslog applies a syslog prefix to each message, no matter what device it arrives from, which contains the date and hostname:
Jan 18 11:07:53 host CEF:Version\|…
However, if an event producer is unable to write syslog messages, it is still
possible to write the events to a file. In this case, begin the message with the format shown below, and omit the syslog prefix:
CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]
<!--
Sample Python functions for CEF creation
This section describes a set of sample Python functions for generating
CEF-formatted CP events.
There are three main functions in this package:
-
fetch_oauth_token()
-
query_events()
-
cef_generator()
Using the functions to demonstrate sample usage
Prerequisite: Python 3.5 or above
Follow these steps:
-
Download the Python code from
https://github.com/centrify/centrify-hparcsight-integration-sample/ -
Install pip packages in requirement.txt.
-
Provide the values for tenant, siem_username, and siem_password in config.ini.
-
Execute sample_usage.py to generate CEF-formatted CP events for one hour:
-
python3.5 sample_usage.py
The following example shows a CEF message for a Self-Service App Launch CIS
Event:
CEF:0|Centrify|Centrify_Cloud|1.0|Cloud.Saas.Application|Cloud.Saas.Application.SelfServiceAppLaunch|5|dhost=AAA0056 duser=cloudadmin@persistent.com01 msg=User cloudadmin@persistent.com01 launched Instagram from 103.6.32.100 shost=103.6.32.100 src=103.6.32.100 rt=1525844566655 deviceProcessName=centrify-syslog-writer dvchost=dinesh-VirtualBox dtz=Africa/Abidjan requestContext=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063 externalId=772a4a904e82da87.W00.0315.1aa20afe647f09c dpriv=WebRole destinationServiceName=CDS suid=c2c7bcc6-9560-44e0-8dff-5be221cd37ee cs1=Instagram cs1Label=applicationId cs2=Instagram cs2Label=applicationName cs3=Web cs3Label=applicationType cs4=103.6.32.100 cs4Label=clientIPAddress cs5=65f79bb1-4f91-4496-9991-d148da16cc3e cs5Label=internalSessionId cs6=0d10a24f4c57434198fb3ad4559cc48b cs6Label=azDeploymentId directoryServiceNameLocalized=Centrify Directory threadType=RestCall azRoleId=WebRole_IN_0 internalTrackingID=d3a0713b610146ca916155efca2be690 authMethod=UserPassword requestIsMobileDevice=False directoryServiceUuid=09B9A9B0-6CE8-465F-AB03-65766D33B05E requestDeviceOS=Windows level=Info
You can customize the usage or the APIs per your application needs.
CEF has a predefined set of keys.
CEF Mapping of CP Events
This section provides detailed information about how the CEF fields have been mapped from the CP event fields in the Python application described above.
CEF header
Header Field |
CP Event Field |
---|---|
Version |
‘0’ |
Device Vendor |
‘Centrify’ |
Device Product |
‘Centrify_Cloud’ |
Device Version |
‘1.0’ |
Device Event Class ID |
Variable — depends on the event. For example: ‘Cloud.Saas.Application’ |
Name |
Variable — depends on the event. For example: Cloud.Saas.Application.SelfServiceAppLaunch’ |
Severity |
Variable — depends on the Level field in event. For example: ‘5’ for Info, ‘10’ for Error. |
CP ArcSight CEF extension
The CEF Extension contains a collection of key-value pairs. The keys are
predefined and are referred to as the ArcSight Extension Dictionary. (CEF Fields)
Common properties in CP events
This section lists the CEF field mapping of CP events, which are part of the CEF extension.
These properties are common to all events of Cloud Suite and Privilege Services.
ArcSight CEF Field |
CP Event Field |
---|---|
The common properties are those listed below in bold. |
|
Destination Host Name |
Tenant |
Destination User Name |
NormalizedUser |
Message |
EventMessage |
Source Host Name |
RequestHostName |
Source Address |
FromIPAddress |
Device Receipt Time |
whenoccurred_epoch_ms (This is the event timestamp in UTC) |
Device Process Name |
‘centrify-syslog-writer’ (can be configured in cef_mapping.ini) |
Device Host Name |
Hostname of machine running the python app |
Device Time Zone |
‘Africa/Abidjan’ |
The keys in the common properties section below are added in the CEF message only if no event-specific CEF mapping is specified for an event in the mapping configuration file, which is enclosed with the Sample Python application for CEF creation. |
|
Device Custom String 1 |
AuthMethod |
Device Custom String1 Label |
'authMethod' |
Device Custom String2 |
RequestIsMobileDevice |
Device Custom String2 Label |
'requestIsMobileDevice' |
Device Custom String3 |
DirectoryServiceUuid |
Device Custom String3 Label |
'directoryServiceUuid' |
Device Custom String4 |
RequestDeviceOS |
Device Custom String4 Label |
'requestDeviceOS' |
Device Custom String5 |
Level |
Device Custom String5 Label |
Level |
--> |
|
Event-specific properties in CP
This section lists the event-specific properties mapped to ArcSight Fields. All events (whether they are listed below or not) will have the first nine common properties, identified in the table above, mapped in an ArcSight.CEF message.
Any CEF key appearing in event-specific mapping will override the CEF key mapping in the common properties section. For example, the
Cloud.Server.ManualAccount.SessionStart event, Destination host (Dhost), and Destination User(duser) will be ‘ComputerName’ and ‘AccountName’, which will overwrite the common properties mapped for dhost and duser.
EventType=Cloud.Core.MfaSummary
ArcSight CEF Field |
CP Event Field |
---|---|
Reason |
MfaReason |
Outcome |
MfaResult |
RequestContext |
RequestUserAgent |
ExternalId |
ID |
Dpriv |
AzRoleName |
DestinationServiceName |
DirectoryServiceName |
Device Custom String 1 |
MfaInitiator |
Device Custom String1 Label |
‘mfaInitiator’ |
Device Custom String2 |
FactorsLocalized |
Device Custom String2 Label |
‘factorsLocalized’ |
Device Custom String3 |
ProfileName |
Device Custom String3 Label |
‘profileName’ |
Device Custom String4 |
FailReason |
Device Custom String4 Label |
‘failReason’ |
Device Custom String5 |
MfaUnlock |
Device Custom String5 Label |
‘mfaUnlock’ |
Device Custom String6 |
ForgotPassword |
Device Custom String6 Label |
‘forgotPassword’ |
Device Custom Number1 |
Factorcount |
Device Custom Number1 Label |
‘factorCount’ |
Device Custom Number2 |
SecurityQuestionAnswerCount |
Device Custom Number2 Label |
‘securityQuestionAnswercount’ |
The remaining fields in an event that are not mapped to CEF keys will still be added in the CEF message with their CP-event field keys. These custom non-CEF keys will not be available for reporting in ArcSight, but they can viewed as part of the raw event message.
EventType=Cloud.Saas.Application.AppLaunch
ArcSight CEF Field |
CP Event Field |
---|---|
RequestContext |
RequestUserAgent |
ExternalId |
ID |
Dpriv |
AzRoleName |
DestinationServiceName |
DirectoryServiceName |
Suid |
UserGuid |
Device Custom String 1 |
ApplicationID |
Device Custom String1 Label |
‘applicationId’ |
Device Custom String2 |
ApplicationName |
Device Custom String2 Label |
‘applicationName’ |
Device Custom String3 |
ApplicationType |
Device Custom String3 Label |
‘applicationType’ |
Device Custom String4 |
TemplateName |
Device Custom String4 Label |
‘templateName’ |
Device Custom String5 |
InternalSessionId |
Device Custom String5 Label |
‘internalSessionId’ |
Device Custom String6 |
AzDeploymentId |
Device Custom String6 Label |
azDeploymentId |
EventType=Cloud.Saas.Application.GatewayAppLaunch
EventType=Cloud.Saas.Application.SelfServiceAppLaunch
ArcSight CEF Field |
CP Event Field |
---|---|
RequestContext |
RequestUserAgent |
ExternalId |
ID |
Dpriv |
AzRoleName |
DestinationServiceName |
DirectoryServiceName |
Suid |
UserGuid |
Device Custom String 1 |
ApplicationID |
Device Custom String1 Label |
‘applicationId’ |
Device Custom String2 |
ApplicationName |
Device Custom String2 Label |
‘applicationName’ |
Device Custom String3 |
ApplicationType |
Device Custom String3 Label |
‘applicationType’ |
Device Custom String4 |
ClientIPAddress |
Device Custom String4 Label |
‘clientIPAddress’ |
Device Custom String5 |
InternalSessionId |
Device Custom String5 Label |
‘internalSessionId’ |
Device Custom String6 |
AzDeploymentId |
Device Custom String6 Label |
azDeploymentId |
EventType=Cloud.Server.ManualAccount.SessionStart
EventType= Cloud.Server.LocalAccount.SessionStart
ArcSight CEF Field |
CP Event Field |
---|---|
Src |
FromIPAddress |
Suser |
NormalizedUser |
Dhost |
ComputerName |
Duser |
AccountName |
RequestContext |
RequestUserAgent |
ExternalId |
ID |
Dpriv |
AzRoleName |
DestinationServiceName |
DirectoryServiceName |
Suid |
UserGuid |
Device Custom String 1 |
UserType |
Device Custom String1 Label |
‘userType’ |
Device Custom String2 |
SessionType |
Device Custom String2 Label |
‘sessionType’ |
Device Custom String3 |
AuthorityName |
Device Custom String3 Label |
‘authorityName’ |
Device Custom String4 |
JumpType |
Device Custom String4 Label |
‘jumpType’ |
Device Custom String5 |
DirectoryServiceNameLocalized |
Device Custom String5 Label |
‘directoryServiceNameLocalized’ |
Device Custom String6 |
AuthoritySource |
Device Custom String6 Label |
‘authoritySource’ |
EventType=Cloud.Server.LocalAccount.PasswordExport
EventType= Cloud.Server.DomainAccount.PasswordExport
ArcSight CEF Field |
CP Event Field |
---|---|
Src |
FromIPAddress |
Suser |
NormalizedUser |
Dhost |
ComputerName |
Duser |
AccountName |
RequestContext |
RequestUserAgent |
ExternalId |
ID |
Dpriv |
AzRoleName |
DestinationServiceName |
DirectoryServiceName |
Suid |
UserGuid |
Device Custom String 1 |
UserType |
Device Custom String1 Label |
‘userType’ |
Device Custom String2 |
AuthorityID |
Device Custom String2 Label |
‘authorityID’ |
Device Custom String3 |
AuthorityName |
Device Custom String3 Label |
‘authorityName’ |
Device Custom String4 |
AzRoleId |
Device Custom String4 Label |
‘azRoleId’ |
Device Custom String5 |
DirectoryServiceNameLocalized |
Device Custom String5 Label |
‘directoryServiceNameLocalized’ |
Device Custom String6 |
CheckedOut |
Device Custom String6 Label |
‘checkedOut’ |
Device Custom Date1 |
WhenDueBack |
Device Custom Date1 Label |
‘whenDueBack’ |
EventType=Cloud.Core.Server.CpsTileLaunch
ArcSight CEF Field |
CP Event Field |
---|---|
RequestContext |
RequestUserAgent |
ExternalId |
ID |
Dpriv |
AzRoleName |
DestinationServiceName |
DirectoryServiceName |
Suid |
UserGuid |
Device Custom String 1 |
UserType |
Device Custom String1 Label |
‘userType’ |
Device Custom String2 |
ApplicationType |
Device Custom String2Label |
‘applicationType’ |
Device Custom String3 |
ApplicationName |
Device Custom String3Label |
‘applicationName’ |
Device Custom String4 |
ApplicationID |
Device Custom String4Label |
‘applicationId’ |
Device Custom String5 |
DirectoryServiceNameLocalized |
Device Custom String5Label |
‘directoryServiceNameLocalized’ |
Device Custom String6 |
InternalTrackingID |
Device Custom String6Label |
‘internalTrackingID’ |
EventType=Cloud.Core.AdaptiveMfa.RiskAnalysis
Only Common properties.
Alternate approach for creating the Common Extension Format (CEF)
In case you are using the CP REST APIs directly in your application and generating your own Cloud Suite syslog messages in a generic non-CEF format having key=value pairs separated by a delimiter, then ArcSight SmartConnector will need to be installed and configured to collect these Cloud Suite syslog.
These logs will need to be parsed into CEF format by creating ArcSight
FlexConnector, to enable Cloud Suite events to be usable for SIEM in ArcSight. The only downside to using a FlexConnector is that ArcSight does not officially certify it.