Cloudera Enterprise 5.15.x | Other versions

Configuring Audit and Log Properties

Cloudera Manager Required Role: Navigator Administrator (or Full Administrator)

Each service (with exceptions noted) that supports auditing has the following properties:
  • Enable Audit Collection - See Enabling Audit Collection and Enabling Solr Auditing.
  • Audit Log properties - See Configuring Audit Logs and Configuring Impala Daemon Logging.
  • Audit Event Filter - A set of rules that capture properties of auditable events and actions to be performed when an event matches those properties. The Cloudera Manager Agent uses this property to filter events out before they are sent to Cloudera Navigator. The default filter settings discard the following events:
    • HDFS
      • Generated by the internal Cloudera and Hadoop users (cloudera-scm, hbase, hive, impala, mapred, solr, spark, and dr.who)
      • Generated by the hdfs user running the listStatus, listCachePools, listCacheDirectives, and getfileinfo operations
      • That affect files in the /tmp directory
    • HBase - That affect the -ROOT-, .META., and acl tables
    • Hive - Generated by Hive MapReduce jobs in the /tmp directory
    • Impala, Solr, Solr, Navigator Metadata Server - No default filter
  • Audit Event Tracker - A set of rules for tracking and coalescing events. This feature is used to define equivalency between different audit events. Tracking works by keeping a reference to events when they first appear, and comparing other incoming events against the tracked events according to the rules defined. When events match, according to a set of configurable parameters, only one entry in the audit list is generated for all the matching events. This property is not supported for the Navigator Metadata Server.
  • Audit Queue Policy - The action to take when the audit event queue is full. The options are Drop or Shutdown. When a queue is full and the queue policy of the service is Shutdown, before shutting down the service, N audits will be discarded, where N is the size of the Cloudera Navigator Audit Server queue.
      Note: If the queue policy is Shutdown, the Impala service is shut down only if Impala is unable to write to the audit log file. It is possible that an event may not appear in the audit event log due to an error in transfer to the Cloudera Manager Agent or database. In such cases Impala will not shut down and will keep writing to the log file. When the transfer problem is fixed the events will be transferred to the database.
    This property is not supported for Hue or the Navigator Metadata Server.

The Audit Event Filter and Audit Event Tracker rules for filtering and coalescing events are expressed as JSON objects. You can edit these rules using a rule editor. The username filters shown here include an optional phrase ((?:/.+)?) to filter events where the user name is constructed with a slash as part of the name, such as a Kerberos principal (username/instance@REALM). It applies only if the regular expression matches one of the usernames listed in the first phrase. The src filter shows a similar optional phrase ((?:/.*)?) that filters for subdirectories under the /tmp directory. The regular expression looks for occurrences of slash followed by zero or more characters; this makes sure that subdirectories are included but other directories whose name starts with "tmp" are not included in the filter.

or in a JSON text field:

For information on the structure of the objects, and the properties for which you can set filters, display the description on the configuration page as follows:
  1. Log in to the Cloudera Manager Admin Console.
  2. Select select any service that supports auditing. Select Cluster > Service-Name
  3. Click the Configuration tab.
  4. Select Service (Service-Wide) for the Scope filter.
  5. Select Cloudera Navigator for the Category filter.
  6. Click for Audit Event Tracker to open the property description.
    For example, the Hive properties are:
    • operation: the Hive operation being performed.
    • userName: the user performing the action.
    • ipAddress: the IP from where the request originated.
    • allowed: whether the operation was allowed or denied.
    • databaseName: the databaseName for the operation.
    • tableName: the tableName for the operation.
    • objectType: the type of object affected by the operation.
    • resourcePath: the path of the resource affected by the operation.

Configuring Service Auditing Properties

To configure service and role auditing properties, perform the procedure below. In addition, for Impala and Solr auditing, perform the steps in Configuring Impala Daemon Logging and Enabling Solr Auditing and for all services perform the steps in Configuring Audit Logs.

For Navigator Metadata Server auditing properties:

  1. Select Clusters > Cloudera Management Service.
  2. Click the Configuration tab.
  3. Select Navigator Metadata Server for the Scope filter.
  4. Select Category > Cloudera Navigator category.
  5. Edit the properties.
  6. Click Save Changes to commit the changes.
  7. Restart the service.

For other services' auditing properties:

  1. Go to a supported service.
  2. Click the Configuration tab.
  3. Select Service (Service-Wide) for the Scope filter.
  4. Select Category > Cloudera Navigator category.
  5. Edit the properties.
  6. Click Save Changes to commit the changes.
  7. Restart the service.

Audit Filter Examples

Use audit filters to control what information is collected by Navigator Audit Server. The following sections describe cases where a filter can allow you to remove noise from the audit events or otherwise improve the quality of the audit trail.

Removing audit events for automated tasks

If you find that the audit events include references to automated tasks against HDFS or other services that you don't want to include in your audit, you can include the tasks in a "discard" filter. For example, to discard all tasks performed against HDFS by the user under which the automated tasks are performed, create an HDFS filter:

Action: Discard
Fields: username: (?:automationuser|automationuser@realm.example.com)

This rule would discard any event performed by the automationuser account, where the user is specified by itself or with a Kerberos realm designation. Note that the filter is not case sensitive. As a performance optimization, the expression is included in parentheses with ?: to indicate that the matched values are not retained.

Removing audit events for low priority operations

You may find that your audit log includes reference to events that you are not interested in tracking. You can prevent these events from being included in the audit log by filtering them out by operation. For example, if the audit event that you don't want to track looks like this:

2170816,hdfs,1,someuser@MYREALM.COM,null,192.168.12.18,1477605498525,getfileinfo,
/user/someuser/.staging/jobx/status,null,null,null

where the HDFS operation getfileinfo ran against the file status for the user someuser on a node in the cluster. The user account is indicated with a Kerberos realm.

To remove the HDFS getfileinfo accesses by this user, the audit filter would reference the user (by itself or with a Kerberos realm name) and the getfileinfo operation:

Action: Discard
Fields: username: (?:username|username@MYREALM.com)
        operation: (?:getfileinfo)

As a performance optimization, the expressions are included in parentheses with ?: to indicate that the matched values are not retained.

To include more than one operation in the same filter, include both operations with an OR operator |. :

Action: Discard
Fields: username: (?:username|username@MYREALM.com)
        operation: (?:getfileinfo|liststatus)
Page generated May 18, 2018.