Cloudera Enterprise 5.15.x | Other versions

Flume MorphlineSolrSink Configuration Options

The MorphlineSolrSink is a Flume sink used to ingest and index documents into Cloudera Search. For Cloudera Manager environments, the Flume agent is configured using procedures described in Configuring the Flume Agents. For unmanaged environments, you can use the standard configuration file flume.conf to configure Flume agents, including their sources, sinks, and channels. For more information about flume.conf, see the Flume User Guide.

Flume Morphline SolrSink supports the following configuration options (required options in bold):

Property Name	Default	Description
`type`		Must be set to the fully qualified class name (FQCN) `org.apache.flume.sink.solr. morphline.MorphlineSolrSink`.
`channel`		Specifies the channel to use for transferring records. For more information, see Flume Channels in the Flume User Guide.
`morphlineFile`		The location of the morphline configuration file. For Cloudera Manager deployments, use: `<agentName>`.sinks.`<sinkName>`. morphlineFile=morphlines.conf For unmanaged deployments, provide the relative or absolute path on the local filesystem to the morphline configuration file. For example, `/etc/flume-ng/conf/ tutorialReadAvroContainer.conf`
`batchSize`	100	The maximum number of Flume events per transaction. The transaction is committed when the specified `batchSize` or `batchDurationMillis` is reached, whichever comes first.
`batchDurationMillis`	1000	The maximum duration for a transaction, in milliseconds. The transaction is committed when the specified `batchSize` or `batchDurationMillis` is reached, whichever comes first.
`indexerClass`	`org.apache.flume.sink.solr. morphline.MorphlineSolrIndexer`	The FQCN of a class implementing `org.apache.flume.sink.solr.morphline.SolrIndexer`.
`morphlineId`	`null`	The name of the morphline to use when there is more than one morphline in a morphline configuration file.

This example shows a section for a MorphlineSolrSink named solrSink for an agent named agent:

agent.sinks.solrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
agent.sinks.solrSink.channel = memoryChannel
agent.sinks.solrSink.batchSize = 100
agent.sinks.solrSink.batchDurationMillis = 1000
agent.sinks.solrSink.morphlineFile = /etc/flume-ng/conf/morphline.conf
agent.sinks.solrSink.morphlineId = morphline1

Note: This example uses a Flume MemoryChannel to easily get started. For production environments, it is often more appropriate to configure a Flume FileChannel instead, which is a high performance transactional persistent queue.

Page generated May 18, 2018.