Configure Kylo & Global Search¶
Kylo supports Global search via a plugin-based design. Three plugins are provided out of the box:
- Elasticsearch (rest client) [default]
- Elasticsearch (transport client)
- Solr
Elasticsearch 5 support¶
Elasticsearch 5 is supported when using NiFi 1.3 (or later) and rest client. Kylo has been tested with version 5.5.1. Please refer to the rest client configuration for additional details.
Elasticsearch (rest client) [default]¶
- Plugin Jar:
- Name:
kylo-search-elasticsearch-rest-<version>.jar
- Default location:
/opt/kylo/kylo-services/plugin/
- Name:
- Plugin Property File:
- Name:
elasticsearch-rest.properties
- Default location:
/opt/kylo/kylo-services/conf/
- Name:
Steps to configure Kylo with Elasticsearch engine (using rest client) are below. Both Elasticsearch versions 2 and 5 are supported via this plugin.
Include
search-esr
profile in existing list of profiles in/opt/kylo/kylo-services/conf/application.properties
spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-esr
Ensure that the plugin is available in
/opt/kylo/kylo-services/plugin
. This comes out-of-the-box at this location by default. It should have ownership askylo:users
and permissions755
.kylo-search-elasticsearch-rest-<version>.jar
Note
It is recommended to have only one search plugin in the /opt/kylo/kylo-services/plugin/ directory. If there is another search plugin (for example, kylo-search-elasticsearch-<version>.jar), move it along with its property file to /opt/kylo/setup/plugins/<plugin-name>/ for later use.
Provide elasticsearch-rest properties
Update cluster properties in
/opt/kylo/kylo-services/conf/elasticsearch-rest.properties
if different from the defaults provided below.search.rest.host=localhost search.rest.port=9200
Create Kylo Indexes
Execute a script to create kylo indexes. If these already exist, Elasticsearch will report an
index_already_exists_exception
. It is safe to ignore this and continue. Change the host and port if necessary./opt/kylo/bin/create-kylo-indexes-es.sh localhost 9200 1 1
Restart Kylo Services
service kylo-services restart
Steps to import updated Index Text Service feed
- Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
2a. [Elasticsearch version 2] Pick the
index_text_service_elasticsearch.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
2b. [Elasticsearch version 5] [This requires NiFi 1.3 or later] Pick the
index_text_service_v2.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.3
- Leave Change the Category field blank (It defaults to System)
- Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
- (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
- Click Import Feed.
- Verify that the feed imports successfully.
Elasticsearch (transport client)¶
- Plugin Jar:
- Name:
kylo-search-elasticsearch-<version>.jar
- Default location:
/opt/kylo/setup/plugins/search-elasticsearch-transport-client/
- Name:
- Plugin Property File:
- Name:
elasticsearch.properties
- Default location:
/opt/kylo/setup/plugins/search-elasticsearch-transport-client/
- Name:
Steps to configure Kylo with Elasticsearch engine (using transport client) are below. Only Elasticsearch version 2 is supported via this plugin.
Include
search-es
profile in existing list of profiles in/opt/kylo/kylo-services/conf/application.properties
spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-es
Ensure that the plugin is available in
/opt/kylo/kylo-services/plugin
. The plugin comes out-of-the-box at another location/opt/kylo/setup/plugins/search-elasticsearch-transport-client/
. It should have ownership askylo:users
and permissions755
.kylo-search-elasticsearch-<version>.jar
Note
It is recommended to have only one search plugin in the /opt/kylo/kylo-services/plugin/ directory. If there is another search plugin (for example, kylo-search-solr-<version>.jar), move it along with its property file to to /opt/kylo/setup/plugins/<plugin-name>/ for later use.
Reference commands to get the plugin, and change ownership and permissions:
cp /opt/kylo/setup/plugins/search-elasticsearch-transport-client/kylo-search-elasticsearch-<version>.jar /opt/kylo/kylo-services/plugin/ cp /opt/kylo/setup/plugins/search-elasticsearch-transport-client/elasticsearch.properties /opt/kylo/kylo-services/conf/ cd /opt/kylo/kylo-services/plugin/ chown kylo:users kylo-search-elasticsearch-<version>.jar chmod 755 kylo-search-elasticsearch-<version>.jar cd /opt/kylo/kylo-services/conf/ chown kylo:users elasticsearch.properties chmod 755 elasticsearch.properties
Provide elasticsearch properties
Update cluster properties in
/opt/kylo/kylo-services/conf/elasticsearch.properties
if different from the defaults provided below.search.host=localhost search.clusterName=demo-cluster search.restPort=9200 search.transportPort=9300
Restart Kylo Services
service kylo-services restart
Steps to import updated Index Text Service feed
- Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
- Pick the
index_text_service_elasticsearch.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
- Leave Change the Category field blank (It defaults to System)
- Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
- (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
- Click Import Feed.
- Verify that the feed imports successfully.
Solr¶
- Plugin Jar:
- Name:
kylo-search-solr-<version>.jar
- Default location:
/opt/kylo/setup/plugins/search-solr/
- Name:
- Plugin Property File:
- Name:
solrsearch.properties
- Default location:
/opt/kylo/setup/plugins/search-solr/
- Name:
Kylo is designed to work with Solr (SolrCloud mode) and tested with v6.5.1. This configuration assumes that you already have a running Solr instance. You can also get it from the official download page.
Steps to configure Kylo with Solr are below:
Include
search-solr
profile in existing list of profiles in/opt/kylo/kylo-services/conf/application.properties
spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-solr
Ensure that the plugin is available in
/opt/kylo/kylo-services/plugin
. The plugin comes out-of-the-box at another location/opt/kylo/setup/plugins/search-solr/
. It should have ownership askylo:users
and permissions755
.kylo-search-solr-<version>.jar
Note
It is recommended to have only one search plugin in the /opt/kylo/kylo-services/plugin/ directory. If there is another search plugin (for example, kylo-search-elasticsearch-<version>.jar), move it along with its property file to /opt/kylo/setup/plugins/<plugin-name>/ for later use.
Reference commands to get the plugin, and change ownership and permissions:
cp /opt/kylo/setup/plugins/search-solr/kylo-search-solr-<version>.jar /opt/kylo/kylo-services/plugin/ cp /opt/kylo/setup/plugins/search-solr/solrsearch.properties /opt/kylo/kylo-services/conf/ cd /opt/kylo/kylo-services/plugin/ chown kylo:users kylo-search-solr-<version>.jar chmod 755 kylo-search-solr-<version>.jar cd /opt/kylo/kylo-services/conf/ chown kylo:users solrsearch.properties chmod 755 solrsearch.properties
Create a folder on the box where Kylo is running to store indexes for Kylo metadata. Ensure that Kylo can write to this folder.
Reference commands to create this folder and give full permissions:
mkdir /tmp/kylosolr chmod 777 /tmp/kylosolr
Provide solr properties
Update cluster properties in
/opt/kylo/kylo-services/conf/solrsearch.properties
if different from the defaults provided below. Thesearch.indexStorageDirectory
should match with the folder location created in previous step.search.host=localhost search.port=8983 search.indexStorageDirectory=/tmp/kylosolr search.zooKeeperPort=9983
Create collections in Solr that Kylo will use.
Reference commands:
bin/solr create -c kylo-datasources -s 1 -rf 1 bin/solr create -c kylo-data -s 1 -rf 1
Configure Kylo collections created in previous step via Admin UI
Reference steps:
- Navigate to Admin UI
Configure collection for datasources
Select
kylo-datasources
collection from the drop down on left nav areaClick Schema on bottom left of nav area
Click Add Field on top of right nav pane
- name: kylo_collection
- type: string
- default value: kylo-datasources
- index: no
- store: yes
Configure collection for data
Select
kylo-data
collection from the drop down on left nav areaClick Schema on bottom left of nav area
Click Add Field on top of right nav pane
- name: kylo_collection
- type: string
- default value: kylo-data
- index: no
- store: yes
Restart Kylo Services
service kylo-services restart
Steps to import updated Index Text Service feed
- Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
- Pick the
index_text_service_solr.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
- Leave Change the Category field blank (It defaults to System)
- Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
- (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
- Click Import Feed.
- Verify that the feed imports successfully.
Ensure that the box running Kylo can connect to the box running Solr (if they are on separate machines). If required, open up these ports:
- 8983
- 9983