Configure Kylo & Global Search¶
Kylo supports Global search via a plugin-based design. Two plugins are provided out of the box:
- Elasticsearch (default)
- Solr
Elasticsearch¶
Steps to configure Kylo with Elasticsearch engine are below:
Include
search-es
profile in existing list of profiles in/opt/kylo/kylo-services/conf/application.properties
spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-es
Ensure that the plugin is available in
/opt/kylo/kylo-services/plugin
. This comes out-of-the-box at this location by default. It should have ownership askylo:users
and permissions755
.kylo-search-elasticsearch-0.8.2.jar
Note
There should be only one search plugin in the above plugin directory. If there is another search plugin (for example, kylo-search-solr-0.8.2.jar), move it to /opt/kylo/setup/plugins/ for later use.
Reference commands to get the plugin, and change ownership and permissions:
cp <location-where-you-saved-the-default-plugin-jar-named kylo-search-elasticsearch-0.8.2.jar> /opt/kylo/kylo-services/plugin/ cd /opt/kylo/kylo-services/plugin/ chown kylo:users kylo-search-elasticsearch-0.8.2.jar chmod 755 kylo-search-elasticsearch-0.8.2.jar
Provide elasticsearch properties
Update cluster properties in
/opt/kylo/kylo-services/conf/elasticsearch.properties
if different from the defaults provided below.search.host=localhost search.clusterName=demo-cluster search.restPort=9200 search.transportPort=9300
Restart Kylo Services
service kylo-services restart
Steps to import updated Index Schema Service feed
- Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
- Pick the
index_schema_service_elasticsearch.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
- Leave Change the Category field blank (It defaults to System)
- Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
- (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
- Click Import Feed.
- Verify that the feed imports successfully.
- If your Hive metastore is in a schema named something other than
hive
, edit the feed and sethive.schema
to the schema name (if not already properly set). This configuration value may be available with the keyconfig.hive.schema
in/opt/kylo/kylo-services/conf/application.properties
. For Cloudera CDH, Hive metastore is generally namedmetastore
.
Steps to import updated Index Text Service feed
- Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
- Pick the
index_text_service_elasticsearch.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
- Leave Change the Category field blank (It defaults to System)
- Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
- (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
- Click Import Feed.
- Verify that the feed imports successfully.
Solr¶
Kylo is designed to work with Solr (SolrCloud mode) and tested with v6.5.1. This configuration assumes that you already have a running Solr instance. You can also get it from the official download page.
Steps to configure Kylo with Solr are below:
Include
search-solr
profile in existing list of profiles in/opt/kylo/kylo-services/conf/application.properties
spring.profiles.include=native,nifi-v1,auth-kylo,auth-file,search-solr
Ensure that the plugin is available in
/opt/kylo/kylo-services/plugin
. The plugin comes out-of-the-box at another location/opt/kylo/setup/plugins
. It should have ownership askylo:users
and permissions755
.kylo-search-solr-0.8.2.jar
Note
There should be only one search plugin in the /opt/kylo/kylo-services/plugin directory. If there is another search plugin (for example, kylo-search-elasticsearch-0.8.2.jar), move it to /opt/kylo/setup/plugins/ for later use.
Reference commands to get the plugin, and change ownership and permissions:
mv /opt/kylo/kylo-services/plugin/kylo-search-elasticsearch-0.8.2.jar /opt/kylo/setup/plugins/ cp /opt/kylo/setup/plugins/kylo-search-solr-0.8.2.jar /opt/kylo/kylo-services/plugin/ cd /opt/kylo/kylo-services/plugin/ chown kylo:users kylo-search-solr-0.8.2.jar chmod 755 kylo-search-solr-0.8.2.jar
Create a folder on the box where Kylo is running to store indexes for Kylo metadata. Ensure that Kylo can write to this folder.
Reference commands to create this folder and give full permissions:
mkdir /tmp/kylosolr chmod 777 /tmp/kylosolr
Provide solr properties
Update cluster properties in
/opt/kylo/kylo-services/conf/solrsearch.properties
if different from the defaults provided below. Thesearch.indexStorageDirectory
should match with the folder location created in previous step.search.host=localhost search.port=8983 search.indexStorageDirectory=/tmp/kylosolr
Create collections in Solr that Kylo will use.
Reference commands:
bin/solr create -c kylo-schema-metadata -s 1 -rf 1 bin/solr create -c kylo-data -s 1 -rf 1
Configure Kylo collections created in previous step via Admin UI
Reference steps:
- Navigate to Admin UI
Configure collection for schema
Select
kylo-schema-metadata
collection from the drop down on left nav areaClick Schema on bottom left of nav area
Click Add Field on top of right nav pane
- name: kylo_collection
- type: string
- default value: kylo-schema-metadata
- index: no
- store: yes
Configure collection for data
Select
kylo-data
collection from the drop down on left nav areaClick Schema on bottom left of nav area
Click Add Field on top of right nav pane
- name: kylo_collection
- type: string
- default value: kylo-data
- index: no
- store: yes
Restart Kylo Services
service kylo-services restart
Steps to import updated Index Schema Service feed
- Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
- Pick the
index_schema_service_solr.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
- Leave Change the Category field blank (It defaults to System)
- Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
- (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
- Click Import Feed.
- Verify that the feed imports successfully.
- If your Hive metastore is in a schema named something other than
hive
, edit the feed and sethive.schema
to the schema name (if not already properly set). This configuration value may be available with the keyconfig.hive.schema
in/opt/kylo/kylo-services/conf/application.properties
. For CDH, Hive metastore is generally namedmetastore
.
Steps to import updated Index Text Service feed
- Feed Manager -> Feeds -> + orange button -> Import from file -> Choose file
- Pick the
index_text_service_solr.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
- Leave Change the Category field blank (It defaults to System)
- Click Yes for these two options (1) Overwrite Feed (2) Replace Feed Template
- (optional) Click Yes for option (3) Disable Feed upon import only if you wish to keep the indexing feed disabled upon import (You can explicitly enable it later if required)
- Click Import Feed.
- Verify that the feed imports successfully.
Ensure that the box running Kylo can connect to the box running Solr (if they are on separate machines). If required, open up these ports:
- 8983
- 9983