Release 0.8.3 (Aug 30, 2017)¶
Highlights¶
- Pluggable JMS implementation with out-of-the-box support for ActiveMQ and Amazon SQS. Refer to JMS Providers for details
- Pluggable REST client for Elasticsearch. This is now used by default in lieu of transport client.
- Cloudera Services Monitor as Kylo plugin. Refer to Service Monitor Plugins for details
- Business domain types for columns. Define rules to auto-apply domain types during feed creation or manually select the domain type to apply predefined standardization and validation rules.
- Column-level tagging. Apply tags to columns and search column tags using Global Search.
- Schema changes for column descriptions. The Hive schema is updated when modifying the column description of a feed. The column description is also available on the Visual Query page when hovering over a column name.
- Alerts improvement. User Interface enhancements and additional alerts capabilities. The Alerts page has been improved and the alerts on the dashboard are now in sync with the alerts page and adhere to entity access controls
- Category-level feed role memberships. Ability to manage feed access control of all feeds under a category by assigning feed role memberships at the category level
- Ability to query/filter Service Level Assessments against the Service Level Agreements
- IE & Safari browser support
- Elasticsearch 5 support
- New angular UI module plugin support. Ability to create entirely new user interface modules and plug them into the UI navigation. Refer to Custom Kylo Module
- Spark Jobserver processors for NiFi. Reuse a SparkContext between multiple Spark jobs for increased performance. Requires an existing Spark Jobserver.
- Pluggable Spark functions. Custom Spark functions can be added to the Visual Query page by providing a json file with the function definitions. Refer to Writing Spark Function Definitions.
- MS SQL support
- Maven Central support
Upgrade Instructions from v0.8.2¶
- Stop NiFi:
service nifi stop
- Uninstall Kylo:
/opt/kylo/remove-kylo.sh
- Install the new RPM:
rpm –ivh <RPM_FILE>
Copy the application.properties file from the 0.8.2 install. If you have customized the application.properties file you will want to copy the 0.8.2 version and add the new properties that were added for this release.
4.1 Find the /bkup-config/TIMESTAMP/kylo-services/application.properties file
- Kylo will backup the application.properties file to the following location, /opt/kylo/bkup-config/YYYY_MM_DD_HH_MM_millis/kylo-services/application.properties, replacing the “YYYY_MM_DD_HH_MM_millis” with a valid time:
4.2 Copy the backup file over to the /opt/kylo/kylo-services/conf folder
### move the application.properties shipped with the .rpm to a backup file mv /opt/kylo/kylo-services/conf/application.properties /opt/kylo/kylo-services/conf/application.properties.0_8_3_template ### copy the backup properties (Replace the YYYY_MM_DD_HH_MM_millis with the valid timestamp) cp /opt/kylo/bkup-config/YYYY_MM_DD_HH_MM_millis/kylo-services/application.properties /opt/kylo/kylo-services/conf
4.3 Add in the new properties to the /opt/kylo/kylo-services/conf/application.properties file
The following properties allow Kylo to inspect the database schema when creating database feeds
#Kylo MySQL controller service configuration nifi.service.kylo_mysql.database_user=root nifi.service.kylo_mysql.password=hadoop
Flow Aggregation Stats
##when getting aggregate stats back for flows if errors are detected kylo will query NiFi in attempt to capture matching bulletins. ## by default this data is stored in memory. Setting this to true will store the data in the MySQL table kylo.ops.mgr.stats.nifi.bulletins.persist=false ## if not perisiting (above flag is false) this is the limit to the number of error bulletins per feed. ## this is a rolling queue that will keep the last # of errors per feed kylo.ops.mgr.stats.nifi.bulletins.mem.size=30
New NiFi version 1.1 profile
Previous versions of Kylo were compatible with Nifi v110 when using the nifiv1.0 profile. If you are using NiFi v1.1 in your environment then going forward you should use the nifi-1.1 profile.
spring.profiles.include=<other-profiles-as-required>,nifi-v1.1
New configuration for JMS
Previous versions of Kylo did not have a profile based method of configured the queue services. With new SQS support, the profile must be stated explicitly. See section 8 for more info.
spring.profiles.include=<other-profiles-as-required>,jms-activemq
4.4 Ensure the property
security.jwt.key
in both kylo-services and kylo-ui application.properties file match. They property below needs to match in both of these files:/opt/kylo/kylo-ui/conf/application.properties
/opt/kylo/kylo-services/conf/application.properties
security.jwt.key=
Update the NiFi nars. Run the following shell script to copy over the new NiFi nars/jars to get new changes to NiFi processors and services.
/opt/kylo/setup/nifi/update-nars-jars.sh <NIFI_HOME> <KYLO_SETUP_FOLDER> <NIFI_LINUX_USER> <NIFI_LINUX_GROUP> Example: /opt/kylo/setup/nifi/update-nars-jars.sh /opt/nifi /opt/kylo/setup nifi users
Backup the Kylo database. Run the following code against your kylo database to export the ‘kylo’ schema to a file. Replace the PASSWORD with the correct login to your kylo database.
mysqldump -u root -pPASSWORD --databases kylo > kylo-0_8_2_backup.sql
- Database updates. Kylo uses liquibase to perform database updates. Two modes are supported.
Automatic updates
By default Kylo is set up to automatically upgrade its database on Kylo services startup. As such, there isn’t anything specific an end user has to do. When Kylo services startup the kylo database will be automatically upgraded to latest version if required. This is configured via an application.properties setting
liquibase.enabled=trueManual updates
Sometimes, however you may choose to disable liquibase and manually apply the upgrade scripts. By disabling liquibase you are in control of how the scripts are applied. This is needed if the kylo database user doesnt have priviledges to make schema changes to the kylo database. Please follow this Database Upgrades on how to manually apply the additional database updates.
Update NiFi to use default ActiveMQ JMS provider. Kylo now supports two JMS providers out-of-the-box: ActiveMQ and Amazon SQS. A particular provider is selected by active Spring profile in
/opt/nifi/ext-config/config.properties
.8.1. Edit
/opt/nifi/ext-config/config.properties
8.2. Add following line to enable ActiveMQ
spring.profiles.active=jms-activemq
Please follow this JMS Providers on how to switch active JMS Provider.
If using Elasticsearch as the search engine, go through steps 9.1 to 9.5. If using Solr, go to step 10 and also refer to Solr plugin section.
9.1. Modify Elasticsearch rest client configuration (if required) in
/opt/kylo/kylo-services/conf/elasticsearch-rest.properties
. The defaults are provided below.search.rest.host=localhost search.rest.port=9200
9.2. Verify
search-esr
profile in existing list of profiles in/opt/kylo/kylo-services/conf/application.properties
spring.profiles.include=<other-profiles-as-required>,search-esr
9.3. Create Kylo Indexes
Execute a script to create kylo indexes. If these already exist, Elasticsearch will report an
index_already_exists_exception
. It is safe to ignore this and continue. Change the host and port if necessary./opt/kylo/bin/create-kylo-indexes-es.sh localhost 9200 1 1
9.4. Import updated Index Text Service feed. This step should be done once Kylo services are started and Kylo is up and running.
9.4.1. [Elasticsearch version 2] Import the feed
index_text_service_elasticsearch.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.0
9.4.2. [Elasticsearch version 5] [This requires NiFi 1.3 or later] Import the feed
index_text_service_v2.feed.zip
file available at/opt/kylo/setup/data/feeds/nifi-1.3
9.5. For additional details, refer to this document under Rest Client section.
If using Solr as the search engine, go through steps 10.1 to 10.5. Also refer to Solr plugin section
10.1. Create the collection in Solr
bin/solr create -c kylo-datasources -s 1 -rf 1
10.2. Navigate to Solr’s Admin UI
10.3. Select the
kylo-datasources
collection from the drop down in the left nav area10.4. Click Schema on bottom left of nav area
10.5. Click Add Field on top of right nav pane
- name: kylo_collection
- type: string
- default value: kylo-datasources
- index: no
- store: yes
If Kerberos has been enabled in
spark.properties
then make the below edits and disable the kylo-spark-shell service. The service will be started as needed by kylo-services.# Changes for kylo-services/conf/spark.properties with Kylo 0.8.3 #spark.shell.server.host = localhost #spark.shell.server.port = 8450 spark.shell.deployMode = local
# RedHat: disable kylo-spark-shell service chkconfig kylo-spark-shell off # Debian: disable kylo-spark-shell service update-rc.d kylo-spark-shell disable
Start NiFi and Kylo
service nifi start /opt/kylo/start-kylo-apps.sh
- Migrate Hive schema indexing to Kylo. The indexing of Hive schemas is now handled internally by Kylo instead of using a special feed.
12.1. Remove the Register Index processor from the
standard_ingest
anddata_transformation
reusable templates12.2. Delete the Index Schema Service feed
- Import updated Index Text Service feed as mentioned in earlier step 9.4. At this point, Kylo should be up and running and hence 9.4 can be completed.