Release 0.8.0 (Apr 19, 2017)¶
- 90+ issues resolved
- Automatic and manual database upgrades. See Database Upgrades
- Support for PostgreSQL as Kylo metadata store
- Join Hive and any JDBC tables in Data Transformation feeds by creating a new Data Source.
- Wrangler can now use standardization and validation functions, and be merged, profiled, and indexed.
- The Feed/Template import provides visual feedback and progress
- Kylo will now encrypt ‘sensitive’ properties and enforce ‘required’ properties.
Upgrade Instructions from v0.7.1¶
Build or download the RPM
- Shut down NiFi:
service nifi stop
- Uninstall Kylo:
- Install the new RPM:
rpm –ivh <RPM_FILE>
Copy the application.properties file from the 0.7.1 install. If you have customized the application.properties file you will want to copy the 0.7.1 version and add the new properties that were added for this release.
4.1 Find the /bkup-config/TIMESTAMP/kylo-services/application.properties file
- Kylo will backup the application.properties file to the following location, /opt/kylo/bkup-config/YYYY_MM_DD_HH_MM_millis/kylo-services/application.properties, replacing the “YYYY_MM_DD_HH_MM_millis” with a valid time:
4.2 Copy the backup file over to the /opt/kylo/kylo-services/conf folder
### move the application.properties shipped with the .rpm to a backup file mv /opt/kylo/kylo-services/application.properties application.properties.0_8_0_template ### copy the backup properties (Replace the YYYY_MM_DD_HH_MM_millis with the valid timestamp) cp /opt/kylo/bkup-config/YYYY_MM_DD_HH_MM_millis/kylo-services/application.properties /opt/kylo/kylo-services/conf
4.3 Add in the 2 new properties to the /opt/kylo/kylo-services/conf/application.properties file
4.4 Ensure the property
security.jwt.keyin both kylo-services and kylo-ui application.properties file match. They property below needs to match in both of these files:
4.5 If using Spark 2 then add the following property to the /opt/kylo/kylo-services/conf/application.properties file
Backup the Kylo database. Run the following code against your kylp database to export the ‘kylo’ schema to a file. Replace the PASSWORD with the correct login to your kylo database.
mysqldump -u root -pPASSWORD --databases kylo >kylo-0_7_1_backup.sql
- Upgrade Kylo database:
/opt/kylo/setup/sql/mysql/kylo/0.8.0/update.sh localhost root <password or blank>
- Additional Database updates. Kylo now uses liquibase to perform database updates. Two modes are supported.
By default Kylo is set up to automatically upgrade its database on Kylo services startup. As such, there isn’t anything specific an end user has to do. When Kylo services startup the kylo database will be automatically upgraded to latest version if required.
Sometimes, however you may choose to disable liquibase and manually apply the upgrade scripts. By disabling liquibase you are in control of how the scripts are applied. This is needed if the kylo database user doesnt have priviledges to make schema changes to the kylo database. Please follow this Database Upgrades on how to manually apply the additional database updates.
Update the NiFi nars. Run the following shell script to copy over the new NiFi nars/jars to get new changes to NiFi processors and services.
Update the NiFi Templates.
- The Data Transformation template now allows you to apply standardization and validation rules to the feed. To take advantage of this you will need to import the new template. The new data transformation template can be found:If you import the new Data Transformation template, be sure to re-initialize your existing Data Transformation feeds if you update them.
Data Transformation Enhancement Changes¶
New to this release is the ability for the data wrangler to connect to various JDBC data sources, allowing you to join Hive tables with, for example, MySQL or Teradata. The JDBC drivers are automatically read from /opt/nifi/mysql/ when Kylo is starting up. When Kylo Spark Shell is run in yarn-client mode then these jars need to be added manually to the run-kylo-spark-shell.sh script:
/opt/kylo/kylo-services/bin/run-kylo-spark-shell.shand append –jars to the
spark-submitcommand-line:spark-submit --jars /opt/nifi/mysql/mariadb-java-client-1.5.7.jar ...
Additional driver locations can be added separating each location with a commaspark-submit --jars /opt/nifi/mysql/mariadb-java-client-1.5.7.jar,/opt/nifi/teradata/terajdbc4.jar ...
Ambari Service Monitor Changes¶
The Ambari Service Monitor is now a Kylo plugin jar. In order for Kylo to report status on Ambari services you will need to do the following:
Modify/Ensure the connection properties are setup. The ambari connection parameters need to be moved out of the main kylo-services application.properties to a new file called
Create a new file
/opt/kylo/kylo-services/conf/ambari.properties. Ensure the owner of the file is kylo
Add and configure the following properties in that file:
ambariRestClientConfig.host=127.0.0.1 ambariRestClientConfig.port=8080 ambariRestClientConfig.username=admin ambariRestClientConfig.password=admin ambari.services.status=HDFS,HIVE,MAPREDUCE2,SQOOP
cp /opt/kylo/setup/plugins/kylo-service-monitor-ambari-0.8.0.jar /opt/kylo/kylo-services/plugin/