Release 0.7.0 (Feb. 13, 2017)¶
- Renamed thinkbig artifacts to kylo
- Our REST API documentation has been updated and reorganized for easier reading. If you have Kylo running you can open http://localhost:8400/api-docs/index.html to view the docs.
- Kylo is now available under the Apache 2 open-source license. Visit our new GitHub page!
- Login to Kylo with our new form-based authentication. A logout option has been added to the upper-right menu in both the Feed Manager and the Operations Manager.
Upgrade Instructions from v0.6.0¶
Build or download the rpm.
- Shut down NiFi:
service nifi stop
useradd -r -m -s /bin/bash kylo
usermod -a -G hdfs kylo
/opt/thinkbig/remove-kylo-datalake-accelerator.sh to uninstall the RPM
- Install the new RPM:
rpm –ivh <RPM_FILE>
Migrate the “thinkbig” database schema to “kylo”.
Kylo versions 0.6 and below use the thinkbig schema in MySQL. Starting from version 0.7, Kylo uses the kylo schema. This guide is intended for installations that are already on Kylo 0.6, and want to upgrade to Kylo 0.7. It outlines the procedure for migrating the current thinkbig schema to kylo schema, so that Kylo 0.7 can work.
6a. Uninstall Kylo 0.6 (Refer to deployment guide and release notes for details).
6b. Install Kylo 0.7 (Refer to deployment guide and release notes for details).
Do not yet start Kylo services.
6c. Log into MySQL instance used by Kylo, and list the schemas:
mysql> show databases
6d. Verify that:
- thinkbig schema exists
- kylo schema does not exist
6e. Navigate to Kylo’s setup directory for MySQL.
6f. Execute the migration script. It takes 3 parameters. For no password, provide the 3rd parameter as ‘’../migrate-schema-thinkbig-to-kylo-mysql.sh <host> <user> <password>
- Step 1 of migration: kylo schema is set up.
- Step 2 of migration: thinkbig schema is migrated to kylo schema.
6g. Start Kylo services. Verify that Kylo starts and runs successfully. At this point, there are two schemas in MySQL: kylo and thinkbig.
Once Kylo is running normally and migration is verified, the thinkbig schema can be dropped.
6h. Navigate to Kylo’s setup directory for MySQL.
6i. Execute the script to drop thinkbig schema. It takes 3 parameters. For no password, provide the 3rd parameter as:
../drop-schema-thinkbig-mysql.sh <host> <user> <password>
6j. Verify that only kylo schema now exists in MySQL.
mysql> show databases
This completes the migration procedure.
Update the database:
/opt/kylo/setup/sql/mysql/kylo/0.7.0/update.sh localhost root <password or blank>
/opt/nifi/current/conf/bootstrap.confChange “java.arg.15=Dthinkbig.nifi.configPath=/opt/nifi/ext-config” to “java.arg.15=Dkylo.nifi.configPath=/opt/nifi/ext-config”.
mv /opt/thinkbig/bkup-config /opt/kylo chown -R kylo:kylo bkup-config
mv /opt/thinkbig/encrypt.key /opt/kyloIf prompted for overwrite, answer ‘yes’.
chown kylo:kylo /opt/kylo/encrypt.key
- Copy the mariadb driver to access MySQL database.
> cp /opt/kylo/kylo-services/lib/mariadb-java-client-*.jar /opt/nifi/mysql > chown nifi:users /opt/nifi/mysql/mariadb-java-client-*.jar
- Start NiFi (wait to start):
service nifi start
- In the standard-ingest template, update the”Validate and Split Records” processor and change the ApplicationJAR value to:
- In the standard-ingest template update the”Profile Data” processor and change the ApplicationJAR value to:
- For the MySQL controller service (type: DBCPConnectionPool), update the properties to use the mariadb driver:
- Database Driver Class Name: org.mariadb.jdbc.Driver
- Database Driver Location(s): file:///opt/nifi/mysql/mariadb-java-client-1.5.7.jar
- For the JMSConnectionFactoryProvider controller service, set the MQ Client Libraries path property value to:
- For the StandardSqoopConnectionService, copy the value of Source Driver to Source Driver (Avoid providing value) then delete the Source Driver property.
- Update, with your custom configuration, the configuration files at:
/opt/kylo/kylo-ui/conf/, /opt/kylo/kylo-services/conf/ /opt/kylo/kylo-spark shell/conf/A backup of the previous version’s configuration is available from /opt/kylo/bkup-config/.
- Modify both of the metadata controller services in NiFi with the new REST endpoint.
- The first one should be under the root process group and is used by our processors. The REST Client URL property should be changed to http://localhost:8400/proxy/v1/metadata.
- The second is under the right-hand menu and is used by our reporting task. The REST Client URL property should be changed to http://localhost:8400/proxy/v1/metadata.
- If using NiFi v0.7 or earlier, modify:
/opt/kylo/kylo-services/conf/application.propertiesChange spring.profiles.active from nifi-v1 to nifi-v0.
- Modify permissions to allow existing NiFi flows to use /tmp/kylo directory.
After re-importing data_ingest.zip in a later step, any new feeds created will use the /tmp/kylo-nifi folder. The below command will allow existing flows to continue using the /tmp/kylo folder.
> chmod 777 /tmp/kylo
- Start kylo apps:
- Re-import the data_ingest.zip template. (New feeds will use the temp location /tmp/kylo-nifi.)
- (Optional) If unused, the mysql driver in /opt/nifi/mysql can be deleted.
> rm /opt/nifi/mysql/mysql-connector-java-*.jar