Developer Getting Started Guide¶
This guide should help you get your local development environment up and running quickly. Development in an IDE is usually done in conjunction with a Hortonworks sandbox in order to have a cluster with which to communicate.
Dependencies¶
To run the Kylo project locally the following tools must be installed:
- Maven 3
- RPM (for install)
- Java 1.8 (or greater)
- Hadoop 2.3+ Sandbox
- Virtual Box or other virtual machine manager
The assumption is that you are installing on a Mac or Linux box. You can do most activities below on a Windows box, except to perform a Maven build with the RPM install. At some point, we could add a Maven profile to allow you to build but skip the final RPM step.
Install Maven 3¶
This project requires Maven to execute a build. Use this link to download to the Maven installation file:
Note
For instructions on installing Apache Maven see the Installing Apache Maven docs at the Apache Maven project site.
Optional - Add Java 8 to Bash Profile¶
To build from the command line, you need to add Java 8 and Maven to your $PATH variable.
Edit ~/.bashrc and add the following:
export MVN_HOME=/Users/<HomeFolderName>/tools/apache-maven-3.3.3
export MAVEN_OPTS="-Xms256m -Xmx512m"
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home
export PATH=$JAVA_HOME/bin:$MVN_HOME/bin:$PATH
To test, run the following:
$ mvn -v
$ java -version
Install the RPM Tool on your Mac¶
The RPM library is required for building the RPM file as part of the Maven build. This can be done using Home Brew or Mac Ports.
$ brew install rpm
Clone Project from Github¶
Clone the Kylo project to your host. You can do this in your IDE or from the command line.
- From the command line,run the “git clone” command.
- cd to the directory you want to install the project to.
- Type “git clone *https://github.com/kyloio/kylo.git”*.
- Import from your IDE using the “*https://github.com/kyloio/kylo.git*” URL.
Import the Project into your IDE¶
Import the project into your favorite IDE as a Maven project.
Note
Configure the project to use Java 8.
Perform a Maven Build¶
Perform a Maven build to download all of the artifacts and verify that everything is setup correctly.
$ mvn clean install
Note
If you receive an OutOfMemoryError try increasing the Java heap space:
$ export MAVEN_OPTS="-Xms2g -Xmx4g"
Tip
For faster Maven builds you can run in offline mode and skip unit testing:
$ mvn clean install -o -DskipTests
Install and Configure the Hortonworks Sandbox¶
Follow the guide below to install and configure the Hortonworks sandbox:
Install the Kylo Applications¶
To install the Kylo apps, NiFi, ActiveMQ, and Elasticsearch in the VM you can use the deployment wizard instructions found here:
Instead of downloading the RPM file copy the RPM file from your project folder after running a Maven build.
$ cd /opt
$ cp /media/sf_kylo/install/target/rpm/kylo/RPMS/noarch/kylo-<version>.noarch.rpm.
$ rpm -ivh kylo-<version>.rpm
Follow the rest of the deployment wizard steps to install the rest of the tools in the VM.
You now have a distribution of the stack running in your Hortonworks sandbox.
Running in the IDE¶
You can run kylo-ui and thinkbig-services in the IDE. If you plan to run the apps in the IDE, you should shut down the services in your sandbox so you aren’t running two instances at the same time.
$ service kylo-services stop
$ service kylo-ui stop
The applications are configured using Spring Boot.
IntelliJ Configuration¶
Install the Spring Boot plugin.
Create the kylo-services application run configuration.
- Open the Run configurations.
- Create a new Spring Boot run configuration.
- Give it a name like “KyloServerApplication”.
- Set “use classpath of module” property to “kylo-service-app” module.
- Set the “Main Class” property to “com.thinkbiganalytics.server.KyloServerApplication”.
- Add “dev” to list of Active Profiles.
- Add a file named “application-dev.properties” to kylo-service-app at kylo/services/service-app/src/main/resources location. Populate properties to override from the standard “application.properties”. Some key properties are:
security.entity.access.controlled=<value> spring.datasource.username=<value> spring.datasource.password=<value> hive.datasource.username=<value> hive.metastore.datasource.username=<value> hive.metastore.datasource.password=<value> modeshape.datasource.username=${spring.datasource.username} modeshape.datasource.password=${spring.datasource.password} nifi.service.kylo_mysql.database_user=<value> nifi.service.kylo_mysql.password=<value> nifi.service.kylo_metadata_service.rest_client_password=<value> modeshape.index.dir=<value>
Create the kylo-ui application run configuration.
- Open the Run configurations.
- Create a new Spring Boot run configuration.
- Give it a name like “KyloDataLakeUiApplication”.
- Set “use classpath of module” property to “kylo-ui-app” module.
- Set the “Main Class” property to “com.thinkbiganalytics.KyloUiApplication”.
Run both applications.
Eclipse Configuration¶
Open Eclipse.
Import the Kylo project.
- File - Import
- Choose “maven” and “Existing Maven Projects” then choose next
- Choose the Kylo root folder. You should see all Maven modules checked
- Click finish
- Import takes a bit - if you get an error about scala plugin, just click finish to ignore it.
Find and open the “com.thinkbiganalytics.server.KyloServerApplication” class.
Right click and choose to debug as a Java application.
Repeat for “com.thinkbiganalytics.KyloUiApplication”.
OPTIONAL: Install the spring tools suite and run as a spring boot option
Note
Consult the Spring Boot documentation for Running Your Application for additional ways to run with spring boot.
Web Development¶
Most of the Kylo UI depends on AngularJS and AngularJS Material but a few parts have been upgraded to Angular 2 and Covalent. New plugins should be written in Typescript and use Angular 2 for future compatibility.
NPM should be used to configure and start your web development environment:
Install NPM in your development environment:
- apt-get install npm (Debian / Ubuntu)
- brew install npm (Mac)
Install the development packages:
$ cd kylo/ui/ui-app
$ npm install
- Start Kylo and the development server:
$ service kylo-services start
$ service kylo-ui start
$ npm run start
- A new browser window will open showing the Kylo UI. Any changes you make will automatically refresh the page with the new changes.
If you will be editing stylesheets then you will need to additionally configure IntelliJ to compile SCSS files:
- Install the File Watchers plugin and restart IntelliJ.
- Go to Preferences -> Tools -> File Watchers.
- Click the + plus sign at the bottom and create a SCSS file watcher.
- Change the Arguments to:
--output $FileDir$ $FilePath$
- Change the Output paths to refresh to:
$FileNameWithoutExtension$.css
Angular Material Notes¶
There are a few notes worth mentioning about using AngularJS Material:
- Do not use
layout-row
andlayout-wrap
with percents. It has been broken on Safari for a while now with current plan to be fixed only in Angular 4.x. - Do not refer to Angular model in plain HTML
style
element, it is broken on IE. Instead use Angularng-style
element which works on all browsers like song-style="{'fill':controller.fillColor}"
- Do not use
flex
element where you don’t have to. Browsers will usually flex elements correctly. This is to minimise the occurrence offlex
being required by Safari while breaking layout on IE.