Apache Knox First encounter

This post setups up an standalone Apache Knox 1.3. I do this to better understand and improve customer's Cloudera / CDP 7 setup where this Knox version is included.  

Note for high speed JDBC connections create complementary entry points using newly emerged Apache Kyuubi https://kyuubi.apache.org/

Download the following

Apache Knox 1.3
Apache Directory Studio

The Apache Directory Studio is an nice UI against LDAP servers. I unpacked it under /opt/ApacheDirectoryStudio/ and started it with java version 17 by altering the /opt/ApacheDirectoryStudio/ApacheDirectoryStudio.ini

-startup
plugins/org.eclipse.equinox.launcher_1.6.0.v20200915-1508.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.2.0.v20200915-1442
/studio-rcp/resources/icons/linux/studio.xpm
-vm
/usr/lib/jvm/java-17-openjdk/bin/java
-vmargs
-Dosgi.requiredJavaVersion=11

Now start the DEMO ldap included in Knox distribution. Later on you can add an topology (entrance point) that points to your own LDAP that works in tandem with this DEMO ldap until you shutdown the DEMO ldap.

cd knox-1.3.0/bin
./ldap.sh start

Use the Apache Directory Studio to inspect the build in DEMO ldap

Knox's DEMO ldap has Bind DN (username on LDAP lingo) "uid=admin,ou=people,dc=hadoop,dc=apache,dc=org" with the password "admin-password" as default password.

Browse around the DEMO ldap that looks like this

Now start the actual Knox service , first time a master secret needs to be generated.

NOTE: java 8 neccesary for the Knox service itself , I used "sudo update-alternatives --config java" and set it to java 8 before doing anything with Knox master secret creation or service start. Or simply hardcode the following line in knox-1.3.0/bin/knox-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre

With the right Java machine configure now run:

cd knox-1.3.0/bin
./knoxcli.sh  create-master --generate
./gateway start

Now enjoy the admin-ui at the url https://localhost:8443/gateway/manager/admin-ui/ with username: admin and password: admin-password

An Thriftserver instance is(was...) neccessary to access Hive or Spark using jdbc. To have jdbc via knox your thriftserver needs to be started in http mode.

spark/sbin/start-thriftserver.sh  --hiveconf hive.server2.thrift.http.port=10001 --hiveconf hive.server2.transport.mode=http --hiveconf hive.server2.http.endpoint=cliservice 

As noted in the intro text, for high speed JDBC connections create complementary entry points using newly emerged Apache Kyuubi https://kyuubi.apache.org/ This is an alternative to hive/spark "bundled" Thriftserver.

Here im adding an topology (connection point) to enable LDAP authenticated jdbc in knox. An Thriftserver (jdbc enabler infront of Spark/hive) needs to be running , se above "Setup Thriftserver"

To have knox access the LDAP it needs to authenticate itself , look out for these propertines in the topoplogy

main.ldapRealm.contextFactory.systemUsername=<LDAP SUBTREE ID>
main.ldapRealm.contextFactory.systemPassword=<LDAP ACOUNT PASSWORD>
main.ldapRealm.contextFactory.authenticationMechanism=simple

Put the file below in knox-1.3.0/conf/topologies/ldap.xml

<topology>

    <gateway>

        <provider>
            <role>authentication</role>
            <name>ShiroProvider</name>
            <enabled>true</enabled>
            <param>
                <!-- 
                session timeout in minutes,  this is really idle timeout,
                defaults to 30mins, if the property value is not defined,, 
                current client authentication would expire if client idles contiuosly for more than this value
                -->
                <name>sessionTimeout</name>
                <value>30</value>
            </param>
            <param>
                <name>main.ldapRealm</name>
                <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value>
            </param>
            <param>
                <name>main.ldapContextFactory</name>
                <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory</name>
                <value>$ldapContextFactory</value>
            </param>
            <param>
                <name>main.ldapRealm.userDnTemplate</name>
                <value>uid={0},ou=Users,dc=ignalina,dc=dk</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.url</name>
                <value>ldap://10.1.1.9:389</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                <value>simple</value>
            </param>
            <param>
                <name>urls./**</name>
                <value>authcBasic</value>
            </param>

            <param>
                <name>main.ldapRealm.contextFactory.systemPassword</name>
                <value>CHANGEME_LDAP_ACOUNT_PASSWORD</value>
            </param>

            <param>
                <name>main.ldapRealm.contextFactory.systemUsername</name>
                <value>cn=admin,dc=ignalina,dc=dk</value>
            </param>


        </provider>

        <provider>
            <role>identity-assertion</role>
            <name>Default</name>
            <enabled>true</enabled>
        </provider>

        <provider>
            <role>hostmap</role>
            <name>static</name>
            <enabled>true</enabled>
            <param>
                <name>localhost</name>
                <value>sandbox,sandbox.hortonworks.com</value>
            </param>
        </provider>

    </gateway>

    <service>
        <role>HIVE</role>
        <url>http://localhost:10001/cliservice</url>
        <param>
            <name>replayBufferSize</name>
            <value>8</value>
        </param>
    </service>

</topology>

Finally try to connect via knox

./beeline -n rickard -p bajsanka -u 'jdbc:hive2://localhost:8443/;ssl=true;sslTrustStore=ca.jks;trustStorePassword=bajsanka;transportMode=http;httpPath=gateway/ldap/hive'

Links
Apache Knox 1.3
Apache Directory Studio
Apache Knox 1.3 offical guide
Knox Tutorial