Hkube

Hkube Cloud Native Open Source Framework for Distributed Pipeline Of Algorithm Built Top of Kubernetes

  • HKube provides an easy way for algorithm developers to simplify the deployment and execution of such pipelines.
  • HKube is tightly coupled to Kubernetes, utilizing many of its features, as well as other CNCF projects, to achieve its design goals.
  • Algorithms in HKube are deployed seamlessly using HKube’s automatic builder. These algorithms are used to create a pipeline, a DAG that can be defined with YAML schema or code.
  • Algorithms in the pipeline can run as a batch of the same algorithm in order to accelerate the performance.
  • HKube will optimize hardware resource utilization by automatically prioritize workload based user priorities and other metrics collected from previous runs. HKube mixes critical and best-effort workloads in order to drive up utilization and save resources and money.
  • HKube helps developers debug their code by allowing running a pipeline on a local machine, debug a single algorithm on your IDE, and by integrating with Jupyter Notebook.
  • The rich UI dashboard provides an easy and intuitive way to define and run algorithms and pipeline, monitor progress and get results.

Features:

  • Distributed pipeline of algorithms
    • Receives DAG graph as input and automatically parallelizes your algorithms over the cluster.
    • Manages the complications of distributed processing, keep your code simple (even single threaded).
  • Language Agnostic – As a container based framework designed to facilitate the use of any language for your algorithm.
  • Batch Algorithms – Run algorithms as a batch – instances of the same algorithm in order to accelerate the running time.
  • Optimize Hardware Utilization
    • Containers automatically placed based on their resource requirements and other constraints, while not sacrificing availability.
    • Mixes critical and best-effort workloads in order to drive up utilization and save resources.
    • Efficient execution and clustering by heuristics which uses pipeline and algorithm metrics with combination of user requirements.
  • Build API – Just upload your code, you don’t have to worry about building containers and integrating them with HKube API.
  • Cluster Debugging
    • Debug a part of a pipeline based on previous results.
    • Debug a single algorithm on your IDE, while the rest of the algorithms running in the cluster.
  • Jupyter Integration – Scale your jupyter running tasks Jupyter with hkube.

Installation

Dependencies

HKube runs on top of Kubernetes so in order to run HKube we have to install it’s prerequisites.

Helm

  1. Add the HKube Helm repository to helm:
  2. Configure a docker registry for builds
    Create a values.yaml file for custom helm values
helm repo add hkube http://hkube.io/helm/
  1. Install HKube chart
build_secret:
 pull secret is only needed if docker hub is not accessible
 pull:
     registry: ''
     namespace: ''
     username: ''
     password: ''
 enter your docker hub / other registry credentials
 push:
     registry: '' # can be left empty for docker hub
     namespace: '' # registry namespace - usually your username
     username: ''
     password: ''
  • Install HKube chart
helm install hkube/hkube  -f ./values.yaml --name my-release

This command installs HKube in a minimal configuration for development. Check production-deployment.

APIs

There are three ways to communicate with HKube: DashboardREST API and CLI.

UI Dashboard

Dashboard is a web-based HKube user interface. Dashboard supports every functionality HKube has to offer.

REST API

HKube exposes it’s functionality with REST API.

CLI

hkubectl is HKube command line tool.

hkubectl [type] [command] [name]
 More information
 hkubectl --help

Download hkubectl latest version.

Check release page for latest version
 latestVersion="v1.1.28"
 curl -Lo hkubectl https://github.com/kube-HPC/hkubectl/releases/download/${latestVersion}/hkubectl-linux \
 && chmod +x hkubectl \

For mac replace with hkubectl-macos For Windows download hkubectl-win.exe

Config hkubectl with your running Kubernetes.

Config
 hkubectl config set endpoint ${KUBERNETES-MASTER-IP}
 hkubectl config set rejectUnauthorized false

Make sure kubectl is configured to your cluster.

HKube requires that certain pods will run in privileged security permissions, consult your Kubernetes installation to see how it’s done.

HKube – Dashborad – http://playground.hkube.io

Leave a Reply

Your email address will not be published. Required fields are marked *