Setup Guide

Pre-requisites :

  • Core Python

  • Pyspark

  • Druid

  • Kafka

  • MongoDB

System Requirements :

  • Java version 1.8. 0 or More

  • Python 3.6

  • Druid

  • Kafka

How to set up

First clone project

git clone

Checkout latest version or repo

  • install python and pip

  • add alias python=python3.8 in ~/.bashrc file

  • Refresh with source .bashrc command [Try python -V && pip -V to confirm the installation]

  • Install virtual environment using pip install virtual env command or sudo apt install python3.8-venv command

  • Create a virtual environment using the python -m venv spark_venv command

  • Activate virtual environment using source spark_venv/bin/activate command

  • Install the required dependency using sudo pip install -r requirements.txt

  • Use the pip list command to recheck if the dependencies are installed correctly

  • Create a config.ini file and add config paths ( In the ml-analytics dir )

Last updated