Build
Overview
To install SunbirdED release-5.0.0
, we will need to run many Jenkins jobs and a couple of manual steps. Some of the jobs need to be run in a specific order. This page details out the list of jobs, their order and the github release tags to be used for each job.
Create Plugin Containers / Folder Structure
We will create few folders in azure for storing sunbird plugins
Our deployment job always tries to delete the folder first before uploading, so we need to have the placeholders before running the jobs
We will create the folders by uploading random files from a random folder on your PC
Create a public container with the name you have set for the variable
sunbird_content_azure_storage_container
and run the below commands
Upload the default T&C file to
sunbird_content_azure_storage_container
container in a folder named terms-and-conditions. You can get a copy of the sample T&C file from here. You are free to edit the HTML as per your requirementsCreate a private container named
label
in AzureRun the below command in your Jenkins VM where you cloned the sunbird-devops repo
Upload Initial Plugins
Upload the initial set of plugins to
sunbird_content_azure_storage_container
containerDownload the initial Plugins from here
Run the below command from the directory where content-plugins.zip is unzipped
Create a container named
public
in your Azure storage account from the Azure portal with public access levelCreate a container named
artifacts
in your Azure storage account from the Azure portal with private access level
Upload Maxmind Database to Azure Blob
Download the Maxmind city database in zip format from Maxmind website
Upload the zip file to the
artifacts
container in Azure
Code Builds
Note:
Jenkins will have many jobs which may not be documented or is not suitable for adoption yet
Jobs listed on this page is sufficient to setup Sunbird and not every job present in Jenkins is required to be run
Jobs can be run in parallel to speed up execution. The only exception is you should NOT run multiple jobs which clone the code from same repo in parallel
If you get errors in some of the build jobs, rerun the job again
The errors are usually due to missing maven jars or maven repo timeouts
If you get errors even after re-running the job, come back to it later, there are other jobs which would generate dependent jars
Build/Core/AdminUtils
release-4.5.0_RC3
Handles mobile device registration, refresh token
Build/Core/Analytics
release-4.8.5_RC2
Location Register, Location Retrieval
Build/Core/APIManager
release-4.10.0_RC1
The API Gateway
Build/Core/ApiManagerEcho
master-RC1
Simply echos the request
Build/Core/Assessment
release-5.0.0_RC1
Handles course assessments
Build/Core/Bot
release-3.9.0_RC1
Frontend for the chatbot
Build/Core/Cassandra
release-4.10.0_RC1
Handles the Cassandra database schema
Build/Core/CassandraTrigger
release-4.6.0_RC1
Creates few Cassandra Triggers
Build/Core/Cert
release-5.0.0_RC2
Handles course certificates
Build/Core/CertRegistryService
release-5.0.0_RC1
Helper for Certificate service
Build/Core/Content
release-5.0.0_RC1
Handles the Contents and Content Metadata
Build/Core/Dial
release-5.0.0_RC1
Handles QR code generation
Build/Core/DiscussionsMiddleware
release-5.0.0_RC2
Middleware for discussion forum
Build/Core/EncService
release-3.8.0_RC1
Encryption service to setup certificate signing keys
Build/Core/Groups
release-5.0.0_RC2
Handles groups functions
Build/Core/Keycloak
release-3.8.0_RC1
User authentication service
Build/Core/KnowledgeMW
release-4.8.0_RC2
Middleware for Content service
Build/Core/Learner
release-5.0.0_RC2
Handles user and organizations
Build/Core/Lms
release-5.0.0_RC2
Handles courses
Build/Core/Nodebb
github_release_tag: release-5.0.0_RC1, nodebb_branch: v1.18.6
Handles the discussion forum
Build/Core/Notification
release-5.0.0_RC2
Handles notifications
Build/Core/Player
release-5.0.0_RC18
Handles the UI elements
Build/Core/Print
release-3.8.0_RC1
Handles PDF generation of certificates
Build/Core/Proxy
release-3.8.0_RC14
Nginx web server / proxy
Build/Core/Report
release-4.10.0_RC6
Handles some of the reports
Build/Core/Router
release-3.9.0_RC1
Backend for chatbot
Build/Core/Search
release-5.0.0_RC1
Handles contents / course search
Build/Core/Taxonomy
release-5.0.0_RC1
Handles the taxonomies
Build/Core/Telemetry
release-3.3.0_RC1
Handles the telemetry generated
Build/Core/Yarn
release-2.6.0
Handles SSO and account merges
Build/KnowledgePlatform/CassandraTrigger
release-3.8.0_RC18
Creates few Cassandra Triggers
Build/KnowledgePlatform/FlinkJobs
release-5.0.0_RC3
Multiple functions like search indexing, video streaming
Build/KnowledgePlatform/Learning
github_release_tag: release-5.0.0_RC1, profile_id: platform-services
Handles frameworks
Build/KnowledgePlatform/Neo4j
release-4.5.0_RC12
Generates neo4j plugins
Build/KnowledgePlatform/SyncTool
release-4.5.0_RC12
Generates a jar to manually sync contents
Build/KnowledgePlatform/Yarn
release-5.0.0_RC1
Multiple functions like publishing contents, issuing certificates
Build/DataPipeline/AdhocScripts
release-4.5.1_RC1
Generates supporting scripts for reporting
Build/DataPipeline/AnalyticsCore
release-5.0.0_RC1
Generates core libraries for reporting
Build/DataPipeline/CoreDataProducts
release-5.0.0_RC1
Generates supporting libraries for reporting
Build/DataPipeline/EdDataProducts
release-5.0.0_RC2
Generates supporting libraries for reporting
Build/DataPipeline/ETLJobs
release-4.10.5_RC2
Extract, Transform, Data Loading jobs
Build/DataPipeline/FlinkPipelineJobs
release-4.9.0_RC2
Handles processing of telemetry data
Build/DataPipeline/Secor
secor-0.29_RC2
Handlles backup of telemetry data to Azure Blob
Build/Lern/LernDataProducts
release-5.0.0_RC3
New Lern BB jobs
Build/Lern/LernFlinkJobs
release-5.0.0_RC4
New Lern BB jobs
Build/Plugins/CollectionEditor
release-3.6.0_RC1
A bunch of plugins to handle collection editing
Build/Plugins/ContentEditor
release-5.0.0_RC3
A bunch of plugins to edit certain types of content
Build/Plugins/ContentPlayer
release-4.6.0_RC1
A bunch of plugins to handle content playback
Build/Plugins/ContentPlugins
release-5.0.0_RC1
A bunch of base plugins to support the content editors and content player
Build/Plugins/GenericEditor
release-5.0.0_RC3
A bunch of plugins to edit certain types of content
Infra Provision
Note:
Jobs can be run in parallel to speed up execution. The only exception is you should NOT run multiple jobs which will provision packages on the same server in parallel
OpsAdministration/Core/Bootstrap
hosts: env, branch_or_tag: release-5.0.0_RC2 , tag: bootstrap_any,node_exporter
Creates the deployer user and installs node exporter and python packages for ansible
OpsAdministration/KnowledgePlatform/Bootstrap
hosts: env, branch_or_tag: release-5.0.0_RC2, tag: bootstrap_any,node_exporter
Creates the deployer user and installs node exporter and python packages for ansible
OpsAdministration/DataPipeline/Bootstrap
hosts: env, branch_or_tag: release-5.0.0_RC2, tag: bootstrap_any,node_exporter
Creates the deployer user and installs node exporter and python packages for ansible
Provision/Core/ApplicationElasticSearch
release-5.0.0_RC2
Installs Elasticsearch used by the applications
Provision/Core/Cassandra
release-5.0.0_RC2
Installs Cassandra database
Provision/Core/CassandraExporter
release-5.0.0_RC2
Installs Cassandra prometheus exporter
Provision/Core/ESMapping
indices_name: all, branch_or_tag: release-5.0.0_RC2
Creates Elasticsearch indices and mappings
Provision/Core/Keycloak
release-5.0.0_RC2
Installs java and other pre-requisites for Keycloak service
Provision/Core/LogES
release-5.0.0_RC2
Installs Elasticsearch used to store application and VM logs
Provision/Core/MongodbCluster
release-5.0.0_RC2
Installs Mongo database
Provision/Core/Postgres
release-5.0.0_RC2
Installs Postgres database
Provision/Core/PostgresDbUpdate
release-5.0.0_RC2
Creates Postgres tables, schema and users
Provision/KnowledgePlatform/Learning
release-5.0.0_RC2
Install tomcat and other pre-requisites for Learning service
Provision/KnowledgePlatform/Neo4j
release-5.0.0_RC2
Installs Neo4j database
Provision/KnowledgePlatform/Yarn
release-5.0.0_RC2
Install Hadoop and other pre-requisites to run Yarn
Provision/DataPipeline/AnalyticsGeoLocationDBSetup
release-5.0.0_RC1, geoip_zip_csv_file_name: Enter the maxmind city database zip file name you uploaded to artifacts
container
Creates tables and schema in Postgres
Provision/DataPipeline/AnalyticsSpark
release-5.0.0_RC1
Install Apache Spark and other pre-requisites for reporting
Provision/DataPipeline/Druid
release-5.0.0_RC1, service: Select All, remote: raw
Installs Druid
Provision/DataPipeline/InfluxDB
release-5.0.0_RC1
Install Influx DB used by monitoring system
Provision/DataPipeline/Kafka
release-5.0.0_RC1
Installs Kafka
Provision/DataPipeline/KafkaIndexer
release-5.0.0_RC1
Install Logstash to read telemetry data from Kafka and index into Log ES
Provision/DataPipeline/Postgres
release-5.0.0_RC1
Installs Postgres database and users
Provision/DataPipeline/PostgresDbUpdate
release-5.0.0_RC1
Creates Postgres tables, schema and users
Provision/DataPipeline/Redis
release-5.0.0_RC1
Install Redis
Provision/DataPipeline/Zookeeper
release-5.0.0_RC1
Installs Zookeeper
ArtifactUpload
Every job in the Build directory has a corresponding job in ArtifactUpload directory with the same name
These jobs are auto triggered and usually run without issues after the corresponding job in the build directory succeeds
The job will fail if your ansible inventory setup is incorrect or incomplete
If the job has failed, fix the ansible variables issue and rerun the job to upload the artifact / docker image
Ensure the artifact upload jobs are successful before proceeding
Code Deploy
Note:
We will run the jobs which are a pre-requisite for other jobs first
The pre-requisite jobs reside in different folders so we will be jumping across folders
Jobs in the Deploy directory can be run in parallel and don’t have any restrictions if the code is from same repo unlike the Build directory
Ensure you don’t run those jobs in parallel which modify the databases (such as cassandra, neo4j etc)
Deploy/Kubernetes/BootstrapMinimal
release-5.0.0_RC2
Creates namespaces, configmaps and secrets
Deploy/DataPipeline/BootstrapMinimal
release-5.0.0_RC1
Creates namespaces, configmaps and secrets
Deploy/Kubernetes/Monitoring
branch_or_tag: release-5.0.0_RC2, tag: all
Deploys the monitoring stack which can be accessed via DOMAIN/grafana/ post nginx-public-ingress deployment
Deploy/Kubernetes/CassandraTrigger
release-5.0.0_RC2
Installs the Cassandra triggers
Deploy/KnowledgePlatform/CassandraTrigger
release-5.0.0_RC2
Installs the Cassandra triggers
Deploy/Kubernetes/CassandraDBUpdate
release-5.0.0_RC2
Creates / Updates Cassandra DB schema
Deploy/KnowledgePlatform/CassandraDbUpdate
release-5.0.0_RC2
Creates / Updates Cassandra DB schema
Deploy/Kubernetes/Cassandra
release-5.0.0_RC2
Creates / Updates Cassandra DB schema
Deploy/Kubernetes/KafkaSetup
release-5.0.0_RC2
Creates / Updates Kafka Topics
Deploy/DataPipeline/KafkaSetup
release-5.0.0_RC1
Creates / Updates Kafka Topics
Deploy/KnowledgePlatform/KafkaSetup
release-5.0.0_RC2
Creates / Updates Kafka Topics
Deploy/KnowledgePlatform/Learning
release-5.0.0_RC2
Deploys the Learning service on tomcat
Deploy/KnowledgePlatform/Neo4j
release-5.0.0_RC2
Deploys the Neo4j plugins
Deploy/KnowledgePlatform/Neo4jDefinitionUpdate
release-5.0.0_RC2
Creates / Updates the Neo4j database
Deploy/KnowledgePlatform/Yarn
release-5.0.0_RC2
Deploys the Samza jobs
Deploy/KnowledgePlatform/FlinkJobs
branch_or_tag: release-5.0.0_RC2, job_names_to_deploy: Select All
Deploys the Flink jobs in Kubernetes cluster
Deploy/KnowledgePlatform/LoggingFileBeatsVM
branch_or_tag: release-5.0.0_RC2, hosts: Select All, tags: default
Deploys filebeat in all the selected VMs
Deploy/Kubernetes/nginx-private-ingress
release-5.0.0_RC2
Deploys nginx with a private load balancer
Deploy/Kubernetes/APIManager
release-5.0.0_RC2
Deploys Kong, the API gateway
Deploy/Kubernetes/OnboardAPIs
release-5.0.0_RC2
Creates the APIs in Kong
Deploy/Kubernetes/OnboardConsumers
release-5.0.0_RC2
Creates the consumers in Kong. Now read the comment mentioned in Core/Secrets.yml against the variable core_vault_sunbird_api_auth_token
Deploy/Kubernetes/Keycloak
release-5.0.0_RC2
Deploys Keycloak
Deploy/Kubernetes/KeycloakRealm
release-5.0.0_RC2
Creates the Sunbird Realm. Now read the comment mentioned in Core/Secrets.yml against the variables core_vault_sunbird_sso_publickey
, adminutil_refresh_token_public_key_kid
, adminutil_refresh_token_secret_key
Deploy/Kubernetes/BootstrapMongodb
release-5.0.0_RC2
Creates the initial token in Mongo database
Deploy/Kubernetes/CertTemplate
branch_or_tag: release-5.0.0_RC2, sunbird_util_branch_or_tag: release-4.10.0_RC1
Uploads the certificate templates into Azure blob
Deploy/Kubernetes/UploadChatbotConfig
branch_or_tag: release-5.0.0_RC2, bot_repo_branch: master
Uploads the chatbot configs to Azure blob
Deploy/Kubernetes/UploadFAQs
branch_or_tag: release-5.0.0_RC2, source_folder: Select All
Uploads the FAQs to Azure blob
Deploy/Kubernetes/UploadSchema
branch_or_tag: release-5.0.0_RC2, kp_branch_or_tag: release-4.9.0_RC8
Uploads the Content schemas to Azure blob
Deploy/Kubernetes/AdminUtils
release-5.0.0_RC2
Deploys Adminutils service
Deploy/Kubernetes/Analytics
release-5.0.0_RC2
Deploys Analytics service
Deploy/Kubernetes/APIManagerEcho
release-5.0.0_RC2
Deploys Echo service
Deploy/Kubernetes/Assessment
release-5.0.0_RC2
Deploys Assessment service
Deploy/Kubernetes/Bot
release-5.0.0_RC2
Deploys Bot service
Deploy/Kubernetes/Cert
release-5.0.0_RC2
Deploys Cert service
Deploy/Kubernetes/CertRegistry
release-5.0.0_RC2
Deploys CertRegistry service
Deploy/Kubernetes/Content
release-5.0.0_RC2
Deploys Content service
Deploy/Kubernetes/Dial
release-5.0.0_RC2
Deploys Dial service
Deploy/Kubernetes/Enc
release-5.0.0_RC2
Deploys Enc service
Deploy/Kubernetes/Groups
release-5.0.0_RC2
Deploys Groups service
Deploy/Kubernetes/KnowledgeMW
release-5.0.0_RC2
Deploys KnowledgeMW service
Deploy/Kubernetes/Learner
release-5.0.0_RC2
Deploys Learner service
Deploy/Kubernetes/Lms
release-5.0.0_RC2
Deploys Lms service
Deploy/Kubernetes/Notification
release-5.0.0_RC2
Deploys Notification service
Deploy/Kubernetes/Player
release-5.0.0_RC2
Deploys Player service
Deploy/Kubernetes/Print
release-5.0.0_RC2
Deploys Print service
Deploy/Kubernetes/Report
release-5.0.0_RC2
Deploys Report service
Deploy/Kubernetes/Router
release-5.0.0_RC2
Deploys Router service
Deploy/Kubernetes/Search
release-5.0.0_RC2
Deploys Search service
Deploy/Kubernetes/Taxonomy
release-5.0.0_RC2
Deploys Taxonomy service
Deploy/Kubernetes/Telemetry
release-5.0.0_RC2
Deploys Telemetry service
Deploy/Kubernetes/Yarn
release-5.0.0_RC2
Deploys Samza jobs
Deploy/Kubernetes/LoggingFileBeatsVM
branch_or_tag: release-5.0.0_RC2, hosts: Select All, tags: default
Deploys filebeat in all the selected VMs
Deploy/Kubernetes/Logging
release-5.0.0_RC2
Choose one chartname each time - fluent-bit
, kibana
, oauth2-proxy
. Deploys Fluent Bit, Kibana and Oauth2 Proxy respectively. Kibana can accessed via DOMAIN/dashboard/
Deploy/Kubernetes/nginx-public-ingress
release-5.0.0_RC2
Deploys the nginx web server. Now you can open visit your domain
Deploy/Kubernetes/Nodebb
release-5.0.0_RC2
Deploys NodeBB service. Now read the comment mentioned in Core/Secrets.yml against the variable discussionsmw_nodebb_authorization_token
Deploy/Kubernetes/DiscussionsMW
release-5.0.0_RC2
Deploys DiscussionsMW service
Deploy/Plugins/CollectionEditor
release-5.0.0_RC2
Deploys CollectionEditor files to Azure blob
Deploy/Plugins/ContentEditor
release-5.0.0_RC2
Deploys ContentEditor files to Azure blob
Deploy/Plugins/ContentPlayer
release-5.0.0_RC2
Deploys ContentPlayer files to Azure blob
Deploy/Plugins/ContentPlugins
release-5.0.0_RC2
Deploys ContentPlugins files to Azure blob
Deploy/Plugins/GenericEditor
release-5.0.0_RC2
Deploys GenericEditor files to Azure blob
Deploy/DataPipeline/AdhocScripts
release-5.0.0_RC1
Deploys AdHoc scripts in Spark VM
Deploy/DataPipeline/AnalyticsCore
release-5.0.0_RC1
Deploys jars in Spark VM
Deploy/DataPipeline/CoreDataProducts
release-5.0.0_RC1
Deploys jars in Spark VM
Deploy/DataPipeline/DruidIngestion
branch_or_tag: release-5.0.0_RC1, ingestion_task_names: Select All
Deploys and starts Druid ingestion tasks. This job will be updated in a couple of days. Please wait until then
Deploy/DataPipeline/EdDataProducts
release-5.0.0_RC1
Deploys jars in Spark VM
Deploy/DataPipeline/ETLJobs
release-5.0.0_RC1
Deploys ETL scripts in Spark VM
Deploy/DataPipeline/ETLDruidContentIndexer
release-5.0.0_RC1
Deploys ETLDruidContentIndexer
Deploy/DataPipeline/ETLUserCacheIndexer
release-5.0.0_RC1
Deploys ETLUserCacheIndexer
Deploy/DataPipeline/Secor
release-5.0.0_RC1
Deploys Secor Statefulsets in Kubernetes cluster
Deploy/DataPipeline/FlinkPipelineJobs
release-5.0.0_RC1
Deploys Flink jobs Kubernetes cluster
Deploy/DataPipeline/GraphitePrometheusExporter
release-5.0.0_RC1
Deploys graphite prometheus exporter in Druid VM
Deploy/Kubernetes/LoggingFileBeatsVM
branch_or_tag: release-5.0.0_RC2, hosts: Select All, tags: default
Deploys filebeat in all the selected VMs
Last updated