Build
Overview
To install SunbirdED release-5.2.0, we will need to run many Jenkins jobs and a couple of manual steps. Some of the jobs need to be run in a specific order. This page details out the list of jobs, their order and the github release tags to be used for each job.
Create Plugin Containers / Folder Structure
We will create few folders in object storage for storing sunbird plugins
Our deployment job always tries to delete the folder first before uploading, so we need to have the placeholders before running the jobs
Create a public object storage with the name you have set for the variable
cloud_storage_content_bucketnameUnder
cloud_storage_content_bucketnamecreate folders with below names.collection-editor
generic-editor
content-editor
v3/preview (under v3 folder create preview folder)
Upload the default T&C file to
cloud_storage_content_bucketnamein a folder named terms-and-conditions. You can get a copy of the sample T&C file from here. You are free to edit the HTML as per your requirementsCreate another object store with the name you have set for the variable
cloud_storage_label_bucketnameGo to cloned sunbird-devops repo and navigate to
cd sunbird-devops/utils/portaland upload all files tocloud_storage_label_bucketname
Upload Initial Plugins
Upload the initial set of plugins to
cloud_storage_content_bucketnamecontainerDownload the initial Plugins from here
Unzip content-plugins.zip and upload all the files and folders to
cloud_storage_content_bucketname/content-pluginsobject storageCreate a object storage with name you have set for the variable
cloud_storage_public_bucketnamewith public access levelCreate a object storage with name you have set for the variable
cloud_storage_artifacts_bucketnamewith private access level
Upload Maxmind Database to Object storage
Download the Maxmind city database in zip format from Maxmind website
Upload the zip file to the
cloud_storage_artifacts_bucketnameobject storage
Build source code
Note:
Jenkins will have many jobs which may not be documented or is not suitable for adoption yet
Jobs listed on this page is sufficient to setup Sunbird and not every job present in Jenkins is required to be run
Jobs can be run in parallel to speed up execution. The only exception is you should NOT run multiple jobs which clone the code from same repo in parallel
If you get errors in some of the build jobs, rerun the job again
The errors are usually due to missing maven jars or maven repo timeouts
If you get errors even after re-running the job, come back to it later, there are other jobs which would generate dependent jars
| JENKINS JOB TO RUN | GITHUB TAG | GITHUB REPO | COMMENTS |
|---|---|---|---|
Build/Core/AdminUtils | release-4.5.0_RC3 | Handles mobile device registration, refresh token | |
Build/Core/Analytics | release-5.1.0_RC2 | Location Register, Location Retrieval | |
Build/Core/APIManager | release-4.10.0_RC1 | The API Gateway | |
Build/Core/ApiManagerEcho | master-RC1 | Simply echos the request | |
Build/Core/Assessment | release-5.2.0_RC2 | Handles course assessments | |
Build/Core/Bot | release-3.9.0_RC1 | Frontend for the chatbot | |
Build/Core/Cassandra | release-4.10.0_RC1 | Handles the Cassandra database schema | |
Build/Core/CassandraTrigger | release-4.6.0_RC1 | Creates few Cassandra Triggers | |
Build/Core/Cert | release-5.0.0_RC2 | Handles course certificates | |
Build/Core/CertRegistryService | release-5.0.0_RC1 | Helper for Certificate service | |
Build/Core/Content | release-5.2.0_RC2 | Handles the Contents and Content Metadata | |
Build/Core/Dial | release-5.2.0_RC1 | Handles QR code generation | |
Build/Core/DiscussionsMiddleware | release-5.0.0_RC2 | Middleware for discussion forum | |
Build/Core/EncService | release-3.8.0_RC1 | Encryption service to setup certificate signing keys | |
Build/Core/Groups | release-5.0.0_RC2 | Handles groups functions | |
Build/Core/Keycloak | release-3.8.0_RC1 | User authentication service | |
Build/Core/KnowledgeMW | release-5.2.0_RC1 | Middleware for Content service | |
Build/Core/Learner | release-5.1.0_RC2 | Handles user and organizations | |
Build/Core/Lms | release-5.1.0_RC2 | Handles courses | |
Build/Core/Nodebb | github_release_tag: release-5.0.0_RC1, nodebb_branch: v1.18.6 | Handles the discussion forum | |
Build/Core/Notification | release-5.0.0_RC2 | Handles notifications | |
Build/Core/Player | release-5.2.0_RC11 | Handles the UI elements | |
Build/Core/Print | release-3.8.0_RC1 | Handles PDF generation of certificates | |
Build/Core/Proxy | release-3.8.0_RC14 | Nginx web server / proxy | |
Build/Core/Report | release-4.10.0_RC6 | Handles some of the reports | |
Build/Core/Router | release-3.9.0_RC1 | Backend for chatbot | |
Build/Core/Search | release-5.0.0_RC2 | Handles contents / course search | |
Build/Core/Taxonomy | release-5.2.0_RC3 | Handles the taxonomies | |
Build/Core/Telemetry | release-3.3.0_RC1 | Handles the telemetry generated | |
Build/KnowledgePlatform/CassandraTrigger | release-3.8.0_RC18 | Creates few Cassandra Triggers | |
Build/KnowledgePlatform/FlinkJobs | release-5.2.1_RC1 | Multiple functions like search indexing, video streaming | |
Build/KnowledgePlatform/Learning | github_release_tag: release-5.2.0_RC3, profile_id: platform-services | Handles frameworks | |
Build/KnowledgePlatform/Neo4j | release-4.1.0_RC2 | Generates neo4j plugins | |
Build/KnowledgePlatform/SyncTool | release-4.5.0_RC12 | Generates a jar to manually sync contents | |
Build/DataPipeline/AdhocScripts | release-4.5.1_RC1 | Generates supporting scripts for reporting | |
Build/DataPipeline/AnalyticsCore | release-5.0.0_RC2 | Generates core libraries for reporting | |
Build/DataPipeline/CoreDataProducts | release-5.0.0_RC1 | Generates supporting libraries for reporting | |
Build/DataPipeline/EdDataProducts | release-5.0.0_RC2 | Generates supporting libraries for reporting | |
Build/DataPipeline/ETLJobs | release-4.10.5_RC2 | Extract, Transform, Data Loading jobs | |
Build/DataPipeline/FlinkPipelineJobs | release-4.9.0_RC2 | Handles processing of telemetry data | |
Build/DataPipeline/Secor | secor-0.29_RC2 | Handlles backup of telemetry data to Object storage | |
Build/Lern/LernDataProducts | release-5.1.0_RC1 | Lern BB data products | |
Build/Lern/LernFlinkJobs | release-5.1.1_RC1 | Lern BB jobs | |
Build/managed-learn/ml-core-service | release-5.2.0_RC3 | ||
Build/managed-learn/ml-project-service | release-5.2.0_RC4 | ||
Build/managed-learn/ml-report-service | release-5.2.0_RC2 | ||
Build/managed-learn/ml-survey-service | release-5.2.0_RC1 | ||
Build/Sunbird-RC/CertificateApi | github_release_tag: release-5.1.0 dockerhub_image_source: dockerhub/sunbird-rc-certificate-api:v0.0.11 | ||
Build/Sunbird-RC/CertificateSign | github_release_tag: release-5.1.0 dockerhub_image_source: dockerhub/sunbird-rc-certificate-signer:v0.0.11 | ||
Build/Sunbird-RC/Registry | github_release_tag: release-5.1.0 dockerhub_image_source: dockerhub/sunbird-rc-core:v0.0.13 | ||
Build/UCI/Utils | release-5.1.0 | ||
Build/UCI/message-rosa | release-5.1.0 | ||
Build/UCI/dao | release-5.1.0 | ||
Build/UCI/Adapter | release-5.1.0 | ||
Build/UCI/Inbound | release-5.1.0 | ||
Build/UCI/odk | release-4.4.0_RC1 | ||
Build/UCI/Outbound | release-5.1.0 | ||
Build/UCI/Orchestrator | release-5.1.0 | ||
Build/UCI/Transformer | release-5.1.0 | ||
Build/UCI/UCI | release-5.0.0_RC3 | ||
Build/Plugins/CollectionEditor | release-3.6.0_RC1 | A bunch of plugins to handle collection editing | |
Build/Plugins/ContentEditor | release-5.0.0_RC3 | A bunch of plugins to edit certain types of content | |
Build/Plugins/ContentPlayer | release-4.6.0 | A bunch of plugins to handle content playback | |
Build/Plugins/ContentPlugins | release-5.2.0_RC1 | A bunch of base plugins to support the content editors and content player | |
Build/Plugins/GenericEditor | release-5.0.0_RC3 | A bunch of plugins to edit certain types of content |
Service Provision
Note:
Jobs can be run in parallel to speed up execution. The only exception is you should NOT run multiple jobs which will provision packages on the same server in parallel
| JENKINS JOB TO RUN | GITHUB TAG | GITHUB REPO | COMMENTS |
|---|---|---|---|
OpsAdministration/Core/Bootstrap | hosts: env, branch_or_tag: release-5.1.0 , tag: bootstrap_any,node_exporter | Creates the deployer user and installs node exporter and python packages for ansible | |
OpsAdministration/KnowledgePlatform/Bootstrap | hosts: env, branch_or_tag: release-5.1.0, tag: bootstrap_any,node_exporter | Creates the deployer user and installs node exporter and python packages for ansible | |
OpsAdministration/DataPipeline/Bootstrap | hosts: env, branch_or_tag: release-5.1.0, tag: bootstrap_any,node_exporter | Creates the deployer user and installs node exporter and python packages for ansible | |
Provision/Core/ApplicationElasticSearch | release-5.1.0 | Installs Elasticsearch used by the applications | |
Provision/Core/Cassandra | release-5.1.0 | Installs Cassandra database | |
Provision/Core/CassandraExporter | release-5.1.0 | Installs Cassandra prometheus exporter | |
Provision/Core/ESMapping | indices_name: all, branch_or_tag: release-5.1.0, es_files: indices,mappings,pipelines | Creates Elasticsearch indices and mappings | |
Provision/Core/Keycloak | release-5.1.0 | Installs java and other pre-requisites for Keycloak service | |
Provision/Core/LogES | release-5.1.0 | Installs Elasticsearch used to store application and VM logs | |
Provision/Core/MongodbCluster | release-5.1.0 | Installs Mongo database | |
Provision/Core/Postgres | release-5.1.0 | Installs Postgres database | |
Provision/Core/PostgresDbUpdate | release-5.1.0 | Creates Postgres tables, schema and users | |
Provision/Core/Graylog | release-5.1.0 tag: all | Installs Graylog, MongoDB and OpenJDK | |
OpsAdministration/Core/GraylogMongoImport | release-5.1.0 tag: all | Installs Graylog dashboards, inputs and other configurations | |
OpsAdministration/Core/GraylogGeoIP | release-5.1.0 graylog_geoip_compressed_file_name: geoip file name in graylog_geoip_file_name: geoip file name within the | Installs the Geo IP database on Graylog. The | |
Provision/KnowledgePlatform/Learning | release-5.2.0 | Install tomcat and other pre-requisites for Learning service | |
Provision/KnowledgePlatform/Neo4j | release-5.2.0 | Installs Neo4j database | |
Provision/DataPipeline/AnalyticsGeoLocationDBSetup | release-5.1.0, geoip_zip_csv_file_name: Enter the maxmind city database zip file name you uploaded to | Creates tables and schema in Postgres | |
Provision/DataPipeline/AnalyticsSpark | release-5.1.0 | Install Apache Spark and other pre-requisites for reporting | |
Provision/DataPipeline/Druid | release-5.1.0, service: Select All, remote: raw | Installs Druid | |
Provision/DataPipeline/InfluxDB | release-5.1.0 | Install Influx DB used by monitoring system | |
Provision/DataPipeline/Kafka | release-5.1.0 | Installs Kafka | |
Provision/DataPipeline/KafkaIndexer | release-5.1.0 | Install Logstash to read telemetry data from Kafka and index into Log ES | |
Provision/DataPipeline/Postgres | release-5.1.0 | Installs Postgres database and users | |
Provision/DataPipeline/PostgresDbUpdate | release-5.1.0 | Creates Postgres tables, schema and users | |
Provision/DataPipeline/Redis | release-5.1.0 | Install Redis | |
Provision/DataPipeline/Zookeeper | release-5.1.0 | Installs Zookeeper |
Artifact Upload
Every job in the Build directory has a corresponding job in ArtifactUpload directory with the same name
These jobs are auto triggered and usually run without issues after the corresponding job in the build directory succeeds
The job will fail if your ansible inventory setup is incorrect or incomplete
If the job has failed, fix the ansible variables issue and rerun the job to upload the artifact / docker image
Ensure the artifact upload jobs are successful before proceeding
Service Deploy
Note:
We will run the jobs which are a pre-requisite for other jobs first
The pre-requisite jobs reside in different folders so we will be jumping across folders
Jobs in the Deploy directory can be run in parallel and don’t have any restrictions if the code is from same repo unlike the Build directory
Ensure you don’t run those jobs in parallel which modify the databases (such as cassandra, neo4j etc)
| JENKINS JOB TO RUN | GITHUB TAG | GITHUB REPO | COMMENTS |
|---|---|---|---|
Deploy/Kubernetes/BootstrapMinimal | release-5.1.0 | Creates namespaces, configmaps and secrets | |
Deploy/DataPipeline/BootstrapMinimal | release-5.1.0 | Creates namespaces, configmaps and secrets | |
Deploy/Kubernetes/Monitoring | branch_or_tag: release-5.1.0, tag: all | Deploys the monitoring stack which can be accessed via DOMAIN/grafana/ post nginx-public-ingress deployment | |
Deploy/Kubernetes/CassandraTrigger | release-5.1.0 | Installs the Cassandra triggers | |
Deploy/KnowledgePlatform/CassandraTrigger | release-5.1.0 | Installs the Cassandra triggers | |
Deploy/Kubernetes/CassandraDBUpdate | release-5.1.0 | Creates / Updates Cassandra DB schema | |
Deploy/KnowledgePlatform/CassandraDbUpdate | release-5.1.0 | Creates / Updates Cassandra DB schema | |
Deploy/Kubernetes/Cassandra | release-5.1.0 | Creates / Updates Cassandra DB schema | |
Deploy/Kubernetes/KafkaSetup | release-5.1.0 | Creates / Updates Kafka Topics | |
Deploy/DataPipeline/KafkaSetup | release-5.1.0 | Creates / Updates Kafka Topics | |
Deploy/KnowledgePlatform/KafkaSetup | release-5.2.0 | Creates / Updates Kafka Topics | |
Deploy/KnowledgePlatform/Learning | release-5.2.0 | Deploys the Learning service on tomcat | |
Deploy/KnowledgePlatform/Neo4j | release-5.2.0 | Deploys the Neo4j plugins | |
Deploy/KnowledgePlatform/Neo4jDefinitionUpdate | release-5.2.0 | Creates / Updates the Neo4j database | |
Deploy/KnowledgePlatform/FlinkJobs | branch_or_tag: release-5.2.1_RC1, job_names_to_deploy: Select All | Deploys the Flink jobs in Kubernetes cluster | |
Deploy/KnowledgePlatform/LoggingFileBeatsVM | branch_or_tag: release-5.1.0, hosts: Select All, tags: default | Deploys filebeat in all the selected VMs | |
Deploy/Kubernetes/nginx-private-ingress | release-5.1.0 | Deploys nginx with a private load balancer Note: Before deploying update | |
Deploy/Kubernetes/APIManager | release-5.1.0 | Deploys Kong, the API gateway | |
Deploy/Kubernetes/OnboardAPIs | release-5.1.0 | Creates the APIs in Kong | |
Deploy/Kubernetes/OnboardConsumers | release-5.1.0 | Deploy only after OnboardAPIs. Creates the consumers in Kong. Now read the comment mentioned in Core/Secrets.yml against the variable | |
Deploy/Kubernetes/KongJWTAdminUtil | release-5.1.0 | Onboards the v2 consumers for mobile, desktop and portal | |
Deploy/Kubernetes/Keycloak | release-5.1.0 | Deploys Keycloak | |
Deploy/Kubernetes/KeycloakRealm | release-5.1.0 | Creates the Sunbird Realm. Now read the comment mentioned in Core/Secrets.yml against the variables | |
Deploy/Kubernetes/BootstrapMongodb | release-5.1.0 | Creates the initial token in Mongo database | |
Deploy/Kubernetes/CertTemplate | branch_or_tag: release-5.1.0, sunbird_util_branch_or_tag: release-4.10.0_RC1 | Uploads the certificate templates into Object storage | |
Deploy/Kubernetes/UploadChatbotConfig | branch_or_tag: release-5.1.0, bot_repo_branch: master | Uploads the chatbot configs to Object storage | |
Deploy/Kubernetes/UploadFAQs | branch_or_tag: release-5.1.0, source_folder: Select All | Uploads the FAQs to Object storage | |
Deploy/Kubernetes/UploadSchema | branch_or_tag: release-5.1.0, kp_branch_or_tag: release-4.9.0_RC8 | Uploads the Content schemas to Object storage | |
Deploy/Kubernetes/AdminUtils | release-5.1.0 | Deploys Adminutils service | |
Deploy/Kubernetes/Analytics | release-5.1.0 | Deploys Analytics service | |
Deploy/Kubernetes/APIManagerEcho | release-5.1.0 | Deploys Echo service | |
Deploy/Kubernetes/Assessment | release-5.1.0 | Deploys Assessment service | |
Deploy/Kubernetes/Bot | release-5.1.0 | Deploys Bot service | |
Deploy/Kubernetes/Cert | release-5.1.0 | Deploys Cert service | |
Deploy/Kubernetes/CertRegistry | release-5.1.0 | Deploys CertRegistry service | |
Deploy/Kubernetes/Content | release-5.1.0 | Deploys Content service | |
Deploy/Kubernetes/Dial | release-5.1.0 | Deploys Dial service | |
Deploy/Kubernetes/Enc | release-5.1.0 | Deploys Enc service | |
Deploy/Kubernetes/Groups | release-5.1.0 | Deploys Groups service | |
Deploy/Kubernetes/KnowledgeMW | release-5.1.0 | Deploys KnowledgeMW service | |
Deploy/Kubernetes/Learner | release-5.2.0 | Deploys Learner service | |
Deploy/Kubernetes/Lms | release-5.2.0 | Deploys Lms service | |
Deploy/Kubernetes/Notification | release-5.1.0 | Deploys Notification service | |
Deploy/Kubernetes/Player | release-5.2.0 | Deploys Player service | |
Deploy/Kubernetes/Print | release-5.1.0 | Deploys Print service | |
Deploy/Kubernetes/Report | release-5.1.0 | Deploys Report service | |
Deploy/Kubernetes/Router | release-5.1.0 | Deploys Router service | |
Deploy/Kubernetes/Search | release-5.1.0 | Deploys Search service | |
Deploy/Kubernetes/Taxonomy | release-5.1.0 | Deploys Taxonomy service | |
Deploy/Kubernetes/Telemetry | release-5.1.0 | Deploys Telemetry service | |
Deploy/Kubernetes/LoggingFileBeatsVM | branch_or_tag: release-5.1.0, hosts: Select All, tags: default | Deploys filebeat in all the selected VMs | |
Deploy/Kubernetes/Logging | release-5.1.0 | Run job twice by choosing the chartname as - | |
Deploy/Kubernetes/nginx-public-ingress | release-5.2.0 | Deploys the nginx web server. Now you can open visit your domain Note: Add new variable mobile_deeplink_url Refer documentation to add assetlinks.json file | |
Deploy/Kubernetes/Nodebb | release-5.1.0 | Deploys NodeBB service. Now read the comment mentioned in Core/Secrets.yml against the variable | |
Deploy/Kubernetes/DiscussionsMW | release-5.1.0 | Deploys DiscussionsMW service | |
Deploy/Lern/LernDataProducts | release-5.1.0_RC1 | ||
Deploy/Lern/LernFlinkJobs | release-5.1.1_RC1 | relational-cache-updater, activity-aggregate-updater jobs need to be deployed | |
Deploy/managed-learn/gotenberg | release-5.1.0 | ||
Deploy/managed-learn/ml-analytics-service | release-5.2.0 | ml_analytics_version: release-5.1.0_RC16 | |
Deploy/managed-learn/ml-core-service | release-5.2.0 | ||
Deploy/managed-learn/ml-projects-service | release-5.2.0 | ||
Deploy/managed-learn/ml-reports-service | release-5.2.0 | ||
Deploy/managed-learn/ml-survey-service | release-5.2.0 | ||
Deploy/Sunbird-RC/CertificateApi | release-5.1.0 | ||
Deploy/Sunbird-RC/CertificateSign | release-5.1.0 | ||
Deploy/Sunbird-RC/Registry | release-5.1.0 | ||
Deploy/Sunbird-RC/UploadRCSchema | branch_or_tag:release-5.1.0, source_folder: sunbird-RC | ||
Deploy/UCI/fusionauth | release-5.1.0 | ||
Deploy/UCI/gql | release-5.1.0 | ||
Deploy/UCI/Inbound | release-5.1.0 | ||
Deploy/UCI/odk | release-5.1.0 | ||
Deploy/UCI/Orchestrator | release-5.1.0 | ||
Deploy/UCI/Outbound | release-5.1.0 | ||
Deploy/UCI/Transformer | release-5.1.0 | ||
Deploy/UCI/UCI | release-5.1.0 | ||
Deploy/Plugins/CollectionEditor | release-5.1.0 | Deploys CollectionEditor files to Object storage | |
Deploy/Plugins/ContentEditor | release-5.1.0 | Deploys ContentEditor files to Object storage | |
Deploy/Plugins/ContentPlayer | release-5.1.0 | Deploys ContentPlayer files to Object storage | |
Deploy/Plugins/ContentPlugins | release-5.1.0 | Deploys ContentPlugins files to Object storage | |
Deploy/Plugins/GenericEditor | release-5.1.0 | Deploys GenericEditor files to Object storage | |
Deploy/DataPipeline/AdhocScripts | release-5.1.0 | Deploys AdHoc scripts in Spark VM | |
Deploy/DataPipeline/AnalyticsCore | release-5.1.0 | Deploys jars in Spark VM | |
Deploy/DataPipeline/CoreDataProducts | release-5.1.0 | Deploys jars in Spark VM | |
Deploy/DataPipeline/DruidIngestion | branch_or_tag: release-5.1.0, ingestion_task_names: Select All | Deploys and starts Druid ingestion tasks. This job will be updated in a couple of days. Please wait until then | |
Deploy/DataPipeline/EdDataProducts | release-5.1.0 | Deploys jars in Spark VM | |
Deploy/DataPipeline/ETLJobs | release-5.1.0 | Deploys ETL scripts in Spark VM | |
Deploy/DataPipeline/ETLDruidContentIndexer | release-5.1.0 | Deploys ETLDruidContentIndexer | |
Deploy/DataPipeline/ETLUserCacheIndexer | release-5.1.0 | Deploys ETLUserCacheIndexer | |
Deploy/DataPipeline/Secor | release-5.1.0 | Deploys Secor Statefulsets in Kubernetes cluster | |
Deploy/DataPipeline/FlinkPipelineJobs | release-5.1.0 | Deploys Flink jobs Kubernetes cluster | |
Deploy/DataPipeline/GraphitePrometheusExporter | release-5.1.0 | Deploys graphite prometheus exporter in Druid VM | |
Deploy/DataPipeline/LoggingFileBeatsVM | branch_or_tag: release-5.1.0, hosts: Select All, tags: default | Deploys filebeat in all the selected VMs |
Last updated