5.2.0 to 6.0.0

Note: This document has been revised to incorporate updates for version 6.0.1, including testing of the cloud-agnostic feature on Oracle Cloud Infrastructure (OCI).

(updated on TBD)

Overview

This page details out the jobs required to be run as part of the upgrade of Sunbird Ed from release 5.2.0 to release 6.0.0. Use the following table to understand the Jenkins jobs that need to be executed in order to successfully complete the upgrade. Any jenkins job configuration or pre-requisites mentioned under manual configuration section needs to be done first before running any of the mentioned jobs. The order of the jobs should also be run as shown below.

1. Login to Jenkins VM

su jenkins

Replace the config.xml for the following mentioned Jenkins path

Jenkins Jobs	Path of Jenkins Server	Update Jenkins job URL
Build/Core/Content	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/Content	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/release-6.0.0/deploy/jenkins/jobs/Build/jobs/Core/jobs/Content/config.xml
Build/Core/Learner	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/Learner	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/Core/jobs/Learner/config.xml
Build/Core/Lms	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/Lms	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/Core/jobs/Lms/config.xml
Build/Core/Analytics	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/Analytics	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/Core/jobs/Analytics/config.xml
Build/DataPipeline/CoreDataProducts	/var/lib/jenkins/jobs/Build/jobs/DataPipeline/jobs/CoreDataProducts	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/DataPipeline/jobs/CoreDataProducts/config.xml
Build/DataPipeline/AnalyticsCore	/var/lib/jenkins/jobs/Build/jobs/DataPipeline/jobs/AnalyticsCore	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/DataPipeline/jobs/AnalyticsCore/config.xml
Build/KnowledgePlatform/FlinkJobs	Build/KnowledgePlatform/FlinkJobs	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/KnowledgePlatform/jobs/FlinkJobs/config.xml
Build/Lern/LernDataProducts	/var/lib/jenkins/jobs/Build/jobs/Lern/jobs/LernDataProducts	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/Lern/jobs/LernDataProducts/config.xml
Build/Lern/LernFlinkJobs	/var/lib/jenkins/jobs/Build/jobs/Lern/jobs/LernFlinkJobs	https://raw.githubusercontent.com/project-sunbird/sunbird-devops/blob/release-6.0.0/deploy/jenkins/jobs/Build/jobs/Lern/jobs/LernFlinkJobs/config.xml

Adding the new variables

Update the following variables in sunbird-devops private repo according to your environment

cd ansible/inventory/dev/core/common.yaml

### Variables added as part of 6.0.0 release - https://github.com/Sunbird-Ed/ml-analytics-service/tree/release-6.0.0/release-notes/6.0.0.md
#### ML Analytics Service #####
ml_analytics_cname_url: # Reference value - https://obj.dev.sunbirded.org/samiksha/

### Variables added as part of 6.0.0 release - https://github.com/Sunbird-Ed/ml-analytics-service/tree/release-6.0.0/release-notes/6.0.0.md
ml_analytics_client_id : project-sunbird-dev-client # Keycloak client group
ml_analytics_username : reportAdmin1 # Username of a user which has roles of "PROGRAM_MANAGER", "PROGRAM_DESIGNER", "REPORT_ADMIN", "REPORT_VIEWER"
ml_analytics_createdBy : fb85a044-d9eb-479b-a55a-faf1bfaea14d # Unique system generated user UUID which is the same user as above
ml_container: samiksha

# provide the s3 compatible endpoint
# for AWS
# cloud_private_storage_account_endpoint: "https://s3.{{ cloud_public_storage_region }}.amazonaws.com"
# for OCI
#cloud_private_storage_account_endpoint: "https://<oci_namespace>.compat.objectstorage.{{cloud_public_storage_region}}.oraclecloud.com"
cloud_private_storage_account_endpoint: "{{ cloud_public_storage_endpoint }}" # Leave Blank for Azure
bb_name: "" #Leave Blank

cd ansible/inventory/dev/core/secrets.yaml

#### ML Analytics Service #####
### Variables added as part of 6.0.0 release - https://github.com/Sunbird-Ed/ml-analytics-service/tree/release-6.0.0/release-notes/6.0.0.md
# Examples values added below
ml_analytics_password : Test@123 # Password of a user which has roles of "PROGRAM_MANAGER", "PROGRAM_DESIGNER", "REPORT_ADMIN", "REPORT_VIEWER"
ml_analytics_authorization_access_token : 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJkOTY3NzRjYzXXXXXXXXXXXXXQ4Y2RiOWQ2Mzg0OSJ9.n4hXxKxl_698yeZPSWhXeGvMOb4esfgPadIZe8jZ0Z4' # Bearer auth token which has access to APIs as defined in the https://github.com/Sunbird-Ed/ml-analytics-service/tree/release-6.0.0/release-notes/6.0.0.md
ml_analytics_client_secret : fd241dce-4ZZ9-47e1-97cf-1c7de7a44216 # Keycloak client secret for creating tokens, mapped to client id supplied to variable ml_analytics_client_id

cd ansible/inventory/dev/KnowledgePlatform/common.yaml

druid_storage_type: ""  # Your cloud service provider name. Supported values are aws, azure, gcloud
# Optional variables - Can be left blank
cloud_storage_report_verfication_bucketname: ""
dp_storage_endpoint_config: ""

Remove the following variables

cd ansible/inventory/dev/core/common.yaml

ml_cloud_config: |
  CLOUD_STORAGE=AZURE
  AZURE_ACCOUNT_NAME={{sunbird_public_storage_account_name}}
  AZURE_ACCOUNT_KEY={{sunbird_public_storage_account_key}}
  AZURE_STORAGE_CONTAINER=samiksha

Build and Deploy for SunbirdED

SunbirdED:

Service	Build job	Build tag	Comments	Deploy Job	Deploy tag
Portal	Build/Core/Player	release-6.0.0_RC42		Deploy/Kubernetes/Player	release-6.0.0
ML Services	Build/managed-learn/ml-projects-service	release-6.0.0_RC9		Deploy/managed-learn/ml-projects-service	release-6.0.0
	Build/managed-learn/ml-core-service	release-6.0.0_RC24		Deploy/managed-learn/ml-core-service	release-6.0.0
	Build/managed-learn/ml-reports-service	release-6.0.0_RC7		Deploy/managed-learn/ml-reports-service	release-6.0.0
	Build/managed-learn/ml-survey-service	release-6.0.0_RC9		Deploy/managed-learn/ml-survey-service	release-6.0.0
ML-Analytics			ml_analytics_version: release-6.0.0_RC14	Deploy/managed-learn/ml-analytics-service	release-5.2.0

Sunbird Lern:

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
Data pipeline	Build/Lern/FlinkJobs	github_release_tag:release-5.3.1_RC2, CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/Lern/FlinkJobs	release-5.3.0_RC5
Lern data products	Build/Lern/LernDataProducts	github_release_tag:release-5.3.1_RC8, CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/Lern/LernDataProducts	release-5.1.0_RC1
Analytics	Build/Core/Analytics	github_release_tag:release-5.1.3_RC1,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Analytics	release-5.2.0-lern
User&Org Service	Build/Core/Learner	github_release_tag:release-5.3.1_RC1,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.0CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Learner	release-5.3.0-lern
Batch Service	Build/Core/Lms	github_release_tag:release-5.3.2_RC1,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.0CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Lms	release-5.3.0-lern

Data Security Policy setup

Configurations to be done by System admin:

1. Setup default 'Data Security Policy' settings using tenant preference API.

curl --location --request PATCH '{{host}}/api/org/v2/preferences/update' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "dataSecurityPolicy",
        "data": {
            "level": "PLAIN_DATASET",
            "dataEncrypted": "No",
            "comments": "Data is not encrypted",
            "job": {
                    "userinfo-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    },
                    "program-user-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    }
                },
            "securityLevels": {
                "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
                "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
                "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
                "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
            }
        }
    }
}'

Sunbird Obsrv

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
Data pipeline	Build/DataPipeline/AnalyticsCore	github_release_tag:release-5.1.3_RC3,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/DataPipeline/AnalyticsCore	release-5.2.0
Core Data products	Build/DataPipeline/CoreDataProducts	github_release_tag:release-5.1.3_RC3,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/DataPipeline/CoreDataProducts	release-5.2.0

Sunbird-Knowlg

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
Knowledge-platform-jobs	Build/KnowledgePlatform/FlinkJobs	github_release_tag:release-5.5.0_RC2,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/KnowledgePlatform/FlinkJobs	release-5.4.1_RC1	Jobs to be deployed: 1. asset-enrichment 2.content-publish 3. post-publish-processor 4. qrcode-image-generator 5. video-stream-generator
content-service	Build/Core/Content	github_release_tag:release-5.5.0_RC2,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.7	Deploy/Kubernetes/Content	release-6.0.0
Plugins	Build/Plugins/ContentPlugins	release-5.2.1_RC3	Deploy/Plugins/ContentPlugins	release-6.0.0
Plugins	Build/Plugins/ContentEditor	release-5.2.1_RC3	Deploy/Plugins/ContentEditor	release6.0.0

Sunbird-InQuiry

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
Assessment	Build/Core/Assessment	inquiry_release_tag:release-5.6.0_RC1,core_release_tag:release-5.2.0_RC2	Deploy/Kubernetes/Assessment	release-6.0.0
InQuiryFlink Job	Build/KnowledgePlatform/InquiryFlinkJob	release-5.7.0_RC4, cloud_store_group_id: org.sunbird, cloud_store_artifact_id: cloud-store-sdk_2.12, cloud_store_version: 1.4.6 "	Deploy/KnowledgePlatform/InquiryFlinkJob	release-5.7.0_RC1