6.0.1 to 7.0.0

Overview

This page details out the jobs required to be run as part of the upgrade of Sunbird Ed from release 6.0.1 to release 7.0.0. Use the following table to understand the Jenkins jobs that need to be executed in order to successfully complete the upgrade. Any jenkins job configuration or pre-requisites mentioned under manual configuration section needs to be done first before running any of the mentioned jobs. The order of the jobs should also be run as shown below.

As part of release-7.0.0 some Jenkins job parameters are added for cloud agnostic changes. Follow the steps below to replace the config.xml for those jobs.

1. Login to Jenkins VM

su jenkins

Replace the config.xml for the following mentioned Jenkins path

Jenkins Jobs	Path of Jenkins Server	Update Jenkins job URL
Build/Core/UserOrg	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/UserOrg	https://github.com/project-sunbird/sunbird-devops/blob/release-7.0.0/deploy/jenkins/jobs/Build/jobs/Core/jobs/UserOrg/config.xml
Deploy/Kubernetes/UserOrg	/var/lib/jenkins/jobs/Deploy/jobs/Kubernetes/jobs/UserOrg	https://github.com/project-sunbird/sunbird-devops/blob/release-7.0.0/deploy/jenkins/jobs/Deploy/jobs/dev/jobs/Kubernetes/jobs/UserOrg/config.xml
Deploy/Kubernetes/InquiryUploadSchema	/var/lib/jenkins/jobs/Deploy/jobs/Kubernetes/jobs/InquiryUploadSchema	https://github.com/project-sunbird/sunbird-devops/blob/release-7.0.0/deploy/jenkins/jobs/Deploy/jobs/dev/jobs/Kubernetes/jobs/InquiryUploadSchema/config.xml
Build/Core/Search	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/UserOrg	https://github.com/project-sunbird/sunbird-devops/blob/release-7.0.0/deploy/jenkins/jobs/Build/jobs/Core/jobs/Search/config.xml
Build/Core/Knowlgmw	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/UserOrg	https://github.com/project-sunbird/sunbird-devops/blob/release-7.0.0/deploy/jenkins/jobs/Build/jobs/Core/jobs/KnowledgeMW/config.xml
Build/Core/Taxonomy	/var/lib/jenkins/jobs/Build/jobs/Core/jobs/Taxonomy	https://github.com/project-sunbird/sunbird-devops/blob/release-7.0.0/deploy/jenkins/jobs/Deploy/jobs/dev/jobs/Kubernetes/jobs/Taxonomy/config.xml

Adding the new variables

Update the following variables in sunbird-devops private repo according to your environment

cd ansible/inventory/dev/KnowledgePlatform/secrets.yaml

core_vault_sunbird_keycloak_user_federation_provider_id: fd241dce-4ZZ9-47e1-97cf-1c7de7a44216 #releated to keycloak

cd ansible/inventory/dev/DataPipeline/common.yaml

druid_storage_type: azure

Build and Deploy for SunbirdED

SunbirdED:

Service	Build job	Build tag	Comments	Deploy Job	Deploy tag
Portal	Build/Core/Player	release-7.0.0_RC23		Deploy/Kubernetes/Player	release-7.0.0
ML Services	Build/managed-learn/ml-projects-service	release-7.0.0_RC2		Deploy/managed-learn/ml-projects-service	release-7.0.0
	Build/managed-learn/ml-core-service	release-7.0.0_RC4		Deploy/managed-learn/ml-core-service	release-7.0.0
	Build/managed-learn/ml-reports-service	release-7.0.0_RC1		Deploy/managed-learn/ml-reports-service	release-7.0.0
	Build/managed-learn/ml-survey-service	release-7.0.0_RC3		Deploy/managed-learn/ml-survey-service	release-7.0.0

Sunbird Lern:

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
OnboardAPIs			Deploy/Kubernetes/OnboardAPIs	release-7.0.0
Cassandra Migration	Build/Core/Cassandra	release-7.0.0_RC3	Deploy/Kubernetes/Cassandra	release-7.0.0	Select the sunbird in cassandra_keyspace_to_migrate while deploying script_repo_branch_or_tag: release-7.0.0_RC3
Kafka Setup			Deploy/Lern/KafkaSetup	release-7.0.0_RC6
ES mapping			Provision/Core/ESMapping	release-7.0.0	Updates the es mapping to accept dynamic framework category to index user data
Data pipeline	Build/Lern/FlinkJobs	github_release_tag:release-7.0.0_RC7,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/Lern/FlinkJobs	release-7.0.0_RC7	Add user-deletion-cleanup and ml-user-delete into job list and deploy it.
Lern data products	Build/Lern/LernDataProducts	github_release_tag:release-7.0.0_RC7,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/Lern/LernDataProducts	release-7.0.0_RC7
Group Service	Build/Core/Groups	github_release_tag:release-7.0.0_RC2,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.0	Deploy/Kubernetes/Groups	release-7.0.0
User&Org Service	Build/Core/UserOrg	github_release_tag:release-7.0.0_RC5,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.0CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Learner	release-7.0.0
Batch Service	Build/Core/Lms	github_release_tag:release-7.0.0_RC2,,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.0CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Lms	release-7.0.0
Discussions Middleware	Build/Core/DiscussionsMiddleware	release-7.0.0_RC1	Deploy/Kubernetes/DiscussionsMW	release-7.0.0

Data Security Policy setup

Configurations to be done by System admin:

1. Setup default 'Data Security Policy' settings using tenant preference API.

curl --location --request PATCH '{{host}}/api/org/v2/preferences/update' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "dataSecurityPolicy",
        "data": {
            "level": "PLAIN_DATASET",
            "dataEncrypted": "No",
            "comments": "Data is not encrypted",
            "job": {
                    "userinfo-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    },
                    "program-user-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    }
                },
            "securityLevels": {
                "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
                "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
                "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
                "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
            }
        }
    }
}'

Sunbird Obsrv

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
Data pipeline	Build/DataPipeline/AnalyticsCore	github_release_tag:release-5.1.3_RC4,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.0	Deploy/DataPipeline/AnalyticsCore	release-5.2.0_RC6
Core Data products	Build/DataPipeline/CoreDataProducts	github_release_tag:release-5.1.3_RC5,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.0	Deploy/DataPipeline/CoreDataProducts	release-5.2.0_RC6
Analytics Service	Build/Core/Analytics	release-5.1.3_RC1	Deploy/Kubernetes/Analytics	release-5.2.0_RC6
Spark Provision			Provision/DataPipeline/AnalyticsSpark	release-5.2.0_RC6
Secor			Deploy/DataPipeline/Secor	release-5.2.0_RC6
FlinkJobs	Build/DataPipeline/FlinkPipelineJobs	release-5.2.0_RC6	Deploy/DataPipeline/FlinkPipelineJobs	release-5.2.0_RC6

Sunbird-Knowlg

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
Knowledge-platform-jobs	Build/KnowledgePlatform/FlinkJobs	github_release_tag:release-5.7.0_RC3,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk_2.12,CLOUD_STORE_VERSION:1.4.6	Deploy/KnowledgePlatform/FlinkJobs	release-5.7.0_RC4
content-service	Build/Core/Content	github_release_tag:release-5.7.0_RC11,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Content	release-7.0.0
Plugins	Build/Plugins/ContentPlugins	release-5.2.1_RC4	Deploy/Plugins/ContentPlugins	release-7.0.0
Plugins	Build/Plugins/ContentEditor	release-5.2.1_RC4	Deploy/Plugins/ContentEditor	release-7.0.0
Search	Build/Core/Search	release-5.7.0_RC8,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Search	release-7.0.0
Taxonomy	Build/Core/Taxonomy	release-5.7.0_RC7,CLOUD_STORE_GROUP_ID:org.sunbird,CLOUD_STORE_ARTIFACT_ID:cloud-store-sdk,CLOUD_STORE_VERSION:1.4.6	Deploy/Kubernetes/Taxonomy	release-7.0.0
Schema Upload			Deploy/Kubernetes/UploadSchema	release-5.7.0_RC9

Sunbird-InQuiry

Service/Component	Build job	Build tag	Deploy Job	Deploy tag	Comments
Assessment	Build/Core/Assessment	inquiry_release_tag:release-7.0.0_RC2,core_release_tag:release-5.6.0_RC3	Deploy/Kubernetes/Assessment	release-7.0.0
InQuiryFlink Job	Build/KnowledgePlatform/InquiryFlinkJob	release-7.0.0_RC7	Deploy/KnowledgePlatform/InquiryFlinkJob	release-7.0.0_RC7	A New Flink Job user-pii-data-updater is added.
InquiryUploadSchema			Deploy/job/dev/job/Kubernetes/job/InquiryUploadSchema/	Schema Should be Uploaded For Question & QuestionSet using release-7.0.0_RC2
InquiryKafkaSetup			Deploy/job/dev/job/KnowledgePlatform/job/InquiryKafkaSetup/	release-7.0.0_RC4	A new kafka topic delete.user has to be created for user-pii-data-updater flink job.