5.0.0 to 5.1.0

5.0.0 to 5.1.0

Overview

This page details out the jobs required to be run as part of the upgrade from Sunbird Ed and CoKreat release 5.0.0 to release 5.1.0. Use the following table to understand the jobs that need to be executed in order to successfully complete the upgrade. Any jenkins job configuration or pre-requisites mentioned under manual configuration section needs to be done first before running any of the mentioned jobs. The order of the jobs should also be run as shown below. They can be run in parallel to speed up the execution.
Create cname for storage account
Create custom domain for storage account and update variables Before starting deployments.
Below are the steps to add custom domain to storage account
For Azure Click Here , AWS Click Here , for GCP Click Here

Build and Deploy for Sunbird

Sunbird_ED:

Service
Build job
Build tag
Deploy Job
Deploy tag
Comments
Mobile App
Mobile/job/Sunbird-IONIC-Mobile-App/
release-5.1.0.1
Portal
Build/Core/Player
Deploy/Kubernetes/Player
release-5.1.0.6
ML Services
Build/managed-learn/ml-projects-service
release-5.1.0_RC32
Deploy/managed-learn/ml-projects-service
release-5.1.0_RC7
release-5.1.0.1
Build/managed-learn/ml-core-service
release-5.1.0_RC11
Deploy/managed-learn/ml-core-service
release-5.1.0_RC7
release-5.1.0.1
Build/managed-learn/ml-reports-service
release-5.1.0_RC10
Deploy/managed-learn/ml-reports-service
release-5.1.0_RC7
release-5.1.0.1
Build/managed-learn/ml-survey-service
release-5.1.0_RC5
Deploy/managed-learn/ml-survey-service
release-5.1.0_RC7
release-5.1.0.1
ML-Analytics
Deploy/managed-learn/ml-analytics-service
release-5.1.0_RC14
release-5.1.0.1
Other
Deploy/Kubernetes/OnboardAPI
5.1.0_RC5
Deploy/Kubernetes/OnboardConsumers
5.1.0_RC5
Deploy/Kubernetes/nginx-public-ingress
5.1.0_RC9
Deploy/DataPipeline/InternalKong
Service
Build Job
Build Tag
Deploy Job
Deploy Tag
Comments
Certificate API
CertificateAPI
v0.0.11
CertificateAPI
release-5.1.0_RC5
First build the service with values -
github_release_tag = "release-5.1.0_RC5", dockerhub_image_source = "dockerhub/sunbird-rc-certificate-api:v0.0.11" and repository_name = "sunbird-rc-certificate-api"
and then deploy
Registry
Registry
v0.0.13
Registry
release-5.1.0_RC5
First build the service with values -
github_release_tag = "release-5.1.0_RC5", dockerhub_image_source = "dockerhub/sunbird-rc-core:v0.0.13" and repository_name = "sunbird-rc-core"
and then deploy
UploadRCSchema
release-5.1.0_RC5
Service/Compon
Build job
Build tag
Deploy Job
Deploy tag
Comments
Sync tool
Build/KnowledgePlatform/SyncTool
release-5.2.0_RC9
migratecspdata - **** Add this command to the command list in the Deploy/KnowledgePlatform/Neo4jElasticSearchSyncTool Jenkins job.
Content service
Build/Core/Content
release-5.2.0_RC5
Deploy/Kubernetes/Content
release-5.1.0_RC6
Taxonomy service
Build/Core/Taxonomy
release-5.2.0_RC5
Deploy/Kubernetes/Taxonomy
release-5.1.0_RC6
****
DIAL service
Build/Core/Dial
release-5.2.0_RC1
Deploy/Kubernetes/Dial
release-5.1.0_RC6
****
Learning service
Build/KnowledgePlatform/Learning
release-5.2.0_RC9
Deploy/KnowledgePlatform/Learning
release-5.2.0_RC9
****
Knowlg middleware
Build/Core/KnowledgeMW
release-5.2.0_RC1
Deploy/Kubernetes/KnowledgeMW
release-5.1.0_RC6
Deploy this before deploying the service and flink-jobs.
Knowlg platform jobs
Build/KnowledgePlatform/FlinkJobs
release-5.2.0_RC11
Deploy/KnowledgePlatform/FlinkJobs
release-5.2.0_RC8
Add the below job list to the job_names_to_deploy list in the Deploy/KnowledgePlatform/FlinkJobs jenkins job
csp-migrator,
live-node-publisher,
live-video-stream-generator,
cassandra-data-migration
Deploy the
assessment-enrichment,
content-publish,
qrcode-image-generator, search-indexer,
csp-migrator,
live-node-publisher,
live-video-stream-generator,
cassandra-data-migration flink jobs.
Interactive editor
Build/Plugins/ContentEditor
release-5.2.0_RC1
Deploy/Plugins/ContentEditor
release-5.1.0_RC5
Content plugins
Build/Plugins/ContentPlugins
release-5.2.0_RC1
Deploy/Plugins/ContentPlugins
release-5.0.0_RC5
File upload editor
Build/Plugins/GenericEditor
release-5.2.0_RC1
Deploy/Plugins/GenericEditor
release-5.0.0_RC5
Service/Component
Build job
Build tag
Deploy Job
Deploy tag
Comments
User & Org service
Build/Core/Learner
release-5.0.1_RC2
Deploy/Kubernetes/Learner
release-5.1.0_RC6
release-5.1.0
LMS
Build/Core/Lms
release-5.0.1_RC3
Deploy/Kubernetes/Lms
release-5.1.0_RC6
Certificate
Build/Core/Cert
release-5.0.1_RC2
Deploy/Kubernetes/Cert
release-5.1.0_RC6
release-5.1.0.1
Lern data products
Build/Lern/LernDataProducts
release-5.0.1_RC4
Deploy/Lern/LernDataProducts
release-5.0.1_RC4
release-5.1.0.1
Lern jobs
Build/Lern/LernFlinkJobs
release-5.0.1_RC5
Deploy/Lern/LernFlinkJobs
release-5.0.1_RC5
release-5.1.0.1
Service to be Build
Build Tag
Service to Deploy
Deploy Tag
Comments
Deploy/Kubernetes/InquiryUploadSchema
schema_repo_branch_or_tag: release-5.2.0_RC2 source_folder: question,questionset
Create jenkins jobs from config
Build/Core/Assessment
release-5.2.0_RC3
Deploy/Kubenetes/Assessment
release-5.1.0_RC4
Update build and deploy job config, job-config
Build/job/KnowledgePlatform/InquiryFlinkJob
release-5.2.0_RC3
Deploy/KnowledgePlatform/InquiryFlinkJob
release-5.2.0_RC3
create build, deploy and artifactupload jobs, job-config
Build/Core/Analytics
release-5.1.0_RC2
Deploy/Kubernetes/Analytics
release-5.1.0_RC3
Service to be Build
Build Tag
Service to Deploy
Deploy Tag
Comments
Build/Core/Analytics
release-5.1.0_RC2
Deploy/Kubernetes/Analytics
release-5.1.0_RC3

Variables to update for Ed and vdn

variable name
description
Value
cloudstorage_replace_absolute_path
Boolean value is required to enable/disable cloud storage agnostic urls in db. Update in core/common.yml and KnowledgePlatform/common.yml
'true'
cloudstorage_base_path
String value is required. The value could be either cloud specific base path or cname configured for current cloud storage provider. Update in core/common.yml and KnowledgePlatform/common.yml
ex: storage url: "https://storagename.blob.core.windows.net" cname url: https://obj.stage.sunbirded.org
valid_cloudstorage_base_urls
Array of String value is required. This array will have list of all cloud path which should be accepted by service. Update in core/common.yml and KnowledgePlatform/common.yml
ex: '["https://storagename.blob.core.windows.net","https://obj.stage.sunbirded.org"]'
cloudstorage_relative_path_prefix
String value is required. This value will be used as prefix for any metadata which holds cloud storage path. Update in Core/common.yml and KnowledgePlatform/common.yml
CONTENT_STORAGE_BASE_PATH
cloudstorage_metadata_list
Array of String value is required. This array will have list of metadata keys which holds cloud storage path. Update in Core/common.yml
["appIcon","posterImage","artifactUrl","downloadUrl","variants","previewUrl","pdfUrl"]
cloud_storage_base_url
storage url, Update in Core/common.yml.
ex: "https://storagename.blob.core.windows.n
cloud_storage_cname_url
cname url, Update in Core/common.yml
ex: "https://obj.stage.sunbirded.org"
lern variables to be updated in inventory
update Lern/common.yml
cloud_service_provider
Update in core, kp, dp common.yml
Supported values are aws, azure, gcloud
lp_vault_youtube_api_key
youtube api key. Add in core/common.yml
copy value from KP secrets file
cloud_storage_url
storage url
ex: "https://storagename.blob.core.windows.n

Add below Api's in Internal Kong configuration

Check if below api's exists or not before adding, if exists replace the existing one if not add it.
- name: createAndUpdateCertificateTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/createOrUpdate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/createOrUpdate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: uploadCertificateTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/uploadTemplate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/uploadTemplate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: projectCertificateReissue
uris: "{{ private_ml_project_prefix }}/api/v1/userProjects/certificateReIssue"
upstream_url: "{{ ml_project_private_url }}/v1/userProjects/certificateReIssue"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: createCertificateTemplateSvg
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/createSvg"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/createSvg"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: createAndUpdateCertificateBaseTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateBaseTemplates/createOrUpdate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateBaseTemplates/createOrUpdate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: certificateBaseTemplateDetailsList
uris: "{{ private_ml_core_prefix }}/api/v1/certificateBaseTemplates/list"
upstream_url: "{{ ml_core_private_url }}/v1/certificateBaseTemplates/list"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"

Execute Below API for ML Analytic Data Product Survey Status Report Configuration Update

curl -X POST \
{{host}}/api/dataset/v1/add \
-H 'authorization: Bearer {{jwt_token}}' \
-H 'content-type: application/json' \
-d '{
"id": "ekstep.analytics.dataset.add",
"ver": "1.0",
"ts": "2016-12-07T12:40:40+05:30",
"params": {
"msgid": "4f04da60-1e24-4d31-aa7b-1daf91c46341"
},
"request": {"dataset":"druid-dataset","datasetSubId":"ml-survey-status-detail-exhaust","datasetConfig":{"type":"ml-survey-status-detail-exhaust","params":{"programId":"program-1","solutionId":"solution-1"}},"datasetType":"druid","visibility":"private","version":"v1","authorizedRoles":["PROGRAM_MANAGER","PROGRAM_DESIGNER"],"validationJson":{"type":"object","properties":{"tag":{"id":"http://api.ekstep.org/dataexhaust/request/tag","type":"string"},"dataset":{"id":"http://api.ekstep.org/dataexhaust/request/dataset","type":"string"},"datasetSubId":{"id":"http://api.ekstep.org/dataexhaust/request/datasetSubId","type":"string"},"requestedBy":{"id":"http://api.ekstep.org/dataexhaust/request/requestedBy","type":"string"},"encryptionKey":{"id":"http://api.ekstep.org/dataexhaust/request/encryptionKey","type":"string"},"datasetConfig":{"id":"http://api.ekstep.org/dataexhaust/request/datasetConfig","type":"object"}},"required":["tag","dataset","datasetConfig"]},"druidQuery":{"id":"ml-survey-status-detail-exhaust","labels":{"user_id" : "UUID","user_type" : "User type","user_sub_tye" : "User sub type","state_name" : "Declared State","district_name" : "District","block_name" : "Block","school_code" : "School ID","school_name" : "School Name","board_name" : "Declared Board","organisation_name" : "Org Name","program_name" : "Program Name","program_externalId" : "Program ID","survey_name" : "Survey Name","survey_externalId" : "Survey ID","survey_submission_id" : "survey_submission_id","submission_status" : "Status of submission","submission_date" : "Submission date"},"dateRange":{"interval":{"startDate":"1901-01-01","endDate":"2101-01-01"},"granularity":"all","intervalSlider":0},"metrics":[{"metric":"total_content_plays_on_portal","label":"total_content_plays_on_portal","druidQuery":{"intervals":"1901-01-01T00:00+00:00/2101-01-01T00:00:00+00:00","dataSource":"ml-survey-status","columns":["user_id", "user_type", "user_sub_type", "state_name", "district_name", "block_name", "school_code", "school_name", "board_name", "organisation_name", "program_name", "program_externalId", "survey_name", "survey_externalId", "survey_submission_id", "submission_status","submission_date"],"queryType":"scan"}}],"output":[{"zip":false,"label":"","dims":["date"],"fileParameters":["id","dims"],"metrics":["user_id", "user_type", "user_sub_type", "state_name", "district_name", "block_name", "school_code", "school_name", "board_name", "organisation_name", "program_name", "program_externalId", "survey_name", "survey_externalId", "survey_submission_id", "submission_status","submission_date"],"type":"csv"}],"sort":["UUID","Program ID","Survey ID","survey_submission_id"],"queryType":"scan"},"supportedFormats":["csv","zip"],"exhaustType":"OnDemand"}}'

Execute Below API for ML Analytic Data Product Project Status Report Configuration Update

curl -X POST \
{{host}}/api/dataset/v1/add \
-H 'authorization: Bearer {{jwt_token}}' \
-H 'content-type: application/json' \
-d '{
"id": "ekstep.analytics.dataset.add",
"ver": "1.0",
"ts": "2016-12-07T12:40:40+05:30",
"params": {
"msgid": "4f04da60-1e24-4d31-aa7b-1daf91c46341"
},
"request": {
"dataset": "druid-dataset",
"datasetSubId": "ml-project-status-exhaust",
"datasetConfig": {
"type": "ml-project-status-exhaust",
"params": {
"programId": "program-1",
"solutionId": "solution-1"
}
},
"datasetType": "druid",
"visibility": "private",
"version": "v1",
"authorizedRoles": [
"PROGRAM_MANAGER",
"PROGRAM_DESIGNER"
],
"validationJson": {"type":"object","properties":{"tag":{"id":"http://api.ekstep.org/dataexhaust/request/tag","type":"string"},"dataset":{"id":"http://api.ekstep.org/dataexhaust/request/dataset","type":"string"},"datasetSubId":{"id":"http://api.ekstep.org/dataexhaust/request/datasetSubId","type":"string"},"requestedBy":{"id":"http://api.ekstep.org/dataexhaust/request/requestedBy","type":"string"},"encryptionKey":{"id":"http://api.ekstep.org/dataexhaust/request/encryptionKey","type":"string"},"datasetConfig":{"id":"http://api.ekstep.org/dataexhaust/request/datasetConfig","type":"object"}},"required":["tag","dataset","datasetConfig"]},
"druidQuery":{"id":"ml-project-status-exhaust","labels":{"block_name":"Block","board_name":"Declared Board","project_title_editable":"Project Title","project_completed_date":"Project completion date of the user","user_type":"User Type","designation":"User sub type","school_code":"School ID","project_created_date":"Project start date of the user","project_last_sync":"Project last Synced date","project_duration":"Project Duration","status_of_project":"Project Status","project_id":"Project ID","project_description":"Project Objective","program_externalId":"Program ID","organisation_name":"Org Name","createdBy":"UUID","school_name":"School Name","district_name":"District","program_name":"Program Name","certificate_status_customised":"Certificate Status","state_name":"Declared State"},"dateRange":{"interval":{"startDate":"1901-01-01","endDate":"2101-01-01"},"granularity":"all","intervalSlider":0},"metrics":[{"metric":"total_content_plays_on_portal","label":"total_content_plays_on_portal","druidQuery":{"intervals":"1901-01-01T00:00+00:00/2101-01-01T00:00:00+00:00","dataSource":"sl-project","columns":["createdBy","user_type","designation","state_name","district_name","block_name","school_name","school_code","board_name","organisation_name","program_name","program_externalId","project_id","project_title_editable","project_description","project_created_date","project_completed_date","project_duration","project_last_sync","status_of_project","certificate_status_customised"],"queryType":"scan"}}],"output":[{"zip":false,"label":"","dims":["date"],"fileParameters":["id","dims"],"metrics":["createdBy","user_type","designation","state_name","district_name","block_name","school_name","school_code","board_name","organisation_name","program_name","program_externalId","project_id","project_title_editable","project_description","project_created_date","project_completed_date","project_duration","project_last_sync","status_of_project","certificate_status_customised"],"type":"csv"}],"sort":["UUID","Program ID","Project ID"],"queryType":"scan"},
"supportedFormats": ["csv","zip"],
"exhaustType": "OnDemand"
}
}'

Add below variables in Core,KnowledgePlatform and Datapipeline inventory for ED and vdn

Refer to variables and comment section from here link to update below variables.
storage_endpoint variables for public, private, management and artifact is not required in current release, give empty string as value (cloud_public_storage_endpoint, cloud_private_storage_endpoint, etc)
// ###### common.yml ###############
cloud_public_storage_accountname: ""
cloud_public_storage_endpoint: ""
cloud_public_storage_region: ""
cloud_public_storage_project: ""
cloud_private_storage_accountname: ""
cloud_private_storage_endpoint: ""
cloud_private_storage_region: ""
cloud_private_storage_project: ""
cloud_management_storage_accountname: ""
cloud_management_storage_endpoint: ""
cloud_management_storage_region: ""
cloud_management_storage_project: ""
cloud_artifact_storage_accountname: ""
cloud_artifact_storage_endpoint: ""
cloud_artifact_storage_region: ""
cloud_artifact_storage_project: ""
Refer to variables and comment section from here link to update below secrets variables
// ########## secrets.yml ###############
cloud_public_storage_secret: ""
cloud_private_storage_secret: ""
cloud_management_storage_secret: ""
cloud_artifact_storage_secret: ""
Refer to variables and comment section from here link to update below variables
// ############# common.yml ##############
cloud_storage_certqr_bucketname: "certqr"
cloud_storage_chatbot_bucketname: "chatbot"
cloud_storage_dial_bucketname: "dial"
cloud_storage_flink_bucketname: "flink-state-backend"
cloud_storage_playercdn_bucketname: "player"
cloud_storage_public_bucketname: "public"
cloud_storage_publicreports_bucketname: "public-reports"
cloud_storage_privatereports_bucketname: "reports"
cloud_storage_samiksha_bucketname: "samiksha"
cloud_storage_schema_bucketname: "schema"
cloud_storage_sourcing_bucketname: "sourcing"
cloud_storage_offlineinstaller_bucketname: "{{env}}-offlineinstaller"
cloud_storage_content_bucketname: "sunbird-content-{{env}}"
cloud_storage_telemetry_bucketname: "telemetry-data-store"
cloud_storage_termsandcondtions_bucketname: "termsandcondtions"
cloud_storage_user_bucketname: "user"
cloud_storage_desktopappcrashlogs_bucketname: "desktopappcrashlogs"
cloud_storage_label_bucketname: "label"
cloud_storage_certservice_bucketname: "{{env}}-e-credentials"
cloud_storage_uci_bucketname: "uci-staging"
cloud_storage_cassandrabackup_bucketname: "cassandra-backup"
cloud_storage_dpcassandrabackup_bucketname: "dp-cassandra-backup"
cloud_storage_dppostgresbackup_bucketname: "dp-postgresql-backup"
cloud_storage_dpredisbackup_bucketname: "dp-redis-backup"
cloud_storage_esbackup_bucketname: "elasticsearch-snapshots"
cloud_storage_influxdbbackup_bucketname: "influxdb-backup"
cloud_storage_jenkinsbackup_bucketname: "jenkins-backup"
cloud_storage_mongobackup_bucketname: "mongodb-backup"
cloud_storage_neo4jbackup_bucketname: "neo4j-backup"
cloud_storage_redisbackup_bucketname: "redis-backup"
cloud_storage_artifacts_bucketname: "{{ artifacts_container }}"
Variables to remove from inventory
delete below variables from core/common.yml if its defined.
upstream_url
plugin_upstream_url
discussion_upstream_url
Migrations
Please execute the migration instructions mentioned by the dependent building blocks

Add new Form

To create a new form to show or remove the onboarding popups\
// Some codecurl --location --request POST '<https://{{env}}>/api/data/v1/form/create' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{API KEY}}' \
--data-raw '{
"request": {
"type": "newUserOnboarding",
"action": "onboarding",
"subType": "global",
"rootOrgId": <please specify your root org id>,
"component": "portal",
"data":{
"fields": {
"shownewUserOnboarding": "true"
}
}
}
}'

CoKreat Deployments and migration

Please follow the instructions mentioned in the below link by CoKreat building block to upgrade CoKreat from 5.0.0 to 5.1.0
Variables lists and build tag is same for Ed and vdn. Use release-5.1.0-vdn_RC1 for knowlg service deployments in VDN