5.0.0 to 5.1.0
Overview
This page details out the jobs required to be run as part of the upgrade from Sunbird Ed and CoKreat release 5.0.0 to release 5.1.0. Use the following table to understand the jobs that need to be executed in order to successfully complete the upgrade. Any jenkins job configuration or pre-requisites mentioned under manual configuration section needs to be done first before running any of the mentioned jobs. The order of the jobs should also be run as shown below. They can be run in parallel to speed up the execution.
Create cname for storage account
Create custom domain for storage account and update variables Before starting deployments.
Below are the steps to add custom domain to storage account
For Azure Click Here , AWS Click Here , for GCP Click Here
Build and Deploy for Sunbird
Sunbird_ED:
Service Build job Build tag Deploy Job Deploy tag Comments Mobile/job/Sunbird-IONIC-Mobile-App/
Build/managed-learn/ml-projects-service
Deploy/managed-learn/ml-projects-service
Build/managed-learn/ml-core-service
Deploy/managed-learn/ml-core-service
Build/managed-learn/ml-reports-service
Deploy/managed-learn/ml-reports-service
Build/managed-learn/ml-survey-service
Deploy/managed-learn/ml-survey-service
Deploy/managed-learn/ml-analytics-service
Deploy/Kubernetes/OnboardAPI
Deploy/Kubernetes/OnboardConsumers
Deploy/Kubernetes/nginx-public-ingress
Deploy/DataPipeline/InternalKong
Service Build Job Build Tag Deploy Job Deploy Tag Comments First build the service with values -
Copy github_release_tag = "release-5.1.0_RC5", dockerhub_image_source = "dockerhub/sunbird-rc-certificate-api:v0.0.11" and repository_name = "sunbird-rc-certificate-api" First build the service with values -
Copy github_release_tag = "release-5.1.0_RC5", dockerhub_image_source = "dockerhub/sunbird-rc-core:v0.0.13" and repository_name = "sunbird-rc-core"
Service/Compon Build job Build tag Deploy Job Deploy tag Comments Build/KnowledgePlatform/SyncTool
migratecspdata - **** Add this command to the command list in the Deploy/KnowledgePlatform/Neo4jElasticSearchSyncTool Jenkins job.
Deploy/Kubernetes/Content
Deploy/Kubernetes/Taxonomy
Build/KnowledgePlatform/Learning
Deploy/KnowledgePlatform/Learning
Deploy/Kubernetes/KnowledgeMW
Deploy this before deploying the service and flink-jobs.
Build/KnowledgePlatform/FlinkJobs
Deploy/KnowledgePlatform/FlinkJobs
Add the below job list to the job_names_to_deploy list in the Deploy/KnowledgePlatform/FlinkJobs jenkins job
csp-migrator,
live-node-publisher,
live-video-stream-generator,
cassandra-data-migration
Deploy the
assessment-enrichment ,
content-publish ,
qrcode-image-generator , search-indexer ,
csp-migrator,
live-node-publisher,
live-video-stream-generator,
cassandra-data-migration flink jobs.
Build/Plugins/ContentEditor
Deploy/Plugins/ContentEditor
Build/Plugins/ContentPlugins
Deploy/Plugins/ContentPlugins
Build/Plugins/GenericEditor
Deploy/Plugins/GenericEditor
Service/Component Build job Build tag Deploy Job Deploy tag Comments Deploy/Kubernetes/Learner
Build/Lern/LernDataProducts
Deploy/Lern/LernDataProducts
Deploy/Lern/LernFlinkJobs
Service to be Build Build Tag Service to Deploy Deploy Tag Comments Deploy/Kubernetes/InquiryUploadSchema
schema_repo_branch_or_tag: release-5.2.0_RC2 source_folder: question,questionset
Create jenkins jobs from config
Deploy/Kubenetes/Assessment
Build/job/KnowledgePlatform/InquiryFlinkJob
Deploy/KnowledgePlatform/InquiryFlinkJob
create build, deploy and artifactupload jobs, job-config
Deploy/Kubernetes/Analytics
Service to be Build Build Tag Service to Deploy Deploy Tag Comments Deploy/Kubernetes/Analytics
Variables to update for Ed and vdn
variable name description Value cloudstorage_replace_absolute_path
Boolean value is required to enable/disable cloud storage agnostic urls in db. Update in core/common.yml and KnowledgePlatform/common.yml
String value is required. The value could be either cloud specific base path or cname configured for current cloud storage provider. Update in core/common.yml and KnowledgePlatform/common.yml
valid_cloudstorage_base_urls
Array of String value is required. This array will have list of all cloud path which should be accepted by service. Update in core/common.yml and KnowledgePlatform/common.yml
ex: '["https://storagename.blob.core.windows.net","https://obj.stage.sunbirded.org"]'
cloudstorage_relative_path_prefix
String value is required. This value will be used as prefix for any metadata which holds cloud storage path. Update in Core/common.yml and KnowledgePlatform/common.yml
CONTENT_STORAGE_BASE_PATH
cloudstorage_metadata_list
Array of String value is required. This array will have list of metadata keys which holds cloud storage path. Update in Core/common.yml
["appIcon","posterImage","artifactUrl","downloadUrl","variants","previewUrl","pdfUrl"]
storage url, Update in Core/common.yml.
ex: "https://storagename.blob.core.windows.n
cname url, Update in Core/common.yml
ex: "https://obj.stage.sunbirded.org"
lern variables to be updated in inventory
Update in core, kp, dp common.yml
Supported values are aws, azure, gcloud
youtube api key. Add in core/common.yml
copy value from KP secrets file
ex: "https://storagename.blob.core.windows.n
Add below Api's in Internal Kong configuration
Check if below api's exists or not before adding, if exists replace the existing one if not add it.
Copy - name: createAndUpdateCertificateTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/createOrUpdate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/createOrUpdate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: uploadCertificateTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/uploadTemplate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/uploadTemplate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: projectCertificateReissue
uris: "{{ private_ml_project_prefix }}/api/v1/userProjects/certificateReIssue"
upstream_url: "{{ ml_project_private_url }}/v1/userProjects/certificateReIssue"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: createCertificateTemplateSvg
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/createSvg"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/createSvg"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: createAndUpdateCertificateBaseTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateBaseTemplates/createOrUpdate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateBaseTemplates/createOrUpdate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: certificateBaseTemplateDetailsList
uris: "{{ private_ml_core_prefix }}/api/v1/certificateBaseTemplates/list"
upstream_url: "{{ ml_core_private_url }}/v1/certificateBaseTemplates/list"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}" Execute Below API for ML Analytic Data Product Survey Status Report Configuration Update
Copy curl -X POST \
{{host}}/api/dataset/v1/add \
-H 'authorization: Bearer {{jwt_token}}' \
-H 'content-type: application/json' \
-d '{
"id": "ekstep.analytics.dataset.add",
"ver": "1.0",
"ts": "2016-12-07T12:40:40+05:30",
"params": {
"msgid": "4f04da60-1e24-4d31-aa7b-1daf91c46341"
},
"request": {"dataset":"druid-dataset","datasetSubId":"ml-survey-status-detail-exhaust","datasetConfig":{"type":"ml-survey-status-detail-exhaust","params":{"programId":"program-1","solutionId":"solution-1"}},"datasetType":"druid","visibility":"private","version":"v1","authorizedRoles":["PROGRAM_MANAGER","PROGRAM_DESIGNER"],"validationJson":{"type":"object","properties":{"tag":{"id":"http://api.ekstep.org/dataexhaust/request/tag","type":"string"},"dataset":{"id":"http://api.ekstep.org/dataexhaust/request/dataset","type":"string"},"datasetSubId":{"id":"http://api.ekstep.org/dataexhaust/request/datasetSubId","type":"string"},"requestedBy":{"id":"http://api.ekstep.org/dataexhaust/request/requestedBy","type":"string"},"encryptionKey":{"id":"http://api.ekstep.org/dataexhaust/request/encryptionKey","type":"string"},"datasetConfig":{"id":"http://api.ekstep.org/dataexhaust/request/datasetConfig","type":"object"}},"required":["tag","dataset","datasetConfig"]},"druidQuery":{"id":"ml-survey-status-detail-exhaust","labels":{"user_id" : "UUID","user_type" : "User type","user_sub_tye" : "User sub type","state_name" : "Declared State","district_name" : "District","block_name" : "Block","school_code" : "School ID","school_name" : "School Name","board_name" : "Declared Board","organisation_name" : "Org Name","program_name" : "Program Name","program_externalId" : "Program ID","survey_name" : "Survey Name","survey_externalId" : "Survey ID","survey_submission_id" : "survey_submission_id","submission_status" : "Status of submission","submission_date" : "Submission date"},"dateRange":{"interval":{"startDate":"1901-01-01","endDate":"2101-01-01"},"granularity":"all","intervalSlider":0},"metrics":[{"metric":"total_content_plays_on_portal","label":"total_content_plays_on_portal","druidQuery":{"intervals":"1901-01-01T00:00+00:00/2101-01-01T00:00:00+00:00","dataSource":"ml-survey-status","columns":["user_id", "user_type", "user_sub_type", "state_name", "district_name", "block_name", "school_code", "school_name", "board_name", "organisation_name", "program_name", "program_externalId", "survey_name", "survey_externalId", "survey_submission_id", "submission_status","submission_date"],"queryType":"scan"}}],"output":[{"zip":false,"label":"","dims":["date"],"fileParameters":["id","dims"],"metrics":["user_id", "user_type", "user_sub_type", "state_name", "district_name", "block_name", "school_code", "school_name", "board_name", "organisation_name", "program_name", "program_externalId", "survey_name", "survey_externalId", "survey_submission_id", "submission_status","submission_date"],"type":"csv"}],"sort":["UUID","Program ID","Survey ID","survey_submission_id"],"queryType":"scan"},"supportedFormats":["csv","zip"],"exhaustType":"OnDemand"}}' Execute Below API for ML Analytic Data Product Project Status Report Configuration Update
Copy curl -X POST \
{{host}}/api/dataset/v1/add \
-H 'authorization: Bearer {{jwt_token}}' \
-H 'content-type: application/json' \
-d '{
"id": "ekstep.analytics.dataset.add",
"ver": "1.0",
"ts": "2016-12-07T12:40:40+05:30",
"params": {
"msgid": "4f04da60-1e24-4d31-aa7b-1daf91c46341"
},
"request": {
"dataset": "druid-dataset",
"datasetSubId": "ml-project-status-exhaust",
"datasetConfig": {
"type": "ml-project-status-exhaust",
"params": {
"programId": "program-1",
"solutionId": "solution-1"
}
},
"datasetType": "druid",
"visibility": "private",
"version": "v1",
"authorizedRoles": [
"PROGRAM_MANAGER",
"PROGRAM_DESIGNER"
],
"validationJson": {"type":"object","properties":{"tag":{"id":"http://api.ekstep.org/dataexhaust/request/tag","type":"string"},"dataset":{"id":"http://api.ekstep.org/dataexhaust/request/dataset","type":"string"},"datasetSubId":{"id":"http://api.ekstep.org/dataexhaust/request/datasetSubId","type":"string"},"requestedBy":{"id":"http://api.ekstep.org/dataexhaust/request/requestedBy","type":"string"},"encryptionKey":{"id":"http://api.ekstep.org/dataexhaust/request/encryptionKey","type":"string"},"datasetConfig":{"id":"http://api.ekstep.org/dataexhaust/request/datasetConfig","type":"object"}},"required":["tag","dataset","datasetConfig"]},
"druidQuery":{"id":"ml-project-status-exhaust","labels":{"block_name":"Block","board_name":"Declared Board","project_title_editable":"Project Title","project_completed_date":"Project completion date of the user","user_type":"User Type","designation":"User sub type","school_code":"School ID","project_created_date":"Project start date of the user","project_last_sync":"Project last Synced date","project_duration":"Project Duration","status_of_project":"Project Status","project_id":"Project ID","project_description":"Project Objective","program_externalId":"Program ID","organisation_name":"Org Name","createdBy":"UUID","school_name":"School Name","district_name":"District","program_name":"Program Name","certificate_status_customised":"Certificate Status","state_name":"Declared State"},"dateRange":{"interval":{"startDate":"1901-01-01","endDate":"2101-01-01"},"granularity":"all","intervalSlider":0},"metrics":[{"metric":"total_content_plays_on_portal","label":"total_content_plays_on_portal","druidQuery":{"intervals":"1901-01-01T00:00+00:00/2101-01-01T00:00:00+00:00","dataSource":"sl-project","columns":["createdBy","user_type","designation","state_name","district_name","block_name","school_name","school_code","board_name","organisation_name","program_name","program_externalId","project_id","project_title_editable","project_description","project_created_date","project_completed_date","project_duration","project_last_sync","status_of_project","certificate_status_customised"],"queryType":"scan"}}],"output":[{"zip":false,"label":"","dims":["date"],"fileParameters":["id","dims"],"metrics":["createdBy","user_type","designation","state_name","district_name","block_name","school_name","school_code","board_name","organisation_name","program_name","program_externalId","project_id","project_title_editable","project_description","project_created_date","project_completed_date","project_duration","project_last_sync","status_of_project","certificate_status_customised"],"type":"csv"}],"sort":["UUID","Program ID","Project ID"],"queryType":"scan"},
"supportedFormats": ["csv","zip"],
"exhaustType": "OnDemand"
}
}' Add below variables in Core,KnowledgePlatform and Datapipeline inventory for ED and vdn
Refer to variables and comment section from here link to update below variables.
storage_endpoint variables for public, private, management and artifact is not required in current release, give empty string as value (cloud_public_storage_endpoint, cloud_private_storage_endpoint, etc)
Copy // ###### common.yml ###############
cloud_public_storage_accountname: ""
cloud_public_storage_endpoint: ""
cloud_public_storage_region: ""
cloud_public_storage_project: ""
cloud_private_storage_accountname: ""
cloud_private_storage_endpoint: ""
cloud_private_storage_region: ""
cloud_private_storage_project: ""
cloud_management_storage_accountname: ""
cloud_management_storage_endpoint: ""
cloud_management_storage_region: ""
cloud_management_storage_project: ""
cloud_artifact_storage_accountname: ""
cloud_artifact_storage_endpoint: ""
cloud_artifact_storage_region: ""
cloud_artifact_storage_project: "" Refer to variables and comment section from here link to update below secrets variables
Copy // ########## secrets.yml ###############
cloud_public_storage_secret: ""
cloud_private_storage_secret: ""
cloud_management_storage_secret: ""
cloud_artifact_storage_secret: ""
Refer to variables and comment section from here link to update below variables
Copy // ############# common.yml ##############
cloud_storage_certqr_bucketname: "certqr"
cloud_storage_chatbot_bucketname: "chatbot"
cloud_storage_dial_bucketname: "dial"
cloud_storage_flink_bucketname: "flink-state-backend"
cloud_storage_playercdn_bucketname: "player"
cloud_storage_public_bucketname: "public"
cloud_storage_publicreports_bucketname: "public-reports"
cloud_storage_privatereports_bucketname: "reports"
cloud_storage_samiksha_bucketname: "samiksha"
cloud_storage_schema_bucketname: "schema"
cloud_storage_sourcing_bucketname: "sourcing"
cloud_storage_offlineinstaller_bucketname: "{{env}}-offlineinstaller"
cloud_storage_content_bucketname: "sunbird-content-{{env}}"
cloud_storage_telemetry_bucketname: "telemetry-data-store"
cloud_storage_termsandcondtions_bucketname: "termsandcondtions"
cloud_storage_user_bucketname: "user"
cloud_storage_desktopappcrashlogs_bucketname: "desktopappcrashlogs"
cloud_storage_label_bucketname: "label"
cloud_storage_certservice_bucketname: "{{env}}-e-credentials"
cloud_storage_uci_bucketname: "uci-staging"
cloud_storage_cassandrabackup_bucketname: "cassandra-backup"
cloud_storage_dpcassandrabackup_bucketname: "dp-cassandra-backup"
cloud_storage_dppostgresbackup_bucketname: "dp-postgresql-backup"
cloud_storage_dpredisbackup_bucketname: "dp-redis-backup"
cloud_storage_esbackup_bucketname: "elasticsearch-snapshots"
cloud_storage_influxdbbackup_bucketname: "influxdb-backup"
cloud_storage_jenkinsbackup_bucketname: "jenkins-backup"
cloud_storage_mongobackup_bucketname: "mongodb-backup"
cloud_storage_neo4jbackup_bucketname: "neo4j-backup"
cloud_storage_redisbackup_bucketname: "redis-backup"
cloud_storage_artifacts_bucketname: "{{ artifacts_container }}" Variables to remove from inventory
delete below variables from core/common.yml if its defined.
upstream_url
plugin_upstream_url
discussion_upstream_url
Migrations
Please execute the migration instructions mentioned by the dependent building blocks
Building Block Migration Instructions link
Add new Form
To create a new form to show or remove the onboarding popups \
Copy // Some codecurl --location --request POST '<https://{{env}}>/api/data/v1/form/create' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{API KEY}}' \
--data-raw '{
"request": {
"type": "newUserOnboarding",
"action": "onboarding",
"subType": "global",
"rootOrgId": <please specify your root org id>,
"component": "portal",
"data":{
"fields": {
"shownewUserOnboarding": "true"
}
}
}
}' CoKreat Deployments and migration
Please follow the instructions mentioned in the below link by CoKreat building block to upgrade CoKreat from 5.0.0 to 5.1.0
Variables lists and build tag is same for Ed and vdn. Use release-5.1.0-vdn_RC1 for knowlg service deployments in VDN
