Component Diagram

GitHub Repository

https://github.com/Sunbird-Ed/ml-analytics-service.giTgithub.com

The Ml-analytics service is constructed upon a framework that incorporates Kafka, MongoDB, Druid, and cloud storage. the ML-Analytics service collects data from MongoDB or Kafka, performs data transformation, and then transfers the refined data to either Cloud Storage or Kafka. This data is then made available in Druid for further analysis needs.

The Analytics Service is composed of three key modules,

1. Observations

The basically do the batch and real-time data ingestion.

• Batch: Operating on a predefined schedule, it reads Observation-related data from MongoDB. The data is then transformed into a flattened structure before being loaded into Druid for further analysis.

• Real-time: The system subscribes to a designated Kafka topic that holds observations. Upon receiving new observations, the system processes them and subsequently publishes the processed data to another Kafka topic. This processed data is then consumed by Druid for real-time analysis.

ML Survey Service

1. Projects

• Batch: Operating on a predefined schedule, it reads project-related data from MongoDB. The data is then transformed into a flattened structure before being loaded into Druid for further analysis.

ML Project Service

1. Survey

The basically do the batch and real-time data ingestion.

• Batch: Operating on a predefined schedule, it reads survey-related data from MongoDB. The data is then transformed into a flattened structure before being loaded into Druid for further analysis.

• Real-time: The system subscribes to a designated Kafka topic that holds a survey. Upon receiving the new survey, the system processes them and subsequently publishes the processed data to another Kafka topic. This processed data is then consumed by Druid for real-time analysis.

ML Survey Service\

1. Migrations

Migration scripts are used to create, update or retire Charts and Reports in the respective environments.