Folder Structure

.

β”œβ”€β”€ LICENSE

β”œβ”€β”€ README.md

β”œβ”€β”€ back-up_reports_configs

β”‚ β”œβ”€β”€ pre_prod_reports

β”‚ └── prod_reports

β”œβ”€β”€ cloud_storage

β”‚ β”œβ”€β”€ __init__.py

β”‚ β”œβ”€β”€ aws.py

β”‚ β”œβ”€β”€ cloud.py

β”‚ β”œβ”€β”€ gcp.py

β”‚ β”œβ”€β”€ ms_azure.py

β”‚ β”œβ”€β”€ oracle.py

β”‚ └── py_cloud_migration.py

β”œβ”€β”€ config.ini

β”œβ”€β”€ config.sample

β”œβ”€β”€ druid_data_product_query_config.txt

β”œβ”€β”€ hp_reports_metrics

β”‚ └── hawkeye_report_metric.py

β”œβ”€β”€ manual_report_scripts

β”‚ β”œβ”€β”€ domain_criteria_report.py

β”‚ β”œβ”€β”€ question_domain_criteria_report.py

β”‚ β”œβ”€β”€ question_report.py

β”‚ └── status_report.py

β”œβ”€β”€ migrations

β”‚ β”œβ”€β”€ lib

β”‚ └── releases

β”œβ”€β”€ observations

β”‚ β”œβ”€β”€ ml_observation_faust-data

β”‚ β”œβ”€β”€ py_observation_batch_del_update.py

β”‚ β”œβ”€β”€ py_observation_evidence_streaming.py

β”‚ β”œβ”€β”€ py_observation_evidence_streaming_batch_refreshment.py

β”‚ β”œβ”€β”€ py_observation_streaming.py

β”‚ β”œβ”€β”€ py_observation_streaming_update_druid_data.py

β”‚ β”œβ”€β”€ pyspark_obs_domain_batch.py

β”‚ β”œβ”€β”€ pyspark_obs_status_batch.py

β”‚ β”œβ”€β”€ pyspark_observation_rollup.py

β”‚ β”œβ”€β”€ pyspark_observation_status_batch.py

β”‚ β”œβ”€β”€ pyspark_observation_streaming_refreshment_batch.py

β”‚ β”œβ”€β”€ pyspark_temp_punjab_entities.py

β”‚ └── pyspark_temp_script_add_cols.py

β”œβ”€β”€ projects

β”‚ β”œβ”€β”€ __pycache__

β”‚ β”œβ”€β”€ py_gather_program.py

β”‚ β”œβ”€β”€ pyspark_project_batch _entire_db.py

β”‚ β”œβ”€β”€ pyspark_project_batch.py

β”‚ β”œβ”€β”€ pyspark_project_deletion_batch.py

β”‚ β”œβ”€β”€ pyspark_project_pagination_batch.py

β”‚ β”œβ”€β”€ pyspark_project_rollup.py

β”‚ β”œβ”€β”€ pyspark_project_wo_pagination.py

β”‚ β”œβ”€β”€ pyspark_started_inProgress_programdashboard.py

β”‚ β”œβ”€β”€ pyspark_submitted_programdashboard.py

β”‚ └── udf_func.py

β”œβ”€β”€ python3 -> /usr/bin/python3

β”œβ”€β”€ release-notes

β”‚ └── 6.0.0.md

β”œβ”€β”€ requirements.txt

β”œβ”€β”€ run.sh

β”œβ”€β”€ run_program.sh

β”œβ”€β”€ run_weekly.sh

β”œβ”€β”€ shell_script_config

β”œβ”€β”€ survey

β”‚ β”œβ”€β”€ ml_survey_evidence_faust-data

β”‚ β”œβ”€β”€ ml_survey_faust-data

β”‚ β”œβ”€β”€ py_survey_batch_del_update.py

β”‚ β”œβ”€β”€ py_survey_evidence_streaming.py

β”‚ β”œβ”€β”€ py_survey_evidence_streaming_batch_refreshment.py

β”‚ β”œβ”€β”€ py_survey_streaming.py

β”‚ β”œβ”€β”€ py_survey_streaming_batch_refreshment.py

β”‚ β”œβ”€β”€ pyspark_survey_rollup.py

β”‚ └── pyspark_survey_status.py

β”œβ”€β”€ trust_review_scripts

β”‚ β”œβ”€β”€ Trust Review Report.pdf

β”‚ β”œβ”€β”€ active_users.py

β”‚ β”œβ”€β”€ config.sample

β”‚ └── ml_submissions_unique_users.py

└── urgent_data_metrics

└── imp_project_metrics.py

\

Module details

  • back-up_reports_configs - In this folder, we store old reports that are no longer in use, solely for reference purposes.

  • cloud_storage - Within this directory, we maintain the "Cloud" class. This class is utilized in scripts to facilitate the transmission of data or files to the cloud.

  • hp_reports_metrics - Within this folder, we have scripts dedicated to generating CSV files specifically for program, solution, and district metrics.

  • lib - Within this folder, we have scripts dedicated to generating and pussing logs to MongoDB

  • manual_report_scripts - Within this folder, we have scripts dedicated to generating data related to observations.

  • migrations - Within this folder, we have JSON data and script related to report creations.

  • observations - Inside this directory, you can find both real-time and batch scripts intended for handling observations.

  • projects - Inside this directory, you can find batch scripts intended for handling projects.

  • release-notes - In this folder, we store documentation detailing the necessary steps and tasks to be carried out for deployment.

  • survey - Inside this directory, you can find both real-time and batch scripts intended for handling Survey.

  • trust_review_scripts -Within this folder, you can find scripts designed to provide metrics relevant to Trust review meetings.

  • config.sample - This file stores all the essential variables needed for our scripts.

  • druid_data_product_query_config.txt -This file contains the necessary URLs (CURLs) responsible for updating the program dashboard's CSV-related data.

  • requirements.txt - This file includes a list of necessary packages required for the successful execution of all our scripts.

  • run.sh - This script outlines the sequence and instructions for running our batch scripts in the intended order.

  • run_program.sh - This script outlines the sequence and instructions for running selected scripts in the intended order.

  • shell_script_config.sample - This file stores all the essential variables needed for our .sh files.