Added new recipe for visit tracking

.gitbook/assets/docker-compose.yml

@@ -0,0 +1,32 @@
+version: '2'
+services:
+  zookeeper:
+    image: confluentinc/cp-zookeeper:latest
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_TICK_TIME: 2000
+    ports:
+      - 22181:2181
+
+  kafka:
+    image: confluentinc/cp-kafka:latest
+    depends_on:
+      - zookeeper
+    ports:
+      - 29092:29092
+    environment:
+      KAFKA_BROKER_ID: 1
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+
+  telemetry-service:
+    image: sunbird/telemetry-service:release-3.3.0_RC1
+    ports:
+      - 9001:9001
+    environment:
+      telemetry_kafka_broker_list: kafka:9092
+      telemetry_kafka_topic: dev.telemetry.ingest
+      telemetry_local_storage_type: kafka

learn/recipes/pageview-tracking.md

@@ -0,0 +1,63 @@
+# Tracking Pageviews using raw Telemetry Data
+
+Telemetry events with eid `IMPRESSION` can be used to create a time-series pageview graph. [Impression events](http://docs.sunbird.org/latest/developer-docs/telemetry/specification/#impression) are generated when a user visits any page. Impression events look like the one below
+
+```JSON
+{
+  "eid": "IMPRESSION",
+  "ets": 1664162202978,
+  "ver": "3.0",
+  "mid": "LOAD_TEST_6c029add-7508-43c4-aefc-9591cb53e5f9",
+  "actor": {
+    "id": "7ec05264-b111-4833-9594-55d807c44084",
+    "type": "User"
+  },
+  "context": {
+    "channel": "99fbe3f8-c07a-4ace-b0ac-f2fa7411da07",
+    "pdata": {
+      "id": "sunbird.app",
+      "pid": "sunbird-portal.collectioneditor",
+      "ver": "1"
+    },
+    "env": "user",
+    "sid": "74489131-fdbb-448c-b36b-98fe8c70971b",
+    "did": "7b238a24-bc15-487d-8a59-b4ad9f37d178",
+    "cdata": [],
+    "rollup": {
+      "l1": "do_21271780182595993616932",
+      "l2": "do_9e90c447-7b87-42d9-898b-6e6fbc990957",
+      "l3": "do_869422f8-20ea-4e43-9f13-26c666c416d2",
+      "l4": "do_3e187467-d966-4e6e-9af1-ea057acce678"
+    }
+  },
+  "object": {
+    "id": "do_21274813398450176015065",
+    "type": "Community",
+    "ver": "3",
+    "rollup": {
+      "l1": "do_21273610197362278414264",
+      "l2": "do_21271780182595993616932",
+      "l3": "do_312469507013812224118164",
+      "l4": "do_21271780182595993616932"
+    }
+  },
+  "edata": {
+    "type": "workflow",
+    "subtype": "Scroll",
+    "pageid": "c5577e38-6c47-4795-952b-143057fdb15a",
+    "uri": "/workspace/content/create",
+    "duration": 9,
+    "visits": []
+  },
+  "tags": []
+}
+```
+
+Here, `ets` is the unix timestamp of when the event ocurred, and `edata.uri` is the relative URL of the page that was visited. Using these 2 attributes from IMPRESSION events, a graph like the below one can be generated. To do this, aggregate the events based on date, and then count the number of events for each date. This data can be used to plot a graph like below. 
+
+![Visits Graph using IMPRESSION events](<../../.gitbook/assets/recipe-visits-bar-graph.png>)
+
+
+Likewise, by aggregating on the `edata.uri` a table representing page wise total views may be generated.
+
+![Page wise total views](<../../.gitbook/assets/recipe-views-pagewise.png>)

use/installation-guide.md

@@ -2,7 +2,32 @@
 
 All components in Sunbird Obsrv can be installed through automation scripts. The automation scripts require [Ansible](https://docs.ansible.com/ansible/latest/index.html) as a prerequisite. Some of the components also require the de-facto package manager for Kubernetes, [helm](https://helm.sh/docs/) as a prerequisite to run the component on [Kubernetes](https://kubernetes.io).
 
-#### Telemetry Service
+## Using Docker
+Using docker, the telemetry API service can be easily tested. This service exposes a single endpoint `POST /v1/telemetry` which will accept the telemetry events in JSON format. The service can print the telemetry to console, write it to a file, or push it to a Kafka topic. 
+
+Use the below commands to first build the docker image for the service.
+```shell
+git clone https://github.com/project-sunbird/sunbird-telemetry-service.git
+cd sunbird-telemetry-service
+./build.sh release-3.3.0 telemetry-service <your dockerhub user>
+```
+
+The telemetry service image would now be available as <your dockerhub user>/telemetry-service:release-3.3.0
+Download this [docker-compose.yml](../.gitbook/assets/docker-compose.yml) file and run the below command to get the service up. In the docker compose file, you may change the telemetry service image to use your own account (eg: change to <your dockerhub user>/telemetry-service:release-3.3.0)
+
+```shell
+docker-compose up -d
+```
+
+Once the service is up you can run the below cURL to send a test event to the telemetry service. If everything is working fine, you should be able to view the event with kafka console consumer (eg: `./bin/kafka-console-consumer.sh --bootstrap-server localhost:29092 --topic dev.telemetry.ingest`)
+
+```shell
+curl -d '{"eid":"IMPRESSION","ets":1664162202978,"ver":"3.0","mid":"LOAD_TEST_6c029add-7508-43c4-aefc-9591cb53e5f9","actor":{"id":"7ec05264-b111-4833-9594-55d807c44084","type":"User"},"context":{"channel":"99fbe3f8-c07a-4ace-b0ac-f2fa7411da07","pdata":{"id":"sunbird.app","pid":"sunbird-portal.collectioneditor","ver":"1"},"env":"user","sid":"74489131-fdbb-448c-b36b-98fe8c70971b","did":"7b238a24-bc15-487d-8a59-b4ad9f37d178","cdata":[],"rollup":{"l1":"","l2":"","l3":"","l4":""}},"object":{"id":"","type":"Community","ver":"3","rollup":{"l1":"","l2":"","l3":"","l4":""}},"edata":{"type":"workflow","subtype":"Scroll","pageid":"c5577e38-6c47-4795-952b-143057fdb15a","uri":"/workspace/content/create","duration":9,"visits":[]},"tags":[]}
+' -H "Content-Type: application/json" -X POST http://localhost:9001/v1/telemetry
+```
+
+## Using Kubernetes
+### Telemetry Service
 
 The Sunbird Obsrv Telemetry service can be deployed onto Kubernetes using the helm chart. The deployment is handled using Ansible to manage the configuration and the commands that are necessary. The deployments can also be integrated into Jenkins, a popular CI/CD tool. The Telemetry Service deployment also has the capability of configurable horizontal scaling using the Horizontal Pod Scaling (HPA) concept of Kubernetes. A sample command to deploy the telemetry service on Kubernetes is provided below.
 
@@ -15,7 +40,7 @@ helm install telemetry-service sunbird-devops/kubernetes/helm_charts/telemetry
 Telemetry Service helm chart
 {% endembed %}
 
-#### Data Pipeline
+### Data Pipeline
 
 Sunbird Obsrv Data Pipeline consists of a series of real-time streaming jobs chained together to unzip, transform and enrich the telemetry data. We use ansible and helm charts to deploy the series of jobs. The list of jobs that need to be deployed and their configurations can be controlled by the [ansible defaults configuration](https://github.com/project-sunbird/sunbird-data-pipeline/blob/release-4.8.0/kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml#L168-L339).
 
@@ -37,7 +62,7 @@ userLocations(state, district, block, cluster, school), rootOrg (orgName), userI
 framework, profileUserTypes (usertype, subusertype)
 ```
 
-#### Data Service
+### Data Service
 
 The Data Service is a collection of data exhaust and report apis. The Data Service can be installed on Kubernetes using Ansible and Helm. A sample command to install the service is provided below. All the required configuration is managed using the [ansible configuration](https://github.com/project-sunbird/sunbird-devops/blob/release-4.8.0/ansible/roles/stack-sunbird/defaults/main.yml#L987-L1015).
 
@@ -50,13 +75,13 @@ helm install telemetry-service sunbird-devops/kubernetes/helm_charts/analytics
 Data Service helm chart
 {% endembed %}
 
-#### Report Service
+### Report Service
 
 {% embed url="https://github.com/project-sunbird/sunbird-devops/tree/release-4.8.0/kubernetes/helm_charts/core/report" %}
 Report Service helm chart
 {% endembed %}
 
-#### Summarisers
+### Summarisers
 
 {% embed url="https://github.com/project-sunbird/sunbird-data-pipeline/tree/release-4.8.0/ansible/roles/data-products-deploy" %}
 Ansbile role for Summarizer data products