MLF workflow could be considered as a composition of three different processes:
- Importing process
- Learning Process
- Forecasting Process
Importing Process
It takes data from input topic and, after some cleaning operations, stores raw data in the Cassandra DB.
Learning Process
It is a configurable scheduled process, structured in three main steps:
- Takes data from Cassandra DB (stored continously by the Importing process)
- Learns from them
- Outputs the results in Cassandra DB.
The process can be scheduled through a job that usually runs one time per day.
The scheduling is handled through a set of configuration variables (→ MLF configuration parameters):
- UPDATE_TIME_UNIT
- UPDATE_SHORT_DURATION
- UPDATE_SHORT_COUNT
- UPDATE_MEDIUM_COUNT
Forecasting Process
It takes data from aninput topic, aggregate them and, with aggregated results, produces forecast in real time for every street for which we have at least one data in a configurable window (generally 1 hour).
Output results are put in another Kafka topic. Forecast-results microservice, one for each customer, reads from the output topic and exposes results via REST API.
Optima can handle results via REST API.
The architecture of the processes is shown in the diagram:
