MLF can be configured through two different types of parameters:
-
MLF API parameters
-
MLF environment variables
MLF API parametersThe MLF API requires the configuration of the file:
/opt/ptv-optima-as/standalone/configuration/optima/optima-configuration.xml
Two specific tags must be configured:
- <mlf>
- <proxy>
<mlf> tag
Select the tag <configuration>/<mlf> of the file optima-configuration.xml.
|
Property |
Type |
Description |
|---|---|---|
|
mlf |
Object |
Contains two tags:
|
Example:
<mlf>
<customerId>4b0e2501-bfa3-49f9-8c7b-5d75f6cf12d7</customerId>
<serverHost>https://mlf01
</mlf>Select the tag <configuration>/<proxy> of the file optima-configuration.xml.
|
Property |
Type |
Description |
|---|---|---|
|
proxy |
Object |
The <url> tag contains the URL of the proxy associated to the MLF service. Here you can make use of Apache Common Expression. |
Example:
<proxy>
<mlf>
<url>${configuration[@name='optima']/mlf/serverHost}/mlf-api-processing-api/ml-forecast/public/forecast/realtime</url>
</mlf>
</proxy>MLF needs to be configured through a wide set of environment variables, listed in the folder config. Some of them have a default value which usually does not need to be changed.
The variables are injected by Kubernetes into the Docker container, using a configuration file with the .yaml extension.
Every variable is involved in one or more MLF processes (→ MLF processes).
The main variables that may be interesting for the Project Manager are:
- HISTORICAL_DATA_TTL_DAYS
- CUSTOMERID
- FORECAST_CRON
-
FORECAST_WINDOW
In the following table you can find all the possible environment variables.
| Name | Description | Microservices (where they apply) |
|---|---|---|
|
LOG_LEVEL |
Data type: String Log level applicable to all processes. Values:
Default value: INFO |
|
|
LOG_FILE |
Data type: String Log file path. Involved in all processes. Default value: Not applicable |
|
|
LICENSE_FILE |
Data type: String License file path. Involved in all processes. Default value: Not applicable |
|
|
ASSOCIATION_WINDOW_FLOW |
Data type: Integer Association window width to consider to link real time flow data to a representative profile. Involved in the Forecasting process. Default value: 60 |
|
|
ASSOCIATION_WINDOW_SPEED |
Data type: Integer Association window width to consider to link real time speed data to a representative profile. Involved in the Forecasting process. Default value: 15 |
|
|
VALIDITY_DURATION |
Data type: Integer Frequency of data that representative are built with. Involved in the:
Default value: 900 |
|
|
FILTER_DATA |
Data type: Boolean Used if raw data must be smoothed before performing the clustering procedure. Involved in:
Values:
Default value: FALSE |
|
|
SERVER.PORT |
Data type: Integer Port number associated to the running microservice. Involved in all processes. Default value: Not applicable (depending on the microservice) |
|
|
KAFKA_BOOTSTRAP_ADDRESS |
Data type: String Combination <KAFKA_HOST:PORT>. Involved in all processes. Default value: Not applicable |
|
|
ENVIRONMENTID |
Data type: String Unique name of the environment where the microservices run. Involved in all processes. Default value: Not applicable |
|
|
UPDATE_TIME_UNIT |
Data type: String Time unit used for configuration variables. Involved in all processes. Default value: HOURS |
|
|
UPDATE_SHORT_DURATION |
Data type: Integer Time period between two consecutive runs of self-learning iterations in the most frequent phase. Involved in the Clustering process. Default value: 24 |
|
|
UPDATE_SHORT_COUNT |
Data type: Integer How many times the self learning procedure has to be run before passing from the most frequent phase to the medium frequent phase. Involved in the Clustering process. Default value: 1 |
|
|
UPDATE_MEDIUM_COUNT |
Data type: Integer How many times the self-learning procedure has to be run before passing from medium to frequent phase to the least frequent phase. Involved in the Clustering process. Default value: 0 |
|
|
DIVERGENCETYPE |
Data type: String Type of distance to perform the clustering procedure. Involved in the Clustering process. Values:
Default value: CORRELATION |
|
|
CARDINALITY_THRESHOLD |
Data type: Integer Minimum number of elements within a cluster. Involved in the Clustering process. Default value: 1 |
|
| CLUSTER_UPDATE_CRON |
Data type: Cron expression Cron expression specifying the precise times and dates when the clustering process should begin. Involved in the Clustering Update Process. Example: 0 15 10 * * ? to say Fire at 10:15 AM every day Default value: 0 0 5 * * * |
|
|
HISTORICAL_DATA_TTL_DAYS |
Data type: Integer Number of days of historical data needed to run the clustering process (learning phase). Involved in the:
Involved in the Clustering process. Example: 120 Default value: 90 |
|
|
DB_CASSANDRA_HOST |
Data type: String Address hosting the Cassandra installation. Involved in all processes. Default value: Not applicable |
|
|
DB_CASSANDRA_NAME |
Data type: String Keyspace name associated to the Cassandra cluster . Involved in all processes. Default value: Not applicable |
|
|
DB_CASSANDRA_DATACENTER |
Data type: String Data-center name associated to the Cassandra cluster. Involved in all processes. Default value: Not applicable |
|
|
DB_CASSANDRA_PORT |
Data type: String Listening port associated to Cassandra. Involved in all processes. Default value: Not applicable |
|
|
DB_CASSANDRA_USERNAME |
Data type: String Username for the connection to the Cassandra cluster. Involved in all processes. Default value: Not applicable |
|
|
DB_CASSANDRA_PASSWORD |
Data type: String Password for the connection to the Cassandra cluster. Involved in all processes. Default value: Not applicable |
|
|
DB_CASSANDRA_TIMEOUT |
Data type: Integer Number of seconds defining a timeout for Cassandra responses. Involved in all processes. Default value: 60 |
|
|
DB_CASSANDRA_SSL |
Data type: Boolean SSL enablement. Values:
Involved in all processes. Default value: FALSE |
|
|
DB_CASSANDRA_LOAD_BALANCER |
Data type: Boolean Load balancer connection enablement. Values:
Involved in all processes. Default value: FALSE |
|
|
DB_CASSANDRA_REPLICATION_FACTOR |
Data type: Integer Set the replication factor used to persist data in Cassandra. Involved in all processes. Default value: 1 |
|
|
DB_CASSANDRA_MAX_TRIALS |
Data type: Integer Max number of consecutive trials that can be performed to persist data in Cassandra when it seems not available. Involved in all processes. Default value: 10 |
|
|
DB_CASSANDRA_TRIALS_DELAY_MS |
Data type: Integer Time period, measured in ms, between two consecutive trials where data persists in Cassandra. Involved in all processes. Default value: 1000 |
|
|
CUSTOMERID |
Data type: UUID Identifier of the customer. It is used only by the forecast-results microservice, that has to dump results on the correct DB for the correct customer. If the Harmonizer is present, CUSTOMERID must match the customer identifier associated to the Harmonizer. Default value: Not applicable |
|
|
FORECAST_SLAVE_MAX_POLL_RECORDS |
Data type: Integer Number of records polled at the same time. It’s an optimization parameter. Involved in the Forecasting process. Default value: 100 |
|
|
FORECAST_PIVOTING |
Data type: Boolean Enables the option of adapting the forecast to real-time data. Values:
Involved in the Forecasting process. Default value: TRUE |
|
|
FORECAST_CRON |
Data type: Cron expression Cron expression specifying when the Forecasting process has to start. Involved in the Forecasting process. Example: 0 15 10 * * ? to say Fire at 10:15 AM every day Default value: 0 * /5 * * * * |
|
|
FORECAST_CHUNK_SIZE |
Data type: Integer Number of streets that must be processed at the same time by the forecast. Involved in the Forecasting process. Default value: 100 |
|
|
AGGREGATION_TIME_UNIT |
Data type: String Time unit supported in the current configuration variables. Involved in the Forecasting process. Default value: MINUTES |
|
|
AGGREGATION_TIME_DURATION |
Data type: Integer Time duration, expressed in AGGREGATION_TIME_UNIT, so that the data within it is aggregated together to produce the forecast. Involved with the Forecasting process. Default value: 60 |
|
|
AGGREGATION_TIME_ADVANCE |
Data type: Integer Time duration, expressed in AGGREGATION_TIME_UNIT, used to specify which is the shift between two aggregation windows. Involved in the Forecasting process. Default value: 300 |
|
|
FORECAST_WINDOW |
Data type: Integer Forecast horizon, expressed in minutes. Involved in the Forecasting process. Example: 60 Default value: 75 |
|
Examples of micro-services files
Here are some examples of the five microservices provided by the MLF system.
Each of them is in a deployment.yaml file and specifies the variable ENVIRONMENTID.
mlf-forecast-slave-processing-worker
nameOverride: mlf-forecast-slave-processing-worker
fullnameOverride: mlf-forecast-slave-processing-worker
service:
labels:
rest-api: "true"
ports:
- name: http
port: 80
targetPort: 9003
deployment:
pullSecrets:
- name: registry-pull-secret
replicaCount: 1
resources:
limits:
cpu: 0.5
memory: 3Gi
requests:
cpu: 0.3
memory: 2Gi
containers:
ports:
- containerPort: 9003
protocol: TCP
readinessProbe:
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 20
httpGet:
path: /healthz/ready
port: 9003
livenessProbe:
initialDelaySeconds: 400
periodSeconds: 10
failureThreshold: 18
httpGet:
path: /healthz/live
port: 9003
platformConfigFilesFrom:
- configMap: mlf-forecast-slave-processing-worker
subPath: application-env.properties
platformSecretFilesFrom:
- secret: mlf-kafka-forecast-secret
- secret: mlf-cassandra-data-secret
env:
- name: ENVIRONMENTID
value: "onpremise"
- name: DB_CASSANDRA_DATACENTER
value: dc1
- name: FORECAST_PIVOTING
value: "false"
- name: ROCKSDB_BOUNDED_MEMORY_ENABLED
value: "false"
- name: FORECAST_CRON
value: "0 */5 * * * *"
- name: FORECAST_WINDOW
value: "75"
nodeSelector:
kubernetes.io/os: "linux"
serviceMonitor:
enabled: false #If `true`, a ServiceMonitor resource for the prometheus-operator is creatednameOverride: mlf-cluster-slave-processing-worker
fullnameOverride: mlf-cluster-slave-processing-worker
service:
labels:
rest-api: "true"
ports:
- name: http
port: 80
targetPort: 9001
deployment:
pullSecrets:
- name: registry-pull-secret
replicaCount: 4
resources:
limits:
cpu: 0.2
memory: 1Gi
requests:
cpu: 0.1
memory: 1Gi
containers:
ports:
- containerPort: 9001
protocol: TCP
readinessProbe:
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 20
httpGet:
path: /healthz/ready
port: 9001
livenessProbe:
initialDelaySeconds: 320
periodSeconds: 10
failureThreshold: 18
httpGet:
path: /healthz/live
port: 9001
platformConfigFilesFrom:
- configMap: mlf-cluster-slave-processing-worker
subPath: application-env.properties
platformSecretFilesFrom:
- secret: mlf-kafka-forecast-secret
- secret: mlf-cassandra-data-secret
env:
- name: ENVIRONMENTID
value: "onpremise"
- name: DB_CASSANDRA_DATACENTER
value: dc1
- name: HISTORICAL_DATA_TTL_DAYS
value: "90"
- name: CLUSTER_UPDATE_CRON
value: "0 0 5 * * *"
nodeSelector:
kubernetes.io/os: "linux"
serviceMonitor:
enabled: false #If `true`, a ServiceMonitor resource for the prometheus-operator is creatednameOverride: mlf-importer-processing-worker
fullnameOverride: mlf-importer-processing-worker
service:
labels:
rest-api: "true"
ports:
- name: http
port: 80
targetPort: 9003
deployment:
pullSecrets:
- name: registry-pull-secret
replicaCount: 1
resources:
limits:
cpu: 0.2
memory: 1Gi
requests:
cpu: 0.1
memory: 1Gi
containers:
ports:
- containerPort: 9003
protocol: TCP
readinessProbe:
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 20
httpGet:
path: /healthz/ready
port: 9003
livenessProbe:
initialDelaySeconds: 300
periodSeconds: 10
failureThreshold: 18
httpGet:
path: /healthz/live
port: 9003
platformConfigFilesFrom:
- configMap: mlf-importer-processing-worker
subPath: application-env.properties
platformSecretFilesFrom:
- secret: mlf-kafka-forecast-secret
- secret: mlf-cassandra-data-secret
env:
- name: ENVIRONMENTID
value: "onpremise"
- name: DB_CASSANDRA_DATACENTER
value: dc1
- name: HISTORICAL_DATA_TTL_DAYS
value: "90"
- name: DB_CASSANDRA_REPLICATION_FACTOR
value: "1"
nodeSelector:
kubernetes.io/os: "linux"
serviceMonitor:
enabled: false #If `true`, a ServiceMonitor resource for the prometheus-operator is creatednameOverride: mlf-forecast-results-processing-api
fullnameOverride: mlf-forecast-results-processing-api
ingress:
enabled: true
http:
- serviceName: mlf-forecast-results-processing-api
servicePort: http
path: /mlf-forecast-results-processing-api/(.*)
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
service:
labels:
rest-api: "true"
ports:
- name: http
port: 80
targetPort: 9007
deployment:
pullSecrets:
- name: registry-pull-secret
replicaCount: 1
resources:
limits:
cpu: 0.2
memory: 1Gi
requests:
cpu: 0.1
memory: 512Mi
containers:
ports:
- containerPort: 9007
protocol: TCP
readinessProbe:
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 20
httpGet:
path: /healthz/ready
port: 9007
livenessProbe:
initialDelaySeconds: 320
periodSeconds: 10
failureThreshold: 18
httpGet:
path: /healthz/live
port: 9007
platformConfigFilesFrom:
- configMap: mlf-forecast-results-processing-api
subPath: application-env.properties
platformSecretFilesFrom:
- secret: mlf-kafka-forecast-secret
- secret: mlf-pg-data-admin-secret
env:
- name: ENVIRONMENTID
value: "onpremise"
- name: CUSTOMERID
value: "4b0e2501-bfa3-49f9-8c7b-5d75f6cf12d7"
- name: TZ
value: "Europe/Rome"
nodeSelector:
kubernetes.io/os: "linux"
serviceMonitor:
enabled: false #If `true`, a ServiceMonitor resource for the prometheus-operator is creatednameOverride: mlf-api-processing-api
fullnameOverride: mlf-api-processing-api
ingress:
enabled: true
http:
- serviceName: mlf-api-processing-api
servicePort: http
path: /mlf-api-processing-api/(.*)
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
service:
# headless: true
labels:
rest-api: "true"
ports:
- name: http
port: 80
targetPort: 9008
annotations:
spec-path: "spec/openapi.json"
# selector:
# app.kubernetes.io/instance: mlf-api-processing-api
# app.kubernetes.io/name: mlf-api-processing-api
deployment:
pullSecrets:
- name: registry-pull-secret
replicaCount: 4
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 0.05
memory: 0.5Gi
containers:
ports:
- containerPort: 9008
protocol: TCP
readinessProbe:
initialDelaySeconds: 60
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 20
httpGet:
path: /healthz/ready
port: 9008
livenessProbe:
initialDelaySeconds: 320
periodSeconds: 10
failureThreshold: 18
httpGet:
path: /healthz/live
port: 9008
platformConfigFilesFrom:
- configMap: mlf-api-processing-api
subPath: application-env.properties
platformSecretFilesFrom:
- secret: mlf-kafka-forecast-secret
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ENVIRONMENTID
value: "onpremise"
- name: HEADLESS_SERVICE_NAME
value: mlf-api-processing-api-headless
- name: ENCODE_OPENLR
value: "false"
nodeSelector:
kubernetes.io/os: "linux"
serviceMonitor:
enabled: false #If `true`, a ServiceMonitor resource for the prometheus-operator is created