Harmonizer configuration

Contains the general configuration parameters of Harmonizer .

Some parameters are directly mapped to the parameters of the standard Jakarta EE ScheduleExpression class.

If timeNow is the current time, Harmonizer runs with the frequency set by the schedule elements.

It collects data from all sources that belong to the time window:

[(timeNow – processingDelay – clusterWindow), (timeNow – processingDelay)].

Harmonizer produces data fusion results that are valid in the temporal window:

[(timeNow + processingDelayResults), (timeNow + processingDelayResults + resultsValidity)].

The setup of the filtering procedure is controlled by the <filterRules> tag, containing one or more rules (<rule> tag) to be applied to filter each source.

Data is filtered according to different conditions ( <condition> tag).

Each rule can contain one or more conditions and a single <action> tag.

A condition represents the logical operation we would like to check in order to trigger the action.

A condition is composed by an attribute (see → Attributes), an operator (see → Operators), and a value.

Conditions are evaluated in the same order as they are specified and all conditions must be satisfied in order to apply the rule (logical AND mode).

Important:  Rules are applied in the top-down order. This means that the order of rules is significant.

Example 1

In this example, six rules are listed and every rule has only one condition.

<configurations>
....
        <harmonizer>
....
            <filterRules>
                <rule name="filterKO">
                    <condition attr="status" operator="maxequal" value="2" />
                    <action attr="reliabilitymultiplier" value="0.0" />
                </rule>
                <rule name="clocDisabledRule">
                    <condition attr="clocDisabled" operator="equal" value="1" />
                    <action attr="reliabilitymultiplier" value="0.0" />
                </rule>
                <rule name="speedzero">
                    <condition attr="speed" operator="min" value="0" />
                    <action attr="reliabilitymultiplier" value="0.0" />
                </rule>
                <rule name="flowzero">
                    <condition attr="flow" operator="min" value="0" />
                    <action attr="reliabilitymultiplier" value="0.0" />
                </rule>
                <rule name="speedmax">
                    <condition attr="speed" operator="maxequal" value="180" />
                    <action attr="reliabilitymultiplier" value="0.0" />
                </rule>
                <rule name="flowmax">
                    <condition attr="flow" operator="maxequal" value="2000" />
                    <action attr="reliabilitymultiplier" value="0.0" effectRelevance="0.8" />
                </rule>
            </filterRules>
....
        </harmonizer>
....
</configurations>
                

As you can see, the <action> tag can have the attribute reliabilitymultiplier with the associated attribute value ∈[0,1], and the optional attribute effectRelevance.

The value is the number multiplied by the reliability of the measurements to set the reliability field in Harmonizer output:

HarmonizedReliability=value*reliability

The reliability of a measure is a value in the range [0-100] which the data provider expresses for each traffic state to define how reliable the given data is according to their own standards. Each traffic state has only one reliability value, which therefore equally affects all related measurements.

Important:  It is possible that the reliability or the HarmonizedReliability of one or more traffic states is 0. In this case, when the Harmonizer examines a specific element (e.g., a single street), such traffic states are discarded, hence they don't contribute to the output at all. Should all traffic states of the same element be equal to 0, the system won't generate any relevant output for that element.

The attribute (effectRelevance) is used to sort the rules, in case of applyStrongestRuleOnly.

Attributes

The complete list of attributes that can be used to build a rule is indicated in the table:

Parameter	Description
reliabilitymultiplier	Mandatory. It is associated to a value ∈[0,1].
effectRelevance	If the effectRelevance is specified AND applyAllRules=false (→ Applying rules: applyAllRules parameter), the effectRelevance overwrites the value of reliabilitymultiplier.
speed	Real value (measured in Km/h). -1: Get the speed defined in the model on the street (strt.sped). strt.sped: Get the speed defined in the model on the street.
flow	Real value (measured in Veh/h). -1: Get the capacity defined in the model on the street (strt.capa). strt.capa: Get the capacity defined into the model on the street.
area	String value. Contains the name of the area defined in the model. The meta-data filed of the strt table must be filled with meta-data having as key "area" and as value the "area name".
hierarchy	Integer value. The filter condition works on the street hierarchy defined in the model in the field strt.hier.
status	Integer value. The filter condition works on the traffic states status.
startTime endTime	Timestamp value. The filter condition works on the traffic states time window start and end time. The format of the Timestamp can be in hh:mm. See also the standard ISO-8601.
clocDisabled	Boolean value. The filter condition affected the count location of a specific street. true: count location disabled. false: count location enabled.

Operators

You can use the following relational operators:

• greaterThanOrEqual (≥)
• greaterThan (>)
• equal (=)
• lessThan (<)
• lessThanOrEqual (≤)
• notEqual (<>)
• isNull

Important:  These operators are equivalent to maxequal (≥), max (>), equal (=), min (<), and minequal (≤) used in Optima v15.5.1602 and earlier. For backward compatibility, you can still use these old operators.

Value

It can be a numeric threshold value you can set or it can be some special predefined key.

In case of special needs (for example, to filter all data whose speed are lower than the free flow speed of the street), the value can be set to -1.

Applying rules: applyAllRules parameter

If there is more than one rule, you can set two different filtering modes based on the attribute applyAllRules of the <FilterRules> node.

applyAllRules=true

In this case all rules matching the specified conditions are applied in the specified top-down sequence.

All the values of the reliabilitymultiplier attribute, associated to the applied rules, must be multiplied to get a unique value (see Example 3 → Example: combination of two rules).

applyAllRules=false

Only the strongest rule is applied.

Each rule has in the <action> node the value for the reliability multiplier.

By default the relevance of a rule is given by (1-value): the rule having this number as the highest one is the strongest rule.

This number can be explicitly overwritten by setting "the relevance" of the rule through the attribute effectRelevance which is associated to the <action> node.

The effectRelevance is a number between 0 and 1.

If there are two or more rules with the same "strength" (the same relevance), ONLY the first rule is run (considering the top-down order).

Applying rules: partitioned by source

You can apply the rules according to a specified source (see → <Sources> node node).

Important:  The rules specified within the tag <filterRules> must be applied to the source defined by the tag <tableName>.

Example 2

                            <configurations>
....
        <harmonizer>
....
            <sources>
                <source>
                    <filterRules>
                        <rule>clocDisabledRule</rule>
                        <rule>speedzero</rule>
                        <rule>flowzero</rule>
                        <rule>speedmax</rule>
                        

                            
                    </filterRules>
 		     <pivoting>true</pivoting>
                    <tableName>trafficstate_here</tableName>

                </source>
            
                <source>
                    <filterRules>
                        <rule>filterRule1</rule>
                        <rule>filterRule2</rule>
                    </filterRules>
 		     <pivoting>true</pivoting>
                    <tableName>trafficstate_here</tableName>                
	         </source>
            </sources>
....
        </harmonizer>
....
</configurations>
                        

Example: combination of two rules

A rule named filterRule1 with six conditions, combined with a rule filterRule2 with two conditions.

Example 3

                            <configurations>
....
        <harmonizer>
....
            <filterRules>
                <rule name="filterRule1">
                    <condition attr="speed" operator="lessThanOrEqual" value="50" />
                    <condition attr="density" operator="greaterThanOrEqual" value="25" />
                    <condition attr="flow" operator="equal" value="street.capacity" />
                    <condition attr="area" operator="equal" value="center" />
                    <condition attr="hierarchy" operator="greaterThan" value="2" />
                    <condition attr="status" operator="equal" value="1" />  
                    <action attr="reliabilitymultiplier" value="0.3" />
                </rule>
                <rule name=" filterRule2">
                    <condition attr="startTime" operator="greaterThan" value="10:00" />
                    <condition attr="endTime" operator="lessThan" value="12:00" />
                    <action attr="reliabilitymultiplier" value="0.9" />
                </rule>
            </filterRules>
....
        </harmonizer>
....
</configurations>

In a specific scenario where:

• Speed is lower than or equal to 50
• Density is lower than or equal to 25
• Flow is equivalent to the reference values over the streets (strt table)
• The measurement is associated to a street with geographic location center
• The measurement is associated to a street with a hierarchy greater than 2
• The measurement is associated to a traffic state with its status equal to 1
• The given time window is 10:00 to 12:00.

The data reliability:

• Is multiplied by the factor 0.3 if the data matches only filterRule1
• Is multiplied by the factor 0.9 if the data matches only filterRule2
• Is multiplied by the factor 0.27 if the data matches both rules (0.27=0.9·0.3)
• Remains unchanged for data that does not match any of the rules.

Example: reliability of a rule

A rule named filterRule3, based on count locations elements:

Example 4

                            <configurations>
....
        <harmonizer>
....
            <filterRules>
                <rule name="filterRule3">
                    <condition attr="clocDisabled" operator="equal" value="true" />
                    <action attr="reliabilitymultiplier" value="0" />
                </rule>
            </filterRules>
....
        </harmonizer>
....
</configurations>

If the count locations associated to a given traffic state are disabled (in the cloc table for the corresponding street disable = true), you can set the corresponding reliability to 0.

Important:  Data with its reliability set to 0 is skipped by the clustering procedure.

Important:   Setting the attribute clocDisabled to true or false is equivalent, on Optima v15.5.1602 and earlier, to set it to 1 or 0 (backward-compatibility).

Special settings for filtering

You can build filtering rules with several special settings.

Important:  The filtering based on geographic locations or streets hierarchy are implemented from Optima v16.0.1604.

Filtering with stored values

The settings:

• value="street.capacity"
• value="street.speed"
• value="street.density")

Mean that the incoming values for flow (or speed or density) are compared with the values stored in the strt table (→ Architecture and Data Model).

Important:  This setting is equivalent to setting value=−1 on Optima v15.5.1602 and earlier versions, which are maintained for backward-compatibility.

Filtering for geographic location

You can filter with data based on the geographic location (attr="area").

To achieve this, the DB must be configured with a JSON name/value pair:

name=area, value=center

This JSON must be stored in the strt table.

Filtering for street hierarchy

You can filter with data based on the geographic location (attr="hierarchy").

To achieve this, the DB must be configured, setting the field strt.hier.

Filtering for time window

You can filter data according to the time window which starts at the startTime and ends at the endTime.

To achieve this you must use a query with the parameters tableName.fdat and tableName.ldat, as long as:

tableName.ldat > startTime and tableName.fdat < endTime.

For example:

trafficstate.ldat > startTime and trafficstate.fdat < endTime.

For further information, see:

• → Harmonizer

• → Real-time traffic data.

The <sources>/<source> node contains elements to configure each datasource.

The currently supported sources are DB tables in the Optima trafficstate format (→ Real-time traffic data).

Important:  The rules specified within the tag <filterRules> must be applied to the source defined by the tag <tableName>.

Example 5

In this example you have tow different sources:

• <tableName>trafficstate_here</tableName>
• <tableName>trafficstate_scats</tableName>

Important:  For every source that you want to connect to get data to be harmonized, you need a specific table named trafficstate_SourceName.

For the first source the pivoting is enabled, while it is disabled for the second source.

Only for the first source can you also get harmonized results for equivalent vehicles according to specific transport systems and ignoring input measures related to equivalent vehicles.

                    <configurations>
....
        <harmonizer>
....
            <sources>
                <source>
                    <filterRules>
                        <rule>clocDisabledRule</rule>
                        <rule>speedzero</rule>
                    </filterRules>
                    <pivoting>true</pivoting>
                    <tableName>trafficstate_here</tableName>
                </source>
            
                <source>
                    <filterRules>
                        <rule>filterRule1</rule>
                        <rule>filterRule2</rule>
                    </filterRules>
                    <pivoting>false</pivoting>
                    <tableName>trafficstate_scats</tableName>
                    <recalculateEquivalents>true</recalculateEquivalents>
                </source>
            </sources>
....
        </harmonizer>
....
</configurations>
                

The main goal of clustering is to group the better available data for the final harmonization process.

The grouping procedure considers all the measurements in a certain clustering window and chooses the center of the cluster as the best point of accumulation where most of the measurements are grouped.

Different parameters are used to specify the width of the clustering window, for each type of traffic parameter (speed, flow, and density):

Clustering is performed for each source and each associated transport system, on each directional street and lane separately.

The parameters are valid globally for all sources, transport systems, directional streets and lanes; thus, you cannot set them for a specific source in a way and for another source in a different way.

Tip:  Clustering cannot be disabled, but if you set all parameters to very high values, the process doesn't cut any data, thus the clustering is virtually disabled and all measured data is forwarded to the harmonization phase.

On Optima you can configure different destinations to write data produced by the Harmonizer.

A destination generally contains an output table of the DB where the following are stored:

• The result of the aggregation process.
• Other parameters.

These parameters control archiving and special processing during the aggregation process.

Important:  A destination must contain a table created according to the tsta table schema of the Optima DB.

You can also configure a destination associated to Kafka when the Harmonizer must work with the MLF module (see → MLF-Harmonizer communication).

Example 6

In the example:

• One source filtered with two rules and pivoting enabled.
• One source filtered with only one rule and pivoting disabled.
• No data is written in the DB table tsta_rtlm (destination), because skipWriteToDb is set at true.
• Nothing is deleted (keepOldData/timeWindow = -1)
• It is necessary to deploy a module by providing an implementation for the HarmonizerInMemorySink, and to register it by setting the tag lookupname with the name of the module.
• One block of preQueries with two queries.
• One block of postQueries with only one query.
• The last <destination> node contains info related to communication with Kafka.

                    <configurations>
....
        <harmonizer>
....
            <sources>
                <source>
                    <filterRules>
                        <rule>filterRule1</rule>
                        <rule>filterRule2</rule>
                    </filterRules>
                    <pivoting>true</pivoting>
                    <tableName>trafficstate_here</tableName>
                    <recalculateEquivalents>false</recalculateEquivalents>
                </source>
            
                <source>
                    <filterRules>
                        <rule>filterRule3</rule>
                    </filterRules>
                    <pivoting>false</pivoting>
                    <tableName>trafficstate_scats</tableName>
                    <recalculateEquivalents>true</recalculateEquivalents>
                </source>
            </sources>
            <destinations>
                <destination>
                    <tableName>tsta_rltm</tableName>
                    <skipWriteToDb>true</skipWriteToDb>
                    <keepOldData><timeWindow>-1</timeWindow></keepOldData>
                    <inmemorytransfer>
                        <lookupnames>
                            <lookupname>theKeyUsedToRegisterInInitialContext</lookupname>
                        </lookupnames>
                    </inmemorytransfer>
                    <preQueries>
                        <query>update TABLE_NAME_A set COLUMN_NAME = val_1;</query>
                        <query>update TABLE_NAME_B set COLUMN_NAME = val_2;</query>
                    </preQueries>
                    <postQueries>
                        <query>update TABLE_NAME_C set COLUMN_NAME = val_3;</query>
                    </postQueries>
                </destination>
		 <destination>
    			<type>KAFKA</type>
    			<bootstrapAddress>192.168.50.85:9092</bootstrapAddress>
    			<outputTopicName>sf-harmonizerdata_${env:ENVIRONMENTID}</outputTopicName>
    			<zoneId>Europe/Paris</zoneId>
		 </destination>									
            </destinations>
....
        </harmonizer>
....
</configurations>