LENS

Special Topics: Time and Events


Dealing with time can be a bit confusing in Lens. Those who are just using feed-forward networks don't need to be too concerned with this because your examples will typically have a single event that is not extended over time. But if you are using recurrent networks or Boltzmann machines, you will need to understand time intervals, ticks and example events.

Time intervals are meant to represent abstract units of time in the network's environment. You might think of a time interval as a second or a tenth of a second of real time. The network's timeIntervals value is the maximum amount of time that the netwrk will be run on any example. To be safe, it should be just a bit longer than the maximum length of any example the network will see. You should not think of time intervals as discrete steps. They are really just a measure of continuous time.

However, actually simulating the network is done in discrete steps called ticks. The activation values are updated in the network once per tick. Typically there will be from 1 to 20 ticks per time interval. The more ticks per interval, the finer the grain at which the continuous network is simulated. With standard and simple recurrent networks, there can be only one tick per time interval, so they needn't be distinguished. But in fully recurrent networks, it is usual to have more than one tick per time interval. In this case, it is helpful to think of an interval as representing a span of real time and the ticks as the granularity at which this time is simulated. The maximum number of intervals per example and the number of ticks per interval can be specified when a network is constructed with addNet or later on with the setTime command.

The dt parameter is used in fully recurrent networks to scale the rate at which unit inputs or outputs can change. It is automatically set to 1 / ticksPerInterval when the network is created or when the ticksPerInterval is changed with the setTime command. However, dt can then be directly changed by the user if a faster or slower time constant is wanted. If you have changed dt and would not like it to be overridden when setTime is called, you can used the -dtfixed flag. Every group also has its own dtScale that is multiplied by the network's dt. Thus, each group can have its own integration rate, but you can still scale the rate of the whole network together. Each unit can also scale its integration rate with its own dtScale parameter. The effective dt for a unit is actually the product of the network's dt, the group's dtScale, and the unit's dtScale. The group and unit dtScales default to 1.0.

Training examples are composed of events. Each event specifies a pattern of inputs and targets. Most examples will just consist of a single event, but multiple events can be used to create examples with inputs and/or targets that change over time. Each event lasts a certain amount of time, expressed as a number of time intervals, which need not be an integer. Parts of time intervals will be rounded down to the closest tick. The event has a minTime, a maxTime, and a graceTime associated with it. These are real values specified in time intervals (or fractions thereof), but for a network with one tick per time interval, they may as well be thought of as integers specified in ticks. The event will always last minTime ticks and will never last more than maxTime.

When running fully recurrent networks, it is common to turn on an event's inputs for some time before the targets appear. This can be accomplished in a single event by using graceTime. This specifies the delay between turning on the inputs and turning on the targets. Before this point, targets will be NaN and the network will receive no error derivatives. If the minTime, maxTime, or graceTime is set to NaN, the example set's default minTime, maxTime, or graceTime, field will be used for the event.

Several conditions can cause an event to end. An event will not last longer than its maxTime. If an event has lasted at least as long as its minTime, the event will end if the groupCriterionReached is true for all groups that have one. The standard criterion function is true during training if all output units are within the trainGroupCrit of their targets (or within the testGroupCrit during testing). If the group's trainGroupCrit is NaN, the network's value will be used. Note that the event's graceTime should generally be shorter than the minTime. Otherwise, the event could end before the targets even appear.

Sometimes you only want the next event to be presented if the network has reached criterion on the current event. This can be accomplished with the groupCritRequired flag. If this is set to true (1), the example will end as soon as the network fails to reach criterion by the end of an event.

An example will be run for enough ticks to process each event or until the tick given by the network's maxTicks, whichever comes first. The maxTicks is just the product of the network's timeIntervals and its ticksPerInterval. The historyLength field determines the size of the units' history arrays, which are used in continuous backpropagation and for drawing the Unit Viewer. By default, this is the same as the maxTicks. However, if you do not want to waste the space consumed by these arrays, you can reduce the history length using the setTime command.


Douglas Rohde

Please direct questions to pdplab support