Asynchronous Temporal Fields for Action Recognition

Sample frame v ∈ V ⊆ V 5:

Get incoming messages 6:

Update Q i and Q I 7:

Find gradients with Eq. 7-9 8:

Backprop gradients through CNN 9:

Send outgoing messages to calculate each partial gradient. This shares ideas with contrastive divergence [11, 39] . Given a single video at test time, we visualize in Figure 4 how the predictions changes as the distribution converges with multiple messages passes.

Message Passing The key thing to note is all the incoming messages are of the form M (z)= j f j (z) where f j is some function from node j; for e.g.,

M (z) = j E Uj ∼Qj [φ XI (U j , z)] = j f j (z)

from Algorithm 1. We use the following approximation during training:

EQUATION (10): Not extracted; please refer to original document.

where

d ∈ [0, 1]

is a discount factor, h is a hyperparameter, and J(•) is an ordering of the messages in that video based on the iteration in which the message was computed. The messages are a weighted combination of stored messages.

4. Experimental Results And Analysis

We analyzed the efficacy of our model on the challenging tasks of video activity classification and temporal localization. In addition, we investigated the different parts of the model, and will demonstrate how they operate together. Dataset Recent years have witnessed an emergence of large-scale datasets containing sequences of common daily activities [43, 64, 61] . For our evaluation, we chose the Charades dataset [43] . This dataset is a challenging benchmark containing 9,848 videos across 157 action classes with 66,500 annotated activities, including nouns (objects), verbs (actions), and scenes. A unique feature of this dataset is the presence of complex co-occurrences of realistic humangenerated activities making it a perfect test-bed for our analysis. We evaluate video classification using the evaluation criteria and code from [43] . Temporal localization is evaluated in terms of per-frame classification using the provided temporal annotations. Implementation details We use a VGG16 network [46] with additional layers to predict the model potentials (Figure 5) . We train both a network on RGB frames, and stacks of optical flow images, following the two-stream architecture [45] . The main challenge in training the network is the increase in the output layer size. For the larger potentials, Figure 5 . The VGG-16 variant predicts the potentials for both RGB and Flow. The network predicts the values of all potentials except one (in this figure we group the frame potentials φX into one layer for clarity). The model is trained end-to-end by passing gradients from the Asynchronous Temporal Field through the network.

we used the following structure to go from fc7 to φ XI : Linear layer (4096 to 100), ReLU, Dropout, Linear layer (100 to the potential values). The input to the RGB network is an image of size 224×224×3 where we crop random location, size, and aspect ratio. We use data augmentation with color jitter and PCA lighting noise. The RGB network was pretrained on ImageNet. The input to the Flow network is a stack of 10 consecutive optical flow frames at 24 FPS starting with the current frame. Since each optical flow has two channels, the input size is 224×224×20 as in [45] . The Flow network was pretrained on UCF101 [48] as in Sigurdsson et al. [43] , and random cropped in the same way as RGB.

We follow the training setup in Charades [43] and consider a frame to have one activity label at a time. Even so, our method is still able to reason about other activities in the video. Convergence of the model is evaluated using the approximate distribution Q i (X) at each frame. The Charades dataset has the property that scenes were chosen at random for each sequence of activities. For this reason, we found reasoning about scenes to reduce the performance, and the weight of that term was lowered in the model.

To obtain annotations for action progress p t , we split each activity annotation into three equally sized parts. All layers of the network are trained with a batch size of 240 and a learning rate of 10 −3 (RGB), 10 −5 (Flow). Learning rate was reduced by a factor of 10 every 30k iterations for RGB, and every 140k iterations for Flow. The value of the message decay parameter d was set to d = 0.9, and the standard deviation σ in (4) was set to 6.25 sec (150 frames).

For testing, we sampled 25 equally spaced frames from the video and synchronously pass messages between the frames until convergence (10 message passes). The predictions of the RGB and Flow networks are combined in a probabilistic fashion by multiplying their probabilistic predictions for each class. More implementation details may be found in the appendix. The networks were implemented in Torch, and the code is available on project page. Diverse batches As highlighted in Section 1, the standard Iterations Figure 6 . Convergence of our method compared to other methods that capture temporal structure. Our asynchronous training method contains more diverse batches, has faster and more stable convergence, and reaches higher accuracy on the test set.

way of sampling batches for temporal models results in high correlation between data points leading to a breakdown of the SGD. To understand the importance of having many diverse examples from multiple videos, we compare the convergence of our method to two alternatives using homogeneous batches: CNN+LSTM from Ng et al. [66] , and a synchronous version of our method, where each batch contains full videos (only three videos fit into each mini-batch).

We do synchronous message passing until convergence before calculating gradients for backprop. Figure 6 shows that our asynchronous training method, containing more diverse training batches, has faster and more stable convergence.

4.1. Video Classification

Given a video, the task here is to verify whether it contains one or several of the 157 activity categories. Classification accuracy is measured with the standard mean average precision (mAP) criterion, where a prediction is given for each video. This task has been shown to be highly challenging, with the state-of-the-art non-ensemble methods reaching an mAP of only 17.2%, particularly as each video in this dataset has a sequence of multiple fine-grained activities with a real-world long-tailed activity distribution.

We trained our models using the provided training split following the procedure outlined in Section 3. To make predictions for the whole video, we marginalize out everything except the activity category for 25 equidistant frames in the video. The score for each activity category is the maximum across all frames following the setup from [43] . In our analysis, we include the provided non-ensemble baselines from [43] as well as the following additional baselines:

Two-Stream++. We reimplemented the network described in [43] , which follows Simonyan et al. [46] , with the same parameters. We added data augmentation and finetuned all layers of the network. The performance of only the RGB stream is included (RGB++). We also consider Two-Stream Extended which is the same network, but the Flow network was trained for 25 times more iterations than the RGB network (two weeks of computation on a Titan X Figure 7 . The classes with the highest positive and negative difference between our method and Two-Stream (no structure). Our method does better on many classes, without doing much worse on any. In particular, activities that have temporal structure, such as Opening/Closing a refrigerator have significantly higher performance, since our model can reason jointly about those. GPU). Combined with the augmentation, we found this to non-trivially increase the accuracy. Two-Stream+LSTM. We followed the method outlined in [66] to jointly train a LSTM on top of the two-stream network. We trained both an RGB and an Optical Flow network using the same setup from [43] . The trained networks from Two-Stream++ were used to initialize the models. Table 1 displays the accuracy obtained by our method along with the baselines. Our proposed approach obtains an mAP of 22.4% substantially outperforming the Twostream Extended baseline at 18.6% mAP, and the IDT baseline at 17.2%. Our method reasons over significantly larger timescales and multiple aspects of the activities. To ascertain this, we highlight in Figure 7 , the activity classes with the highest positive and negative difference between our method and the Two-Stream network. It is interesting to note that two of those activities are opening and closing a refrigerator, that arguably have a significant causal structure (an open refrigerator was opened at some point), which our model harnesses to significantly increase the accuracy.

Ablation studies To study the contribution of different model parts, we also train ablated versions of our model separately choosing the best hyperparameters for each version. In addition to our model with only RGB or Flow, we also consider dropping φ XX (i.e., no sequential informa- Table 2 . Temporal localization results (mAP %) on the Charades [43] dataset. Our proposed method outperforms the LSTM model, and is also more tractable to train at a large-scale. tion), φ XI (i.e., no intent), both (i.e., only semantic information), and further dropping φ X (i.e., dropping all structure). Figure 8 shows that semantic reasoning improves over the baseline. Further, while both φ XI and φ XX capture temporal information, they are complementary.

4.2. Temporal Localization

To measure the ability of the methods to temporally localize and understand when exactly activities happen, we adapt the benchmark of [43] to evaluate with the same mAP metric but on individual frames. That is, instead of having a single prediction per video, evaluation is now split into 25 equidistant timepoints having zero or more activities, and the models make a prediction for each of those * . We find this way of evaluating localization robust to annotation ambiguity, and informative for challenging datasets. All hyperparameters were kept equal between localization and classification experiments. All baselines are run on 75 frames across the video, and then every third frame selected for a total of 25 frames. We also considered methods with postprocessing where the model predictions for the 75 frames are averaged across 30 frames to obtain more spatial consistency, and then 25 frames selected as before. Table 2 shows that our method outperforms the alternatives, including the LSTM model which has been shown to be a powerful temporal modeling tool, but challenging to train on top of a two-stream network due to correlations between consecutive samples. These results demonstrate the our method is tractable way of training end-to-end structured models to understand activities. Interestingly, our method still benefits from adding post-processing, significantly more than the LSTM baseline, likely since our method is reasoning on larger time-scales. This suggests Category: Sitting in a chair Category: Reading a book Category: Holding a book Action: sit Action: hold Object: book Figure 9 . Model predictions for a sample video. We see the interplay between categories, objects and actions over time. For example, model becomes confident about the action sit early, which aids the understanding of Sitting in a chair once the chair becomes visible, and helps predicting Reading a book. Darker colors represent higher likelihood, and we average predictions to correspond to each frame.

Cluster 1

Cluster 2 Cluster 3 Figure 10 . To visualize the learned intent, we cluster videos based on intent. In Cluster 1, the model captures the intent of get up from lying down. In Cluster 2, folding clothes is followed by putting them away, and Cluster 3 shows cleaning with a broom/vacuum/towel, followed by picking up things.

that our model could further benefit from joint training with additional kernels in the temporal term.

Qualitative visualization A key advantage of our model is the structured understanding of videos in terms of multiple aspects, such as action sequences, objects, and even intentions. To visualize this, we display predictions over time in Figure 9 for the three most confident activity categories, two most confident actions, and the most confident object. More examples are presented in the Appendix.

Interpretation of Intent In our model, the intent I is a continuous distribution over the latent variables. To get an insight into how our model learns the intent, we ran a simple experiment that clustered videos in the dataset that have the most similar inferred intent distributions. The first cluster in Figure 10 shows the model captures the simple intent that the person intends to get up from lying down. In the videos, these actions are 10-20 seconds apart, demonstrating that the intent helps reason over large time scales. In order to further analyze the 'intent' variable, we plot the t-SNE embedding of the intent variable for the videos in the test set. We see that there is clear clustering of similar videos in Fig. 11a . We also annotated 10 types of intent (100 videos total). More details are presented in the Appendix. We observe that the intent representation preserves some of the intent types in Fig. 11b . Quantitatively, even without mitigating outliers, the average distance (in 10 −3 ) between pairs of videos within an intent type was 6.02 compared to 7.25 (σ=1.06) for any points, and the difference is significant for 5 of 10 intent types (p=0.1). This tentatively suggest that the intent captures interesting structure in the data, and we hope this will encourage future work.

5. Conclusion

In this paper, we have presented a deep-structured model using a fully-connected temporal CRF that not only models semantic aspects of activities but also reasons about longterm temporal relations. We also presented an asynchronous stochastic inference algorithm that circumvents a key bottleneck in the large-scale end-to-end model learning. Using our proposed method, we have demonstrated impressive activity classification and temporal localization results on a challenging dataset of realistic activities.

6. Appendix

This appendix contains the following additional content:

1. Description of the CRF.

6.1. Description Of The Crf

We create a CRF which predicts activity, object, etc., for every frame in the video. For reasoning about time, we create a fully-connected temporal CRF, referred to as Asynchronous Temporal Field in the text. That is, unlike a linear-chain CRF for temporal modelling (the discriminative counterpart to Hidden Markov Models), each node depends on the state of every other node in the graph. We incorporate intention as another latent variable which is connected to all the action nodes.

In this work we encode multiple components of an activity. Each video with T frames is represented as {X 1 , . . . , X T , I} where X t is a set of frame-level random variables for time step t and I is a random variable that represent global intent in the entire video. As discussed in the paper, for clarity of derivation X t includes all frame level variables (C t , O t , A t , P t , S t )

Mathematically we consider a random field {X, I} over all the random variables in our model ({X 1 , . . . , X T , I}). We now list the complete description of the CRF.

CRF Variables: CRF Potentials:

• Random field {X, I} = {X 1 , . . . , X T , I} • Frame X t = {C t , O t , A t , P t , S t }, X t ∈ X , X = C×O×A×P×S -Category C t ∈ C, C = {1,

• φ X : X → R, equivalently: φ X : C×O×A×P×S → R • φ X decomposes as follows: φ X (C t , O t , A t , P t , S t )=φ(O t , P t )+φ(A t , P t )+φ(O t , S t )+φ(C t , O t , A t , P t ) -φ(O t , P t ) : O×P → R -φ(A t , P t ) : A×P → R -φ(O t , S t ) : O×S → R -φ(C t , O t , A t , P t ) : B → R,

here B is all configurations of C t , O t , A t , P t that exist in the training data.

• φ XI : X ×I → R (specifically we parametrize this as φ XI : O×I → R)

• φ XX : X ×X → R (specifically we parametrize this as φ XI : O×O → R) Figure 12 . The model captures interactions between all frames Xt and the intent I, that is, a fully-connected model. Here shown for T = 5. We visualize some of the potentials of the model, and where they fit into the graph. All φ i XI share the same parameters, but we calculate the gradients with respect for each of them separately below. For efficient inference, we use a mean-field approximation presented below. A mean-field approximation is a simpler distribution that is fit to the original distribution when needed.

The complete distribution of the model is:

EQUATION (11): Not extracted; please refer to original document.

where φ XX (x i , x j ) is the potential between frame i and frame j, and φ XI (x i , I) is the potential between frame i and the intent. For notational clarity φ X (x i ) incorporates all potentials for C t , O t , A t , P t , S t . The model is presented in Figure 12 .

6.2. Derivation Of The Update Equations

Given an input video V ={V 1 , . . . , V T }, our goal is to estimate the maximum a posteriori labeling of the random field by marginalizing over the intent I, I P (X, I|V ) as discussed in the paper. In the following derivations we omit the conditioning on V and write P (X, I) and φ(X, I).

Before we present the update equations and gradients, we define the following messages which will be used in the final version of the following equations for clarity in their presentation. Messages are a term used for cached computations sent between different functions in a dynamic programming fashion. In the following derivations, X * is used to explicitly denote the ground truth used for training. Plain X is used to refer to the variable.

Outgoing Messages (Messages that are calculated from a single frame)

EQUATION (17): Not extracted; please refer to original document.

Incoming Messages (Messages that are calculated from messages from multiple frames and used for the computation of a single frame)

EQUATION (19): Not extracted; please refer to original document.

EQUATION (20): Not extracted; please refer to original document.

EQUATION (21): Not extracted; please refer to original document.

EQUATION (22): Not extracted; please refer to original document.

EQUATION (23): Not extracted; please refer to original document.

EQUATION (24): Not extracted; please refer to original document.

EQUATION (25): Not extracted; please refer to original document.

Instead of computing the exact distribution P (X, I) presented above, the structured variational approximation finds the distribution Q(X, I) among a given family of distributions that best fits the exact distribution in terms of KL-divergence. By choosing a family of tractable distributions, it is possible to make inference involving the ideal distribution tractable. Here we use Q(X, I) = Q I (I) i Q i (x i ), the structured mean-field approximation. More details on mean-field approximation are presented section 11.5 generic update equation for Q (Equation 11.54 in [18] ) is:

EQUATION (26): Not extracted; please refer to original document.

where X −i refers to all variables except x i . Using Eq. 11 along with Eq. 26 we get the following update equations:

EQUATION (27): Not extracted; please refer to original document.

EQUATION (28): Not extracted; please refer to original document.

∝ exp H i (I) + H i (I) (Here i refers to the frame of interest, but any choice of i holds)

where Q i is marginal distribution with respect to each of the frames, and Q I is the marginal with respect to the intent.

6.3. Details Of The Learning Algorithm

Training a deep CRF model requires calculating derivatives of the objective in terms of each of the potentials in the model, which in turn requires inference of P (X, I|V ). The network is trained to maximize the log-likelihood of the data:

EQUATION (31): Not extracted; please refer to original document.

= log

EQUATION (32): Not extracted; please refer to original document.

EQUATION (33): Not extracted; please refer to original document.

where we explicitly write out the partition function Z(V), andP () is the unnormalized version of P (). Again, we use X * to explicitly refer to the ground truth labels. As before, V is omitted from the following derivations. The goal is to update the parameters of the model, for which we need gradients with respect to the parameters. Similar to SGD, we find the gradient with respect to one part of the parameters at a time, specifically with respect to one potential in one frame. That is, φ i X (x) instead of φ X (x). The partial derivatives of this loss with respect to each of the potentials are as follows.

6.3.1 Updating The Frame Potential Φ X

The frame potential φ X (x i ) incorporates the interplay between activity category, object, action, progress and scene, and could be written explicitly as φ X (C t , O t , A t , P t , S t ). In practice this potential is composed of unary, pairwise, and tertiary potentials directly predicted by a CNN. We found predicting only the following terms to be sufficient without introducing too many additional parameters:

φ X (C t , O t , A t , P t , S t )=φ(O t , P t )+φ(A t , P t )+φ(O t , S t ) + φ(C t , O t , A t , P t )

where we only model the assignments seen in the training set, and assume others are not possible.

Let us first derive the update equation for φ X as a whole, and then demonstrate how to update each of the individual potentials. In the following derivation, we simply take the partial derivative where appropriate and iteratively use the chain rule.

∂l(X * ) ∂φî X (x) = 1 IP (X * , I) IP (X * , I) ∂ i φ i X (x * i ) ∂φî X (x) − ∂ log Z ∂φî X (x) (34) = 1x =x * − 1 Z X I ∂P (X, I)

∂φî X (x) (Denominator and numerator cancel)

EQUATION (35): Not extracted; please refer to original document.

1x =x Q(X, I) (Using the mean-field)

EQUATION (38): Not extracted; please refer to original document.

where we use X * to refer to the ground truth labels, andX to refer to the variables we are taking the partial derivative with respect to. We note that

∂( i φ i X (x * i )) ∂φî X (x)

= 1x =x * . Intuitively this implies the partial gradient is the difference between the ground truth and the model prediction. This equation is easily extended to update each of the individual potentials as follows:

EQUATION (44): Not extracted; please refer to original document.

where we marginalize out the variables that are not a part of each potential. Again, X t incorporates all the frame variables {C t , O t , A t , P t , S t }. These partial derivatives are passed down the CNN (backprop) to update the parameters of the network.

6.3.2 Updating The Frame-Intent Potential Φ Xi

Similarly to φ X we proceed as follows:

EQUATION (53): Not extracted; please refer to original document.

This equation can be interpreted in that it captures the difference between the distribution of the intent given the ground truth, and the predicted distribution of the intent.

6.3.3 Updating The Frame-Frame Potential Φ Xx

The pairwise potentials φ XX (x i , x j ) for two time points i and j in our model have the form:

EQUATION (55): Not extracted; please refer to original document.

where µ models the asymmetric affinity between frames, w are kernel weights, and each k (m) is a Gaussian kernel that depends on the videoframes v i and v j which are omitted from this notation for convenience, but the probability and the potentials are conditioned on V. In this work we use a single kernel that prioritises short-term interactions:

EQUATION (56): Not extracted; please refer to original document.

The parameters of the general asymmetric compatibility function µ(x i , x j ) are learned from the data, and σ is a hyperparameter chosen by cross-validation. The parameters of µ are learned as follows, and this could be extended to a more general form of φ XX :

EQUATION (62): Not extracted; please refer to original document.

This update equation consists of two symmetric parts, one for influence from frames before, and one for influence from frames after. Intuitively, this captures the difference in the true affinity between frame i and all frames j on the one hand, and on the other hand the predicted affinity, where the affinity is weighted by the kernel.

6.4. Additional Implementation Details

A more detailed algorithmic description of the model is presented in Algorithm 3. More details can be found on the project page https://github.com/gsig/temporal-fields/. Training time Training the models in this paper took a while: The RGB stream of the Two-Stream model converged after only 0.2 epochs (20% of the total data, randomly selected) of the training data, but training the Flow stream needed 4.0 epochs to reach the best performance. Our model needed 0.7 epochs for the RGB stream and 8. for each example in mini-batch do

4:

Sample frame v ∈ V ⊆ V that has index i 5:

Calculate messages with Eq. 18-25, approximated by Eq. 9 (from paper) 6 :

Alternate updating Q i and Q I until convergence 7:

Find gradients with Eqs. 40 Update CNN using accumulated gradients hardware and model. Our learning rate schedule was chosen by finding the largest learning rate that did not cause divergence, and then making sure the learning rate was decayed by a factor of 100 over the course of training. Investigations into training these kinds of models faster are likely to yield substantial benefits. Training Deep Models with Latent Variables One of the pursuits of this work was introducing latent variables into a deep framework, the intent. The gradient for the frame-intent potential, contains predictions of the model on both sides, which is a common problem in deep reinforcement learning, where a variety of tricks such as target fixing, double Q-learning, and gradient clipping, are used to combat the instability caused by this. In this work we found that simply severing the dependency of the frame-intent variable on the input data got rid of the instability, and still gave acceptable performance on the RGB stream, however we found that this did not give good performance on the Flow stream.

In order to train the network with the frame-intent potential depending on the input data, we experimented with a variety of techniques from the reinforcement learning literature. Only two methods were found to help: Alternating target and prediction networks, and regularization. For alternating target and prediction networks, the network predicts two frameintent potentials, and then the network randomly chooses which to use as the target, and which to use as the source, and backprop only through one of them. For regularization, we enforce the frame-intent potential to be close to zero, similar to weight decay (set to 4 • 10 −4 ). Regularization was found to be give slightly better performance, and easy to implement/tune, and was used in this work.

6.5. Details About Intent Analysis

To analyze the learned intent variable, we defined 10 types of intent: getting something to eat, clean the living space, getting dressed, getting something from storage, get informed, get out of bed, leave the house, photograph something, relaxing, working. To identify videos corresponding to the intent, we used keyword related to the intent (such as closet and clothes for getting dressed) and manually verified that the content of the video matched the intent. The analysis demonstrates that the latent intent variables captures non-trivial structure of the label space, but precisely understanding goal-oriented behavior compared to simple activity analysis remains important future work.

6.6. Additional Visualizations Of Output Predictions

Due to space constraints in the full paper, we present here additional visualizations from the model. In Figure 13 we present in the same way as Figure 9 (from the paper). That is, we present the 3 most confident categories, 2 most confident actions, and 1 most confident object. For example, in the first row we can see that once the light turns on in the room and the couch becomes visible the category Sitting on a sofa/couch fires, which in turn increases the likelihood of sitting in the next few frames. Furthermore, in Figure 14 we present similar visualizations, but only the 6 most confident categories, to further understand the interplay between the activity categories. In the first row, we can see a video of a person walking towards the camera, and we can see how one after the other the model recognizes cup, phone, and sandwich, and reasons about these connected activities. Finally, in Figure 15 we present a breakdown of the mean average precision (mAP) by our model for each class of the dataset, sorted by the mAP of our model. Figure 13 . Visualizations of the model predictions for the 3 most confident categories, 2 most confident actions, and 1 most confident object. Darker colors indicate higher likelihood. Figure 14 . Visualizations of the model predictions for the 6 most confident categories. Darker colors indicate higher likelihood.