betaFeedback

WorksComputer Science

Coupled Scene Graph Anticipation: When With-Constraint Recall Hides Structural Inconsistency

Abstract

Scene Graph Anticipation (SGA) predicts object relationships in future video frames. Continuous-time SGA models evolve each subject-object pair with an independent Neural ODE, so relationships sharing an entity cannot influence one another during integration, and nothing prevents mutually exclusive predictions, such as "sitting on" and "standing on", for the same pair. We introduce CoupledSGA, which links such relationships through a Relationship Interaction Graph with graph attention inside the ODE vector field, and adds a Structural Consistency Loss (LSC\mathcal{L}_{SC}) based on a matrix of mutually exclusive predicates. Developing CoupledSGA revealed a limitation of the evaluation protocol. With-Constraint evaluation keeps one predicate per group for each subject-object pair, so its R@K and mR@K cannot capture the within-group violations targeted by LSC\mathcal{L}_{SC}: we prove they are unchanged whenever each group's winning predicate and its score are held fixed. We further show that the gradient of LSC\mathcal{L}_{SC} depends on predicate scores rather than ground-truth labels: for two-predicate rules it amplifies the ordering the model already holds, and when all scores in a group are equal it suppresses predicates in more exclusion rules more strongly, explaining why the penalty can lower recall while improving consistency. On Action Genome, activating LSC\mathcal{L}_{SC} after two warm-up epochs reduces R@20 from 42.21% to 35.18% while lowering the No-Constraint contradiction rate from 45.92% to 4.07% (matched-budget R@20: 40.48% CoupledSGA vs. 44.78% SceneSayerODE). CoupledSGA is a testbed, not a state-of-the-art claim: With-Constraint recall captures the cost of enforcing structural consistency but not the improvement. We recommend reporting No-Constraint results alongside a contradiction rate for methods that constrain predicate co-activation.

This work is licensed under a Creative Commons Attribution 4.0 International License.

ICVGIP 2026, Kolkata, India

© 2026 Copyright held by the owner/author(s).

CCS Concepts

• Computing methodologies → Computer vision; Neural networks; Machine learning.

ACM Reference Format: Nongmaithem Zeno, Anushttha Srivastava, Anshit Sinha, Apoorv Sisodia, Suman Chowdhury, Prerana Mukherjee, and Rajesh P. Barnwal. 2026. Coupled Scene Graph Anticipation: When With-Constraint Recall Hides Structural Inconsistency. In Indian Conference on Computer Vision, Graphics, and Image Processing (ICVGIP 2026), December 21–24, 2026, Kolkata, India. ACM, New York, NY, USA, 9 pages.

Introduction

Scene Graph Anticipation (SGA) [16] aims to predict the future relationships between entities in a video, representing them as ⟨subject,predicate,object⟩\langle\text{subject}, \text{predicate}, \text{object}\rangle triplets. Its evaluation protocols inherit established metrics from Scene Graph Generation (SGG) [21, 24] and Video Scene Graph Generation (VidSGG) [6, 7]. In particular, Recall@K (R@K) reduces the penalty associated with correct predictions that are not annotated in the ground truth [21], while mean Recall@K (mR@K) mitigates the dominance of frequent predicate classes [5]. These metrics are commonly reported under two evaluation protocols: With Constraint, which restricts the number of predicates that each subject–object pair can contribute to the evaluation, and No Constraint, which permits multiple predicates per pair [15, 16]. Although these protocols provide complementary views of predictive performance, their ability to measure structural consistency has received less attention.

Structural consistency is important in applications that require reliable predictions before an interaction occurs [8, 9]. For example, a person cannot simultaneously be sitting on and standing on the same chair in the same relational context. Nevertheless, a scene graph anticipation model may assign high scores to both predicates for the same subject–object pair. For the Spatial and Contacting groups of Action Genome [11], predicates are decoded using independent sigmoid outputs. Consequently, mutually exclusive predicates are not necessarily forced to compete, and the decoder can produce structurally inconsistent predictions. Whether such inconsistencies affect the reported evaluation results depends on the predictions retained by the evaluation protocol.

In the evaluation implementation released with SceneSayer [16], the With-Constraint protocol retains the highest-scoring predicate from each predicate group for a given subject–object pair, whereas the No-Constraint protocol allows multiple predicates to be retained. As a result, With-Constraint evaluation may discard evidence of a contradiction before the predictions are scored. For example, sitting on and standing on both belong to the Contacting group, so retaining only the highest-scoring predicate prevents both from appearing in the evaluated output. Thus, a prediction may violate a structural exclusion rule before filtering, yet the violation may remain invisible to the reported recall metrics after filtering.

This issue emerged during the development of CoupledSGA, illustrated in Figure 1. CoupledSGA extends SceneSayerODE, the Neural Ordinary Differential Equation (Neural ODE) variant of SceneSayer [16], in which each subject–object pair is modelled by an independent Neural ODE [4]. Because the trajectories are integrated independently, relationships that share an entity cannot directly influence one another during temporal evolution. CoupledSGA introduces two complementary mechanisms. First, a Relationship Interaction Graph (RIG) connects relationships that share an entity, and graph attention [3] within the ODE vector field enables their representations to interact during integration. Second, a Structural Consistency Loss (LSC\mathcal{L}_{\mathrm{SC}}) penalises the simultaneous activation of predicate pairs specified as mutually exclusive in an exclusion matrix. The RIG facilitates interaction between related relationships, whereas LSC\mathcal{L}_{\mathrm{SC}} explicitly regularises predicate co-activation.

CoupledSGA pipeline

Figure 1. CoupledSGA. Observed frames give one embedding per subject–object relationship. The Relationship Interaction Graph connects relationships that share an entity. Graph attention inside the ODE vector field couples their trajectories during integration. Decoded predicates are regularised by the Structural Consistency Loss LSC\mathcal{L}_{\mathrm{SC}}.

The central finding of this work concerns the limitations of the evaluation protocol rather than a claim of improved predictive performance. Every exclusion rule considered in our matrix relates two predicates within the same predicate group. Consequently, the With-Constraint protocol cannot retain both predicates involved in any of these violations for the same subject–object pair. We formally prove that, when pruning predicate and its score in each group remain fixed, With-Constraint R@K and mR@K are unchanged regardless of whether a within-group exclusion rule is violated (Proposition 1). Therefore, these metrics alone cannot quantify the structural improvements targeted by LSC\mathcal{L}_{\mathrm{SC}}. This result has an important implication: improvements in structural consistency cannot be established solely through With-Constraint recall.

We further analyse the gradient of LSC\mathcal{L}_{\mathrm{SC}} with respect to predicate scores and show that the penalty depends on the scores of mutually exclusive predicates rather than their ground-truth labels (Proposition 2). For a two-predicate exclusion rule, the loss reinforces the ordering that the model already assigns to the two predicates. When all scores in a group are equal, predicates involved in a greater number of exclusion rules are suppressed more strongly, irrespective of whether they are correct. These properties explain why enforcing structural consistency may reduce recall, particularly when the penalty is introduced before the model has learned reliable predicate rankings.

Experiments on Action Genome are consistent with this analysis. Under a matched training budget, CoupledSGA achieves an R@20 of 40.48%, compared with 44.78% for SceneSayerODE. In a controlled comparison, activating LSC\mathcal{L}_{\mathrm{SC}} after two warm-up epochs reduces R@20 from 42.21% to 35.80%. Under the No-Constraint protocol, the same change reduces the contradiction rate from 45.92% to 4.07%, while coupling alone reduces it from 54.40% to 45.92%. These results are not intended to establish state-of-the-art performance. Rather, they illustrate the trade-off between recall and structural consistency and show that With-Constraint recall captures the cost of enforcing consistency without reflecting the corresponding structural improvement.

The findings motivate a more informative evaluation practice for SGA methods that explicitly regularise predicate co-activation. In addition to standard With-Constraint metrics, evaluations should report No-Constraint results and a contradiction rate, defined as the proportion of (subject–object pair, predicate group) instances in which the two highest-scoring predicates form an excluded pair. Reporting these complementary measures would help distinguish predictive recall from structural validity and provide a clearer assessment of the effects of consistency constraints.

The contributions of this work are summarised as follows:

  1. An analysis of evaluation limitations. We prove that With-Constraint R@KR@K and mR@KR@K cannot detect within-group structural violations when the winning predicate and its score in each group remain unchanged (Section 5).

  1. A coupled SGA framework. We introduce CoupledSGA, which uses a Relationship Interaction Graph and graph attention within a Neural ODE vector field to couple relationships that share entities, together with a Structural Consistency Loss to penalise mutually exclusive predicate co-activation (Section 4).

  1. An analysis of the consistency penalty. We derive the gradient of LSC\mathcal{L}_{\mathrm{SC}} and establish how its label-independent behaviour can reinforce existing predicate-score orderings in two-predicate rules and, when all scores in a group are equal, disproportionately suppress predicates involved in multiple exclusion rules (Section 5).

  1. An empirical evaluation and reporting recommendation. Experiments on Action Genome demonstrate the trade-off between recall and structural consistency, motivating the joint reporting of recall metrics and contradiction rates (Section 6).

Related Work

Video scene graphs and anticipation. Static SGG extracted relationships from single images using message passing [21] and frequency priors [24]. STTran [6] added factorised spatial and temporal attention for video, and DSG-DETR [7] modelled long-term dependencies over object tracklets. These models label frames they have observed, so their numbers are not a target for anticipation. SceneSayer [16] introduced SGA and solved it with per-pair Neural ODEs and Neural SDEs. Action anticipation [8, 9] predicts coarse labels rather than relational structure.

Recent SGA. ImparTail [15] applies curriculum-driven loss masking to VidSGG and SGA and reports SGA results under both With-Constraint and No-Constraint settings. FDSG [22] forecasts entity labels, boxes and relationships jointly. FORESCENE [1] predicts full future graphs with a latent diffusion model. The last two extend SGA beyond the fixed object set and the predicate-only prediction we study here, and none of the three is a direct numerical baseline for our matched PredCLS budget. We note them because our evaluation argument applies to any of them that adds a co-activation constraint.

Coupled dynamics on graphs. Graph Neural ODEs [17], CG-ODE [10] and Graph-Coupled Oscillator Networks [18] model interacting units in continuous time. They couple over the entity graph and predict a continuous state. We couple over a graph of relationships, which is where mutual exclusion is defined, and our target is a discrete symbolic output.

Structural constraints and scheduled objectives. Symbolic structure has been added to visual predictors through knowledge graphs [23], probabilistic logic [13] and energy-based penalties on invalid subgraphs [20]. TEMPURA [14] addresses long-tailed predicates with memory-guided training. Our schedule borrows the idea of curriculum learning [2], which presents easier training examples before harder ones, and applies it to an auxiliary objective; ImparTail [15] uses a curriculum to mask head-class predicate losses. Our penalty does not act on the class distribution, and it is narrower than the symbolic approaches: it constrains only which predicates may be simultaneously active on one pair.

Task and Evaluation Protocol

Scene Graph Anticipation

Let I1,…,ITI_1,\ldots,I_T be observed frames. SGA predicts the relationships holding in KK later frames IT+1,…,IT+KI_{T+1},\ldots,I_{T+K}. A relationship pair is an ordered pair of entities (s,o)(s,o); a predicate is a relation class assigned to that pair. We write PTP_T for the set of pairs active at the last observed frame and N=∣PT∣N=|P_T|.

We use Predicate Classification (PredCLS): entity identities and boxes are given for all frames, and only the predicate must be predicted. The released Action Genome annotations [11] define 26 predicates in three predicate groups: Attention (3 classes, decoded with a softmax and so single-label by construction), Spatial (6 classes, multi-label) and Contacting (17 classes, multi-label). Spatial and Contacting are decoded with independent sigmoids, so nothing forces the classes inside a group to compete.

The With-Constraint Emission Rule

Our main result depends on the exact scoring rule, so we state it before anything else. Under With Constraint, for each pair ii and each predicate group gg, the evaluator keeps only the top-scoring predicate of that group:

p^ig=arg⁡max⁡u∈gy^i,u(1)\hat{p}_i^g = \arg\max_{u\in g}\hat{y}_{i,u} \tag*{(1)}

where y^i,u\hat{y}_{i,u} is the predicted score of predicate uu for pair ii. The kept triplets are ranked by score, and R@K is the fraction of ground-truth triplets in the top KK. Under No Constraint, several predicates per group are kept, so co-activated predicates can both appear. We use the emission rule implemented in the Action Genome evaluation code released with prior SGA work [16]; the results in § 6 concern the rule itself and not any historical label attached to it.

Two Notions of Consistency

Let M∈{0,1}C×CM \in\{0,1\}^{C\times C} be a symmetric, zero-diagonal exclusion matrix over the CC Spatial and Contacting classes, with Muv=1M_{uv}=1 when uu and vv cannot both hold of one pair. Call (u,v)(u,v) an excluded pair, and call the rule within-group when uu and vv belong to the same predicate group. We keep two notions separate throughout, because conflating them is easy and misleading.

Structural validity is a property of a decoded graph: no pair is assigned both members of any excluded pair. It is evaluated after emission.

Co-activation mass is a property of the continuous scores, before emission:

ci=∑u<vMuvy^i,uy^i,v≥0.(2)c_i=\sum_{u<v}M_{uv}\hat{y}_{i,u}\hat{y}_{i,v}\ge0. \tag*{(2)}

It is zero exactly when, for every excluded pair, at least one of the two scores is zero.

The two do not coincide. A decoded graph can be structurally valid while ci>0c_i>0, and a small cic_i does not certify validity. Our loss minimises co-activation mass; what a downstream consumer cares about is validity. Keeping the distinction visible is what makes the analysis in § 6 precise.

CoupledSGA

Relation Encoder

For each observed frame tt and pair (s,o)(s,o), a spatial-temporal backbone [6] pools RoIAlign features for the subject box, the object box and their union, concatenates a relative-box encoding and learned class embeddings, and refines the result with spatial attention across pairs and temporal attention across frames. This yields hso(t)∈Rdh_{so}^{(t)}\in\mathbb{R}^{d} with d=1936d=1936. Stacking the pairs active at frame TT gives the initial state H(T)∈RN×dH(T)\in\mathbb{R}^{N\times d} for integration.

Relationship Interaction Graph

The RIG is a graph whose nodes are the NN relationship pairs. Two nodes are joined when the underlying pairs share an entity:

Aij={1,if si=sj ∨ si=oj ∨ oi=sj ∨ oi=oj,0,otherwise,(3)A_{ij}= \begin{cases} 1, & \text{if }s_i=s_j\ \lor\ s_i=o_j\ \lor\ o_i=s_j\ \lor\ o_i=o_j,\\ 0, & \text{otherwise,} \end{cases} \tag*{(3)}

with self-loops added, A~=A+IN\tilde{A}=A+I_N. We also learn a scalar weight per edge from the two incident embeddings,

eij=MLP⁡edge([hi(T)∥hj(T)]),(4)e_{ij}=\operatorname{MLP}_{\mathrm{edge}}\left([h_i(T)\Vert h_j(T)]\right), \tag*{(4)}

which enters the attention score in Equation (6). Action Genome annotates person–object relations, so most pairs in a frame share the same actor and the RIG is close to complete. Topology alone therefore carries little information on this dataset. Which neighbours matter is decided by eije_{ij} and by the attention weights, not by the adjacency. On multi-agent data, the same rule would produce a block-structured graph.

Coupled Graph Neural ODE

We integrate the relationship states from TT to the prediction horizon by solving dH(t)/dt=fθ(H(t),A~,e~,t)dH(t)/dt=f_\theta(H(t),\tilde{A},\tilde{e},t), with H(T)H(T) as the initial condition. The point of CoupledSGA is that fθf_\theta does not act row by row. At every solver step, each relationship updates its state from its RIG neighbours:

dhi(t)dt=σ(∑j∈N(i)αij(t)Wvhj(t))−γhi(t),(5)\frac{d h_i(t)}{dt}=\sigma\left(\sum_{j\in\mathcal{N}(i)}\alpha_{ij}(t)W_vh_j(t)\right)-\gamma h_i(t), \tag*{(5)}

where N(i)\mathcal{N}(i) are the neighbours of ii in A~\tilde{A}, WvW_v is a shared value projection, σ\sigma is SiLU, and −γhi(t)-\gamma h_i(t) with γ=1\gamma=1 damps the state, which helps keep trajectories bounded over the rollout. The attention weights are GATv2 [3] scores, extended with the learned edge weight and a time embedding:

αij(t)=softmax⁡j∈N(i)(LeakyReLU⁡(a⊤[Wqhi(t)∥Wkhj(t)∥ϕ(t)+weeij])),(6)\alpha_{ij}(t)=\operatorname{softmax}_{j\in\mathcal{N}(i)}\left(\operatorname{LeakyReLU}\left(a^\top[W_qh_i(t)\Vert W_kh_j(t)\Vert\phi(t)+w_e e_{ij}]\right)\right), \tag*{(6)}

with ϕ(t)\phi(t) a Fourier embedding of the integration time and wew_e a scalar parameter. Including ϕ(t)\phi(t) lets the coupling strength change along the horizon rather than staying fixed at its value at t=Tt=T. We use two layers and four heads, averaged to preserve dd.

We solve the system with explicit Euler at a fixed step τ=0.5\tau= 0.5, so the number of derivative evaluations is S=K/τS = K/\tau, set by the horizon and not by the scene.

Cost. Each solver step costs O(Nd2)O(Nd^2) for the projections and O(∣ERIG∣d)O(|E_{\mathrm{RIG}}|d) for message passing, against O(Nd2)O(Nd^2) per step for an uncoupled model. The marginal cost of coupling is the second term. With d=1936d = 1936 and few active pairs per frame, it is dominated by the projections. This is an analytical statement; we do not report measured runtime.

Structural Consistency Loss

Let Y^∈[0,1]N×C\hat{Y} \in[0,1]^{N \times C} hold the sigmoid scores of the Spatial and Contacting heads at a predicted timestep, C=23C = 23. Summing the co-activation mass of (2) over pairs gives

LSC=1NTr⁡(Y^MY^T)=2N∑i=1Nci≥0,(7)\mathcal{L}_{\mathrm{SC}} = \frac{1}{N}\operatorname{Tr}\left(\hat{Y}M\hat{Y}^{\mathsf{T}}\right) = \frac{2}{N}\sum_{i=1}^{N} c_i \ge0, \tag*{(7)}

which is zero exactly when every excluded pair has at least one score of zero. This is the co-activation condition of Section 3.3, not discrete structural validity. The loss adds no cost at inference. Table 1 lists the rules in MM. Two columns matter later: every rule is within-group, and not contacting appears in three rules while every other predicate appears in one.

GroupExcluded pairDeg.
Spatialabove / beneath1 / 1
Spatialin front of / behind1 / 1
Contactingsitting on / standing on1 / 1
Contactingwiping / drinking from1 / 1
Contactingholding / not contacting1 / 3
Contactingtouching / not contacting1 / 3
Contactingcarrying / not contacting1 / 3

Table 1. Exclusion rules in MM. Deg. is the number of rules the predicates of that row appear in. Every rule is within-group, which Proposition 1 uses; the uneven degrees are what Corollary 2 uses.

Schedule. Section 5 shows that LSC\mathcal{L}_{\mathrm{SC}} reinforces the ordering the network already holds. That argues for delaying it until the ordering is reliable, so we gate it by epoch: λSC(e)=0\lambda_{\mathrm{SC}}^{(e)} = 0 for e<estarte < e_{\mathrm{start}} and λSC\lambda_{\mathrm{SC}} afterwards. The full objective is

Ltotal=Lobs+Lant+β1Lrecon+β2Lbbox+λSC(e)LSC,(8)\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{obs}} + \mathcal{L}_{\mathrm{ant}} + \beta_1\mathcal{L}_{\mathrm{recon}} + \beta_2\mathcal{L}_{\mathrm{bbox}} + \lambda_{\mathrm{SC}}^{(e)}\mathcal{L}_{\mathrm{SC}}, \tag*{(8)}

where Lobs\mathcal{L}_{\mathrm{obs}} and Lant\mathcal{L}_{\mathrm{ant}} are cross-entropy and binary cross-entropy terms on observed and predicted predicates, Lrecon\mathcal{L}_{\mathrm{recon}} is an L1L_1 latent reconstruction term and Lbbox\mathcal{L}_{\mathrm{bbox}} is a smooth-L1L_1 box term (β1=1\beta_1 = 1, β2=0.1\beta_2 = 0.1). Algorithm 1 gives the loop.

Why With-Constraint Recall Misses Structural Violations

This section contains the paper’s two analytical results. Both follow from definitions already given, and neither depends on our experiments.

Algorithm 1 CoupledSGA training

Require: observed frames, entities, horizon KK, step τ\tau, threshold estarte_{\mathrm{start}}, weight λSC\lambda_{\mathrm{SC}}

  1. for e=0e = 0 to E−1E - 1 do

  2. λSC(e)←λSC\lambda_{\mathrm{SC}}^{(e)} \leftarrow\lambda_{\mathrm{SC}} if e≥estarte \ge e_{\mathrm{start}} else 0

  3. for each video batch do

  4. encode observed frames; form H(T)H(T)

  5. build A~\tilde{A} and eije_{ij}; see Eqs. (3), (4)

  6. for k=1k = 1 to KK do

  7. integrate Eq. 5 by Euler steps of size τ\tau to T+kT + k

  8. decode Y^T+k\hat{Y}_{T+k}

  9. end for

  10. update by AdamW on Ltotal\mathcal{L}_{\mathrm{total}}; see Eq. (8)

  11. end for

  12. end for

The Evaluator Discards the Evidence

A contradiction needs two predicates to be present at once. With Constraint keeps one predicate per group, so the evaluator removes the evidence of the contradiction before recall is computed.

Proposition 1 (Within-group violations are invisible). Let (u,v)(u,v) be an excluded pair with u,vu,v in the same predicate group gg. For any score matrix Y^\hat{Y}, the With-Constraint output of (1) contains at most one of uu and vv for any pair ii. Consequently, R@KR@K and mR@KmR@K computed under With Constraint depend on the scores of group gg only through its arg max and the score of that arg max: changing the score of the other member of an excluded pair, with the arg max and its score held fixed, leaves both metrics unchanged.

Proof. (1) keeps a single element of gg per pair, namely arg maxw∈gy^i,w_{w \in g} \hat{y}_{i,w}. Since u,v∈gu,v \in g, at most one of them is kept. The scored candidate set therefore never contains both, for any values of y^i,u\hat{y}_{i,u} and y^i,v\hat{y}_{i,v}. It holds only the arg max of each group and its score, so a statistic computed from that set cannot depend on the score of a predicate that is not kept. □\square

Every rule in Table 1 is within-group. The quantity LSC\mathcal{L}_{\mathrm{SC}} minimises is thus one that With-Constraint recall does not respond to. The violations are nonetheless real and do appear under No Constraint, where more than one predicate per group is kept. A study that claims to improve predicate co-activation consistency therefore cannot support that claim with With-Constraint recall alone.

The Penalty Does Not Know Which Predicate Is Correct

Proposition 2 (Gradient of the exclusion penalty). Define the conflict mass of predicate uu at pair ii as mi,u=∑vMuvy^i,vm_{i,u} = \sum_{v} M_{uv}\hat{y}_{i,v}. Then

∂LSC∂y^i,u=2Nmi,u.(9)\frac{\partial\mathcal{L}_{\mathrm{SC}}}{\partial\hat{y}_{i,u}} = \frac{2}{N}m_{i,u}. \tag*{(9)}

Writing y^i,u=σ(zi,u)\hat{y}_{i,u} = \sigma(z_{i,u}) for the head logits and gi,u=y^i,u(1−y^i,u)g_{i,u} = \hat{y}_{i,u}(1-\hat{y}_{i,u}), a gradient step of size η\eta on LSC\mathcal{L}_{\mathrm{SC}} changes the logit gap of an excluded pair (u,v)(u,v) by

Δ(zi,u−zi,v)=2ηN(mi,vgi,v−mi,ugi,u).(10)\Delta(z_{i,u} - z_{i,v}) = \frac{2\eta}{N}(m_{i,v}g_{i,v} - m_{i,u}g_{i,u}). \tag*{(10)}

No term depends on the ground-truth label.

Proof. MM is symmetric with zero diagonal, so differentiating (7) gives (9). The chain rule through the sigmoid gives

∂LSC∂zi,u=2Nmi,ugi,u,Δzi,u=−2ηNmi,ugi,u\frac{\partial\mathcal{L}_{\mathrm{SC}}}{\partial z_{i,u}} = \frac{2}{N}m_{i,u}g_{i,u}, \qquad\Delta z_{i,u} = -\frac{2\eta}{N}m_{i,u}g_{i,u}

under gradient descent. Subtracting the two updates gives (10). (7) contains no label term. □

Two consequences follow, and they cover the two kinds of rows in Table 1.

Corollary 1 (Two-predicate rules: the ordering is reinforced). If uu and vv are each other’s only excluded partner, then mi,u=y^i,vm_{i,u} = \hat{y}_{i,v} and mi,v=y^i,um_{i,v} = \hat{y}_{i,u}, and (10) reduces to

Δ(zi,u−zi,v)=2ηNy^i,uy^i,v(y^i,u−y^i,v),(11)\Delta(z_{i,u}-z_{i,v}) = \frac{2\eta}{N}\hat{y}_{i,u}\hat{y}_{i,v}(\hat{y}_{i,u}-\hat{y}_{i,v}), \tag*{(11)}

which has the sign of y^i,u−y^i,v\hat{y}_{i,u}-\hat{y}_{i,v}. The penalty widens whichever gap already exists.

Concretely, if the model scores sitting on at 0.8 and standing on at 0.2 for one pair, the penalty pushes the 0.2 down harder than the 0.8. It would do exactly the same if standing on were the correct label. The loss removes a contradiction by amplifying whichever side is currently ahead.

Corollary 2 (Multi-partner rules: a bias by degree). Let uu and vv be excluded partners with y^i,u=y^i,v\hat{y}_{i,u} = \hat{y}_{i,v}, so gi,u=gi,vg_{i,u} = g_{i,v}. Then (10) has the sign of mi,v−mi,um_{i,v}-m_{i,u}: the tie is broken toward the predicate with the smaller conflict mass. If all scores in the group are equal, mi,um_{i,u} is proportional to the number of rules uu appears in, so ties break toward predicates of lower degree in MM.

In our matrix, not contacting has degree three and its three partners have degree one (Table 1). The penalty therefore breaks ties away from not contacting and toward the contact predicates, independently of which is correct. This is a property of the quadratic form, not of the data. It suggests that an exclusion penalty of this shape should be normalised by degree, or replaced by a within-group softmax over excluded sets. We did not test either variant.

Hypothesis 1 (Expected effect on With-Constraint recall). Under With Constraint, R@KR@K depends on the scores only through the within-group arg max and the ranking of kept triplets. By Corollaries 1 and 2, LSC\mathcal{L}_{\mathrm{SC}} moves the arg max for reasons unrelated to the label. Applied while the ordering is unreliable, it consolidates whichever errors are present; applied once the ordering is largely correct, its effect on the arg max is small. We therefore hypothesise that its contribution to With-Constraint recall is non-positive early in training and near zero later; this does not follow from the gradient alone, since recall also depends on the supervised terms.

We separate three things here. The intended role of LSC\mathcal{L}_{\mathrm{SC}} is to reduce co-activation mass. Its unintended optimisation effect is to reinforce the current ordering and to penalise high-degree predicates. Hypothesis 1 is a prediction about recall, not a measurement: it says that enabling the penalty early should depress With-Constraint recall relative to leaving it off. Section 6.3 measures the effect of enabling it after two warm-up epochs.

Experiments

Setup

We use Action Genome [11], built on Charades [19]. Compute was one NVIDIA RTX 3060 with 12 GB, which bounds what these experiments can establish. We fix an 800-video subset and train every model on it with identical features, optimiser and evaluation code, so the comparison is like-for-like even though the absolute numbers are not comparable to published full-split results. Table 2 lists the conditions.

The 600 training and 200 test videos are drawn from the official Action Genome train and test partitions, respectively, so no test video is seen during training.1

We compare against SceneSayerODE, the Neural ODE variant of SceneSayer [16], since it differs from CoupledSGA in precisely the property we study: whether the vector field acts on each pair independently. We report R@K, mR@K averaged over the 26 predicate classes, and their harmonic mean hR@K, for cutoffs K∈{10,20,50}K \in\{10, 20, 50\} (unrelated to the horizon K=3K = 3). Proposition 1 says that none of these responds to the violations LSC\mathcal{L}_{\mathrm{SC}} targets. We report them because R@K and mR@K are the standard SGA metrics [16], and hR@K combines the two. Section 6.3 adds a structural measurement under No Constraint, and Section 6.4 returns to what should accompany them.

@page 6/9 starts_mid=1 ends_mid=0 No term depends on the ground-truth label.

Proof. MM is symmetric with zero diagonal, so differentiating (7) gives (9). The chain rule through the sigmoid gives

∂LSC∂zi,u=2Nmi,ugi,u,Δzi,u=−2ηNmi,ugi,u\frac{\partial\mathcal{L}_{\mathrm{SC}}}{\partial z_{i,u}} = \frac{2}{N}m_{i,u}g_{i,u}, \qquad\Delta z_{i,u} = -\frac{2\eta}{N}m_{i,u}g_{i,u}

under gradient descent. Subtracting the two updates gives (10). (7) contains no label term. □

Two consequences follow, and they cover the two kinds of rows in Table 1.

Corollary 1 (Two-predicate rules: the ordering is reinforced). If uu and vv are each other’s only excluded partner, then mi,u=y^i,vm_{i,u} = \hat{y}_{i,v} and mi,v=y^i,um_{i,v} = \hat{y}_{i,u}, and (10) reduces to

Δ(zi,u−zi,v)=2ηNy^i,uy^i,v(y^i,u−y^i,v),(12)\Delta(z_{i,u}-z_{i,v}) = \frac{2\eta}{N}\hat{y}_{i,u}\hat{y}_{i,v}(\hat{y}_{i,u}-\hat{y}_{i,v}), \tag*{(12)}

which has the sign of y^i,u−y^i,v\hat{y}_{i,u}-\hat{y}_{i,v}. The penalty widens whichever gap already exists.

Concretely, if the model scores sitting on at 0.8 and standing on at 0.2 for one pair, the penalty pushes the 0.2 down harder than the 0.8. It would do exactly the same if standing on were the correct label. The loss removes a contradiction by amplifying whichever side is currently ahead.

Corollary 2 (Multi-partner rules: a bias by degree). Let uu and vv be excluded partners with y^i,u=y^i,v\hat{y}_{i,u} = \hat{y}_{i,v}, so gi,u=gi,vg_{i,u} = g_{i,v}. Then (10) has the sign of mi,v−mi,um_{i,v}-m_{i,u}: the tie is broken toward the predicate with the smaller conflict mass. If all scores in the group are equal, mi,um_{i,u} is proportional to the number of rules uu appears in, so ties break toward predicates of lower degree in MM.

In our matrix, not contacting has degree three and its three partners have degree one (Table 1). The penalty therefore breaks ties away from not contacting and toward the contact predicates, independently of which is correct. This is a property of the quadratic form, not of the data. It suggests that an exclusion penalty of this shape should be normalised by degree, or replaced by a within-group softmax over excluded sets. We did not test either variant.

Hypothesis 1 (Expected effect on With-Constraint recall). Under With Constraint, R@KR@K depends on the scores only through the within-group arg max and the ranking of kept triplets. By Corollaries 1 and 2, LSC\mathcal{L}_{\mathrm{SC}} moves the arg max for reasons unrelated to the label. Applied while the ordering is unreliable, it consolidates whichever errors are present; applied once the ordering is largely correct, its effect on the arg max is small. We therefore hypothesise that its contribution to With-Constraint recall is non-positive early in training and near zero later; this does not follow from the gradient alone, since recall also depends on the supervised terms.

We separate three things here. The intended role of LSC\mathcal{L}_{\mathrm{SC}} is to reduce co-activation mass. Its unintended optimisation effect is to reinforce the current ordering and to penalise high-degree predicates. Hypothesis 1 is a prediction about recall, not a measurement: it says that enabling the penalty early should depress With-Constraint recall relative to leaving it off. Section 6.3 measures the effect of enabling it after two warm-up epochs.

Dataset / taskAction Genome, PredCLS
Split600 train / 200 test videos
Prediction horizonK=3K = 3 frames
Sequence cap20 frames
OptimiserAdamW [12]
Learning rate1×10−51 \times10^{-5}, weight decay 10−410^{-4}
Gradient clippingnorm 5.0
Epochs4
Runs1 (Table 3); 3 seeds (Table 4)
ODE solverexplicit Euler, τ=0.5\tau= 0.5
BaselineSceneSayerODE [16]
ProtocolWith Constraint; No Constraint (Table 4)

Table 2 (PDF p. 6). Experimental conditions, shared by every run.

MethodKKR@KmR@KhR@K
SceneSayerODE [16]1043.2210.7917.27
SceneSayerODE [16]2044.7811.4718.26
SceneSayerODE [16]5044.7811.4818.27
CoupledSGA (ours)1038.969.5115.29
CoupledSGA (ours)2040.4810.1016.16
CoupledSGA (ours)5040.4910.1016.17

Table 3 (PDF p. 6). Matched-budget comparison (Action Genome, PredCLS, With Constraint). Both models use the conditions of Table 2. These are small-subset results under a four-epoch budget and are not a full-benchmark comparison.

Experiments

Setup

We use Action Genome [11], built on Charades [19]. Compute was one NVIDIA RTX 3060 with 12 GB, which bounds what these experiments can establish. We fix an 800-video subset and train every model on it with identical features, optimiser and evaluation code, so the comparison is like-for-like even though the absolute numbers are not comparable to published full-split results. Table 2 lists the conditions.

The 600 training and 200 test videos are drawn from the official Action Genome train and test partitions, respectively, so no test video is seen during training.1

We compare against SceneSayerODE, the Neural ODE variant of SceneSayer [16], since it differs from CoupledSGA in precisely the property we study: whether the vector field acts on each pair independently. We report R@K, mR@K averaged over the 26 predicate classes, and their harmonic mean hR@K, for cutoffs K∈{10,20,50}K \in\{10, 20, 50\} (unrelated to the horizon K=3K = 3). Proposition 1 says that none of these responds to the violations LSC\mathcal{L}_{\mathrm{SC}} targets. We report them because R@K and mR@K are the standard SGA metrics [16], and hR@K combines the two. Section 6.3 adds a structural measurement under No Constraint, and Section 6.4 returns to what should accompany them.

Illustration comparing the uncoupled baseline and CoupledSGA

Figure 2. Illustration of the target failure mode; the scores shown are illustrative and are not model outputs. Left: an uncoupled ODE evolves each relationship alone, and nothing in it prevents high scores on mutually exclusive predicates such as sitting on and standing on. Right: CoupledSGA couples the trajectories and penalises co-activation. By Proposition 1, this difference does not appear in With-Constraint recall (Table 3); the contradiction rate of Table 4 measures it.

Matched-Budget Comparison

On our matched 600/200 split, CoupledSGA does not outperform SceneSayerODE under With-Constraint recall. Table 3 gives the numbers. The baseline leads by 4.30 points of R@20, 1.37 points of mR@20 and 2.10 points of hR@20, and by similar margins at K=10K = 10.

The gap mixes coupling and penalty. The CoupledSGA row of Table 3 is taken after LSC\mathcal{L}_{\mathrm{SC}} was switched off, from a run in which it was active (λSC=0.1\lambda_{\mathrm{SC}} = 0.1) in epochs 0–1. The difference in Table 3 therefore does not isolate the coupled vector field. § 8.3 separates the two effects with controlled runs.

Capacity and budget are confounded. The GATv2 field adds three d×dd \times d projections per head per layer at d=1936d = 1936, many times the parameters of the row-wise field it replaces. Four epochs at a learning rate of 10−510^{-5} is a short schedule for that many new parameters, and the added capacity is what is least likely to be trained under it. We cannot separate a real deficit of the coupled formulation from an undertrained one, and we do not claim to.

K=50K = 50 is saturated. R@20 and R@50 agree to within 0.01 points for both models, because each pair contributes one triplet per group, and a frame with few active pairs yields a candidate set that top-20 already exhausts. Only K=10K = 10 discriminates at this scale.1

Controlled Comparison under Both Protocols

Section 6.2 leaves two questions open: how much of the gap in Table 3 is due to coupling and how much to the penalty, and whether LSC\mathcal{L}_{\mathrm{SC}} reduces contradictions at all. We answer both with three runs trained under the conditions of Table 2: SceneSayerODE (A), CoupledSGA with λSC=0\lambda_{\mathrm{SC}} = 0 throughout (B), and CoupledSGA with λSC=0.1\lambda_{\mathrm{SC}} = 0.1 from estart=2e_{\mathrm{start}} = 2 (C), which is the schedule of Section 4.4. B and C share the architecture, settings and budget, and differ only in whether LSC\mathcal{L}_{\mathrm{SC}} is active in epochs 2–3; A and B differ only in coupling.

Dataset / taskAction Genome, PredCLS
Split600 train / 200 test videos
Prediction horizonK=3K = 3 frames
Sequence cap20 frames
OptimiserAdamW [12]
Learning rate1×10−51 \times10^{-5}, weight decay 10−410^{-4}
Gradient clippingnorm 5.0
Epochs4
Runs1 (Table 3); 3 seeds (Table 4)
ODE solverexplicit Euler, τ=0.5\tau= 0.5
BaselineSceneSayerODE [16]
ProtocolWith Constraint; No Constraint (Table 4)

Table 2. Experimental conditions, shared by every run.

MethodKKR@KmR@KhR@K
SceneSayerODE [16]1043.2210.7917.27
SceneSayerODE [16]2044.7811.4718.26
SceneSayerODE [16]5044.7811.4818.27
CoupledSGA (ours)1038.969.5115.29
CoupledSGA (ours)2040.4810.1016.16
CoupledSGA (ours)5040.4910.1016.17

Table 3. Matched-budget comparison (Action Genome, PredCLS, With Constraint). Both models use the conditions of Table 2. These are small-subset results under a four-epoch budget and are not a full-benchmark comparison.

Each run is scored twice. R@20 is computed under With Constraint, as in Table 3. Contradictions are measured under No Constraint, where they can reach the output, by the contradiction rate (CR), a rank-based proxy for structural validity. For pair ii and group gg (Spatial or Contacting), let u1i,gu_1^{i,g} and u2i,gu_2^{i,g} be the two highest-scoring predicates of gg. Then

CR=100∣I∣∑(i,g)∈IMu1i,gu2i,g (%),(13)\mathrm{CR} = \frac{100}{|\mathcal{I}|}\sum_{(i,g)\in\mathcal{I}} M_{u_1^{i,g}u_2^{i,g}} \,(\%), \tag*{(13)}

where I\mathcal{I} collects all (pair, group) instances over the anticipated frames. CR uses ranks only, so it needs no threshold. Table 4 reports the results. Its runs were trained separately from those of Table 3; SceneSayerODE reaches 44.15 R@20 here (mean of three seeds) and 44.78 there under identical conditions.

ModelLSC\mathcal{L}_{\mathrm{SC}}R@20 (%) ↑CR (%) ↓
ASceneSayerODE—44.15 ± 0.5154.40 ± 1.21
BCoupledSGAoff42.21 ± 0.7545.92 ± 1.44
CCoupledSGAepochs 2–335.18 ± 1.104.07 ± 0.46

Table 4. Controlled comparison (Action Genome, PredCLS; conditions of Table 2). R@20 is computed under With Constraint and the contradiction rate CR (Equation 12) under No Constraint, on anticipated frames. Runs B and C differ only in whether LSC\mathcal{L}_{\mathrm{SC}} (λSC=0.1\lambda_{\mathrm{SC}} = 0.1) is active in epochs 2–3; A and B differ only in coupling. Mean ± standard deviation over three seeds.

The loss reduces contradictions. Between B and C, enabling LSC\mathcal{L}_{\mathrm{SC}} lowers CR from 45.92% to 4.07%, roughly elevenfold, and lowers R@20 by 7.03 points, from 42.21 to 35.18. This is Proposition 1 made measurable: With-Constraint recall registers the cost of the loss but not its structural gain, and Table 4 reports both.

Coupling alone reduces contradictions. Without any knowledge of excluded pairs, the RIG lowers CR from 54.40% to 45.92% (A to B), at a cost of 1.94 points of R@20.

The schedule does not avoid the cost. Hypothesis 1 predicts that the cost shrinks once the ordering within excluded pairs is largely correct. The measured cost suggests that, at our budget, two warm-up epochs did not reach that regime.

Implications for Evaluation

Figure 2 illustrates the failure mode the loss targets. With-Constraint recall shows only the cost of the penalty, because it does not respond to co-activation; the benefit appears only under No Constraint (Table 4). What follows is a recommendation for any SGA study that constrains which predicates may be simultaneously active.

Report results under No Constraint as well, since that is where co-activated predicates reach the scored output at all. Alongside recall, report two quantities computable from stored predictions without further training. A contradiction rate (Equation (13)) is the fraction of (pair, group) instances whose two highest-scoring predicates form an excluded pair; it compares ranks, so it needs no threshold. A structural consistency score is the fraction of predicted graphs containing no excluded co-activation; it is stricter and closer to what a downstream consumer needs, since consumers read graphs rather than single edges. We did not find either reported alongside With-Constraint recall in the SGA studies cited here. We report the first in Table 4 and leave the second to future work.

Discussion and Limitations

The empirical claims are bounded by scale. All results come from an 800-video subset and a four-epoch budget; Table 3 comes from a single seed and Table 4 from three seeds. Propositions 1 and 2 do not depend on scale. The four-epoch budget leaves the capacity confound of Section 6.2 unresolved, and two warm-up epochs were not enough for the schedule of Section 4.4 to avoid the recall cost (Section 6.3). Structural validity is assessed only through the top-two contradiction rate; the structural consistency score of Section 6.4 is not reported. The exclusion matrix is hand-curated, and its coverage of the Action Genome ontology is unknown, so the contradiction rate inherits that coverage. Explicit Euler at τ=0.5\tau=0.5 is first-order, and its truncation error is not quantified against an adaptive solver.

Two failure patterns recur in the predicted graphs. Under heavy occlusion, the backbone features degrade before the RIG is built, and coupling then spreads the resulting noise across neighbouring trajectories instead of damping it. For the rarest contacting predicates, which have fewer than ten training instances in the 600-video split, neither model predicts transitions reliably.

Because Action Genome is built from Charades [19], whose activity videos participants recorded in their own homes, models that forecast human interactions could be repurposed for surveillance. We use only existing annotations and collect no new data.

Conclusion

We set out to remove independent relation trajectories from continuous-time Scene Graph Anticipation, and built CoupledSGA: a graph over relations, coupled into the vector field of a Neural ODE, with a penalty on mutually exclusive predicates. Building it exposed a problem with the measurement rather than the model.

What we prove: under the With-Constraint emission rule, a scored output can never contain both members of a within-group exclusion rule, so recall computed under that protocol cannot register those violations. What we derive: the gradient of a quadratic exclusion penalty is label-free, it widens whichever ordering a two-predicate rule already has, and it breaks ties against predicates that appear in many rules. What we observe: on a matched 600/200 split, CoupledSGA reaches 40.48% R@20 against 44.78% for SceneSayerODE, and in a controlled comparison, enabling the penalty after two warm-up epochs lowers R@20 from 42.21% to 35.18%. The second, which isolates the penalty, has the sign the analysis predicts. Under No Constraint, the same change lowers the contradiction rate from 45.92% to 4.07%, and coupling alone lowers it from 54.40% to 45.92%. On this measure, the penalty reduces contradictions at a recall cost, and With-Constraint recall reports only the cost.

Two next steps follow: a graph-level structural consistency score reported alongside the contradiction rate, and the degree-normalised form of the penalty suggested by Corollary 2.

Acknowledgments

Suman Chowdhury and Rajesh P. Barnwal acknowledge partial financial support from CSIR-CMERI, Durgapur, under Project MLP002613. Suman Chowdhury also acknowledges financial support from the Council of Scientific and Industrial Research (CSIR), India, through a Senior Research Fellowship (GATE), Grant No. GATE31/0019(15628)/2022-EMR-I. AI-based tools were used for copy-editing only; the authors take full responsibility for the content.

References

References

  1. [1]Antonio Alliegro, Francesca Pistilli, Tatiana Tommasi, and Giuseppe Averta. 2025. FORECasting Human Activity via Latent SCENE Graphs Diffusion. arXiv:2503.06182 [cs.CV]
  2. [2]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum Learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML). ACM, New York, NY, USA, 41–48. doi:10.1145/1553374.1553380DOI
  3. [3]Shaked Brody, Uri Alon, and Eran Yahav. 2022. How Attentive Are Graph Attention Networks?. In International Conference on Learning Representations (ICLR). OpenReview.net.
  4. [4]Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. 2018. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 31. Curran Associates, Inc., Red Hook, NY, USA, 6571–6583.
  5. [5]Tianyu Chen, Weihão Yu, Riquan Chen, and Liang Lin. 2019. Knowledge-Embedded Routing Network for Scene Graph Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 6163–6171.arxiv.org/abs/1903.03326
  6. [6]Juncheng Wang, Wenting Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. 2021. Spatial-Temporal Transformer for Dynamic Scene Graph Generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Piscataway, NJ, USA, 16372–16382.
  7. [7]Shengyu Feng, Hesham Mostafa, Marcel Nassar, Sombed Majumdar, and Subarna Tripathi. 2023. Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, Piscataway, NJ, USA, 5130–5139.arxiv.org/abs/2112.09828
  8. [8]Antonino Furnari and Giovanni Maria Farinella. 2019. What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Piscataway, NJ, USA, 6252–6261.arxiv.org/abs/1905.09035
  9. [9]Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. 2019. Predicting the Future: A Jointly Learnt Model for Action Anticipation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Piscataway, NJ, USA, 5562–5571.arxiv.org/abs/1912.07148
  10. [10]Zijie Huang, Yizhou Sun, and Wei Wang. 2021. Coupled Graph ODE for Learning Interactive System Dynamics. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, USA, 705–715. doi:10.1145/3447548.3467385DOI
  11. [11]Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. 2020. Action Genome: Actions as Compositions of Spatio-Temporal Scene Graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 10236–10247.
  12. [12]Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations (ICLR). OpenReview.net.
  13. [13]Robin Manhaeve, Sebastian Dumančić, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. 2018. DeepProbLog: Neural Probabilistic Logic Programming. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 31. Curran Associates, Inc., Red Hook, NY, USA, 3749–3759.arxiv.org/abs/1805.10872
  14. [14]Sayak Nag, Kyle Min, Subarna Tripathi, and Amit K. Roy-Chowdhury. 2023. Unbiased Scene Graph Generation in Videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 2283–22813.arxiv.org/abs/2304.00733
  15. [15]Rohith Peddi, Saurabh, Ayush Abhay Shrivastava, Parag Singla, and Vibhav Gogate. 2025. Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 8648–8657.arxiv.org/abs/2411.13059
  16. [16]Rohith Peddi, Saksham Singh, Saurabh, Parag Singla, and Vibhav Gogate. 2024. Towards Scene Graph Anticipation. In Computer Vision – ECCV 2024 (Lecture Notes in Computer Science, Vol. 15146). Springer, Cham, 159–175. doi:10.1007/978-3-031-73223-2_10DOI
  17. [17]Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkoo Park. 2019. Graph Neural Ordinary Differential Equations. arXiv:1911.07532 [cs.LG]
  18. [18]T. Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. 2022. Graph-Coupled Oscillator Networks. In Proceedings of the 39th International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research, Vol. 162). PMLR, 18888–18909.arxiv.org/abs/2202.02296
  19. [19]Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. In Computer Vision – ECCV 2016 (Lecture Notes in Computer Science, Vol. 9905). Springer, Cham, 510–526. doi:10.1007/978-3-319-46448-0_31
  20. [20]Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Elezabeth, Gerard Medioni, and Leonid Sigal. 2021. Energy-Based Learning for Scene Graph Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 13936–13945.arxiv.org/abs/2103.02221
  21. [21]Dai Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. 2017. Scene Graph Generation by Iterative Message Passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 5410–5419.arxiv.org/abs/1701.02426
  22. [22]Yi Yang, Yuren Cong, Hao Cheng, Bodo Rosenhahn, and Michael Ying Yang. 2025. FDSG: Forecasting Dynamic Scene Graphs. arXiv:2506.01487 [cs.CV]arxiv.org/abs/2506.01487
  23. [23]Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. 2020. Bridging Knowledge Graphs to Generate Scene Graphs. In Computer Vision – ECCV 2020. Springer, Cham, 606–623. doi:10.1007/978-3-030-58592-1_36DOI
  24. [24]Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neural Motifs: Scene Graph Parsing with Global Context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA, 5831–5840.arxiv.org/abs/1711.06640

Paper details

vlnc:W_v39ne3ws
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 9

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

AI-based tools

View statement
Looking for related papers…
Contents