What Order Do DeepSeek-OCR 2's Causal Flow Queries Actually Learn?

Code: DeepSeek-OCR2-query-ablation

Flattening a 2D image into a 1D sequence using a fixed raster scan introduces an unsuitable inductive bias that overlooks semantic relationships. DeepSeek-OCR 2 aims to use DeepEncoder V2 to dynamically reorder visual tokens according to semantics, moving toward visual encoding that more closely resembles human perception. However, the DeepSeek-OCR 2 paper does not analyze what information the causal query tokens carry or what order they actually learn. In this post, I use a few small experiments to explore their role and the ordering they learn.

I examine this question with two practical probes: (1) inspecting attention activation patterns in the encoder, and (2) replacing or shuffling query tokens to observe sensitivity to ordering and changes in the output. Across the few examples examined here, the order learned by the query tokens appears closer to raster-scan order.

DeepSeek-OCR 2 Architecture

DeepEncoder V2 is the core component of DeepSeek-OCR 2. It introduces a set of causally constrained queries that resample image tokens on a 2D grid into a 1D causal sequence of query tokens. The overall idea is similar to Q-Former’s use of queries to aggregate visual features. The main distinction is causality:

• In the encoder, query tokens at each position can attend only to preceding query tokens, rather than future ones, creating a forward flow of information.
• In the decoder, query tokens from the encoder are likewise subject to the constraint of attending only to preceding context, making subsequent generation more dependent on this unidirectional flow of visual information.

The paper calls this design causal visual flow: the aim is for the model to develop an internal path for visual information that better reflects reading order and structural reasoning.

DeepSeek-OCR 2 architecture.

Attention mask in DeepEncoder.

Attention Activation Patterns in DeepEncoder

Experimental setup. I selected two types of example images—a presentation slide and a Chinese Spring Festival couplet—and resized and padded each input to 1024 × 1024. I then used SAM to encode each image into 256 image tokens and passed them to the Qwen2 LLM vision encoder. To reduce interference from other factors, I retained only the global-view features.

Observation method. I recorded the attention weights from query tokens to image tokens in the network’s final two layers. After renormalizing these weights, I plotted attention heatmaps and calculated the attention “centroid” to roughly indicate which image region each query was looking at.

The two videos below show the results. Red boxes indicate regions activated by the attention weights, with darker colors indicating higher weights. Blue dots mark the attention centroids. Green boxes are bounding rectangles around the aggregated regions of high activation.

The main observations are:

1. Almost every query token assigns relatively high attention to the first image token, resembling the commonly observed attention sink phenomenon.
2. The overall activation trajectory is fairly close to raster-scan order.
3. Near the end of the sequence, a few query tokens “look back” at the beginning of the image, refocusing their attention on earlier image tokens.

Query Replacement Analysis in the Decoder

To test how sensitive the decoder is to the order of the query-token sequence, I applied several interventions before decoding. All experiments used deterministic decoding (do_sample=False):

• Operation 1: Replace the last 128 query tokens with the first 128 query tokens from another image.
• Operation 2: Copy the first 128 query tokens into the last 128 positions.
• Operation 3: Randomly shuffle the query-token sequence.
• Operation 4: Swap the first 128 and last 128 query tokens.

Results: The Couplet Image

Baseline (no intervention on the query tokens)

Chinese OCR outputs below are reproduced verbatim to preserve the experimental results.

马年吉祥

金马奔腾开放景

玉羊启泰展宏图

豆包AI生成

Operation 1: Replace the last 128 query tokens with the first 128 query tokens from another image.

# 马年吉祥

- **Human Factors**
  - the process molds to the needs of the people and team, not the other way around
  - key traits must exist among the people on an agile team and the team itself:
    - Competence.
    - Common focus.
    - Collaboration.
    - 五羊启泰

- 金马奔腾
- 玉羊启泰

Operation 2: Copy the first 128 query tokens into the last 128 positions.

马年吉祥
金马奔腾
玉羊启瑞
玉羊启瑞

Operation 3: Randomly shuffle the sequence.

马年吉祥

金马奔腾开放景

玉羊启泰展宏图

豆AI生成

Operation 4: Swap the first 128 and last 128 query tokens.

马年吉祥

金马奔腾开盛景

玉羊启泰展宏图

豆包AI生成

Results: Image 2

Baseline

- **Human Factors**

  - the process molds to the needs of the people and team, not the other way around
  - key traits must exist among the people on an agile team and the team itself:
    - **Competence.**
    - **Common focus.**
    - **Collaboration.**
    - **Decision-making ability.**
    - **Fuzzy problem-solving ability.**
    - **Mutual trust and respect.**
    - **Self-organization.**

Operation 1: Replace the last 128 query tokens with the first 128 query tokens from another image.

# 马年吉祥

- the process molds to the needs of the people and team, not the other way around
- key traits must exist among the people on an agile team and the team itself:
  - Competence.
  - Common focus.
  - Collaboration.
  - 玉羊启瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞启瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞瑞启

Operation 2: Copy the first 128 query tokens into the last 128 positions.

- **Human Factors**

  - the process molds to the needs of the people and team, not the other way around
  - key traits must exist among the people on an agile team and the team itself:
    - Competence.
    - Common focus.
    - Collaboration.
    - Collaboration.
    - Common focus.
    - Collaboration.
    - Common focus.
    - Collaboration.
    - Common focus.
    - Collaboration.
    - Common focus.

Operation 3: Randomly shuffle the sequence.

- **Human Factors**

  - the process of the people and their way around
    - key traits must be present in the people and the way they work
    - self-organisation

  - competence
  - focus
  - collaboration
  - decision-making ability
  - mutual respect
  - common sense
  - fuzzy thinking
  - self-organisation

8

Operation 4: Swap the first 128 and last 128 query tokens.

- **Human Factors**

  - the process molds to the needs of the people and team, not the other way around

  - key traits must exist among the people on an agile team and the team itself:
    - Competence.
    - Common focus.
    - Collaboration.
    - Decision-making ability.
    - Fuzzy problem-solving ability.
    - Mutual trust and respect.
    - Self-organization.

Experimental Observations

• In most of the cases examined, query-token order still corresponds closely to raster-scan order.
• For the couplet example, shuffling has little effect on the final OCR result. For the other example, the effects are more evident as slight semantic perturbations than as a breakdown in structure.

These observations suggest that the query tokens may encode some form of positional signal or traversal prior, allowing the decoder to recover the output without relying entirely on a strict sequence order.

Conclusion

Taken together, these two sets of observations lead me to tentatively favor the interpretation that the order learned by the query tokens along DeepSeek-OCR 2’s causal flow is largely consistent with raster-scan order. Its benefits may therefore not come from explicitly learning to reorder visual tokens into human reading order.