graph LR
%% Conceptual Stage
C2((("sleep(x)")))
%% Lemma Stage (Centralized)
subgraph Lemma_Stage ["Lemma Stage"]
direction LR
F1["V⁰"] --- F2["pres"] --- L1(["sleep"]) --- F3["3"] --- F4["pl"]
end
%% Lexeme Stage
subgraph Lexeme_Stage ["Lexeme Stage"]
direction TB
LX1["/sl:ip/"]
LX2["/s/"]
LX3["/slEpt/"]
LX4["/IN/"]
end
%% Realization Stage
subgraph Realization_Stage ["Realization Stage"]
direction TB
R1["[sli:p]"]
R2["[sli:ps]"]
R3["[slEpt]"]
R4["[IN]"]
end
%% Strict Linear Connections
C2 --> L1
L1 --> LX1 & LX2 & LX3 & LX4
LX1 --> R1
LX1 --> R2
LX1 --> R3
LX2 --> R2
LX3 --> R3
LX4 --> R4
unaccusativity syntax or picture difficulty?
psycholinguistics
ai
experiments
CLIP
unaccusativity
i am obssessed with early advance planning, but let me make sure about some picture saliency
Prelude
What?
During my PhD, I’ve become fairly obsessed with production studies. I find them extremely interesting, especially the way they combine what we know from theoretical linguistics with creative experimental methods. Not to mention that the theoretical framing of production work is quite poor and the main theories used need a major overhaul. One of the most interesting papers in this area is Shota Momma’s work on advanced verb planning. Similar work was also done in German and Basque by Sebastian Sauppe’s group.
Let me set the scene. He and his colleagues ran multiple picture description experiments where participants saw images like:
- “The octopus below the spoon is swimming” (unergative)
- “The octopus below the spoon is boiling” (unaccusative)
If you’re not a syntax nerd, here’s the ultra-compressed version: verbs like “swim” and “bark” (unergatives) are different from verbs like “sink” and “melt” (unaccusatives), even though they both describe single-argument events. The difference has to do with argument structure—where the subject comes from in the underlying syntax. It has been argued that the subjects of unaccusatives are actually ‘deep objects’ for lack of a better term, and they structurally start in the same position as any other object.
They showed that these two verb types behave differently in production experiments. Speakers plan them differently. They tested this by showing related or unrelated words superimposed on the pictures. They found that when the verbs were related, participants slowed down before they started speaking—but only with unaccusatives. His theoretical claim was that unaccusative verbs are planned earlier in the sentence production process—possibly right at the beginning, along with the subject.
Why though?
Here’s the thing. Another thing that made me very excited about the production endeavor is that there are probably so many possible confounds that require checking. And I love this song and dance in psycholinguistics, where I can stress-test findings and see how stable they are. It’s especially important when you find an unexpected result—like participants taking longer to start speaking when they won’t say the verb for at least 3 more seconds—my first instinct, and I hope yours, is to wonder: “Is this real, or is something else going on?”
This post is built on a very specific worry: What if unaccusative scenes themselves, and not the syntax of them, created the results? One interesting finding in Shota Momma’s papers was that unergative planning was seemingly invisible. He has shown that there are reasons to believe that it happens while saying the second NP. But quantitatively, the signature of unergative planning seems to be more dissolved throughout the sentence, while the unaccusative planning is strikingly clear.
This creates the following question: is it possible that participants, simply because the picture was more difficult to understand or the subject was more involved in the action, spent more time initially to either understand the event or to extract the subject from the event, and during this time a deterministic analysis of the written word kicked in and slowed them down when it was related? Since the unergative subjects are more easily dissociable from the event, since nothing is happening to them in those pictures, it takes less time, and since it’s less of a resource-heavy process, no additional process interferes with it. This has several predictions. First, in follow-up experiments where the unergative pictures are hard to ‘retrieve’ from the scene, one should see similar onset effects. Second, if there is some sort of picture-difficulty metric, the advance planning should align with that metric item-wise.
The second prediction is going to be the basis of this blog post, where we will find a way to quantify the picture difficulty.1
I make assumptions
I assume the following ‘two-way’ distinction with respect to lexical verbs. However, one needs to admit that unaccusativity is not stable all the time. Many such unaccusative verbs can be used as unergatives given some adverbial modification or different contexts. This would create some minor infelicity in English, but that is not the case for many languages. For example, Laz can make any verb ‘agentive’ with a small prefix. Imagine a Laz-type English where you have “I cried” vs. “I do-cried,” where the second one means that you made yourself cry or you deliberately cried. Or a better example might be: imagine if English “jump” were decomposable into a prefix “do-” and “fall.” So, for now I only assume that these properties are lexical properties of the verb, but one needs to admit that these are event-related ones.
- Unergative actions (swimming, barking, running): The action is performed by the agent. You can see the octopus swimming—the action is somewhat separable from what happens to the entity.
- Unaccusative actions (boiling, melting, sinking): Something is happening to the entity. The octopus isn’t “doing” boiling—it’s undergoing a change of state. The action and the entity are less separable.
Another assumption I make is about CLIP/VLM. The input that CLIP takes is a written sentence and a picture. I am fully aware that the way CLIP assesses pictures is nowhere near how humans do.2 I am also aware that in human speech, the scenes are what is encoded and the speech is the decoding. CLIP works differently. CLIP is a two-encoder model. Given two inputs of a picture and a text, it creates two separate vectors and checks how similar those vectors are. Thus, it does not give us anything about human cognition. But it gives us a way to quantify relevant metrics. Below what I assume to be the models of human speech production based on Levelt’s work and CLIP’s architecture.
Levelt’s Speech Production Model:
CLIP Architecture:
flowchart TD
A[Picture] --> B[Image Encoder]
C[Text] --> D[Text Encoder]
B --> E[Image Embedding]
D --> F[Text Embedding]
E --> G[Similarity Score]
F --> G
style A fill:#e1f5dd
style C fill:#e1f5dd
style B fill:#d4e9f7
style D fill:#d4e9f7
style E fill:#fff3cd
style F fill:#fff3cd
style G fill:#f8d7da
Multimodal LLMs:
More recently, multimodal large language models have emerged that work quite differently from CLIP. Instead of creating separate embeddings and comparing them, these models integrate visual and textual information into a unified representation and can generate natural language descriptions or answers about images.
I have to say, writing their code is also a bit funny. You basically have to build a pipeline where you create a ‘chat template’ and ask them to give you an output. I am not sure that is how you are supposed to use them, but it works.3
Models like Qwen3-Omni take both images and text as input, process them through vision encoders and language models together, and generate coherent text outputs. Unlike CLIP’s similarity metric, multimodal LLMs can provide richer, more nuanced interpretations of visual scenes and answer complex questions about them. We will use both of them and compare here.
flowchart TD
A[Picture] --> B[Vision Encoder]
C[Text Prompt] --> D[Tokenizer]
B --> E[Visual Tokens]
D --> F[Text Tokens]
E --> G[Unified LLM]
F --> G
G --> H[Generated Text Output]
style A fill:#e1f5dd
style C fill:#e1f5dd
style B fill:#d4e9f7
style D fill:#d4e9f7
style E fill:#fff3cd
style F fill:#fff3cd
style G fill:#ffd4e5
style H fill:#f8d7da
Lastly, these experiments were conducted as a extended-PWI experiment, where participants were provided with a picture with a superimposed text on it. Neither the pictures, nor the tasks I improvise here does not have any relation to picture word interference task. It would be indeed interesting if we have an understanding how PWI would look like interms of LLM tasks. However it is far from what I would like to achieve here. If I have that idea I will probably submit a paper or an abstract somewhere :).
Predictions
If unaccusative actions (like “boiling” or “melting”) are genuinely harder to see in pictures, or if the subjects are harder to visually identify in the scenes, we’d expect: - Lower similarity scores between the images and their target sentences - Evidence that models struggle to “ground” the sentence/entity in the visual input, in the form of subject saliency.
If that’s the case, we have a problem—the onset latency effect might just be about picture difficulty.4
But if the similarity scores are comparable or higher for unaccusatives, then we can rule out the perceptual confound for now and be more confident that the effects reflect genuine linguistic processing.
Model Base
CLIP
CLIP (Contrastive Language-Image Pre-training) is a neural network trained on 400 million image-text pairs from the internet. It learns to match images with their corresponding text descriptions by projecting both into a shared embedding space.
Setting Up
Let’s start by loading the packages we’ll need. I’m going to build this up step by step, just like I did when I first ran this analysis.
import os
import torch
import clip
from PIL import Image
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from transformers import AutoModelForCausalLM, AutoTokenizer
# Set up plotting style
sns.set_style("whitegrid")
# plt.rcParams['figure.figsize'] = (10, 6)First, we need to load the CLIP model. I’m using the ViT-B/32 variant, which is a good balance between performance and computational efficiency:
# Load two decoder CLIP model
# Note: We use CPU for everything if MPS is detected to avoid moondream2 issues
if torch.cuda.is_available():
device = "cuda"
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
device = "cpu"
else:
device = "cpu"
model_clip, preprocess = clip.load("ViT-B/32", device=device, jit=False)
print(f"Using device: {device}")
print(f"CLIP model loaded successfully!")Now let’s also load a multimodal LLM for comparison. We’ll use Qwen-VL-Chat, a powerful vision-language model:
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
import transformers
import torch
from transformers.generation.beam_search import BeamSearchScorer
transformers.BeamSearchScorer = BeamSearchScorer
# Load Qwen-VL-Chat model
model_id = "Qwen/Qwen-VL-Chat"
model_vlm = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.float32
).to('cpu')
tokenizer_vlm = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
# Create the streamer
streamer = TextStreamer(tokenizer_vlm, skip_prompt=True)The Data Structure
My experimental materials consist of 24 scenes:
- 12 unergative scenes (swimming, running, barking, etc.)
- 12 unaccusative scenes (boiling, shrinking, sinking, etc.)
Each scene pairs a character (octopus, ballerina, chef, etc.) with an action. Let’s create a dataframe with our materials:
# Unergative scenes
df_unerg = pd.DataFrame({
"Filename": [
"./pictures/octopus_swim.jpg",
"./pictures/ballerina_run.jpg",
"./pictures/boy_float.jpg",
"./pictures/chef_yell.jpg",
"./pictures/clown_walk.jpg",
"./pictures/cowboy_wink.jpg",
"./pictures/dog_bark.jpg",
"./pictures/monkey_sleep.jpg",
"./pictures/penguin_sneeze.jpg",
"./pictures/pirate_cough.jpg",
"./pictures/rabbit_smile.jpg",
"./pictures/snail_crawl.jpg",
],
"Sentence": [
"The octopus is swimming.",
"The ballerina is running.",
"The boy is floating.",
"The chef is yelling.",
"The clown is walking.",
"The cowboy is winking.",
"The dog is barking.",
"The monkey is sleeping.",
"The penguin is sneezing.",
"The pirate is coughing.",
"The rabbit is smiling.",
"The snail is crawling.",
]
})
# Unaccusative scenes
df_unacc = pd.DataFrame({
"Filename": [
"./pictures/octopus_boil.jpg",
"./pictures/ballerina_shrink.jpg",
"./pictures/boy_yawn.jpg",
"./pictures/chef_drown.jpg",
"./pictures/clown_grow.jpg",
"./pictures/cowboy_fall.jpg",
"./pictures/dog_spin.jpg",
"./pictures/monkey_trip.jpg",
"./pictures/penguin_bounce.jpg",
"./pictures/pirate_sink.jpg",
"./pictures/rabbit_shake.jpg",
"./pictures/snail_melt.jpg",
],
"Sentence": [
"The octopus is boiling.",
"The ballerina is shrinking.",
"The boy is yawning.",
"The chef is drowning.",
"The clown is growing.",
"The cowboy is falling.",
"The dog is spinning.",
"The monkey is tripping.",
"The penguin is bouncing.",
"The pirate is sinking.",
"The rabbit is shaking.",
"The snail is melting.",
]
})Computing Similarity Scores
Now for the main event. For each image-sentence pair, we’ll compute CLIP’s similarity score. This tells us how well the model thinks the image matches the text.
def compute_clip_similarity(df, model, preprocess, device):
"""
Compute CLIP similarity scores for image-text pairs.
Parameters:
-----------
df : pandas.DataFrame
DataFrame with 'Filename' and 'Sentence' columns
model : CLIP model
Loaded CLIP model
preprocess : function
CLIP preprocessing function
device : str
'cuda' or 'cpu'
Returns:
--------
pandas.DataFrame
Original dataframe with added 'CLIP_Similarity' column
"""
similarity_scores = []
for _, row in df.iterrows():
img_path = row['Filename']
text = row['Sentence']
# Preprocess image and tokenize text
img = preprocess(Image.open(img_path)).unsqueeze(0).to(device)
text_tokenized = clip.tokenize([text]).to(device)
# Compute similarity
with torch.no_grad():
logits_per_image, _ = model(img, text_tokenized)
similarity_score = logits_per_image.item()
similarity_scores.append(similarity_score)
# Add scores to dataframe
df_copy = df.copy()
df_copy['CLIP_Similarity'] = similarity_scores
return df_copy
def compute_subject_salience(df, model, preprocess, device):
"""
Compute CLIP similarity scores for subject noun alone.
This measures how visually salient/easy to identify the subject is.
Parameters:
-----------
df : pandas.DataFrame
DataFrame with 'Filename' and 'Sentence' columns
model : CLIP model
Loaded CLIP model
preprocess : function
CLIP preprocessing function
device : str
'cuda' or 'cpu'
Returns:
--------
pandas.DataFrame
Original dataframe with added 'Subject_Salience' column
"""
subject_scores = []
for _, row in df.iterrows():
img_path = row['Filename']
sentence = row['Sentence']
# Extract subject noun (assumes format "The X is ...")
# Extract word after "The " and before " is"
subject = sentence.split("The ")[1].split(" is")[0]
# Preprocess image and tokenize subject
img = preprocess(Image.open(img_path)).unsqueeze(0).to(device)
text_tokenized = clip.tokenize([subject]).to(device)
# Compute similarity
with torch.no_grad():
logits_per_image, _ = model(img, text_tokenized)
similarity_score = logits_per_image.item()
subject_scores.append(similarity_score)
df_copy = df.copy()
df_copy['Subject_Salience'] = subject_scores
return df_copyWe can also use a multimodal LLM to verify the image-sentence match in a different way. Instead of computing similarity scores, we’ll ask the model to rate how well the sentence describes the image:
def compute_qwen_scores(df, model, tokenizer, streamer=None):
"""
Compute verification scores using Qwen-VL-Chat multimodal LLM.
Parameters:
-----------
df : pandas.DataFrame
DataFrame with 'Filename' and 'Sentence' columns
model : Qwen-VL-Chat model
Loaded Qwen model
tokenizer : AutoTokenizer
Qwen tokenizer
streamer : TextStreamer, optional
Streamer for real-time output
Returns:
--------
pandas.DataFrame
Original dataframe with added 'VLM_Score' and 'VLM_Response' columns
"""
import re
scores = []
responses = []
for idx, row in df.iterrows():
img_path = row['Filename']
sentence = row['Sentence']
# Create query for Qwen-VL-Chat
query = tokenizer.from_list_format([
{'image': img_path},
{'text': f'Rate how well this sentence describes the image: "{sentence}"\nScore from 1-10 (1=mismatch, 10=perfect match). Reply with just the number.'},
])
# Generate response
with torch.no_grad():
response, _ = model.chat(tokenizer, query=query, history=None, streamer=streamer)
# Extract numeric score
try:
match = re.search(r'(\d+(?:\.\d+)?)', response)
score = float(match.group(1)) if match else 5.0
score = min(10.0, max(1.0, score)) # Clamp to 1-10
except:
score = 5.0
scores.append(score)
responses.append(response)
df_copy = df.copy()
df_copy['VLM_Score'] = scores
df_copy['VLM_Response'] = responses
return df_copyLet’s run this on both datasets. To avoid re-computing the slow VLM scores on every render, we cache results to a CSV file:
import os
CACHE_FILE = "./cached_scores.csv"
if os.path.exists(CACHE_FILE):
df_all = pd.read_csv(CACHE_FILE)
else:
# Compute CLIP similarities
df_unerg_clip = compute_clip_similarity(df_unerg, model_clip, preprocess, device)
df_unacc_clip = compute_clip_similarity(df_unacc, model_clip, preprocess, device)
# Compute subject salience scores
df_unerg_subj = compute_subject_salience(df_unerg, model_clip, preprocess, device)
df_unacc_subj = compute_subject_salience(df_unacc, model_clip, preprocess, device)
# Compute Qwen-VL scores
df_unerg_vlm = compute_qwen_scores(df_unerg, model_vlm, tokenizer_vlm, streamer=streamer)
df_unacc_vlm = compute_qwen_scores(df_unacc, model_vlm, tokenizer_vlm, streamer=streamer)
# Combine CLIP scores with VLM scores and subject salience
df_unerg_scored = df_unerg_clip.copy()
df_unerg_scored['Subject_Salience'] = df_unerg_subj['Subject_Salience']
df_unerg_scored['VLM_Score'] = df_unerg_vlm['VLM_Score']
df_unerg_scored['VLM_Response'] = df_unerg_vlm['VLM_Response']
df_unerg_scored['VerbType'] = 'Unergative'
df_unacc_scored = df_unacc_clip.copy()
df_unacc_scored['Subject_Salience'] = df_unacc_subj['Subject_Salience']
df_unacc_scored['VLM_Score'] = df_unacc_vlm['VLM_Score']
df_unacc_scored['VLM_Response'] = df_unacc_vlm['VLM_Response']
df_unacc_scored['VerbType'] = 'Unaccusative'
# Combine for analysis
df_all = pd.concat([df_unerg_scored, df_unacc_scored], ignore_index=True)
# Save to cache
df_all.to_csv(CACHE_FILE, index=False)
print(df_all.head()) Filename Sentence CLIP_Similarity \
0 ./octopus_swim.jpg The octopus is swimming. 29.137495
1 ./ballerina_run.jpg The ballerina is running. 27.731918
2 ./boy_float.jpg The boy is floating. 20.843243
3 ./chef_yell.jpg The chef is yelling. 27.878561
4 ./clown_walk.jpg The clown is walking. 27.077477
Subject_Salience VLM_Score VLM_Response VerbType
0 28.454519 8.0 8 Unergative
1 25.250607 7.0 7 Unergative
2 21.628622 1.0 1 Unergative
3 28.490120 8.0 8 Unergative
4 26.241133 8.0 8 Unergative Descriptive Results
Let’s start by looking at the descriptive statistics across all three metrics:
# Create comparison plot with all three metrics
fig, axes = plt.subplots(1, 3, figsize=(8, 5))
# CLIP full sentence results
sns.pointplot(data=df_all, x='VerbType', y='CLIP_Similarity',
hue='VerbType', palette=['#3498db', '#e74c3c'],
ax=axes[0], errorbar='ci', capsize=0.1,
linestyle='none', markers='o', legend=False)
sns.stripplot(data=df_all, x='VerbType', y='CLIP_Similarity',
color='black', alpha=0.5, size=8, ax=axes[0], jitter=0.2)
axes[0].set_xlabel('Verb Type', fontsize=14, fontweight='bold')
axes[0].set_ylabel('CLIP Similarity Score', fontsize=14, fontweight='bold')
axes[0].set_title('Full Sentence Similarity',
fontsize=16, fontweight='bold', pad=20)
for verb_type in ['Unergative', 'Unaccusative']:
mean_val = df_all[df_all['VerbType'] == verb_type]['CLIP_Similarity'].mean()
axes[0].text(0 if verb_type == 'Unergative' else 1, mean_val + 1,
f'M = {mean_val:.2f}', ha='center', fontsize=12, fontweight='bold')
# Subject salience results
sns.pointplot(data=df_all, x='VerbType', y='Subject_Salience',
hue='VerbType', palette=['#3498db', '#e74c3c'],
ax=axes[1], errorbar='ci', capsize=0.1,
linestyle='none', markers='o', legend=False)
sns.stripplot(data=df_all, x='VerbType', y='Subject_Salience',
color='black', alpha=0.5, size=8, ax=axes[1], jitter=0.2)
axes[1].set_xlabel('Verb Type', fontsize=14, fontweight='bold')
axes[1].set_ylabel('Subject Salience Score', fontsize=14, fontweight='bold')
axes[1].set_title('Subject Noun Identifiability',
fontsize=16, fontweight='bold', pad=20)
for verb_type in ['Unergative', 'Unaccusative']:
mean_val = df_all[df_all['VerbType'] == verb_type]['Subject_Salience'].mean()
axes[1].text(0 if verb_type == 'Unergative' else 1, mean_val + 0.5,
f'M = {mean_val:.2f}', ha='center', fontsize=12, fontweight='bold')
# VLM results
sns.pointplot(data=df_all, x='VerbType', y='VLM_Score',
hue='VerbType', palette=['#3498db', '#e74c3c'],
ax=axes[2], errorbar='ci', capsize=0.1,
linestyle='none', markers='o', legend=False)
sns.stripplot(data=df_all, x='VerbType', y='VLM_Score',
color='black', alpha=0.5, size=8, ax=axes[2], jitter=0.2)
axes[2].set_xlabel('Verb Type', fontsize=14, fontweight='bold')
axes[2].set_ylabel('Qwen-VL Match Score (1-10)', fontsize=14, fontweight='bold')
axes[2].set_title('Scene Verification (Qwen-VL)',
fontsize=16, fontweight='bold', pad=20)
for verb_type in ['Unergative', 'Unaccusative']:
mean_val = df_all[df_all['VerbType'] == verb_type]['VLM_Score'].mean()
axes[2].text(0 if verb_type == 'Unergative' else 1, mean_val + 0.3,
f'M = {mean_val:.2f}', ha='center', fontsize=12, fontweight='bold')
plt.tight_layout()
plt.savefig('./model_comparison_plot.png', dpi=300, bbox_inches='tight')
plt.show()
A Deeper Dive with Bayesian Analysis
While the plots above give us a good first look, they don’t tell the whole story. To really understand the strength of the evidence, we need to go beyond just comparing averages. This is where Bayesian analysis comes in.
Instead of just getting a single number for the difference, a Bayesian regression gives us a full range of plausible values for the effect of VerbType on our scores, along with a measure of our certainty.
For the nerds out there, I used Pyro to run three separate models: two simple linear regressions for the CLIP and Subject Salience scores, and an ordered logistic regression for the VLM scores (since they are on a 1-10 scale). In all models, the key parameter is beta, which represents the estimated difference between unaccusative and unergative verbs.
Here’s the code to set up and run the models:
import torch
import pyro
import pyro.distributions as dist
from pyro.infer import MCMC, NUTS
# Prepare data for Pyro
# We'll center the scores and code VerbType numerically
df_pyro = df_all.copy()
df_pyro['VerbType_num'] = df_pyro['VerbType'].map({'Unergative': -0.5, 'Unaccusative': 0.5})
df_pyro['CLIP_centered'] = df_pyro['CLIP_Similarity'] - df_pyro['CLIP_Similarity'].mean()
df_pyro['Subject_centered'] = df_pyro['Subject_Salience'] - df_pyro['Subject_Salience'].mean()
vlm_score_tensor = torch.tensor(df_pyro['VLM_Score'].values, dtype=torch.long)
# Convert to tensors
verb_type_tensor = torch.tensor(df_pyro['VerbType_num'].values, dtype=torch.float32)
clip_tensor = torch.tensor(df_pyro['CLIP_centered'].values, dtype=torch.float32)
subject_tensor = torch.tensor(df_pyro['Subject_centered'].values, dtype=torch.float32)
# --- Model for CLIP Similarity ---
def clip_model(verb_type, obs=None):
intercept = pyro.sample('intercept', dist.Normal(0., 10.))
beta = pyro.sample('beta', dist.Normal(0., 10.))
sigma = pyro.sample('sigma', dist.HalfNormal(10.))
mu = intercept + beta * verb_type
with pyro.plate('data', len(verb_type)):
pyro.sample('obs', dist.Normal(mu, sigma), obs=obs)
# --- Model for Subject Salience ---
def subject_model(verb_type, obs=None):
intercept = pyro.sample('intercept', dist.Normal(0., 10.))
beta = pyro.sample('beta', dist.Normal(0., 10.))
sigma = pyro.sample('sigma', dist.HalfNormal(10.))
mu = intercept + beta * verb_type
with pyro.plate('data', len(verb_type)):
pyro.sample('obs', dist.Normal(mu, sigma), obs=obs)
# --- Model for VLM Score (Ordered Logistic) ---
k_categories = vlm_score_tensor.max().item() + 1
k_cutpoints = k_categories - 1
def vlm_model(verb_type, obs=None):
alpha = pyro.sample('alpha', dist.Normal(0., 10.))
beta = pyro.sample('beta', dist.Normal(0., 10.))
with pyro.plate("cutpoints_plate", k_cutpoints):
raw_cutpoints = pyro.sample('raw_cutpoints', dist.Normal(torch.arange(k_cutpoints).float(), 1.))
cutpoints = torch.sort(raw_cutpoints)[0]
latent_propensity = alpha + beta * verb_type
with pyro.plate('data', len(verb_type)):
pyro.sample('obs', dist.OrderedLogistic(latent_propensity, cutpoints), obs=obs)
# Run the MCMC samplers
mcmc_clip = MCMC(NUTS(clip_model), num_samples=2000, warmup_steps=1000)
mcmc_clip.run(verb_type_tensor, clip_tensor)
clip_samples = mcmc_clip.get_samples()
mcmc_subject = MCMC(NUTS(subject_model), num_samples=2000, warmup_steps=1000)
mcmc_subject.run(verb_type_tensor, subject_tensor)
subject_samples = mcmc_subject.get_samples()
mcmc_vlm = MCMC(NUTS(vlm_model), num_samples=2000, warmup_steps=1000, num_chains=1)
mcmc_vlm.run(verb_type_tensor, vlm_score_tensor)
vlm_samples = mcmc_vlm.get_samples()Warmup: 0%| | 0/3000 [00:00, ?it/s]Warmup: 0%| | 1/3000 [00:00, 8.44it/s, step size=2.25e-01, acc. prob=1.000]Warmup: 0%| | 11/3000 [00:00, 55.64it/s, step size=2.88e-01, acc. prob=0.818]Warmup: 1%| | 17/3000 [00:00, 51.72it/s, step size=3.89e-01, acc. prob=0.818]Warmup: 1%| | 24/3000 [00:00, 56.64it/s, step size=1.87e-01, acc. prob=0.803]Warmup: 1%| | 31/3000 [00:00, 53.91it/s, step size=5.88e-02, acc. prob=0.788]Warmup: 1%| | 37/3000 [00:00, 54.37it/s, step size=7.46e-02, acc. prob=0.792]Warmup: 1%|▏ | 43/3000 [00:00, 49.39it/s, step size=1.19e-01, acc. prob=0.797]Warmup: 2%|▏ | 51/3000 [00:00, 56.92it/s, step size=2.91e-01, acc. prob=0.805]Warmup: 2%|▏ | 59/3000 [00:01, 61.01it/s, step size=1.61e-01, acc. prob=0.800]Warmup: 2%|▏ | 69/3000 [00:01, 69.87it/s, step size=1.52e-01, acc. prob=0.800]Warmup: 3%|▎ | 80/3000 [00:01, 75.01it/s, step size=8.08e-02, acc. prob=0.796]Warmup: 3%|▎ | 91/3000 [00:01, 81.66it/s, step size=2.05e-01, acc. prob=0.802]Warmup: 3%|▎ | 104/3000 [00:01, 94.02it/s, step size=9.92e-01, acc. prob=0.793]Warmup: 4%|▍ | 124/3000 [00:01, 123.35it/s, step size=3.00e-01, acc. prob=0.790]Warmup: 5%|▌ | 156/3000 [00:01, 178.72it/s, step size=8.12e-01, acc. prob=0.785]Warmup: 6%|▌ | 185/3000 [00:01, 209.58it/s, step size=3.78e-01, acc. prob=0.785]Warmup: 7%|▋ | 217/3000 [00:01, 240.58it/s, step size=1.21e+00, acc. prob=0.788]Warmup: 8%|▊ | 249/3000 [00:02, 263.04it/s, step size=2.95e+00, acc. prob=0.789]Warmup: 9%|▉ | 276/3000 [00:02, 224.43it/s, step size=5.73e-01, acc. prob=0.785]Warmup: 10%|█ | 302/3000 [00:02, 231.71it/s, step size=6.43e-01, acc. prob=0.786]Warmup: 11%|█ | 336/3000 [00:02, 259.74it/s, step size=9.18e-01, acc. prob=0.787]Warmup: 12%|█▏ | 372/3000 [00:02, 286.06it/s, step size=6.24e-01, acc. prob=0.786]Warmup: 13%|█▎ | 402/3000 [00:02, 283.75it/s, step size=4.88e-01, acc. prob=0.787]Warmup: 14%|█▍ | 431/3000 [00:02, 280.25it/s, step size=7.66e-01, acc. prob=0.788]Warmup: 15%|█▌ | 460/3000 [00:02, 278.32it/s, step size=6.40e-01, acc. prob=0.787]Warmup: 16%|█▋ | 489/3000 [00:02, 251.86it/s, step size=3.31e-01, acc. prob=0.787]Warmup: 17%|█▋ | 517/3000 [00:03, 257.48it/s, step size=1.02e+00, acc. prob=0.788]Warmup: 18%|█▊ | 552/3000 [00:03, 282.20it/s, step size=5.31e-01, acc. prob=0.788]Warmup: 19%|█▉ | 581/3000 [00:03, 280.45it/s, step size=1.01e+00, acc. prob=0.789]Warmup: 20%|██ | 610/3000 [00:03, 275.98it/s, step size=8.60e-01, acc. prob=0.789]Warmup: 21%|██▏ | 638/3000 [00:03, 269.13it/s, step size=1.14e+00, acc. prob=0.790]Warmup: 22%|██▏ | 670/3000 [00:03, 280.28it/s, step size=1.10e+00, acc. prob=0.790]Warmup: 24%|██▎ | 709/3000 [00:03, 310.67it/s, step size=1.12e+00, acc. prob=0.791]Warmup: 25%|██▍ | 742/3000 [00:03, 314.70it/s, step size=1.46e+00, acc. prob=0.791]Warmup: 26%|██▌ | 777/3000 [00:03, 324.95it/s, step size=9.77e-01, acc. prob=0.791]Warmup: 27%|██▋ | 813/3000 [00:04, 333.64it/s, step size=1.39e+00, acc. prob=0.792]Warmup: 28%|██▊ | 847/3000 [00:04, 335.24it/s, step size=1.21e+00, acc. prob=0.792]Warmup: 29%|██▉ | 883/3000 [00:04, 341.83it/s, step size=8.37e-01, acc. prob=0.792]Warmup: 31%|███ | 918/3000 [00:04, 340.31it/s, step size=1.10e+00, acc. prob=0.792]Warmup: 32%|███▏ | 955/3000 [00:04, 347.64it/s, step size=2.57e+00, acc. prob=0.792]Warmup: 33%|███▎ | 990/3000 [00:04, 323.04it/s, step size=1.54e+00, acc. prob=0.792]Sample: 34%|███▍ | 1028/3000 [00:04, 337.27it/s, step size=8.07e-01, acc. prob=0.865]Sample: 35%|███▌ | 1063/3000 [00:04, 336.06it/s, step size=8.07e-01, acc. prob=0.873]Sample: 37%|███▋ | 1101/3000 [00:04, 348.63it/s, step size=8.07e-01, acc. prob=0.872]Sample: 38%|███▊ | 1137/3000 [00:04, 342.06it/s, step size=8.07e-01, acc. prob=0.876]Sample: 39%|███▉ | 1172/3000 [00:05, 330.85it/s, step size=8.07e-01, acc. prob=0.883]Sample: 40%|████ | 1212/3000 [00:05, 350.27it/s, step size=8.07e-01, acc. prob=0.883]Sample: 42%|████▏ | 1248/3000 [00:05, 346.21it/s, step size=8.07e-01, acc. prob=0.884]Sample: 43%|████▎ | 1287/3000 [00:05, 358.38it/s, step size=8.07e-01, acc. prob=0.883]Sample: 44%|████▍ | 1325/3000 [00:05, 362.72it/s, step size=8.07e-01, acc. prob=0.886]Sample: 45%|████▌ | 1362/3000 [00:05, 337.96it/s, step size=8.07e-01, acc. prob=0.887]Sample: 47%|████▋ | 1401/3000 [00:05, 329.18it/s, step size=8.07e-01, acc. prob=0.889]Sample: 48%|████▊ | 1436/3000 [00:05, 330.63it/s, step size=8.07e-01, acc. prob=0.887]Sample: 49%|████▉ | 1474/3000 [00:05, 341.78it/s, step size=8.07e-01, acc. prob=0.886]Sample: 50%|█████ | 1509/3000 [00:06, 334.17it/s, step size=8.07e-01, acc. prob=0.887]Sample: 51%|█████▏ | 1544/3000 [00:06, 337.08it/s, step size=8.07e-01, acc. prob=0.888]Sample: 53%|█████▎ | 1586/3000 [00:06, 359.25it/s, step size=8.07e-01, acc. prob=0.889]Sample: 54%|█████▍ | 1623/3000 [00:06, 355.25it/s, step size=8.07e-01, acc. prob=0.886]Sample: 56%|█████▌ | 1667/3000 [00:06, 379.48it/s, step size=8.07e-01, acc. prob=0.887]Sample: 57%|█████▋ | 1706/3000 [00:06, 354.69it/s, step size=8.07e-01, acc. prob=0.887]Sample: 58%|█████▊ | 1742/3000 [00:06, 346.44it/s, step size=8.07e-01, acc. prob=0.888]Sample: 60%|█████▉ | 1786/3000 [00:06, 369.37it/s, step size=8.07e-01, acc. prob=0.888]Sample: 61%|██████ | 1824/3000 [00:06, 359.23it/s, step size=8.07e-01, acc. prob=0.887]Sample: 62%|██████▏ | 1861/3000 [00:07, 353.59it/s, step size=8.07e-01, acc. prob=0.886]Sample: 63%|██████▎ | 1900/3000 [00:07, 360.82it/s, step size=8.07e-01, acc. prob=0.887]Sample: 65%|██████▍ | 1937/3000 [00:07, 347.54it/s, step size=8.07e-01, acc. prob=0.886]Sample: 66%|██████▌ | 1977/3000 [00:07, 361.22it/s, step size=8.07e-01, acc. prob=0.887]Sample: 67%|██████▋ | 2015/3000 [00:07, 364.92it/s, step size=8.07e-01, acc. prob=0.888]Sample: 68%|██████▊ | 2052/3000 [00:07, 342.00it/s, step size=8.07e-01, acc. prob=0.887]Sample: 70%|██████▉ | 2091/3000 [00:07, 353.30it/s, step size=8.07e-01, acc. prob=0.887]Sample: 71%|███████ | 2127/3000 [00:07, 345.82it/s, step size=8.07e-01, acc. prob=0.886]Sample: 72%|███████▏ | 2162/3000 [00:07, 329.45it/s, step size=8.07e-01, acc. prob=0.885]Sample: 73%|███████▎ | 2199/3000 [00:07, 337.28it/s, step size=8.07e-01, acc. prob=0.886]Sample: 75%|███████▍ | 2241/3000 [00:08, 357.73it/s, step size=8.07e-01, acc. prob=0.886]Sample: 76%|███████▋ | 2289/3000 [00:08, 388.36it/s, step size=8.07e-01, acc. prob=0.886]Sample: 78%|███████▊ | 2331/3000 [00:08, 394.22it/s, step size=8.07e-01, acc. prob=0.886]Sample: 79%|███████▉ | 2373/3000 [00:08, 400.25it/s, step size=8.07e-01, acc. prob=0.887]Sample: 80%|████████ | 2414/3000 [00:08, 392.26it/s, step size=8.07e-01, acc. prob=0.887]Sample: 82%|████████▏ | 2457/3000 [00:08, 402.07it/s, step size=8.07e-01, acc. prob=0.887]Sample: 83%|████████▎ | 2498/3000 [00:08, 396.26it/s, step size=8.07e-01, acc. prob=0.886]Sample: 85%|████████▍ | 2538/3000 [00:08, 378.26it/s, step size=8.07e-01, acc. prob=0.886]Sample: 86%|████████▌ | 2577/3000 [00:08, 364.23it/s, step size=8.07e-01, acc. prob=0.885]Sample: 87%|████████▋ | 2614/3000 [00:09, 361.63it/s, step size=8.07e-01, acc. prob=0.886]Sample: 88%|████████▊ | 2651/3000 [00:09, 355.81it/s, step size=8.07e-01, acc. prob=0.886]Sample: 90%|████████▉ | 2690/3000 [00:09, 365.41it/s, step size=8.07e-01, acc. prob=0.886]Sample: 91%|█████████ | 2733/3000 [00:09, 382.46it/s, step size=8.07e-01, acc. prob=0.885]Sample: 92%|█████████▏| 2772/3000 [00:09, 353.26it/s, step size=8.07e-01, acc. prob=0.886]Sample: 94%|█████████▎| 2808/3000 [00:09, 349.93it/s, step size=8.07e-01, acc. prob=0.886]Sample: 95%|█████████▍| 2844/3000 [00:09, 350.28it/s, step size=8.07e-01, acc. prob=0.886]Sample: 96%|█████████▌| 2881/3000 [00:09, 355.64it/s, step size=8.07e-01, acc. prob=0.886]Sample: 97%|█████████▋| 2918/3000 [00:09, 359.34it/s, step size=8.07e-01, acc. prob=0.886]Sample: 98%|█████████▊| 2955/3000 [00:10, 361.91it/s, step size=8.07e-01, acc. prob=0.886]Sample: 100%|█████████▉| 2992/3000 [00:10, 360.12it/s, step size=8.07e-01, acc. prob=0.886]Sample: 100%|██████████| 3000/3000 [00:10, 295.60it/s, step size=8.07e-01, acc. prob=0.886]
Warmup: 0%| | 0/3000 [00:00, ?it/s]Warmup: 0%| | 10/3000 [00:00, 53.76it/s, step size=5.43e-02, acc. prob=0.723]Warmup: 1%| | 18/3000 [00:00, 55.30it/s, step size=5.46e-02, acc. prob=0.755]Warmup: 1%| | 24/3000 [00:00, 51.78it/s, step size=9.21e-02, acc. prob=0.772]Warmup: 1%| | 30/3000 [00:00, 43.47it/s, step size=1.44e-01, acc. prob=0.782]Warmup: 1%|▏ | 38/3000 [00:00, 50.95it/s, step size=2.58e-01, acc. prob=0.791]Warmup: 2%|▏ | 50/3000 [00:00, 68.54it/s, step size=2.92e-01, acc. prob=0.794]Warmup: 2%|▏ | 58/3000 [00:00, 69.07it/s, step size=1.63e-01, acc. prob=0.790]Warmup: 2%|▏ | 69/3000 [00:01, 73.48it/s, step size=1.29e-01, acc. prob=0.789]Warmup: 3%|▎ | 77/3000 [00:01, 73.90it/s, step size=1.23e-01, acc. prob=0.790]Warmup: 3%|▎ | 92/3000 [00:01, 94.29it/s, step size=1.64e-01, acc. prob=0.792]Warmup: 3%|▎ | 104/3000 [00:01, 100.38it/s, step size=8.34e-01, acc. prob=0.788]Warmup: 4%|▍ | 132/3000 [00:01, 149.04it/s, step size=5.31e-01, acc. prob=0.788]Warmup: 5%|▌ | 157/3000 [00:01, 177.10it/s, step size=5.46e-01, acc. prob=0.786]Warmup: 6%|▌ | 178/3000 [00:01, 184.90it/s, step size=1.07e+00, acc. prob=0.789]Warmup: 7%|▋ | 207/3000 [00:01, 214.12it/s, step size=1.28e+00, acc. prob=0.790]Warmup: 8%|▊ | 243/3000 [00:01, 254.33it/s, step size=3.37e-01, acc. prob=0.788]Warmup: 9%|▉ | 269/3000 [00:02, 244.60it/s, step size=2.10e+00, acc. prob=0.789]Warmup: 10%|▉ | 294/3000 [00:02, 232.63it/s, step size=7.05e-01, acc. prob=0.787]Warmup: 11%|█ | 320/3000 [00:02, 240.11it/s, step size=9.90e-01, acc. prob=0.788]Warmup: 12%|█▏ | 356/3000 [00:02, 273.74it/s, step size=1.10e+00, acc. prob=0.789]Warmup: 13%|█▎ | 398/3000 [00:02, 315.56it/s, step size=7.52e-01, acc. prob=0.789]Warmup: 15%|█▍ | 436/3000 [00:02, 334.05it/s, step size=1.17e+00, acc. prob=0.790]Warmup: 16%|█▌ | 470/3000 [00:02, 319.05it/s, step size=1.41e-01, acc. prob=0.787]Warmup: 17%|█▋ | 503/3000 [00:02, 285.69it/s, step size=7.70e-01, acc. prob=0.788]Warmup: 18%|█▊ | 533/3000 [00:02, 269.75it/s, step size=1.28e+00, acc. prob=0.789]Warmup: 19%|█▉ | 569/3000 [00:03, 292.95it/s, step size=9.14e-01, acc. prob=0.789]Warmup: 20%|██ | 600/3000 [00:03, 295.56it/s, step size=4.97e-01, acc. prob=0.788]Warmup: 21%|██ | 635/3000 [00:03, 307.39it/s, step size=5.31e-01, acc. prob=0.789]Warmup: 22%|██▏ | 668/3000 [00:03, 311.64it/s, step size=9.64e-01, acc. prob=0.790]Warmup: 24%|██▍ | 713/3000 [00:03, 349.96it/s, step size=9.07e-01, acc. prob=0.790]Warmup: 25%|██▍ | 749/3000 [00:03, 348.70it/s, step size=1.00e+00, acc. prob=0.790]Warmup: 26%|██▌ | 786/3000 [00:03, 353.90it/s, step size=5.23e-01, acc. prob=0.790]Warmup: 28%|██▊ | 826/3000 [00:03, 366.62it/s, step size=6.15e-01, acc. prob=0.790]Warmup: 29%|██▉ | 863/3000 [00:03, 352.96it/s, step size=9.52e-01, acc. prob=0.791]Warmup: 30%|███ | 901/3000 [00:03, 360.52it/s, step size=1.02e+00, acc. prob=0.791]Warmup: 31%|███▏ | 944/3000 [00:04, 377.67it/s, step size=7.42e-01, acc. prob=0.791]Warmup: 33%|███▎ | 982/3000 [00:04, 328.84it/s, step size=2.11e+00, acc. prob=0.792]Sample: 34%|███▍ | 1019/3000 [00:04, 337.49it/s, step size=7.32e-01, acc. prob=0.937]Sample: 35%|███▌ | 1054/3000 [00:04, 325.39it/s, step size=7.32e-01, acc. prob=0.922]Sample: 36%|███▋ | 1090/3000 [00:04, 334.24it/s, step size=7.32e-01, acc. prob=0.916]Sample: 38%|███▊ | 1125/3000 [00:04, 338.02it/s, step size=7.32e-01, acc. prob=0.922]Sample: 39%|███▊ | 1160/3000 [00:04, 332.30it/s, step size=7.32e-01, acc. prob=0.922]Sample: 40%|████ | 1200/3000 [00:04, 349.79it/s, step size=7.32e-01, acc. prob=0.918]Sample: 41%|████ | 1236/3000 [00:04, 337.04it/s, step size=7.32e-01, acc. prob=0.915]Sample: 42%|████▏ | 1274/3000 [00:05, 345.39it/s, step size=7.32e-01, acc. prob=0.916]Sample: 44%|████▎ | 1309/3000 [00:05, 343.00it/s, step size=7.32e-01, acc. prob=0.913]Sample: 45%|████▍ | 1344/3000 [00:05, 340.01it/s, step size=7.32e-01, acc. prob=0.910]Sample: 46%|████▌ | 1379/3000 [00:05, 335.72it/s, step size=7.32e-01, acc. prob=0.912]Sample: 47%|████▋ | 1413/3000 [00:05, 336.91it/s, step size=7.32e-01, acc. prob=0.912]Sample: 48%|████▊ | 1448/3000 [00:05, 336.44it/s, step size=7.32e-01, acc. prob=0.912]Sample: 49%|████▉ | 1483/3000 [00:05, 339.66it/s, step size=7.32e-01, acc. prob=0.912]Sample: 51%|█████ | 1520/3000 [00:05, 347.28it/s, step size=7.32e-01, acc. prob=0.913]Sample: 52%|█████▏ | 1555/3000 [00:05, 347.35it/s, step size=7.32e-01, acc. prob=0.912]Sample: 53%|█████▎ | 1590/3000 [00:06, 318.58it/s, step size=7.32e-01, acc. prob=0.912]Sample: 54%|█████▍ | 1624/3000 [00:06, 323.46it/s, step size=7.32e-01, acc. prob=0.912]Sample: 55%|█████▌ | 1658/3000 [00:06, 326.31it/s, step size=7.32e-01, acc. prob=0.912]Sample: 56%|█████▋ | 1692/3000 [00:06, 326.64it/s, step size=7.32e-01, acc. prob=0.912]Sample: 57%|█████▊ | 1725/3000 [00:06, 325.83it/s, step size=7.32e-01, acc. prob=0.912]Sample: 59%|█████▊ | 1759/3000 [00:06, 328.12it/s, step size=7.32e-01, acc. prob=0.911]Sample: 60%|█████▉ | 1792/3000 [00:06, 326.01it/s, step size=7.32e-01, acc. prob=0.909]Sample: 61%|██████ | 1830/3000 [00:06, 340.82it/s, step size=7.32e-01, acc. prob=0.909]Sample: 62%|██████▏ | 1866/3000 [00:06, 346.01it/s, step size=7.32e-01, acc. prob=0.909]Sample: 63%|██████▎ | 1902/3000 [00:06, 348.91it/s, step size=7.32e-01, acc. prob=0.910]Sample: 65%|██████▍ | 1937/3000 [00:07, 344.33it/s, step size=7.32e-01, acc. prob=0.911]Sample: 66%|██████▌ | 1974/3000 [00:07, 348.19it/s, step size=7.32e-01, acc. prob=0.912]Sample: 67%|██████▋ | 2010/3000 [00:07, 349.29it/s, step size=7.32e-01, acc. prob=0.912]Sample: 68%|██████▊ | 2045/3000 [00:07, 347.36it/s, step size=7.32e-01, acc. prob=0.912]Sample: 69%|██████▉ | 2082/3000 [00:07, 353.90it/s, step size=7.32e-01, acc. prob=0.911]Sample: 71%|███████ | 2118/3000 [00:07, 345.38it/s, step size=7.32e-01, acc. prob=0.911]Sample: 72%|███████▏ | 2153/3000 [00:07, 332.83it/s, step size=7.32e-01, acc. prob=0.911]Sample: 73%|███████▎ | 2187/3000 [00:07, 324.33it/s, step size=7.32e-01, acc. prob=0.911]Sample: 74%|███████▍ | 2224/3000 [00:07, 335.54it/s, step size=7.32e-01, acc. prob=0.912]Sample: 75%|███████▌ | 2258/3000 [00:08, 301.98it/s, step size=7.32e-01, acc. prob=0.911]Sample: 76%|███████▋ | 2289/3000 [00:08, 275.96it/s, step size=7.32e-01, acc. prob=0.911]Sample: 77%|███████▋ | 2318/3000 [00:08, 278.48it/s, step size=7.32e-01, acc. prob=0.911]Sample: 78%|███████▊ | 2347/3000 [00:08, 273.69it/s, step size=7.32e-01, acc. prob=0.910]Sample: 79%|███████▉ | 2377/3000 [00:08, 278.92it/s, step size=7.32e-01, acc. prob=0.910]Sample: 80%|████████ | 2406/3000 [00:08, 281.25it/s, step size=7.32e-01, acc. prob=0.910]Sample: 81%|████████ | 2435/3000 [00:08, 275.91it/s, step size=7.32e-01, acc. prob=0.910]Sample: 82%|████████▏ | 2463/3000 [00:08, 264.78it/s, step size=7.32e-01, acc. prob=0.910]Sample: 83%|████████▎ | 2490/3000 [00:08, 233.30it/s, step size=7.32e-01, acc. prob=0.910]Sample: 84%|████████▍ | 2515/3000 [00:09, 220.45it/s, step size=7.32e-01, acc. prob=0.910]Sample: 85%|████████▍ | 2538/3000 [00:09, 219.29it/s, step size=7.32e-01, acc. prob=0.909]Sample: 86%|████████▌ | 2566/3000 [00:09, 234.54it/s, step size=7.32e-01, acc. prob=0.909]Sample: 87%|████████▋ | 2598/3000 [00:09, 255.28it/s, step size=7.32e-01, acc. prob=0.909]Sample: 88%|████████▊ | 2634/3000 [00:09, 281.02it/s, step size=7.32e-01, acc. prob=0.910]Sample: 89%|████████▉ | 2664/3000 [00:09, 284.47it/s, step size=7.32e-01, acc. prob=0.910]Sample: 90%|████████▉ | 2695/3000 [00:09, 290.85it/s, step size=7.32e-01, acc. prob=0.910]Sample: 91%|█████████ | 2730/3000 [00:09, 301.55it/s, step size=7.32e-01, acc. prob=0.910]Sample: 92%|█████████▏| 2762/3000 [00:09, 303.54it/s, step size=7.32e-01, acc. prob=0.910]Sample: 93%|█████████▎| 2796/3000 [00:10, 310.93it/s, step size=7.32e-01, acc. prob=0.909]Sample: 94%|█████████▍| 2828/3000 [00:10, 305.29it/s, step size=7.32e-01, acc. prob=0.910]Sample: 95%|█████████▌| 2860/3000 [00:10, 308.89it/s, step size=7.32e-01, acc. prob=0.910]Sample: 96%|█████████▋| 2891/3000 [00:10, 305.62it/s, step size=7.32e-01, acc. prob=0.909]Sample: 98%|█████████▊| 2928/3000 [00:10, 323.05it/s, step size=7.32e-01, acc. prob=0.909]Sample: 99%|█████████▊| 2961/3000 [00:10, 314.98it/s, step size=7.32e-01, acc. prob=0.909]Sample: 100%|█████████▉| 2996/3000 [00:10, 325.05it/s, step size=7.32e-01, acc. prob=0.909]Sample: 100%|██████████| 3000/3000 [00:10, 280.55it/s, step size=7.32e-01, acc. prob=0.909]
Warmup: 0%| | 0/3000 [00:00, ?it/s]Warmup: 0%| | 8/3000 [00:00, 70.51it/s, step size=6.28e-02, acc. prob=0.709]Warmup: 1%| | 16/3000 [00:00, 51.32it/s, step size=5.19e-02, acc. prob=0.749]Warmup: 1%| | 22/3000 [00:00, 45.51it/s, step size=2.88e-02, acc. prob=0.752]Warmup: 1%| | 27/3000 [00:00, 31.97it/s, step size=6.72e-02, acc. prob=0.771]Warmup: 1%| | 32/3000 [00:00, 34.60it/s, step size=6.86e-02, acc. prob=0.774]Warmup: 1%| | 36/3000 [00:00, 34.50it/s, step size=6.47e-02, acc. prob=0.776]Warmup: 1%|▏ | 42/3000 [00:01, 39.48it/s, step size=1.15e-01, acc. prob=0.784]Warmup: 2%|▏ | 49/3000 [00:01, 46.25it/s, step size=1.03e-01, acc. prob=0.784]Warmup: 2%|▏ | 55/3000 [00:01, 48.91it/s, step size=7.70e-02, acc. prob=0.783]Warmup: 2%|▏ | 61/3000 [00:01, 50.31it/s, step size=1.12e-01, acc. prob=0.787]Warmup: 2%|▏ | 67/3000 [00:01, 50.41it/s, step size=5.87e-02, acc. prob=0.783]Warmup: 2%|▏ | 73/3000 [00:01, 44.95it/s, step size=1.17e-01, acc. prob=0.789]Warmup: 3%|▎ | 78/3000 [00:01, 38.52it/s, step size=6.81e-02, acc. prob=0.786]Warmup: 3%|▎ | 83/3000 [00:02, 34.01it/s, step size=5.18e-02, acc. prob=0.785]Warmup: 3%|▎ | 87/3000 [00:02, 30.29it/s, step size=5.40e-02, acc. prob=0.785]Warmup: 3%|▎ | 91/3000 [00:02, 29.91it/s, step size=1.21e-01, acc. prob=0.790]Warmup: 3%|▎ | 97/3000 [00:02, 35.28it/s, step size=1.38e-01, acc. prob=0.792]Warmup: 3%|▎ | 102/3000 [00:02, 27.49it/s, step size=5.10e-02, acc. prob=0.769]Warmup: 4%|▎ | 106/3000 [00:02, 23.99it/s, step size=7.94e-02, acc. prob=0.774]Warmup: 4%|▎ | 109/3000 [00:03, 23.35it/s, step size=1.97e-01, acc. prob=0.778]Warmup: 4%|▍ | 113/3000 [00:03, 25.02it/s, step size=1.37e-01, acc. prob=0.777]Warmup: 4%|▍ | 116/3000 [00:03, 25.63it/s, step size=7.91e-01, acc. prob=0.783]Warmup: 4%|▍ | 119/3000 [00:03, 26.18it/s, step size=1.58e-01, acc. prob=0.778]Warmup: 4%|▍ | 124/3000 [00:03, 31.68it/s, step size=5.54e-01, acc. prob=0.782]Warmup: 4%|▍ | 130/3000 [00:03, 38.76it/s, step size=3.95e-01, acc. prob=0.782]Warmup: 5%|▍ | 136/3000 [00:03, 43.41it/s, step size=1.56e-01, acc. prob=0.779]Warmup: 5%|▍ | 141/3000 [00:03, 34.79it/s, step size=1.57e-01, acc. prob=0.780]Warmup: 5%|▍ | 149/3000 [00:04, 43.65it/s, step size=7.69e-02, acc. prob=0.782]Warmup: 5%|▌ | 156/3000 [00:04, 47.97it/s, step size=7.96e-02, acc. prob=0.778]Warmup: 5%|▌ | 162/3000 [00:04, 41.28it/s, step size=6.78e-02, acc. prob=0.779]Warmup: 6%|▌ | 167/3000 [00:04, 41.01it/s, step size=2.18e-01, acc. prob=0.781]Warmup: 6%|▌ | 172/3000 [00:04, 42.35it/s, step size=1.61e-01, acc. prob=0.781]Warmup: 6%|▌ | 177/3000 [00:04, 44.20it/s, step size=1.00e-01, acc. prob=0.781]Warmup: 6%|▌ | 182/3000 [00:04, 39.40it/s, step size=7.72e-02, acc. prob=0.781]Warmup: 6%|▌ | 187/3000 [00:05, 38.01it/s, step size=3.21e-01, acc. prob=0.784]Warmup: 6%|▋ | 193/3000 [00:05, 39.89it/s, step size=9.95e-02, acc. prob=0.782]Warmup: 7%|▋ | 198/3000 [00:05, 37.51it/s, step size=1.51e-01, acc. prob=0.783]Warmup: 7%|▋ | 203/3000 [00:05, 37.05it/s, step size=8.11e-02, acc. prob=0.782]Warmup: 7%|▋ | 210/3000 [00:05, 42.15it/s, step size=1.64e-01, acc. prob=0.784]Warmup: 7%|▋ | 216/3000 [00:05, 45.77it/s, step size=1.66e-01, acc. prob=0.784]Warmup: 7%|▋ | 223/3000 [00:05, 49.67it/s, step size=1.20e-01, acc. prob=0.784]Warmup: 8%|▊ | 229/3000 [00:05, 47.74it/s, step size=1.63e-01, acc. prob=0.785]Warmup: 8%|▊ | 236/3000 [00:06, 52.98it/s, step size=1.18e-01, acc. prob=0.784]Warmup: 8%|▊ | 242/3000 [00:06, 51.78it/s, step size=2.36e-01, acc. prob=0.786]Warmup: 8%|▊ | 248/3000 [00:06, 53.54it/s, step size=1.56e-01, acc. prob=0.786]Warmup: 9%|▊ | 258/3000 [00:06, 65.46it/s, step size=1.40e-01, acc. prob=0.781]Warmup: 9%|▉ | 265/3000 [00:06, 64.80it/s, step size=8.96e-02, acc. prob=0.781]Warmup: 9%|▉ | 272/3000 [00:06, 63.61it/s, step size=1.73e-01, acc. prob=0.782]Warmup: 9%|▉ | 281/3000 [00:06, 70.04it/s, step size=1.40e-01, acc. prob=0.782]Warmup: 10%|▉ | 289/3000 [00:06, 60.07it/s, step size=1.99e-01, acc. prob=0.783]Warmup: 10%|▉ | 297/3000 [00:07, 59.41it/s, step size=1.07e-01, acc. prob=0.782]Warmup: 10%|█ | 304/3000 [00:07, 61.58it/s, step size=2.98e-01, acc. prob=0.784]Warmup: 11%|█ | 316/3000 [00:07, 72.86it/s, step size=2.38e-01, acc. prob=0.784]Warmup: 11%|█ | 324/3000 [00:07, 70.59it/s, step size=2.69e-01, acc. prob=0.784]Warmup: 11%|█ | 333/3000 [00:07, 72.84it/s, step size=9.16e-02, acc. prob=0.783]Warmup: 11%|█▏ | 341/3000 [00:07, 66.98it/s, step size=1.55e-01, acc. prob=0.784]Warmup: 12%|█▏ | 349/3000 [00:07, 69.29it/s, step size=1.52e-01, acc. prob=0.784]Warmup: 12%|█▏ | 357/3000 [00:07, 67.52it/s, step size=9.74e-02, acc. prob=0.783]Warmup: 12%|█▏ | 365/3000 [00:07, 69.95it/s, step size=1.68e-01, acc. prob=0.784]Warmup: 12%|█▏ | 373/3000 [00:08, 71.96it/s, step size=2.69e-01, acc. prob=0.785]Warmup: 13%|█▎ | 381/3000 [00:08, 71.42it/s, step size=2.43e-01, acc. prob=0.785]Warmup: 13%|█▎ | 389/3000 [00:08, 70.05it/s, step size=1.80e-01, acc. prob=0.785]Warmup: 13%|█▎ | 397/3000 [00:08, 64.41it/s, step size=9.86e-02, acc. prob=0.784]Warmup: 13%|█▎ | 404/3000 [00:08, 65.28it/s, step size=9.11e-02, acc. prob=0.784]Warmup: 14%|█▎ | 411/3000 [00:08, 62.54it/s, step size=1.31e-01, acc. prob=0.785]Warmup: 14%|█▍ | 422/3000 [00:08, 73.81it/s, step size=1.83e-01, acc. prob=0.786]Warmup: 14%|█▍ | 430/3000 [00:08, 71.01it/s, step size=1.42e-01, acc. prob=0.785]Warmup: 15%|█▍ | 438/3000 [00:09, 62.02it/s, step size=1.07e-01, acc. prob=0.785]Warmup: 15%|█▍ | 445/3000 [00:09, 51.37it/s, step size=1.89e-01, acc. prob=0.786]Warmup: 15%|█▌ | 458/3000 [00:09, 68.12it/s, step size=3.86e-02, acc. prob=0.783]Warmup: 16%|█▌ | 466/3000 [00:09, 58.68it/s, step size=8.37e-02, acc. prob=0.783]Warmup: 16%|█▌ | 473/3000 [00:09, 57.15it/s, step size=3.53e-01, acc. prob=0.784]Warmup: 16%|█▌ | 480/3000 [00:09, 46.10it/s, step size=2.13e-01, acc. prob=0.784]Warmup: 16%|█▌ | 486/3000 [00:10, 45.36it/s, step size=1.04e-01, acc. prob=0.784]Warmup: 16%|█▋ | 491/3000 [00:10, 44.48it/s, step size=1.41e-01, acc. prob=0.784]Warmup: 17%|█▋ | 496/3000 [00:10, 42.15it/s, step size=2.34e-01, acc. prob=0.784]Warmup: 17%|█▋ | 502/3000 [00:10, 45.40it/s, step size=1.08e-01, acc. prob=0.784]Warmup: 17%|█▋ | 507/3000 [00:10, 46.38it/s, step size=2.43e-01, acc. prob=0.784]Warmup: 17%|█▋ | 514/3000 [00:10, 52.30it/s, step size=1.10e-01, acc. prob=0.784]Warmup: 17%|█▋ | 522/3000 [00:10, 57.69it/s, step size=1.63e-01, acc. prob=0.784]Warmup: 18%|█▊ | 528/3000 [00:10, 56.86it/s, step size=1.49e-01, acc. prob=0.784]Warmup: 18%|█▊ | 535/3000 [00:10, 59.43it/s, step size=1.83e-01, acc. prob=0.784]Warmup: 18%|█▊ | 544/3000 [00:11, 67.04it/s, step size=2.31e-01, acc. prob=0.785]Warmup: 18%|█▊ | 552/3000 [00:11, 65.80it/s, step size=7.76e-02, acc. prob=0.784]Warmup: 19%|█▊ | 559/3000 [00:11, 65.77it/s, step size=1.16e-01, acc. prob=0.784]Warmup: 19%|█▉ | 566/3000 [00:11, 64.99it/s, step size=1.60e-01, acc. prob=0.784]Warmup: 19%|█▉ | 573/3000 [00:11, 60.12it/s, step size=1.57e-01, acc. prob=0.785]Warmup: 19%|█▉ | 580/3000 [00:11, 58.54it/s, step size=1.17e-01, acc. prob=0.784]Warmup: 20%|█▉ | 588/3000 [00:11, 60.83it/s, step size=1.51e-01, acc. prob=0.785]Warmup: 20%|█▉ | 595/3000 [00:11, 62.22it/s, step size=1.54e-01, acc. prob=0.785]Warmup: 20%|██ | 602/3000 [00:12, 56.93it/s, step size=1.43e-01, acc. prob=0.785]Warmup: 20%|██ | 611/3000 [00:12, 64.61it/s, step size=2.77e-01, acc. prob=0.786]Warmup: 21%|██ | 620/3000 [00:12, 70.65it/s, step size=1.54e-01, acc. prob=0.785]Warmup: 21%|██ | 628/3000 [00:12, 61.70it/s, step size=9.72e-02, acc. prob=0.785]Warmup: 21%|██ | 635/3000 [00:12, 63.63it/s, step size=1.74e-01, acc. prob=0.785]Warmup: 21%|██▏ | 644/3000 [00:12, 69.79it/s, step size=1.78e-01, acc. prob=0.785]Warmup: 22%|██▏ | 652/3000 [00:12, 68.13it/s, step size=3.46e-01, acc. prob=0.786]Warmup: 22%|██▏ | 662/3000 [00:12, 74.06it/s, step size=1.68e-01, acc. prob=0.785]Warmup: 22%|██▏ | 670/3000 [00:13, 74.11it/s, step size=2.48e-01, acc. prob=0.786]Warmup: 23%|██▎ | 681/3000 [00:13, 83.50it/s, step size=2.62e-01, acc. prob=0.786]Warmup: 23%|██▎ | 690/3000 [00:13, 67.05it/s, step size=1.49e-01, acc. prob=0.786]Warmup: 23%|██▎ | 699/3000 [00:13, 70.18it/s, step size=1.81e-01, acc. prob=0.786]Warmup: 24%|██▎ | 707/3000 [00:13, 68.48it/s, step size=1.68e-01, acc. prob=0.786]Warmup: 24%|██▍ | 715/3000 [00:13, 70.88it/s, step size=1.41e-01, acc. prob=0.786]Warmup: 24%|██▍ | 723/3000 [00:13, 67.03it/s, step size=1.12e-01, acc. prob=0.786]Warmup: 24%|██▍ | 731/3000 [00:13, 69.09it/s, step size=1.70e-01, acc. prob=0.786]Warmup: 25%|██▍ | 740/3000 [00:13, 73.05it/s, step size=1.79e-01, acc. prob=0.786]Warmup: 25%|██▍ | 748/3000 [00:14, 65.81it/s, step size=1.95e-01, acc. prob=0.787]Warmup: 25%|██▌ | 757/3000 [00:14, 71.91it/s, step size=1.90e-01, acc. prob=0.787]Warmup: 26%|██▌ | 765/3000 [00:14, 71.36it/s, step size=1.91e-01, acc. prob=0.787]Warmup: 26%|██▌ | 773/3000 [00:14, 58.90it/s, step size=8.26e-02, acc. prob=0.786]Warmup: 26%|██▌ | 780/3000 [00:14, 55.82it/s, step size=1.27e-01, acc. prob=0.786]Warmup: 26%|██▌ | 787/3000 [00:14, 58.35it/s, step size=1.00e-01, acc. prob=0.786]Warmup: 26%|██▋ | 794/3000 [00:14, 59.92it/s, step size=2.02e-01, acc. prob=0.787]Warmup: 27%|██▋ | 801/3000 [00:15, 55.37it/s, step size=1.38e-01, acc. prob=0.787]Warmup: 27%|██▋ | 808/3000 [00:15, 56.67it/s, step size=1.26e-01, acc. prob=0.787]Warmup: 27%|██▋ | 817/3000 [00:15, 64.36it/s, step size=1.72e-01, acc. prob=0.787]Warmup: 28%|██▊ | 826/3000 [00:15, 70.22it/s, step size=3.12e-01, acc. prob=0.788]Warmup: 28%|██▊ | 834/3000 [00:15, 71.91it/s, step size=1.38e-01, acc. prob=0.787]Warmup: 28%|██▊ | 843/3000 [00:15, 75.98it/s, step size=2.88e-01, acc. prob=0.788]Warmup: 28%|██▊ | 851/3000 [00:15, 71.92it/s, step size=2.04e-01, acc. prob=0.787]Warmup: 29%|██▊ | 860/3000 [00:15, 76.05it/s, step size=2.14e-01, acc. prob=0.788]Warmup: 29%|██▉ | 868/3000 [00:15, 73.31it/s, step size=1.37e-01, acc. prob=0.787]Warmup: 29%|██▉ | 876/3000 [00:16, 72.12it/s, step size=1.73e-01, acc. prob=0.787]Warmup: 29%|██▉ | 884/3000 [00:16, 69.52it/s, step size=8.85e-02, acc. prob=0.787]Warmup: 30%|██▉ | 892/3000 [00:16, 58.45it/s, step size=1.12e-01, acc. prob=0.787]Warmup: 30%|██▉ | 899/3000 [00:16, 55.71it/s, step size=1.51e-01, acc. prob=0.788]Warmup: 30%|███ | 906/3000 [00:16, 58.68it/s, step size=2.00e-01, acc. prob=0.788]Warmup: 30%|███ | 915/3000 [00:16, 64.15it/s, step size=1.14e-01, acc. prob=0.787]Warmup: 31%|███ | 922/3000 [00:16, 59.86it/s, step size=9.56e-02, acc. prob=0.787]Warmup: 31%|███ | 929/3000 [00:17, 54.03it/s, step size=1.06e-01, acc. prob=0.787]Warmup: 31%|███ | 935/3000 [00:17, 54.44it/s, step size=1.06e-01, acc. prob=0.787]Warmup: 31%|███▏ | 943/3000 [00:17, 58.47it/s, step size=1.34e-01, acc. prob=0.788]Warmup: 32%|███▏ | 949/3000 [00:17, 55.87it/s, step size=3.70e-01, acc. prob=0.788]Warmup: 32%|███▏ | 958/3000 [00:17, 58.23it/s, step size=6.68e-02, acc. prob=0.787]Warmup: 32%|███▏ | 964/3000 [00:17, 55.39it/s, step size=2.77e-01, acc. prob=0.787]Warmup: 32%|███▏ | 970/3000 [00:17, 52.09it/s, step size=7.70e-02, acc. prob=0.787]Warmup: 33%|███▎ | 976/3000 [00:17, 45.97it/s, step size=1.60e-01, acc. prob=0.787]Warmup: 33%|███▎ | 982/3000 [00:18, 48.49it/s, step size=1.07e-01, acc. prob=0.787]Warmup: 33%|███▎ | 989/3000 [00:18, 53.29it/s, step size=2.41e-01, acc. prob=0.788]Warmup: 33%|███▎ | 997/3000 [00:18, 59.73it/s, step size=1.85e-01, acc. prob=0.787]Sample: 34%|███▎ | 1005/3000 [00:18, 64.27it/s, step size=1.81e-01, acc. prob=0.786]Sample: 34%|███▍ | 1014/3000 [00:18, 68.54it/s, step size=1.81e-01, acc. prob=0.723]Sample: 34%|███▍ | 1022/3000 [00:18, 70.60it/s, step size=1.81e-01, acc. prob=0.761]Sample: 34%|███▍ | 1031/3000 [00:18, 73.98it/s, step size=1.81e-01, acc. prob=0.793]Sample: 35%|███▍ | 1039/3000 [00:18, 72.62it/s, step size=1.81e-01, acc. prob=0.809]Sample: 35%|███▍ | 1047/3000 [00:18, 69.96it/s, step size=1.81e-01, acc. prob=0.802]Sample: 35%|███▌ | 1055/3000 [00:19, 68.25it/s, step size=1.81e-01, acc. prob=0.814]Sample: 35%|███▌ | 1062/3000 [00:19, 64.00it/s, step size=1.81e-01, acc. prob=0.810]Sample: 36%|███▌ | 1070/3000 [00:19, 66.75it/s, step size=1.81e-01, acc. prob=0.815]Sample: 36%|███▌ | 1081/3000 [00:19, 76.67it/s, step size=1.81e-01, acc. prob=0.813]Sample: 36%|███▋ | 1089/3000 [00:19, 70.15it/s, step size=1.81e-01, acc. prob=0.809]Sample: 37%|███▋ | 1097/3000 [00:19, 69.91it/s, step size=1.81e-01, acc. prob=0.804]Sample: 37%|███▋ | 1105/3000 [00:19, 69.49it/s, step size=1.81e-01, acc. prob=0.802]Sample: 37%|███▋ | 1113/3000 [00:19, 71.52it/s, step size=1.81e-01, acc. prob=0.800]Sample: 37%|███▋ | 1121/3000 [00:20, 65.24it/s, step size=1.81e-01, acc. prob=0.801]Sample: 38%|███▊ | 1129/3000 [00:20, 68.61it/s, step size=1.81e-01, acc. prob=0.804]Sample: 38%|███▊ | 1138/3000 [00:20, 71.67it/s, step size=1.81e-01, acc. prob=0.807]Sample: 38%|███▊ | 1146/3000 [00:20, 63.00it/s, step size=1.81e-01, acc. prob=0.809]Sample: 38%|███▊ | 1153/3000 [00:20, 57.80it/s, step size=1.81e-01, acc. prob=0.802]Sample: 39%|███▊ | 1160/3000 [00:20, 56.65it/s, step size=1.81e-01, acc. prob=0.803]Sample: 39%|███▉ | 1166/3000 [00:20, 57.13it/s, step size=1.81e-01, acc. prob=0.801]Sample: 39%|███▉ | 1174/3000 [00:20, 61.68it/s, step size=1.81e-01, acc. prob=0.799]Sample: 39%|███▉ | 1181/3000 [00:21, 63.08it/s, step size=1.81e-01, acc. prob=0.802]Sample: 40%|███▉ | 1190/3000 [00:21, 69.03it/s, step size=1.81e-01, acc. prob=0.806]Sample: 40%|███▉ | 1198/3000 [00:21, 69.35it/s, step size=1.81e-01, acc. prob=0.801]Sample: 40%|████ | 1206/3000 [00:21, 66.64it/s, step size=1.81e-01, acc. prob=0.801]Sample: 40%|████ | 1214/3000 [00:21, 68.62it/s, step size=1.81e-01, acc. prob=0.800]Sample: 41%|████ | 1221/3000 [00:21, 64.94it/s, step size=1.81e-01, acc. prob=0.799]Sample: 41%|████ | 1228/3000 [00:21, 65.07it/s, step size=1.81e-01, acc. prob=0.797]Sample: 41%|████ | 1235/3000 [00:21, 63.40it/s, step size=1.81e-01, acc. prob=0.797]Sample: 41%|████▏ | 1244/3000 [00:21, 69.64it/s, step size=1.81e-01, acc. prob=0.801]Sample: 42%|████▏ | 1252/3000 [00:22, 67.74it/s, step size=1.81e-01, acc. prob=0.802]Sample: 42%|████▏ | 1259/3000 [00:22, 62.10it/s, step size=1.81e-01, acc. prob=0.803]Sample: 42%|████▏ | 1267/3000 [00:22, 65.92it/s, step size=1.81e-01, acc. prob=0.805]Sample: 43%|████▎ | 1276/3000 [00:22, 70.70it/s, step size=1.81e-01, acc. prob=0.809]Sample: 43%|████▎ | 1285/3000 [00:22, 72.11it/s, step size=1.81e-01, acc. prob=0.808]Sample: 43%|████▎ | 1293/3000 [00:22, 73.46it/s, step size=1.81e-01, acc. prob=0.810]Sample: 43%|████▎ | 1301/3000 [00:22, 74.19it/s, step size=1.81e-01, acc. prob=0.810]Sample: 44%|████▎ | 1309/3000 [00:22, 72.98it/s, step size=1.81e-01, acc. prob=0.809]Sample: 44%|████▍ | 1317/3000 [00:22, 68.31it/s, step size=1.81e-01, acc. prob=0.808]Sample: 44%|████▍ | 1324/3000 [00:23, 68.11it/s, step size=1.81e-01, acc. prob=0.807]Sample: 44%|████▍ | 1332/3000 [00:23, 68.85it/s, step size=1.81e-01, acc. prob=0.807]Sample: 45%|████▍ | 1341/3000 [00:23, 72.24it/s, step size=1.81e-01, acc. prob=0.809]Sample: 45%|████▍ | 1349/3000 [00:23, 71.73it/s, step size=1.81e-01, acc. prob=0.810]Sample: 45%|████▌ | 1357/3000 [00:23, 69.21it/s, step size=1.81e-01, acc. prob=0.810]Sample: 46%|████▌ | 1365/3000 [00:23, 69.42it/s, step size=1.81e-01, acc. prob=0.809]Sample: 46%|████▌ | 1373/3000 [00:23, 70.50it/s, step size=1.81e-01, acc. prob=0.809]Sample: 46%|████▌ | 1381/3000 [00:23, 54.71it/s, step size=1.81e-01, acc. prob=0.810]Sample: 46%|████▋ | 1388/3000 [00:24, 57.09it/s, step size=1.81e-01, acc. prob=0.807]Sample: 46%|████▋ | 1395/3000 [00:24, 54.60it/s, step size=1.81e-01, acc. prob=0.805]Sample: 47%|████▋ | 1402/3000 [00:24, 57.05it/s, step size=1.81e-01, acc. prob=0.805]Sample: 47%|████▋ | 1409/3000 [00:24, 59.91it/s, step size=1.81e-01, acc. prob=0.806]Sample: 47%|████▋ | 1416/3000 [00:24, 61.34it/s, step size=1.81e-01, acc. prob=0.806]Sample: 47%|████▋ | 1423/3000 [00:24, 59.42it/s, step size=1.81e-01, acc. prob=0.805]Sample: 48%|████▊ | 1430/3000 [00:24, 50.64it/s, step size=1.81e-01, acc. prob=0.806]Sample: 48%|████▊ | 1436/3000 [00:24, 50.37it/s, step size=1.81e-01, acc. prob=0.806]Sample: 48%|████▊ | 1443/3000 [00:25, 53.65it/s, step size=1.81e-01, acc. prob=0.807]Sample: 48%|████▊ | 1449/3000 [00:25, 50.25it/s, step size=1.81e-01, acc. prob=0.807]Sample: 49%|████▊ | 1456/3000 [00:25, 54.01it/s, step size=1.81e-01, acc. prob=0.807]Sample: 49%|████▉ | 1463/3000 [00:25, 57.73it/s, step size=1.81e-01, acc. prob=0.807]Sample: 49%|████▉ | 1471/3000 [00:25, 59.41it/s, step size=1.81e-01, acc. prob=0.807]Sample: 49%|████▉ | 1478/3000 [00:25, 58.67it/s, step size=1.81e-01, acc. prob=0.808]Sample: 50%|████▉ | 1486/3000 [00:25, 63.76it/s, step size=1.81e-01, acc. prob=0.808]Sample: 50%|████▉ | 1494/3000 [00:25, 67.55it/s, step size=1.81e-01, acc. prob=0.810]Sample: 50%|█████ | 1502/3000 [00:26, 69.14it/s, step size=1.81e-01, acc. prob=0.810]Sample: 50%|█████ | 1509/3000 [00:26, 65.08it/s, step size=1.81e-01, acc. prob=0.811]Sample: 51%|█████ | 1516/3000 [00:26, 63.00it/s, step size=1.81e-01, acc. prob=0.810]Sample: 51%|█████ | 1523/3000 [00:26, 61.61it/s, step size=1.81e-01, acc. prob=0.810]Sample: 51%|█████ | 1530/3000 [00:26, 57.92it/s, step size=1.81e-01, acc. prob=0.810]Sample: 51%|█████ | 1536/3000 [00:26, 55.91it/s, step size=1.81e-01, acc. prob=0.810]Sample: 51%|█████▏ | 1542/3000 [00:26, 56.12it/s, step size=1.81e-01, acc. prob=0.810]Sample: 52%|█████▏ | 1549/3000 [00:26, 56.99it/s, step size=1.81e-01, acc. prob=0.811]Sample: 52%|█████▏ | 1555/3000 [00:26, 53.71it/s, step size=1.81e-01, acc. prob=0.811]Sample: 52%|█████▏ | 1563/3000 [00:27, 59.77it/s, step size=1.81e-01, acc. prob=0.811]Sample: 52%|█████▏ | 1570/3000 [00:27, 61.94it/s, step size=1.81e-01, acc. prob=0.811]Sample: 53%|█████▎ | 1577/3000 [00:27, 60.77it/s, step size=1.81e-01, acc. prob=0.810]Sample: 53%|█████▎ | 1585/3000 [00:27, 64.01it/s, step size=1.81e-01, acc. prob=0.810]Sample: 53%|█████▎ | 1592/3000 [00:27, 57.79it/s, step size=1.81e-01, acc. prob=0.811]Sample: 53%|█████▎ | 1599/3000 [00:27, 58.32it/s, step size=1.81e-01, acc. prob=0.811]Sample: 54%|█████▎ | 1605/3000 [00:27, 56.04it/s, step size=1.81e-01, acc. prob=0.809]Sample: 54%|█████▎ | 1611/3000 [00:27, 52.81it/s, step size=1.81e-01, acc. prob=0.810]Sample: 54%|█████▍ | 1617/3000 [00:28, 53.97it/s, step size=1.81e-01, acc. prob=0.810]Sample: 54%|█████▍ | 1624/3000 [00:28, 57.75it/s, step size=1.81e-01, acc. prob=0.810]Sample: 54%|█████▍ | 1630/3000 [00:28, 55.77it/s, step size=1.81e-01, acc. prob=0.809]Sample: 55%|█████▍ | 1638/3000 [00:28, 59.47it/s, step size=1.81e-01, acc. prob=0.809]Sample: 55%|█████▍ | 1648/3000 [00:28, 66.32it/s, step size=1.81e-01, acc. prob=0.809]Sample: 55%|█████▌ | 1657/3000 [00:28, 68.49it/s, step size=1.81e-01, acc. prob=0.810]Sample: 56%|█████▌ | 1665/3000 [00:28, 70.78it/s, step size=1.81e-01, acc. prob=0.811]Sample: 56%|█████▌ | 1673/3000 [00:28, 71.20it/s, step size=1.81e-01, acc. prob=0.810]Sample: 56%|█████▌ | 1681/3000 [00:29, 64.62it/s, step size=1.81e-01, acc. prob=0.810]Sample: 56%|█████▋ | 1688/3000 [00:29, 64.61it/s, step size=1.81e-01, acc. prob=0.810]Sample: 56%|█████▋ | 1695/3000 [00:29, 61.03it/s, step size=1.81e-01, acc. prob=0.808]Sample: 57%|█████▋ | 1702/3000 [00:29, 57.53it/s, step size=1.81e-01, acc. prob=0.809]Sample: 57%|█████▋ | 1708/3000 [00:29, 58.08it/s, step size=1.81e-01, acc. prob=0.809]Sample: 57%|█████▋ | 1714/3000 [00:29, 53.15it/s, step size=1.81e-01, acc. prob=0.809]Sample: 57%|█████▋ | 1720/3000 [00:29, 48.47it/s, step size=1.81e-01, acc. prob=0.809]Sample: 58%|█████▊ | 1726/3000 [00:29, 49.71it/s, step size=1.81e-01, acc. prob=0.809]Sample: 58%|█████▊ | 1732/3000 [00:30, 48.07it/s, step size=1.81e-01, acc. prob=0.808]Sample: 58%|█████▊ | 1737/3000 [00:30, 48.02it/s, step size=1.81e-01, acc. prob=0.808]Sample: 58%|█████▊ | 1743/3000 [00:30, 49.81it/s, step size=1.81e-01, acc. prob=0.809]Sample: 58%|█████▊ | 1749/3000 [00:30, 48.41it/s, step size=1.81e-01, acc. prob=0.808]Sample: 59%|█████▊ | 1756/3000 [00:30, 51.23it/s, step size=1.81e-01, acc. prob=0.810]Sample: 59%|█████▉ | 1764/3000 [00:30, 58.54it/s, step size=1.81e-01, acc. prob=0.810]Sample: 59%|█████▉ | 1770/3000 [00:30, 54.53it/s, step size=1.81e-01, acc. prob=0.810]Sample: 59%|█████▉ | 1777/3000 [00:30, 58.38it/s, step size=1.81e-01, acc. prob=0.810]Sample: 59%|█████▉ | 1783/3000 [00:30, 58.74it/s, step size=1.81e-01, acc. prob=0.810]Sample: 60%|█████▉ | 1790/3000 [00:31, 59.65it/s, step size=1.81e-01, acc. prob=0.811]Sample: 60%|█████▉ | 1797/3000 [00:31, 62.51it/s, step size=1.81e-01, acc. prob=0.811]Sample: 60%|██████ | 1806/3000 [00:31, 68.16it/s, step size=1.81e-01, acc. prob=0.811]Sample: 60%|██████ | 1813/3000 [00:31, 65.18it/s, step size=1.81e-01, acc. prob=0.811]Sample: 61%|██████ | 1820/3000 [00:31, 63.79it/s, step size=1.81e-01, acc. prob=0.810]Sample: 61%|██████ | 1828/3000 [00:31, 67.27it/s, step size=1.81e-01, acc. prob=0.810]Sample: 61%|██████ | 1836/3000 [00:31, 69.61it/s, step size=1.81e-01, acc. prob=0.810]Sample: 61%|██████▏ | 1844/3000 [00:31, 71.28it/s, step size=1.81e-01, acc. prob=0.810]Sample: 62%|██████▏ | 1852/3000 [00:31, 72.80it/s, step size=1.81e-01, acc. prob=0.811]Sample: 62%|██████▏ | 1861/3000 [00:32, 76.44it/s, step size=1.81e-01, acc. prob=0.811]Sample: 62%|██████▏ | 1869/3000 [00:32, 73.02it/s, step size=1.81e-01, acc. prob=0.810]Sample: 63%|██████▎ | 1877/3000 [00:32, 65.62it/s, step size=1.81e-01, acc. prob=0.811]Sample: 63%|██████▎ | 1887/3000 [00:32, 73.60it/s, step size=1.81e-01, acc. prob=0.809]Sample: 63%|██████▎ | 1897/3000 [00:32, 79.77it/s, step size=1.81e-01, acc. prob=0.810]Sample: 64%|██████▎ | 1906/3000 [00:32, 76.94it/s, step size=1.81e-01, acc. prob=0.809]Sample: 64%|██████▍ | 1914/3000 [00:32, 68.38it/s, step size=1.81e-01, acc. prob=0.808]Sample: 64%|██████▍ | 1922/3000 [00:32, 66.09it/s, step size=1.81e-01, acc. prob=0.808]Sample: 64%|██████▍ | 1929/3000 [00:33, 66.12it/s, step size=1.81e-01, acc. prob=0.808]Sample: 65%|██████▍ | 1937/3000 [00:33, 65.90it/s, step size=1.81e-01, acc. prob=0.808]Sample: 65%|██████▍ | 1945/3000 [00:33, 69.54it/s, step size=1.81e-01, acc. prob=0.809]Sample: 65%|██████▌ | 1953/3000 [00:33, 69.29it/s, step size=1.81e-01, acc. prob=0.809]Sample: 65%|██████▌ | 1961/3000 [00:33, 69.15it/s, step size=1.81e-01, acc. prob=0.809]Sample: 66%|██████▌ | 1969/3000 [00:33, 71.08it/s, step size=1.81e-01, acc. prob=0.809]Sample: 66%|██████▌ | 1977/3000 [00:33, 70.06it/s, step size=1.81e-01, acc. prob=0.808]Sample: 66%|██████▌ | 1986/3000 [00:33, 73.61it/s, step size=1.81e-01, acc. prob=0.809]Sample: 66%|██████▋ | 1994/3000 [00:33, 65.65it/s, step size=1.81e-01, acc. prob=0.808]Sample: 67%|██████▋ | 2001/3000 [00:34, 61.89it/s, step size=1.81e-01, acc. prob=0.807]Sample: 67%|██████▋ | 2008/3000 [00:34, 62.59it/s, step size=1.81e-01, acc. prob=0.808]Sample: 67%|██████▋ | 2016/3000 [00:34, 66.39it/s, step size=1.81e-01, acc. prob=0.808]Sample: 67%|██████▋ | 2024/3000 [00:34, 69.04it/s, step size=1.81e-01, acc. prob=0.808]Sample: 68%|██████▊ | 2032/3000 [00:34, 64.45it/s, step size=1.81e-01, acc. prob=0.809]Sample: 68%|██████▊ | 2039/3000 [00:34, 64.74it/s, step size=1.81e-01, acc. prob=0.809]Sample: 68%|██████▊ | 2048/3000 [00:34, 70.50it/s, step size=1.81e-01, acc. prob=0.809]Sample: 69%|██████▊ | 2056/3000 [00:34, 62.81it/s, step size=1.81e-01, acc. prob=0.809]Sample: 69%|██████▉ | 2064/3000 [00:35, 64.47it/s, step size=1.81e-01, acc. prob=0.809]Sample: 69%|██████▉ | 2071/3000 [00:35, 63.26it/s, step size=1.81e-01, acc. prob=0.809]Sample: 69%|██████▉ | 2078/3000 [00:35, 62.64it/s, step size=1.81e-01, acc. prob=0.808]Sample: 70%|██████▉ | 2087/3000 [00:35, 68.84it/s, step size=1.81e-01, acc. prob=0.808]Sample: 70%|██████▉ | 2096/3000 [00:35, 72.62it/s, step size=1.81e-01, acc. prob=0.808]Sample: 70%|███████ | 2104/3000 [00:35, 71.54it/s, step size=1.81e-01, acc. prob=0.808]Sample: 70%|███████ | 2112/3000 [00:35, 68.74it/s, step size=1.81e-01, acc. prob=0.808]Sample: 71%|███████ | 2119/3000 [00:35, 68.89it/s, step size=1.81e-01, acc. prob=0.808]Sample: 71%|███████ | 2127/3000 [00:35, 67.78it/s, step size=1.81e-01, acc. prob=0.808]Sample: 71%|███████ | 2134/3000 [00:36, 67.69it/s, step size=1.81e-01, acc. prob=0.808]Sample: 71%|███████▏ | 2142/3000 [00:36, 69.15it/s, step size=1.81e-01, acc. prob=0.808]Sample: 72%|███████▏ | 2149/3000 [00:36, 63.08it/s, step size=1.81e-01, acc. prob=0.808]Sample: 72%|███████▏ | 2157/3000 [00:36, 67.02it/s, step size=1.81e-01, acc. prob=0.809]Sample: 72%|███████▏ | 2164/3000 [00:36, 65.78it/s, step size=1.81e-01, acc. prob=0.809]Sample: 72%|███████▏ | 2171/3000 [00:36, 59.89it/s, step size=1.81e-01, acc. prob=0.808]Sample: 73%|███████▎ | 2178/3000 [00:36, 59.75it/s, step size=1.81e-01, acc. prob=0.808]Sample: 73%|███████▎ | 2185/3000 [00:36, 61.95it/s, step size=1.81e-01, acc. prob=0.808]Sample: 73%|███████▎ | 2192/3000 [00:36, 61.70it/s, step size=1.81e-01, acc. prob=0.809]Sample: 73%|███████▎ | 2199/3000 [00:37, 61.68it/s, step size=1.81e-01, acc. prob=0.809]Sample: 74%|███████▎ | 2207/3000 [00:37, 66.39it/s, step size=1.81e-01, acc. prob=0.809]Sample: 74%|███████▍ | 2214/3000 [00:37, 65.26it/s, step size=1.81e-01, acc. prob=0.809]Sample: 74%|███████▍ | 2221/3000 [00:37, 65.10it/s, step size=1.81e-01, acc. prob=0.810]Sample: 74%|███████▍ | 2228/3000 [00:37, 65.85it/s, step size=1.81e-01, acc. prob=0.810]Sample: 75%|███████▍ | 2236/3000 [00:37, 67.96it/s, step size=1.81e-01, acc. prob=0.810]Sample: 75%|███████▍ | 2243/3000 [00:37, 68.02it/s, step size=1.81e-01, acc. prob=0.810]Sample: 75%|███████▌ | 2251/3000 [00:37, 71.25it/s, step size=1.81e-01, acc. prob=0.810]Sample: 75%|███████▌ | 2259/3000 [00:37, 70.46it/s, step size=1.81e-01, acc. prob=0.809]Sample: 76%|███████▌ | 2267/3000 [00:38, 68.14it/s, step size=1.81e-01, acc. prob=0.809]Sample: 76%|███████▌ | 2275/3000 [00:38, 70.17it/s, step size=1.81e-01, acc. prob=0.810]Sample: 76%|███████▌ | 2283/3000 [00:38, 69.83it/s, step size=1.81e-01, acc. prob=0.809]Sample: 76%|███████▋ | 2291/3000 [00:38, 68.51it/s, step size=1.81e-01, acc. prob=0.809]Sample: 77%|███████▋ | 2301/3000 [00:38, 73.15it/s, step size=1.81e-01, acc. prob=0.809]Sample: 77%|███████▋ | 2309/3000 [00:38, 70.55it/s, step size=1.81e-01, acc. prob=0.810]Sample: 77%|███████▋ | 2317/3000 [00:38, 70.70it/s, step size=1.81e-01, acc. prob=0.809]Sample: 78%|███████▊ | 2325/3000 [00:38, 70.82it/s, step size=1.81e-01, acc. prob=0.809]Sample: 78%|███████▊ | 2333/3000 [00:39, 69.44it/s, step size=1.81e-01, acc. prob=0.809]Sample: 78%|███████▊ | 2342/3000 [00:39, 73.89it/s, step size=1.81e-01, acc. prob=0.809]Sample: 78%|███████▊ | 2351/3000 [00:39, 77.24it/s, step size=1.81e-01, acc. prob=0.809]Sample: 79%|███████▊ | 2361/3000 [00:39, 82.09it/s, step size=1.81e-01, acc. prob=0.809]Sample: 79%|███████▉ | 2370/3000 [00:39, 82.66it/s, step size=1.81e-01, acc. prob=0.810]Sample: 79%|███████▉ | 2379/3000 [00:39, 79.97it/s, step size=1.81e-01, acc. prob=0.809]Sample: 80%|███████▉ | 2388/3000 [00:39, 76.96it/s, step size=1.81e-01, acc. prob=0.809]Sample: 80%|███████▉ | 2396/3000 [00:39, 76.53it/s, step size=1.81e-01, acc. prob=0.809]Sample: 80%|████████ | 2404/3000 [00:39, 74.07it/s, step size=1.81e-01, acc. prob=0.809]Sample: 80%|████████ | 2414/3000 [00:40, 80.16it/s, step size=1.81e-01, acc. prob=0.809]Sample: 81%|████████ | 2423/3000 [00:40, 69.86it/s, step size=1.81e-01, acc. prob=0.808]Sample: 81%|████████ | 2431/3000 [00:40, 67.13it/s, step size=1.81e-01, acc. prob=0.808]Sample: 81%|████████▏ | 2440/3000 [00:40, 71.69it/s, step size=1.81e-01, acc. prob=0.808]Sample: 82%|████████▏ | 2448/3000 [00:40, 72.80it/s, step size=1.81e-01, acc. prob=0.808]Sample: 82%|████████▏ | 2456/3000 [00:40, 74.71it/s, step size=1.81e-01, acc. prob=0.808]Sample: 82%|████████▏ | 2465/3000 [00:40, 76.49it/s, step size=1.81e-01, acc. prob=0.808]Sample: 82%|████████▏ | 2473/3000 [00:40, 74.69it/s, step size=1.81e-01, acc. prob=0.808]Sample: 83%|████████▎ | 2481/3000 [00:41, 59.12it/s, step size=1.81e-01, acc. prob=0.808]Sample: 83%|████████▎ | 2488/3000 [00:41, 59.10it/s, step size=1.81e-01, acc. prob=0.808]Sample: 83%|████████▎ | 2495/3000 [00:41, 60.67it/s, step size=1.81e-01, acc. prob=0.808]Sample: 83%|████████▎ | 2502/3000 [00:41, 60.11it/s, step size=1.81e-01, acc. prob=0.808]Sample: 84%|████████▎ | 2509/3000 [00:41, 61.38it/s, step size=1.81e-01, acc. prob=0.808]Sample: 84%|████████▍ | 2516/3000 [00:41, 59.89it/s, step size=1.81e-01, acc. prob=0.808]Sample: 84%|████████▍ | 2523/3000 [00:41, 55.11it/s, step size=1.81e-01, acc. prob=0.809]Sample: 84%|████████▍ | 2531/3000 [00:41, 61.09it/s, step size=1.81e-01, acc. prob=0.809]Sample: 85%|████████▍ | 2538/3000 [00:41, 60.86it/s, step size=1.81e-01, acc. prob=0.810]Sample: 85%|████████▍ | 2545/3000 [00:42, 57.60it/s, step size=1.81e-01, acc. prob=0.810]Sample: 85%|████████▌ | 2551/3000 [00:42, 56.66it/s, step size=1.81e-01, acc. prob=0.810]Sample: 85%|████████▌ | 2557/3000 [00:42, 57.04it/s, step size=1.81e-01, acc. prob=0.810]Sample: 85%|████████▌ | 2563/3000 [00:42, 57.68it/s, step size=1.81e-01, acc. prob=0.810]Sample: 86%|████████▌ | 2569/3000 [00:42, 52.89it/s, step size=1.81e-01, acc. prob=0.810]Sample: 86%|████████▌ | 2576/3000 [00:42, 53.00it/s, step size=1.81e-01, acc. prob=0.810]Sample: 86%|████████▌ | 2582/3000 [00:42, 51.99it/s, step size=1.81e-01, acc. prob=0.811]Sample: 86%|████████▋ | 2588/3000 [00:42, 50.73it/s, step size=1.81e-01, acc. prob=0.811]Sample: 86%|████████▋ | 2594/3000 [00:43, 51.84it/s, step size=1.81e-01, acc. prob=0.811]Sample: 87%|████████▋ | 2601/3000 [00:43, 56.49it/s, step size=1.81e-01, acc. prob=0.811]Sample: 87%|████████▋ | 2608/3000 [00:43, 57.00it/s, step size=1.81e-01, acc. prob=0.811]Sample: 87%|████████▋ | 2617/3000 [00:43, 59.23it/s, step size=1.81e-01, acc. prob=0.812]Sample: 87%|████████▋ | 2623/3000 [00:43, 53.92it/s, step size=1.81e-01, acc. prob=0.811]Sample: 88%|████████▊ | 2629/3000 [00:43, 55.28it/s, step size=1.81e-01, acc. prob=0.812]Sample: 88%|████████▊ | 2635/3000 [00:43, 54.89it/s, step size=1.81e-01, acc. prob=0.811]Sample: 88%|████████▊ | 2644/3000 [00:43, 63.99it/s, step size=1.81e-01, acc. prob=0.811]Sample: 88%|████████▊ | 2652/3000 [00:43, 67.13it/s, step size=1.81e-01, acc. prob=0.811]Sample: 89%|████████▊ | 2660/3000 [00:44, 68.46it/s, step size=1.81e-01, acc. prob=0.811]Sample: 89%|████████▉ | 2669/3000 [00:44, 73.27it/s, step size=1.81e-01, acc. prob=0.811]Sample: 89%|████████▉ | 2677/3000 [00:44, 68.75it/s, step size=1.81e-01, acc. prob=0.811]Sample: 90%|████████▉ | 2685/3000 [00:44, 71.36it/s, step size=1.81e-01, acc. prob=0.811]Sample: 90%|████████▉ | 2693/3000 [00:44, 65.20it/s, step size=1.81e-01, acc. prob=0.811]Sample: 90%|█████████ | 2700/3000 [00:44, 60.59it/s, step size=1.81e-01, acc. prob=0.810]Sample: 90%|█████████ | 2707/3000 [00:44, 60.69it/s, step size=1.81e-01, acc. prob=0.811]Sample: 90%|█████████ | 2715/3000 [00:44, 64.58it/s, step size=1.81e-01, acc. prob=0.810]Sample: 91%|█████████ | 2725/3000 [00:45, 70.12it/s, step size=1.81e-01, acc. prob=0.811]Sample: 91%|█████████ | 2734/3000 [00:45, 71.80it/s, step size=1.81e-01, acc. prob=0.810]Sample: 91%|█████████▏| 2742/3000 [00:45, 73.08it/s, step size=1.81e-01, acc. prob=0.810]Sample: 92%|█████████▏| 2750/3000 [00:45, 72.92it/s, step size=1.81e-01, acc. prob=0.811]Sample: 92%|█████████▏| 2758/3000 [00:45, 73.36it/s, step size=1.81e-01, acc. prob=0.811]Sample: 92%|█████████▏| 2766/3000 [00:45, 69.76it/s, step size=1.81e-01, acc. prob=0.811]Sample: 92%|█████████▏| 2774/3000 [00:45, 71.31it/s, step size=1.81e-01, acc. prob=0.811]Sample: 93%|█████████▎| 2782/3000 [00:45, 72.36it/s, step size=1.81e-01, acc. prob=0.812]Sample: 93%|█████████▎| 2790/3000 [00:45, 69.11it/s, step size=1.81e-01, acc. prob=0.812]Sample: 93%|█████████▎| 2798/3000 [00:46, 70.40it/s, step size=1.81e-01, acc. prob=0.812]Sample: 94%|█████████▎| 2807/3000 [00:46, 74.71it/s, step size=1.81e-01, acc. prob=0.812]Sample: 94%|█████████▍| 2815/3000 [00:46, 72.65it/s, step size=1.81e-01, acc. prob=0.812]Sample: 94%|█████████▍| 2823/3000 [00:46, 67.80it/s, step size=1.81e-01, acc. prob=0.812]Sample: 94%|█████████▍| 2832/3000 [00:46, 72.64it/s, step size=1.81e-01, acc. prob=0.813]Sample: 95%|█████████▍| 2840/3000 [00:46, 67.61it/s, step size=1.81e-01, acc. prob=0.812]Sample: 95%|█████████▍| 2847/3000 [00:46, 66.76it/s, step size=1.81e-01, acc. prob=0.813]Sample: 95%|█████████▌| 2854/3000 [00:46, 65.47it/s, step size=1.81e-01, acc. prob=0.813]Sample: 95%|█████████▌| 2863/3000 [00:47, 70.65it/s, step size=1.81e-01, acc. prob=0.813]Sample: 96%|█████████▌| 2871/3000 [00:47, 70.06it/s, step size=1.81e-01, acc. prob=0.812]Sample: 96%|█████████▌| 2881/3000 [00:47, 75.18it/s, step size=1.81e-01, acc. prob=0.812]Sample: 96%|█████████▋| 2889/3000 [00:47, 69.11it/s, step size=1.81e-01, acc. prob=0.813]Sample: 97%|█████████▋| 2898/3000 [00:47, 73.20it/s, step size=1.81e-01, acc. prob=0.812]Sample: 97%|█████████▋| 2906/3000 [00:47, 71.31it/s, step size=1.81e-01, acc. prob=0.812]Sample: 97%|█████████▋| 2914/3000 [00:47, 67.69it/s, step size=1.81e-01, acc. prob=0.812]Sample: 97%|█████████▋| 2922/3000 [00:47, 69.80it/s, step size=1.81e-01, acc. prob=0.812]Sample: 98%|█████████▊| 2930/3000 [00:47, 69.52it/s, step size=1.81e-01, acc. prob=0.812]Sample: 98%|█████████▊| 2938/3000 [00:48, 71.33it/s, step size=1.81e-01, acc. prob=0.812]Sample: 98%|█████████▊| 2946/3000 [00:48, 66.70it/s, step size=1.81e-01, acc. prob=0.811]Sample: 98%|█████████▊| 2954/3000 [00:48, 66.65it/s, step size=1.81e-01, acc. prob=0.812]Sample: 99%|█████████▊| 2962/3000 [00:48, 69.24it/s, step size=1.81e-01, acc. prob=0.812]Sample: 99%|█████████▉| 2969/3000 [00:48, 69.07it/s, step size=1.81e-01, acc. prob=0.812]Sample: 99%|█████████▉| 2977/3000 [00:48, 70.88it/s, step size=1.81e-01, acc. prob=0.812]Sample: 100%|█████████▉| 2987/3000 [00:48, 75.06it/s, step size=1.81e-01, acc. prob=0.812]Sample: 100%|█████████▉| 2995/3000 [00:48, 72.42it/s, step size=1.81e-01, acc. prob=0.812]Sample: 100%|██████████| 3000/3000 [00:48, 61.29it/s, step size=1.81e-01, acc. prob=0.812]After running the analysis, we can extract the posterior distributions for our beta parameter in each model. Let’s see what they tell us.
# Get posterior samples and print results
clip_beta_mean = clip_samples['beta'].mean().item()
clip_beta_hdi = torch.quantile(clip_samples['beta'], torch.tensor([0.025, 0.975]))
print(f"\nCLIP Similarity - Bayesian Regression:")
print(f" Beta (VerbType effect): {clip_beta_mean:.3f}")
print(f" 95% HDI: [{clip_beta_hdi[0]:.3f}, {clip_beta_hdi[1]:.3f}]")
print(f" P(beta < 0): {(clip_samples['beta'] < 0).float().mean():.3f}")
subject_beta_mean = subject_samples['beta'].mean().item()
subject_beta_hdi = torch.quantile(subject_samples['beta'], torch.tensor([0.025, 0.975]))
print(f"\nSubject Salience - Bayesian Regression:")
print(f" Beta (VerbType effect): {subject_beta_mean:.3f}")
print(f" 95% HDI: [{subject_beta_hdi[0]:.3f}, {subject_beta_hdi[1]:.3f}]")
print(f" P(beta < 0): {(subject_samples['beta'] < 0).float().mean():.3f}")
vlm_beta_mean = vlm_samples['beta'].mean().item()
vlm_beta_hdi = torch.quantile(vlm_samples['beta'], torch.tensor([0.025, 0.975]))
print(f"\nVLM Score - Ordered Logistic Regression:")
print(f" Beta (VerbType effect): {vlm_beta_mean:.3f}")
print(f" 95% HDI: [{vlm_beta_hdi[0]:.3f}, {vlm_beta_hdi[1]:.3f}]")
print(f" P(beta < 0): {(vlm_samples['beta'] < 0).float().mean():.3f}")
CLIP Similarity - Bayesian Regression:
Beta (VerbType effect): -2.305
95% HDI: [-5.213, 0.613]
P(beta < 0): 0.942
Subject Salience - Bayesian Regression:
Beta (VerbType effect): -1.428
95% HDI: [-4.651, 1.908]
P(beta < 0): 0.812
VLM Score - Ordered Logistic Regression:
Beta (VerbType effect): -2.293
95% HDI: [-4.128, -0.599]
P(beta < 0): 0.994The plot below visualizes the results of our Bayesian models. For each of our three metrics, it shows the estimated effect of the verb type. Because we coded unaccusatives as +0.5 and unergatives as -0.5, the “Beta” (β) value represents the difference between the two.
The vertical gray line at zero is our baseline for “no effect”. If the colored whisker line (the posterior distribution) for a metric crosses this line, it means we can’t be very confident in the direction of the effect. However, this ‘confidence’ of course will be quantified by the probability values shown on the right side of the plot.
If the distribution is shifted to the left of zero (Negative β), it means that unaccusative scenes scored lower than unergative ones for that metric. This is the “danger zone” for our stimuli, as it would suggest they are harder for the models to process. As you can see, both the full sentence similarity and the VLM verification scores are shifted heavily to the left, indicating that the unaccusative pictures are indeed less clear or representative.
If the distribution were on the right side (Positive β), it would mean unaccusatives scored higher, suggesting they were actually easier for the models. None of our metrics show this result.
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
# Data dictionary from your MCMC samples
beta_data = {
'Full Scene (CLIP)': clip_samples['beta'].numpy(),
'Subject Salience (CLIP)': subject_samples['beta'].numpy(),
'Scene Verification (VLM)': vlm_samples['beta'].numpy()
}
# Adjust figure size for better vertical separation
fig, ax = plt.subplots(figsize=(7, 3))
sns.set_style("whitegrid", {'axes.grid': True, 'grid.color': '.95'})
labels = list(beta_data.keys())
colors = ['#3498db', '#9b59b6', '#e74c3c']
for i, label in enumerate(labels):
samples = beta_data[label]
mean_val = samples.mean()
# 1. Calculate multiple intervals for the "stacking" effect
hdi_95 = np.percentile(samples, [2.5, 97.5])
hdi_80 = np.percentile(samples, [10, 90])
hdi_50 = np.percentile(samples, [25, 75])
# 2. Plot the stacked lines (Bottom to Top: thinnest/widest first)
# 95% Interval - Thin
ax.hlines(i, hdi_95[0], hdi_95[1], color=colors[i], linewidth=1.5, alpha=0.4, zorder=1)
# 80% Interval - Medium
ax.hlines(i, hdi_80[0], hdi_80[1], color=colors[i], linewidth=5.0, alpha=0.7, zorder=2)
# 50% Interval - Thick
ax.hlines(i, hdi_50[0], hdi_50[1], color=colors[i], linewidth=10.0, alpha=1.0, zorder=3)
# 3. Plot the Mean point
ax.plot(mean_val, i, 'o', color='white', markersize=8, zorder=4)
# 4. Perfectly Aligned Statistics
p_dir = (samples < 0).mean() if mean_val < 0 else (samples > 0).mean()
prob_text = f"$P(\\beta {'<' if mean_val < 0 else '>' } 0) = {p_dir:.2f}$"
# Locked to y-coordinate 'i' and x-coordinate 3.0 (outside plot area)
ax.text(3.0, i, prob_text, va='center', ha='left',
fontsize=13, fontweight='bold', color=colors[i])
# 5. Descriptive Annotations (The "How to Read" Guide)
ax.axvline(x=0, color='black', linestyle='-', linewidth=1.5, alpha=0.6, zorder=0)
# Arrow pointing Left (Negative Beta)
ax.annotate('', xy=(-5, -1.0), xytext=(-0.5, -1.0),
arrowprops=dict(arrowstyle="->", color='gray', lw=1.5))
ax.text(-2.75, -1.4, "Lower Scores for\nUnaccusatives", ha='center', color='gray', fontweight='bold')
# Arrow pointing Right (Positive Beta)
ax.annotate('', xy=(2.5, -1.0), xytext=(0.5, -1.0),
arrowprops=dict(arrowstyle="->", color='gray', lw=1.5))
ax.text(1.5, -1.4, "Lower Scores for\nUnergatives", ha='center', color='gray', fontweight='bold')
# 6. Final Layout Polish
ax.set_yticks(range(len(labels)))
ax.set_yticklabels(labels, fontweight='bold', fontsize=12)
ax.set_xlabel('Posterior Beta Weight (Unaccusative vs. Unergative)', fontsize=13, labelpad=45)
# Lock limits so text and arrows don't shift
ax.set_xlim(-6, 3)
ax.set_ylim(-1.5, len(labels) - 0.5)
sns.despine(left=True, bottom=True)
plt.subplots_adjust(right=0.75, bottom=0.2) # Make room for text on right and guide on bottom
plt.savefig('./model_pyro.png', dpi=300, bbox_inches='tight')
plt.show()
So, what do these models tell us? The results are quite clear: all three of our metrics point in the same direction, suggesting that the unaccusative scenes are in some way more challenging than the unergative ones.
Here’s a quick summary of the findings from our Bayesian analysis:
- Scene Verification (VLM): A strong negative effect (β ≈ -2.3, P(β<0) = 0.99). The VLM is very confident that the unaccusative sentences are a worse description of their corresponding images.
- Full Scene (CLIP): A similarly strong negative effect (β ≈ -2.3, P(β<0) = 0.92). CLIP also finds a lower visual-textual fit for unaccusative scenes.
- Subject Salience (CLIP): A moderate negative effect (β ≈ -1.5, P(β<0) = 0.83). The evidence is weaker here, with more uncertainty, but it still suggests that the subject is slightly harder to identify in unaccusative scenes.
Even before we get to the human data, the models are sending a clear signal: these two sets of pictures might not be perceived equally. The unaccusative scenes seem to have a lower “visual-textual fit”, which means we have to be careful not to mistake this perceptual difficulty for a purely linguistic effect.
One interpretation is that the production effects we see in humans might be at least partially due to the visual complexity of the pictures. However, given the broader context of the sentence production literature, this seems unlikely to be the whole story. Other studies have found similar advance planning effects in experiments that did not involve these pictures or pictures at all, such as sentence recall tasks. This suggests that the effect is not just about visual processing, but is tied to the linguistic structure of the sentences themselves.
Conclusion
The Finding
The analysis reveals a consistent pattern across all three metrics: unaccusative scenes are rated as more difficult or less representative by the models compared to unergative scenes.
Scene Verification (VLM): The Qwen-VL model, which was asked to explicitly rate the match between the sentence and the image, showed a strong negative effect for unaccusatives. It consistently gave lower scores to unaccusative pairs, with a high degree of certainty (P(β<0) = 0.99). This suggests that from a generative, “common sense” perspective, the unaccusative sentences are poorer descriptions of their corresponding images.
Full Scene Similarity (CLIP): The standard CLIP similarity score also revealed a strong negative effect for unaccusatives (P(β<0) = 0.92). This indicates that the overall visual-textual fit is lower for unaccusative scenes.
Subject Salience (CLIP): Even the salience of the subject noun was moderately lower in unaccusative scenes (P(β<0) = 0.83). While the evidence is weaker here, it suggests that the subject may be slightly harder to identify in the context of an unaccusative event.
In short, the models are telling us that the unaccusative pictures are not as clear-cut as the unergative ones.
What This Means
This computational analysis provides a crucial piece of context for the human experimental results. The key takeaway is that the unaccusative stimuli seem to be inherently more complex or ambiguous than the unergative stimuli.
This doesn’t invalidate the syntactic hypothesis about advance planning, but it does add a layer of nuance. The increased processing cost observed in human speakers for unaccusative sentences might not be solely due to a syntactic operation. Instead, it could be a combination of factors:
- Perceptual/Conceptual Difficulty: The visual scenes for unaccusative events might be harder to parse, conceptualize, and map onto a linguistic description. The AI models, particularly the VLM, seem to be picking up on this.
- Syntactic Planning: The syntactic structure of unaccusatives may still require earlier planning, as originally hypothesized.
The most likely scenario is that these two factors are intertwined. The very nature of unaccusative events (a change of state happening to a patient) makes them visually more complex, and this complexity might be what triggers the earlier, more resource-intensive syntactic planning.
We can be more confident that the experimental effects are not just due to simple visual confounds like a hidden subject, but we must also acknowledge that the “difficulty” is not purely syntactic. It’s a property of the entire event, from perception to syntax.
Final Thoughts
If you’re running experiments with visual stimuli, I highly recommend giving this a try. Vision-language models like CLIP and multimodal LLMs like Qwen are freely available and give us principled ways to ask: “Are these pictures doing what we think they’re doing?” The ability to triangulate across different model architectures—from similarity-based (CLIP) to generative (Qwen)—provides an unprecedented level of confidence in our experimental materials.
This analysis didn’t end up changing my theoretical interpretation of the experimental findings, I think existing evidence for advance planning require more than random LLM analysis, which probably has nothing to do how humans represent concepts, sentences, or visual areas.
One thing that I need to do a baseline analysis in which I ran the similar analysis for a randomly assigned sentence-picture pairs to show that, when randomized the clip similarity changes dramatically. I will soon run that and upload the results as a comment.
If you want to run this analysis yourself, you can use the following Colab notebook: 
You can download the cached_scores here: cached_scores.csv, and the pictures here: pictures.zip
References
Momma, S., & Ferreira, V. (2019). Beyond linear order: The role of argument structure in speaking. Cognitive Psychology, 114, 101228.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., … & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning (pp. 8748-8763). PMLR.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., … & Zhou, J. (2023). Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.
Session Info
For reproducibility, here’s my setup:
import sys
print(f"Python: {sys.version}")
print(f"PyTorch: {torch.__version__}")
print(f"CLIP: (installed from https://github.com/openai/CLIP)")
print(f"Transformers: (for Qwen-VL-Chat)")Python: 3.13.11 (main, Dec 5 2025, 16:06:33) [Clang 17.0.0 (clang-1700.4.4.1)]
PyTorch: 2.9.1
CLIP: (installed from https://github.com/openai/CLIP)
Transformers: (for Qwen-VL-Chat)Footnotes
This would not be possible without Alper teaching me about these models.↩︎
However, an interesting sidenote is that we do not really know if human cognition is also propositional.↩︎
It works very slowly because they are extremely resource hungry. The reason this post waited this much was because I was waiting for results to come in.↩︎
There are of course other ways to test this. For example Griffin & Bock (2000) used a free-production task where participants were not given an initial word to use with the pictures. They quantified how many different words they used for each picture and named that variable ‘codability’ of the picture and tested if codability was related to onset latency. Egurtzegi et al. (2022) used a similar approach.↩︎