betaFeedback

WorksComputer Science

Grip, Trace, Write: Force-Aware Imitation and Reinforcement Learning for High-DoF Robotic Handwriting

Abstract

Most robotic handwriting rigidly mounts the pen and reduces writing to trajectory tracking. A freely held pen instead couples grasp stability, tip motion, and contact force. We present Grip, Trace, Write, a force-aware simulation environment and learning framework for high-DoF handwriting. A two-branch actor separates pen retention from arm-wrist writing while preserving their closed-loop coupling. Training learns a grasp controller, initializes writing from inverse-kinematics demonstrations by behavior cloning, and improves writing through reinforcement learning with rewards on ink, force, time, and retention. Across diverse tasks, the policy improves completion, precision, force regulation, and efficiency over behavior cloning and inverse kinematics, and generalizes without fine-tuning to unseen English and Chinese content. These results establish force-aware handwriting as a testbed for dexterous tool use.

Introduction

Writing with a freely held pen is a compact testbed for dexterous tool use. It requires a hand to stabilize a slender object through a precision grip while the arm and wrist transport its tip and regulate its contact with a surface. Human manual dexterity similarly combines stable multi-finger grasping, fine motion, and force-sensitive interaction with tools [1].

This coupling makes handwriting fundamentally different from moving a rigidly mounted pen along a geometric path. A desired tip trajectory alone cannot determine whether ink is deposited: insufficient force produces hovering or broken strokes, whereas excessive force can distort the trace, alter the pen pose, or destabilize the grasp. With a freely held pen, finger–pen contact, pen–sheet contact, and arm motion therefore affect one another. Existing systems commonly avoid this coupling by rigidly mounting the pen or brush to an arm [2, 3, 4, 5], or address only limited writing behaviors with multi-fingered hands [6, 7]. They typically treat writing as trajectory tracking [8], with contact force uncontrolled or maintained around a fixed setpoint, and evaluate only a small set of strokes or glyphs [9, 7]. Consequently, sustained force-regulated writing with a freely held pen and a high-DoF hand remains unresolved.

We address this problem in a force-aware simulation environment for robotic handwriting. A learned grasp MLP first establishes a stable hand–pen interface. Given a requested 5D cursor, an inverse-kinematics (IK) controller generates nominal arm–wrist commands while the learned grasp holds the pen; these rollouts initialize a trajectory-conditioned writing policy through behavior cloning (BC). BC provides broad coverage of the writing repertoire but only imitates nominal IK actions. We therefore improve writing through RL with rewards defined on task outcomes: ordered ink coverage, off-target ink, force tracking, elapsed time, and pen retention. This separation makes the roles explicit: grasp learning resolves tool stabilization, IK-guided BC makes high-DoF writing tractable, and RL corrects contact-induced errors that action imitation does not optimize.

Our contributions are: (1) a force-aware simulation environment for high-DoF robotic handwriting with a freely held pen, contact dynamics, and diverse tracing and writing tasks; (2) a two-branch controller that separates pen retention from arm–wrist writing while retaining their closed-loop physical coupling; and (3) a grasp-to-writing learning procedure that uses IK-guided BC for broad initialization and task-outcome RL to improve trace completion, ink precision, contact-force accuracy, and execution efficiency.

Related work. Prior robot-writing and drawing systems typically rigidly mount a pen or brush to an arm, reducing the problem to end-effector trajectory generation or ink modeling [2, 3, 4, 5]. Writing with a pen held in a multi-fingered hand remains rare and limited to single strokes or simple shapes, using synergies [6], online Jacobian estimation [8], or tactile RL with compliant wrist control [7]. Learned in-hand policies reorient and spin objects without external contact [10, 11, 12], track a held stick’s tip without touching a surface [9], or bring a held tool into contact without regulating force [13]. Initializing from demonstrations and refining with RL is effective for dexterous control [14, 15]; we apply it to the full writing repertoire with outcome rewards on ink, force, and pen retention.

Method

Environment and control setup

Our MuJoCo setup [16] mounts a high-DoF anthropomorphic hand (Figure 1(a)(Figure 1b)) on a UR5e and places a free pen in a tripod grasp without virtual assistance. The hand has 25 finger joints, in addition to 2 wrist and 7 arm joints. The suite combines three retention conditions (static, moving, and contact scribbling), three tracing families (line, circle, and varied shapes), and handwriting of digits, capital letters, English words, and Chinese characters. Training covers all 10 digits, 26 capital letters, 15 English words, and 118 Chinese characters with several fonts. Every episode resamples placement, scale, slant, rotation, and target-force profile. Zero-shot tests add 20 held-out English words, 10 four-character Chinese idioms, and 122 Chinese characters with at least seven strokes. Appendix § 7 gives complete task and interface definitions.

Hand kinematics and two-branch actor

Figure 1. Hand kinematics and two-branch actor. (a)(b) Spheres mark the finger joints, colored by finger, with rods along the hinge axes; the skin is rendered translucent. The hand has 25 finger joints, in addition to 2 wrist and 7 arm joints. (c) Actor model reads both the same state and trajectory. A fixed output mask selects finger actions from the grasp branch and UR5e arm, wrist, and stop actions from the writing branch. The branches interact through the held pen and live state.

Two-branch policy

The actor has separate grasp and writing MLPs (Figure 1(c)). A fixed mask takes finger actions from the grasp trunk and arm, wrist, and stop actions from the writing trunk. This split follows the structure of the problem: pen retention can be learned from holding interactions before any glyph is specified, whereas tip transport can use IK once a stable grasp exists. Both trunks still read the same live state, so the mask separates supervision and action roles without breaking the physical coupling among hand, pen, and arm. The transferred grasp controller is folded into the new input representation; its preservation during final RL is described below.

Training workflow

Grasp BC and RL. Stable pen retention is a prerequisite for meaningful handwriting learning. When the pen slips, missing or misplaced ink can arise from either failed finger contact or incorrect arm motion, so the writing objective does not clearly identify which subsystem should change. We therefore learn the grasp branch before the writer. BC on tripod-hold demonstrations quickly initializes a valid multi-finger contact configuration; pen-hold RL then improves retention under static, arm-driven, and environment-driven motion, covering contact disturbances absent from the demonstrations. Transferring this controller gives subsequent IK data collection and writing optimization a consistent, physically stable hand–pen interface.

All-task writing BC. Once the grasp is reliable, the remaining problem is to map a requested trajectory to arm and wrist motion under that particular hand–pen contact. We therefore run IK while the transferred grasp controls the fingers. This produces demonstrations in the same compliant, freely held system that the learned writer will encounter, avoiding a mismatch with data collected using an idealized fixed pen. Clean rollouts teach nominal motion; perturbing the executed action moves the system into nearby off-nominal states while retaining the clean IK command as the recovery target [17]. Because traces, glyphs, words, and Chinese writing are all presented through the same trajectory interface, we pool them to learn a geometry-conditioned controller rather than separate content-specific policies. The writing branch minimizes

LBC=E(x,a∗)∑d=19wd(μd(x)−ad∗sd)2,wd=clip⁡[(median⁡jσjmax⁡(σd,10−3))2,1,100].(1)\mathcal{L}_{\mathrm{BC}}=\mathbb{E}_{(x,a^*)}\sum_{d=1}^{9}w_d\left(\mu_d(x)-\frac{a_d^*}{s_d}\right)^2,\qquad w_d=\operatorname{clip}\left[\left(\frac{\operatorname{median}_j\sigma_j}{\max(\sigma_d,10^{-3})}\right)^2,1,100\right]. \tag*{(1)}

where sds_d scales action dimension dd and σd\sigma_d is its label standard deviation. Inverse-variance weighting balances coordinates with different natural ranges, and fitting the stop output on the clone’s own rollouts teaches termination under its induced state distribution. This stage gives the complete system a broad, stable writing initialization, but it still optimizes agreement with IK rather than the physical trace.

Full-task outcome RL. BC solves initialization, but leaves two gaps: matching IK commands does not directly optimize ink and force, and small execution errors can carry the clone beyond the demonstrated states. We therefore refine the composed controller with PPO on the complete task mixture. The grasp mean remains frozen so drawing reward cannot trade away the reliable contact strategy, and frozen observation statistics prevent normalization drift from changing the transferred policy. The writing mean and Gaussian exploration parameters remain trainable, while both branches stay closed-loop through their shared state and the physical pen. Let CtC_t be ordered coverage within 1 mm, PtP_t normalized off-target ink, EtE_t running force MAE, τt\tau_t normalized elapsed time, ptp_t the pen tip, and qtq_t the first unfinished target point at pressing depth. The outcome reward is

rt=λCΔCt−λPΔPt−λEΔEt−λτΔτt+λq(∥pt−1−qt∥−∥pt−qt∥)−Dt,(2)r_t = \lambda_C \Delta C_t - \lambda_P \Delta P_t - \lambda_E \Delta E_t - \lambda_\tau\Delta\tau_t + \lambda_q \left(\lVert p_{t-1} - q_t \rVert- \lVert p_t - q_t \rVert\right) - D_t, \tag*{(2)}

where all λ\lambda weights are positive and DtD_t penalizes drops and premature termination. Each term targets a distinct failure mode: ordered coverage rewards forward stroke progress, off-target ink rejects indiscriminate scribbling, and the pressing-depth point qtq_t prevents progress by hovering above the sheet. Force and time distinguish meaningful contact from mere touching and favor efficient completion, while DtD_t preserves the pen-retention prerequisite established in the first stage. Crucially, there is no BC or IK-timing term, so IK provides an initialization rather than a performance ceiling. Appendix § 6.§ 2 gives the numerical weights, penalty construction, PPO schedule, and optimization details.

Experiments

Protocol. We compare IK, all-task BC, and final RL on eight primary tasks with 50 matched seeds each. Policies execute deterministic means and their own stop outputs. We report ordered completion and precision within 1 mm, force MAE, time, shape distances, and pen drops; generalization tests use unseen English and Chinese content without fine-tuning.

Writing performance

Table 1 shows that RL raises mean completion to 94.8%, from 91.3% for BC and 92.0% for IK. BC itself is below IK in both completion and precision, confirming its role as an initialization rather than the final controller. Relative to BC, RL raises precision from 79.0% to 87.6%, reduces force MAE by 53% from 0.773 to 0.362 N, and reduces time from 8.19 to 7.67 s. The IK and BC traces exhibit visible stroke jitter, which leaves ink outside the target and depresses precision; outcome RL suppresses this effect (Appendix F, [6]). RL improves completion on seven of eight primary tasks and precision and force error on all eight. Its gains on unseen English words, Chinese idioms, and Chinese characters show execution of new requested trajectories rather than memorization. Appendix § 8 provides complete metrics and diagnostics.

ControllerMetricLineCircleShapeDigitsLettersWordCursiveChinese wordIdiomChineseMean
IKCompletion (%) ↑98.298.894.990.692.785.483.491.787.792.187.792.0
Precision (%) ↑95.796.482.675.882.271.561.678.771.182.171.780.6
Force MAE (N) ↓0.5580.5370.5290.5310.5330.5820.5260.4970.5850.5800.5240.537
Time (s) ↓3.164.477.385.055.9122.278.488.1918.9627.7112.448.11
BCCompletion (%) ↑96.797.693.690.292.585.782.491.587.291.686.991.3
Precision (%) ↑93.493.681.174.280.871.460.677.070.279.870.379.0
Force MAE (N) ↓0.5900.7630.7070.8660.8010.9430.7530.7640.7920.6050.8560.773
Time (s) ↓3.104.547.265.266.0122.318.598.4518.8428.4812.528.19
RLCompletion (%) ↑95.599.797.195.095.889.886.896.491.692.987.394.8
Precision (%) ↑96.699.090.686.589.380.369.689.380.989.483.087.6
Force MAE (N) ↓0.3100.3250.3690.3480.3800.4380.3960.3310.4330.4540.4020.362
Time (s) ↓3.134.737.094.665.6520.587.727.7817.4827.8411.287.67

Table 1. Task-level and zero-shot results with 50 matched seeds per task. Completion, precision, force MAE, and time are reported on separate rows. The mean covers the eight primary tasks; the three unseen columns are zero-shot evaluations. Bold marks the best controller for each task and metric.

Conclusion

We present a force-aware grasp-to-writing system. A transferred grasp trunk retains the pen, all-task IK-guided BC initializes writing, and outcome RL refines the controller. RL improves completion, precision, force, and time over BC and IK, including on unseen English and Chinese content.

References

Implementation and evaluation details

Reading the supplementary material. This appendix provides implementation details, task definitions, training settings, ablations, and extended results for the reported trajectory-conditioned system.

Figure 3 shows the simulated scene. Hand and environment settings are listed in Table 2; network, training, and evaluation settings are listed separately in Table 3. The physical control contract is therefore distinguished from choices made by the learning algorithm.

ParameterHand and environment configuration
Hand and armHigh-DoF anthropomorphic robotic hand mounted on a UR5e assembly
Actuated coordinatesSeven arm coordinates, two wrist joints, and 25 finger joints
Physical action interfaceSeven integrated arm-position increments; 27 speed–torque-limited wrist/finger duties
Control frequency25 Hz
Tool and initial graspFree rigid-body pen, initially seated in a tripod grasp; retained through contact
Grip assistanceNo virtual grip assistance in the reported writing configuration
Starting height2–6 mm in the reported writing configuration
Trajectory advancementTip-driven pointer; pen-down proximity tolerance 1.5 mm; skip timeout 0.25 s
Task setLine, circle, shape, digits, letters, word, cursive, and Chinese writing
Content splitShared training/evaluation content pool; primary numerical comparison uses the eight-task suite

Table 2. Hand, environment, and task-interface settings for the reported writing study. These specify the physical control problem; model architecture and optimization settings are given separately in Table 3.

Simulated setup: UR5e arm, tripod grasp, and sheet with ink written so far

Figure 3. Simulated setup. (a) A UR5e arm with a 2-DoF wrist and a 25-DoF anthropomorphic hand writes on paper with a free pen. (b) The tripod grasp. (c) Top view mid-episode; gray marks the remaining target.

ParameterModel, training, and evaluation configuration
Model architecture and observation processing
Input representation137 state entries plus 16 trajectory samples of five entries each; samples every two trajectory indices; no image encoder
ActorSeparate grasp and writing MLPs, each 217 →\to 256 →\to 256 with tanh activations; masked linear heads
Policy output34 physical-action entries plus one learned stop output; stop threshold 0.5
CriticIndependent 217 →\to 256 →\to 256 →\to 1 MLP
NormalizationVector statistics frozen during final RL; normalized observations clipped to [−10,10][-10, 10]
Behavior cloning
Demonstration noiseClean and noisy demonstrations, σ=0.01,0.02\sigma= 0.01, 0.02
BC objectiveArm/wrist loss with inverse-variance weighting
BC optimizerAdam, learning rate 3×10−43 \times10^{-4}, batch 256, 60 epochs per candidate
Reinforcement learning
Grasp preservationGrasp-mean parameters frozen; Gaussian log-standard-deviations remain trainable
Rollout collection512 environments ×\times 1024 control steps; asynchronous execution
Actor updatePPO clip 0.2; full-rollout gradient accumulation; up to four passes; approximate KL target 0.03 on 16,384 randomly sampled transitions
Optimizer and returnsAdam 10−510^{-5} with linear decay; minibatch 2048; γ=1\gamma= 1, GAE λ=0.95\lambda= 0.95; reward scale 0.001
Critic updateThree initial critic-only iterations, followed by two critic passes per rollout
Initial explorationLog-standard-deviations: arm/wrist −4.6-4.6, fingers −3.9-3.9, stop −3.0-3.0; floor −6.0-6.0
Recorded budget39,845,888 transitions; 193.8 minutes for the reported final RL run
Evaluation protocol
Policy executionDeterministic learned mean and the network’s own stop output
Evaluation samplePrimary comparison: 50 seeds ×\times eight tasks

Table 3. Model and learning settings, separated from the hand/environment parameters in Table 2. BC settings describe the available collection/training recipe, not a reconstructed per-shard manifest of the final clone. Evaluation settings specify measurement rather than training updates.

Data and checkpoint provenance. The main comparison uses an eight-task evaluation with 50 episodes with different seeds per task for IK, all-task BC, and final RL. Intermediate evaluations in the learning plot are snapshots of the same final RL run.

Evaluation-suite changes. The development code merged the original content pools, and new evaluation-only word, idiom, and Chinese-character pools were constructed for the zero-shot tests. A later script-font update and the folding of cursive into the word task change the distribution relative to the primary eight-task evaluation.

Training and checkpoint details

The reported handwriting method transfers a learned grasp, collects IK writing demonstrations over the complete repertoire, fits one BC policy on the pooled data, and refines the composed policy with full-task outcome RL. Figure 2 shows this separation.

Figure 2. Training pipeline. Panels 1–3 correspond to the three stages in § 2.3: grasp BC and hold RL, all-task writing BC with the learned grasp, and full-task outcome RL on the composed policy.

ComponentTraining conditionTraining and transfer
Grasp controller240 teacher episodes for BC, then pen-hold PPO under motion and contactTransfer the selected learned grasp into the MLP
Writing data and BCTraces, glyphs, words/cursive, and Chinese togetherIK arm/wrist plus learned grasp supplies clean/noisy demonstrations; fit one all-task clone
Composed-policy RLFull writing mixture with grasp and contact dynamics activeBoth branches execute; refine outcomes while retaining grasp-mean weights

Table 4. Transferred grip controller, all-task writing demonstration collection and BC, and full-task composed-policy RL.

Recorded grasp lineage. The reported handwriting checkpoint transfers an existing grasp policy initialized from 240 teacher episodes and followed by 8M PPO steps over static hold, moving hold, and environment-driven scribble. The later trajectory-based implementation folds the blank-map encoder contribution and constant input entries into MLP biases.

All-task writing BC. The clone is fit once on the complete pool of traces, glyphs, words, cursive, and Chinese characters, and initializes final RL. IK supplies demonstrations but does not act during learned-policy evaluation. Figure 4 shows training progress for each stage.

Figure 4. Training across the pipeline. (a) Pen-hold RL from the BC grasp: fraction of training episodes that keep the pen (thin: per iteration; thick: 15-iteration mean) and 24-seed evaluations over static hold, moving hold, and scribble. (b) Per-task completion of all-task BC and its IK teacher (50 seeds). (c, d) Writing RL completion and force MAE over environment steps; BC at step 0, IK dashed.

Outcome-based RL for writing. BC provides a broad initialization, but matching IK commands does not directly optimize the written trace or contact force. We therefore refine the writing branch with PPO while executing the full grasp–pen–arm system.

Detailed policy, imitation, and RL implementation

Trajectory-conditioned control. The input comprises 137 retained state entries (proprioception, tactile feedback, task and geometric state) and 16 future trajectory samples. Each sample contains tip-relative displacement, relative height, requested force, and nominal time offset. Samples are spaced by two trajectory indices. A tip-driven pointer advances through the request with a 1.5 mm proximity tolerance and a 0.25 s skip timeout; the request therefore does not advance solely according to wall-clock time. Pen-down input waypoints specify the sheet surface, rather than exposing IK’s below-surface pressing offset. The policy thus receives explicit task geometry.

Separate grasp and writing features. Let xt∈R217x_t \in\mathbb{R}^{217} be the normalized input and mm select the arm, wrist, and stop dimensions. The Gaussian mean is

μ(xt)=(1−m)⊙gθg(xt)+m⊙wθw(xt).(3)\mu(x_t) = (1-m) \odot g_{\theta_g}(x_t) + m \odot w_{\theta_w}(x_t). \tag*{(3)}

The grasp branch inherits the learned holding controller; the writing branch is initialized through imitation. The selected outputs form 34 physical controls and one stop entry, requesting termination above 0.5. A separate 217→256→256→1217 \to256 \to256 \to1 critic has the same hidden activations. The implementation contains 383,082 parameters including inactive head rows and Gaussian log-standard-deviations. During RL, grasp-mean parameters and observation statistics remain fixed; log-standard-deviations remain trainable, including those for fingers.

Imitation initialization. IK supplies arm/wrist commands while a learned holding policy supplies finger commands. The collection recipe gathers clean rollouts and two noise-injected sets (σ=0.01,0.02\sigma= 0.01, 0.02 in executed action units), retaining clean IK labels. We use DART-style noise injection [17] with fixed variance. The delivered arm/wrist clone trains on the pooled demonstrations from all tasks and uses a weighted squared-error loss,

LBC=E(x,a∗)∑d=19wd(μd(x)−ad∗/sd)2,wd=clip⁡[(median⁡jσjmax⁡(σd,10−3))2,1,100],(4)\mathcal{L}_{\mathrm{BC}} = \mathbb{E}_{(x,a^*)} \sum_{d=1}^{9} w_d\left(\mu_d(x)-a_d^*/s_d\right)^2,\qquad w_d = \operatorname{clip}\left[\left(\frac{\operatorname{median}_j \sigma_j}{\max(\sigma_d,10^{-3})}\right)^2,1,100\right], \tag*{(4)}

where sds_d is the action scale and σd\sigma_d the demonstration-label standard deviation. The recorded recipe uses Adam, learning rate 3×10−43 \times10^{-4}, batch 256, and 60 epochs per candidate. The writing branch is merged with the grasp branch and the stop output is fitted on rollouts. The resulting all-task clone is the BC baseline below.

Rewarding drawing outcomes. The final reward configuration uses

rt=λCΔCt−λPΔPt−λEΔEt−λτΔτt+λq(∥pt−1−qt∥−∥pt−qt∥)−Dt.(5)r_t = \lambda_C \Delta C_t - \lambda_P \Delta P_t - \lambda_E \Delta E_t - \lambda_\tau\Delta\tau_t + \lambda_q\left(\lVert p_{t-1}-q_t\rVert-\lVert p_t-q_t\rVert\right)-D_t. \tag*{(5)}

Here CtC_t is online ordered target coverage within 1 mm; PtP_t accumulates off-target ink normalized

WeightValueReward term
λC\lambda_C1000Ordered target coverage
λP\lambda_P500Off-target ink
λE\lambda_E200Contact-force MAE
λτ\lambda_\tau300Normalized elapsed time
λq\lambda_q5000Progress toward the guidance point

Table 5. Weights in the final drawing-outcome reward.

by expected pen-down samples, capped at 2; EtE_t is running contact-force MAE; and Δτt\Delta\tau_t is elapsed time divided by nominal task duration. The guidance point qtq_t is the first uncovered glyph point at or after the coverage pointer, placed 1.5 mm below the sheet. The same qtq_t is used on both sides of the distance difference. DtD_t accounts for revoking accumulated coverage reward on a drop and charging remaining time for specified premature terminations. Coverage is an online surrogate for final alignment; RL has no IK-action or timing loss.

Stable policy refinement. We optimize the PPO clipped objective [18] using gradients accumulated over each full 512×1024512 \times1024 rollout. Up to four actor passes use Adam at 10−510^{-5} with linear decay. Proposed parameter displacements are rescaled if approximate KL on 16,384 randomly sampled rollout transitions exceeds 0.03; this is an empirical budget, not a hard KL guarantee. The critic receives two minibatch passes (batch 2048), after three initial critic-only iterations. We use γ=1\gamma= 1, GAE λ=0.95\lambda= 0.95, clip 0.2, reward scale 0.001, and initial arm/wrist noise exp⁡(−4.6)≈0.01\exp(-4.6) \approx0.01. The recorded run collects 39.85M transitions in 193.8 minutes.

Task catalogue, force-annotated strokes, and the 5D cursor

This section describes the task interface used by the reported system. It distinguishes the three holding tasks used for grasp learning from the seven writing tasks in the current suite and the three evaluation-only generalization tasks. The primary comparison used eight writing tasks, with cursive separate from word. Task geometry, target force, and trajectory timing jointly define the requested behavior. The current source catalogue is not a claim that archived checkpoints were retrained on every subsequent task change.

Tasks used in grasp and writing training

Task IDNominal durationRequested behavior and configuration
L0: grasp training; arm/wrist motion supplied by the environment
L0_hold_static6 sRetain the seated pen while the hand is still; no target ink strokes.
L0_hold_moving8 sRetain the pen during imposed transport and wrist disturbance: 11 waypoints, motion scale 2.2, arm-motion box half-extents (0.12, 0.08, 0.085) m, wrist shake amplitude 4°.
L0_scribble10 sRetain the pen during an environment-driven paper-contact path: 14 waypoints, 3 mm commanded press depth, 8° wrist wobble, and 15 N impact limit. This is a contact-retention task, not glyph matching.
L1: geometric tracing
L1_trace_line8 sA 60–120 mm straight line with randomized orientation and placement; flat force profile.
L1_trace_circle10 sA circle 40–90 mm in diameter; flat force profile.
L1_trace_shape12 sA sampled shape, 40–90 mm requested height: line, circle, square, triangle, spiral, sine, figure-eight, or zigzag; flat or ramp force profile.
L2: handwriting
L2_write_digits10 sOne digit, 50–65 mm requested height, with a recorded-pressure profile. Script-selected digits use the sans face in the current suite.
L2_write_letters10 sOne capital letter, 50–65 mm requested height, sampled font, and recorded-pressure profile.
L2_write_word(8+3n)(8+3n) sAn nn-character word from the content pool; requested height 20–35 mm, sampled font, slant 0 or 0.3, and recorded-pressure profile.
L2_write_chinese12 sOne Chinese character from the at-most-six-stroke pool, requested height 50–65 mm, and recorded-pressure profile.
E2: evaluation-only content, excluded from the training task list
E2_write_word_unseen(8+3n)(8+3n) sOne of 20 held-out English words, with the word-task geometry and force profile.
E2_write_idiom48 sOne of ten four-character Chinese idioms composed of trained characters; character height 50–65 mm before width fitting.
E2_write_chinese_unseen12 sOne of 122 characters outside the trained pool; every character has at least seven strokes, extending both content and stroke complexity.

Table 6. Task catalogue relevant to this paper, from the current samplers. Dimensions are requested geometric ranges before width fitting. Nominal duration paces the trajectory; it is not necessarily the final elapsed episode time. All listed L1/L2/E2 writing samplers use the pen configuration.

Figure 5 shows the retention tasks and one target per tracing and writing task.

Figure 5. Task suite. Top: the three pen-retention tasks. Bottom: one target per tracing and writing task, drawn to scale with its size; * marks held-out content evaluated zero-shot.

Placement and variation. Stroke templates are normalized, scaled, optionally sheared, rotated, and translated on the sheet. The common placement routine samples rotation in [−0.18,0.18][-0.18,0.18] radians unless overridden, respects a 30 mm margin, and reduces scale when needed to fit the width.

Content, font, geometry, and force variation depend on the particular task; they should not all be assumed independent for every sampler. The current font pool contains block, sans, rounded sans, and script. Current content pools merge earlier training/test pools.

Nominal time versus episode horizon. The duration field sets the nominal trajectory budget. For writing with end_mode=policy, the implementation permits up to twice this nominal duration, with earlier termination possible through the policy’s stop output, dropping, or applicable termination rules. Thus a word’s (8+3n) s(8 + 3n)\,\mathrm{s} configuration should not be interpreted as its measured writing time or as an unconditional hard timeout.

Additional registered tasks. The broader environment also registers finger-only scribbling and pen pickup at L0; brush strokes, Chinese brush writing, and brush layouts at L3; sketch hatching at L4; shape/rainbow/scene painting at L5; and chopstick holding, cylinder, cube, block, and sphere tasks at L6. These are outside the reported writing comparisons. The pen-writing idiom result uses the dedicated evaluation-only E2 sampler rather than the unrelated brush-layout task.

What a task configuration specifies

A task is sampled as a TaskSpec, rather than specified only by its natural-language instruction. Its ordered strokes, force annotations, metadata, and timing determine the actual control request.

FieldType / unitMeaning
task_id, levelIdentifierSelect the task family and level.
instructionTextHuman-readable description. The policy MLP does not encode this string as language.
strokesList of Nj×2N_j \times2 arrays, mOrdered centerline points in paper coordinates relative to the sheet center. List boundaries distinguish pen-down strokes.
forcesList of NjN_j arrays, NRequested pressing force at each corresponding stroke point.
widthsList of NjN_j arrays, mmTarget mark widths derived from the force arrays and the tool’s ink model.
durationSecondsNominal schedule duration used to pace the requested strokes and transits.
tool, startEnumerationsTool identity and seated-grasp versus table-start initialization. The reported writing tasks use a seated pen.
hold_motionMode or nullSelects externally driven static/moving/scribble behavior for holding tasks.
hold_scale, hold_waypointsScale / countSet imposed-motion amplitude and number of waypoints in holding tasks.
hold_box, hold_jitter_degm / degreesArm transport bounds and wrist-disturbance amplitude.
scribble_depth_m, scribble_wrist_degm / degreesCommanded contact depth and wrist wobble for driven scribble.
impact_limit_NNContact-impact limit used by the relevant task’s success criterion.
metaDictionarySampled text, font, slant, shape, or other task-specific metadata.

Table 7. Task-definition fields. Reward weights, optimizer settings, and network architecture are separate from this schema. Stored legacy success thresholds should not be confused with the paper’s reported 1 mm completion metric.

Force-annotated target strokes. For stroke jj, write its configuration as

Sj={(xj,i,yj,i)}i=1Nj,Fj={fj,i}i=1Nj,S_j = \{(x_{j,i}, y_{j,i})\}_{i=1}^{N_j}, \qquad F_j = \{f_{j,i}\}_{i=1}^{N_j},
Wj={wj,i}i=1Nj,wj,i=inkWidth⁡(tool⁡,fj,i).(6)W_j = \{w_{j,i}\}_{i=1}^{N_j}, \qquad w_{j,i} = \operatorname{inkWidth}(\operatorname{tool}, f_{j,i}). \tag*{(6)}

The force arrays are target commands, not measurements of the force the robot actually achieves. Their correspondence with stroke points makes force part of the requested writing content, alongside path and order. Width is derived from the same force values, rather than sampled as an unrelated visual decoration.

The default pen force-profile generator samples a base force between 0.9 and 2.5 N. A flat profile holds one sampled force along a stroke. A ramp modulates it monotonically along normalized arc length. A recorded-pressure profile samples a stored Wacom pressure curve, interpolates it along the stroke’s arc length, and rescales it into 0.9–2.5 N. The general generator also supports a swell profile; it is not used by the listed L1/L2 samplers. All generated values are clipped to [0.15, 4.0] N. In particular, the base-force range is not a strict bound on the ramp or swell profile before clipping.

From force to the target image and observed ink. The environment can rasterize a task map using

I∗(x,y)≈255f∗(x,y)Fmax⁡,Fmax⁡=4 N,(7)I^{*}(x,y) \approx255 \frac{f^{*}(x,y)}{F_{\max}}, \qquad F_{\max}=4\,\mathrm{N}, \tag*{(7)}

with integer clipping to [1,255][1,255] on target pixels and zero background. Stroke thickness follows widths. The stored intensity is a force encoding; display software may invert it to render larger values as darker ink. Actual ink instead depends on simulated contact force and tip speed, so target intensity is not a guarantee of the deposited appearance. The current deposition model makes force affect width and darkness, with speed-dependent adjustments. Although these maps are useful for visualization and evaluation, the traj2 actor consumes the numerical trajectory window, not map pixels.

The 5D cursor and its lookahead window

Absolute task-space request. The environment expands the ordered 2D strokes into a time-indexed 3D schedule containing approach, descent, pen-down motion, lifted transitions, and a final pen-up phase. Each requested sample can be represented as a five-dimensional cursor

ck⋆=(xk⋆,yk⋆,hk⋆,fk⋆,tk⋆).(8)c_k^\star= (x_k^\star, y_k^\star, h_k^\star, f_k^\star, t_k^\star). \tag*{(8)}

Here x,yx,y are horizontal target coordinates, hh is height above the sheet, ff is requested force, and tt is nominal due time. “5D” therefore means position, force, and time; it is not a five-joint action or a five-DoF end-effector pose. The source stores these components across timeline/height/force arrays and constructs the five-entry representation for the observation.

For a pen-down request, exposed height is zero and force is positive. For a pen-up/transit request, force is zero and the requested height follows the lift trajectory. The teacher’s 1.5 mm below-surface pressing target is not exposed as a negative target height: the task specifies surface contact and desired force separately. At overlapping strokes, the current implementation obtains timeline force through the spatial target-force lookup; it does not supply an additional stroke-specific force identifier to the actor.

EntryAbsolute requestPolicy entryInterpretation
1xk⋆x_k^\starxk⋆−pt,xx_k^\star- p_{t,x}Horizontal displacement, m
2yk⋆y_k^\staryk⋆−pt,yy_k^\star- p_{t,y}Horizontal displacement, m
3hk⋆h_k^\starhk⋆−hth_k^\star- h_tHeight error relative to the sheet, m
4fk⋆f_k^\starfk⋆f_k^\starRequested force, N; not force error or measured contact force
5tk⋆t_k^\startk⋆−tt_k^\star- tDue-time offset, s; negative when the nominal due time has passed

Table 8. Semantics of one five-dimensional trajectory/cursor sample. Normalization is applied after assembling the policy vector.

What the policy actually sees. The actor observes a window, not just one current cursor. Let ktk_t be the current task pointer and KK the last valid timeline index. With 16 points and stride 2,

kt,i=min⁡(kt+2i,K),i=0,…,15,(9)k_{t,i} = \min(k_t + 2i, K), \qquad i = 0,\ldots,15, \tag*{(9)}
zt,i=(xkt,i⋆−pt,x,ykt,i⋆−pt,y,hkt,i⋆−ht,fkt,i⋆,tkt,i⋆−t).(10)z_{t,i} = (x_{k_{t,i}}^\star- p_{t,x}, y_{k_{t,i}}^\star- p_{t,y}, h_{k_{t,i}}^\star- h_t, f_{k_{t,i}}^\star, t_{k_{t,i}}^\star- t). \tag*{(10)}

The resulting 16×5=8016 \times5 = 80 values are flattened and appended to the 137 retained state entries, yielding 217 inputs. At 25 Hz, adjacent samples are 0.08 s apart in nominal schedule time and the last is 1.2 s beyond the first. This is schedule spacing, not a guaranteed 1.2-second horizon relative to the current physical execution, because the pointer can lag. Near the end, indices are clamped to the last point. For holding tasks without a writing timeline, the trajectory window is zero.

Pointer advancement versus teacher timing. The reported configuration uses adv_mode=tip. A pen-down timeline point passes the advancement check when planar tip error is at most 1.5 mm and tip height is at most 0.8 mm above the sheet. The height condition is a geometric contact proxy, not a requirement that measured normal force exceed a threshold. Pen-up points use a 6 mm planar tolerance and a 15 mm vertical-error tolerance. If no point advances, the current point is skipped after the 0.25 s wait threshold. Skipping does not itself deposit ink or earn completion credit. Consequently, pointer progress, actual stroke coverage, and measured force remain distinct quantities.

Extended results and diagnostics

Metric definitions and task-level effects

Ordered completion uses monotone alignment between the requested stroke sequence and deposited ink, with a 1 mm threshold. Precision is the fraction of ink samples within 1 mm of the target. Symmetric Chamfer distance measures geometric proximity, while the reported matched-stroke discrete Fréchet distance permits reversal when pairing and averages matched pairs. Force MAE compares eligible contact-force samples with the requested force. Elapsed time includes learned stopping or failure termination, and drop rate is the percentage of episodes ending in a detected pen drop.

Case study: RL suppressing stroke jitter

Figure 6. Matched episodes across IK, BC, and final RL (top), and their evolution from BC initialization through RL (bottom). Blue is deposited ink, gray is the target, and red marks target points not covered within 1 mm. Numbers report completion | ink precision. The high-frequency deviations in IK and BC leave off-target ink; RL produces smoother, more target-aligned strokes.

The gains extend across the full repertoire. Digits improve from 90.6% IK completion to 95.0%, Chinese from 91.7% to 96.4%, and words from 85.4% to 89.8%. The line is the exception: its completion is 97.5% for RL and 98.2% for IK. Seed 26254 produces no ink for both BC and RL after an initial tip excursion of roughly 50 mm and a hover termination. Empty ink receives a 50 mm Chamfer cap, increasing its influence on the aggregate. We retain this episode in all reported means.

Progress during final RL

Figure 7. Fifty-seed evaluations during final RL. The initial point is BC and dashed lines indicate IK on the same seeds. Snapshot timing makes the horizontal axis approximate by at most one rollout. All checkpoints come from one training run.

The transition from imitation to RL is nonmonotone: early evaluations can lose completion or shape quality before later checkpoints improve completion, ink precision, force control, and elapsed time together. This motivates evaluating physical outcomes throughout refinement rather than treating a rising return as sufficient evidence of better writing.

Figure 8. Twelve-seed task monitoring and training return/episode length during final RL. This monitoring set differs from the 50-seed aggregate evaluation and is descriptive rather than a repeated-training statistic.

The 12-seed BC monitoring mean is 90.2%, whereas the 50-seed BC mean is 91.3%. These baselines are not interchanged when quantifying improvement. Intermediate checkpoints are snapshots of one final RL run, not independent training seeds.

References

  1. [1]T. L. Kivell, N. Baraki, V. Lockwood, E. M. Williams-Hatala, and B. A. Wood. Form, function and evolution of the human hand. American Journal of Biological Anthropology, 181:6–57, 2023.DOI
  2. [2]S. Mueller, N. Huebel, M. Waibel, and R. D’Andrea. Robotic calligraphy—learning how to write single strokes of Chinese and Japanese characters. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1734–1739. IEEE, 2013. doi:10.1109/IROS.2013.6696583.
  3. [3]R. Wu, W. Fang, F. Chao, X. Gao, C. Zhou, L. Yang, C.-M. Lin, and C. Shang. Towards deep reinforcement learning based Chinese calligraphy robot. In IEEE International Conference on Robotics and Biomimetics, pages 507–512. IEEE, 2018. doi:10.1109/ROBIO.2018.8664813.DOI
  4. [4]Y. Luo, Z. Wu, and Z. Lian. CalliRewrite: Recovering handwriting behaviors from calligraphy images without supervision. In IEEE International Conference on Robotics and Automation, pages 8671–8678. IEEE, 2024. doi:10.1109/ICRA57147.2024.10610332.DOI
  5. [5]P. Schaldenbrand, J. McCann, and J. Oh. FRIDA: A collaborative robot painter with a differentiable, real2sim2real planning environment. In IEEE International Conference on Robotics and Automation, pages 11712–11718. IEEE, 2022. doi:10.1109/ICRA48891.2023.10160702.DOI
  6. [6]E. Rombokas, M. Malhotra, and Y. Matsuoka. Task-specific demonstration and practiced synergies for writing with the ACT hand. In IEEE International Conference on Robotics and Automation, pages 5363–5368. IEEE, 2011. doi:10.1109/ICRA.2011.5980218.DOI
  7. [7]C. Zhao, L. Xie, B. Huang, S. Wang, and D. Ma. Tactile-driven dexterous in-hand writing via extrinsic contact sensing. IEEE Robotics and Automation Letters, 10(9):8914–8921, 2025. doi:10.1109/LRA.2025.3589168.DOI
  8. [8]K. Stewart, Y. Toshimitsu, and R. K. Katzschmann. Rapid learning of dexterous in-hand pen writing through real-time Jacobian estimation. arXiv, 2026. URL https://arxiv.org/abs/2609.11775.
  9. [9]W. Hu, B. Huang, W. W. Lee, S. Yang, Y. Zheng, and Z. Li. Dexterous in-hand manipulation of slender cylindrical objects through deep reinforcement learning with tactile sensing. Robotics Auton. Syst., 186:104904, 2023. doi:10.2139/ssrn.4674022.DOI
  10. [10]OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020.arxiv.org/abs/1808.00177
  11. [11]A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. J. Liu, D. Makoviychuk, K. V. Wyk, A. Zhurkevich, B. Sundaralingam, et al. DeXtreme: Transfer of agile in-hand manipulation from simulation to reality. In IEEE International Conference on Robotics and Automation, pages 5977–5984. IEEE, 2022. doi:10.1109/ICRA48891.2023.10160216. arXiv:2210.13702.DOI
  12. [12]J. Wang, Y. Yuan, H. Che, H. Qi, Y. Ma, J. Malik, and X. Wang. Lessons from learning to spin “pens”. In Conference on Robot Learning, 2024. doi:10.48550/arXiv.2407.18902. arXiv:2407.18902.DOI
  13. [13]K. Kedia, T. Lum, J. Bohg, and C. K. Liu. SimToolReal: An object-centric policy for zero-shot dexterous tool manipulation. In Robotics: Science and Systems XXII. Robotics: Science and Systems Foundation, 2026. doi:10.15607/RSS.2026.XXII.151.
  14. [14]A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems Conference. Robotics: Science and Systems Foundation, 2017. doi:10.15607/RSS.2018.XIV.049. arXiv:1709.10087.DOI
  15. [15]Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570–587. Springer, 2022.
  16. [16]E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.DOI
  17. [17]M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. DART: Noise injection for robust imitation learning. In Conference on Robot Learning (CoRL), 2017. URL https://proceedings.mlr.press/v78/laskey17a.html.arxiv.org/abs/1703.09327
  18. [18]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. URL https://arxiv.org/abs/1707.06347.

Paper details

vlnc:W_2pdzkap3
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents