Learning Bimanual Napkin Folding from Demonstrations

Linique was a six-person UBC capstone project focused on bimanual napkin folding from camera images and joint state.

The central result was that model scale alone did not solve the task. Cloth folding is temporally ambiguous: two observations can look nearly identical while demanding different actions because one belongs to pickup, another to alignment, and another to release. Our results suggest two practical ways to handle that ambiguity: collect enough demonstrations to cover those transitions, or represent task stage and progress explicitly during training.

Why trajectory replay fails on cloth

A linen napkin has no fixed kinematic model. Its next state depends on friction against the table, bending stiffness, self-contact, wrinkles, grasp point, and tension between the two arms. A few millimetres of pickup error can turn the same joint trajectory into a different fold.

This also made simulation a poor shortcut. A useful cloth simulator would need a finely discretized finite-element, particle, or equivalent deformable model; identified warp and weft stiffness, bending, damping, and friction; and stable handling of self-collision plus table and gripper contact. Those parameters are difficult to measure, expensive to simulate, and highly consequential: a small mismatch changes where the napkin buckles or catches, producing a different sequence of contacts. The resulting sim-to-real gap was larger than the variation we were trying to learn. We therefore collected demonstrations and tested the mechanism on the physical system instead of training in simulation.

Our system had:

• two SO-101 follower arms and two leader arms for teleoperation;
• three RGB observations: overhead, left, and right;
• 12 joint positions in and 12 target joint positions out;
• a 30 Hz control and recording loop; and
• a roughly 914 × 607 mm working area under a camera and lighting frame.

The archived project does not define a numerical fold-quality threshold. We judged pickup and fold completion during physical rollouts, which is adequate for prototyping but not for a defensible success-rate comparison. A future evaluation should define grasp, final geometry, cycle time, and consecutive autonomous successes before training begins.

An early data-collection prototype. Operators moved the two leader arms while the follower arms reproduced the motion and the system recorded synchronized observations. The later rig refined the camera, lighting, and mechanical integration.

Designing the interaction before training the policy

Cloth made the end effector part of the learning problem. A fast grasp that dragged or crumpled the napkin produced a different initial condition for the policy. We therefore ran a manual test matrix over four end-effector designs, two napkin states, multiple operators, and front- and side-edge approaches. Each observation measured time to grasp and lift one inch, plus whether the napkin crumpled.

The table uses the original design labels from the test worksheet. “Flat” and “folded” are separate pickup conditions, not autonomous-policy trials.

EES was slower than the fastest geometry, but it was the only design with no recorded crumpling. Testing that directly on the napkin gave us a more useful design criterion than trying to identify a complete cloth and contact model.

What one demonstration contains

During teleoperation, the data system synchronized three camera streams with both arms’ joint positions and commanded actions. Episodes were stored in the LeRobot format so that collection, inspection, training, and inference shared the same schema.

The published dataset contains 499 episodes: 400 full-task demonstrations and 99 pickup-focused demonstrations. Its meta/info.json reports 544,906 recorded frames. The LeRobot viewer below opens episode 0 with synchronized camera video, robot state, actions, and episode statistics; the episode selector can be used to explore the rest of the dataset.

The collection software also had to survive ordinary robotics failures: cameras reconnecting under different device paths, drifting exposure, truncated MJPEG frames, a saturated USB bus, interrupted episodes, and visualization competing with control timing. Stable by-path device mappings, physically separating USB traffic, fixed exposure, health checks, and resumable writes were as important as the model code.

Motor “effort” was not torque

The SO-101’s Feetech motors do not provide calibrated joint torque. They expose a raw Present_Load value, surfaced in parts of the stack as “effort.” It is a coarse, controller-internal estimate—not a force/torque sensor.

LeRobot did not initially include that value in the robot observation and dataset path we were using. We forked LeRobot, changed the motor and robot source to read and log effort alongside position, and collected data to test whether a policy could infer contact or napkin tension from it. Logging Present_Load confirmed that the value was too coarse to recover either quantity reliably.

The production repository retains a motor-health utility that reads signed load, voltage, temperature, and motor error flags. For a future version, actual fingertip force sensing would be substantially more useful than this proxy.

A calibration bug that affected every arm differently

The wrist-roll joint can rotate continuously, so it cannot be calibrated against hard stops like the other joints. We found that the calibration and position-processing path could assign different zero points and mappings to nominally identical SO-101 arms. A leader and follower could therefore report matching numbers while their wrists were physically misaligned—or reach different numbers at the same pose.

We spent several weeks reproducing the problem and working with the LeRobot team through Discord. We contributed hardware observations and test cases while the underlying issue was traced to the STS3215 configuration and multi-turn position handling. The problem was later documented publicly as the SO-101 wrist-roll calibration issue, and the upstream motor fix forced the servo back into single-turn feedback so readings wrap consistently from 0 to 4095. The upstream change corrected the same wrist-roll behavior for other SO-101 users.

In the interim, treating calibration as local device state was too fragile. We stored the four arm-specific JSON files inside robo-ops and explicitly passed the repository calibration directory into every robot and teleoperator constructor. Keeping the files in the repository ensured that each computer loaded the calibration associated with that code and dataset. The current implementation is visible in lib/robots.py.

CPU starvation looked like bad control

The 30 Hz loop has 33.3 ms to read up to four motor buses and three cameras, prepare observations, run the policy, send actions, optionally record, and update visualization. If that work takes longer, sleep(max(period - elapsed, 0)) silently becomes a zero-length sleep. Nothing crashes; the robot simply runs slower and less consistently. At first, that looked like policy instability or USB trouble rather than CPU starvation.

We isolated the problem by timing each part of the control loop. The inference loop now measures observation, policy, and action latency separately; reports actual versus target frequency and p95 loop time; and warns when control rate falls below 95% of 30 Hz. Those timers made it possible to distinguish an overloaded host from a slow model or motor bus. The later production architecture moved synchronous robot control to a dedicated thread and exposed camera JPEGs through a shared frame buffer instead of letting web streaming own the control cadence.

ACT exposed the stage-transition problem

Our first baseline was ACT, an action-chunking transformer. It predicted chunks of 100 actions from the camera observations and robot state; temporal ensembling smoothed overlapping predictions during execution.

The ACT training path also exposed a tooling gap. It accepted one fixed optimizer_lr, but did not provide a learning-rate scheduler or automatic search. We therefore made learning rate environment-specific, ran manual sweeps through Weights & Biases, and saved optimizer state for resumable jobs. The later SARM trainer added scheduler construction, stepping, checkpointing, and learning-rate logging. Without scheduling or a manual sweep, a 100,000-step run could spend most of its compute at an unsuitable learning rate.

The optimization loss decreased and plateaued:

The lower loss confirmed that ACT fitted the demonstrations, but physical rollouts were still required to evaluate folding.

Physical rollouts exposed a sharper distinction:

1. A pickup-only policy could usually establish a grasp in our informal trials.
2. Given a human-assisted initial grasp, a fold-only policy could repeatedly execute the later motion.
3. A unified pickup-and-fold policy was much less reliable.

These ACT observations were not logged with a consistent rollout count, so percentages would be misleading. We used them to locate the failure at the transition between pickup and folding. Isolating the two stages removed the most ambiguous temporal boundary. In the combined task, a similar-looking cloth geometry could call for approaching, maintaining tension, forming a crease, or releasing, depending on what had happened in the preceding seconds. A low average imitation loss could hide precisely this multimodal transition.

A state machine joining two policies would be a useful baseline, although it would encode the stage boundary by hand. An end-to-end policy must instead infer task phase and progress from the demonstrations. With 499 episodes, we found it more practical to provide that temporal structure explicitly than to expect the policy to discover it from data alone.

SmolVLA and stage-aware reward modeling

We subsequently trained SmolVLA and X-VLA checkpoints on the same task family. Pretrained visual representations helped SmolVLA partially disambiguate the sequence, but it still completed only about 40% of physical trials. Because the models were not evaluated under one consistent rollout protocol, their results are not directly comparable.

We then implemented SARM as a separate stage-and-progress model. It does not command the robot. It observes a temporal window of video and robot state, classifies the current stage—pickup, align, fold, or release—and estimates progress within that stage. This turns one sparse success bit at the end of a rollout into a dense estimate of progress from 0 to 1. Training samples also include short rewinds, which force the model to use temporal order when two individual frames look similar.

That progress model can then reweight demonstrations for reward-aligned behavior cloning before training a downstream SmolVLA- or PI0-class policy. SARM provides explicit stage and progress labels that the end-to-end policy did not reliably infer from 499 episodes. Learning the same structure without SARM would require a substantially larger dataset covering variation, failures, recoveries, and stage transitions.

The policy still cannot observe contact, grip force, or cloth tension directly because its inputs are limited to RGB images and joint positions. Force-aware fingers would improve contact observability, while controlled initial-state variation, corrective demonstrations, and explicit failure examples would expand the states represented in training.

Running the same experiment on four kinds of compute

We trained locally, on Modal and RunPod, and on UBC’s Sockeye Slurm cluster. In practice, data transfer, environment setup, and artifact recovery often took longer than the GPU run itself.

• Local was best for iteration because the data, cameras, code, and weights were already together.
• Modal / RunPod made burst GPU access easy, at the cost of uploads, cold starts, duplicated environments, and artifact management.
• Sockeye provided V100s and Slurm reproducibility, but compute nodes had no internet, project storage was read-only inside jobs, and writable scratch storage was purgeable. Dependencies and weights had to be staged; Weights & Biases ran offline; outputs had to be copied back.

For this dataset size, a slower local GPU could complete an experiment iteration sooner once remote packaging, upload, and artifact recovery were included. That shaped the repository into separable collection, repair, visualization, training, and inference tools rather than one notebook tied to one machine.

Team and ownership

The project crossed three disciplines, and the division of work matters:

• I was responsible for the software and robot-learning stack: hardware interfaces, calibration integration, synchronized collection and repair, Rerun visualization, policy training and inference, timing and USB reliability, the production control service, and moving experiments between local, cloud, and HPC environments. Most of that implementation is available in robo-ops.
• Sloan Sobie and Dawson March, both Mechanical Engineering students, focused primarily on mechanical design, fabrication, end effectors, and integration of the physical system.
• Genevieve Merz, Cameron Powell, and Jaden Legate, all Sauder students, led market discovery through restaurant and venture interviews, customer-problem validation, and product positioning.

The engineering and market-research work ran in parallel, so the prototype was tested against the restaurant workflow it was intended to support.

Poster, report, and source

We presented the poster below at the IEEE Vancouver Section AGM. Select it to open the full PDF.

Open the poster · poster source

The 55-page final technical report contains the full requirements, mechanical and software design, verification results, market research, and appendices. (Written in Typst.)

The implementation is in the public robo-ops repository, the modified robotics framework is in the Linique LeRobot fork, and the rest of the project is under the Linique GitHub organization.

See also

• HTTP 418: Autonomous Clue-Finding Robot in Simulation