maoBi

Essays have autocomplete. Sketches don't. Smartboards cost $3,000+ and still can't read ink. MaoBi is the free layer that turns rough sketches into real, working components.

Comment

Inspiration

In the figments of your imagination, a world appears. Picturesque scenes that flow like a river. And every second, it fades. Artists, engineers, thinkers, we all race to pin that flowing thought to paper. Drawing is the least restrictive way to capture an idea: no syntax, no menus, no blank-page paralysis. But drawing is slow. By the time the hand catches up, the palace in your mind is a fragment of what it was.

For text, we stopped accepting that trade years ago. Our essays have autocomplete. Our code has autocomplete. Shouldn't our imagination?

So we built MaoBi (毛笔, "brush pen"), an autocomplete for drawing: digital paper with magic in it. Suggestions bloom in like wet ink, soft, translucent, unmistakably a suggestion, and dry into crisp, editable structure when you accept.

What it does

Sketch a rough webpage. A box for a navbar, a letter b for a button, a purple-shaded circle and hit Enter, and real, editable components appear exactly where you drew them.

  • Draw anything, it gets crisp. Every enclosed shape stays a shape: a wobbly box becomes a clean rect, any doodle becomes your silhouette, smoothed, never replaced. Shade inside an outline and it fills with that color; two colors make a gradient.
  • Glyphs add function, and only glyphs. A single letter alone inside a box makes a working component: b button, n navbar, f form, i image, v video, ? placeholder. No glyph, no surprises. Plain shapes stay plain.
  • Words and colors are details. "Login" inside a b box labels the button; "purple" fills it; "rainbow" makes a gradient.
  • Decoratives & diagrams. A dark rect with scattered dots becomes a procedural night sky. Six diagram types from a sketched skeleton: bar chart, pie, Venn, timeline, atomic structure, and the full 118-element periodic table.
  • A real canvas. Automatic layers on overlap, an endlessly growing page, photos dropped into any drawn enclosure. Non-rectangular frames crop the photo to your drawn silhouette.
  • Frame, the finale. One press sends the wireframe to Claude and returns a complete, responsive, interactive single-file website. Download the HTML and go. The sketch was the spec.

When the model isn't confident, it abstains. Your ink stays ink. Knowing when to stay quiet is a feature.

How we built it

Two models, strict separation of powers. No model is ever trusted with geometry:

ink screenshot + stroke manifest (per-stroke colors)
  → Gemini vision       DESCRIBES only: kind, glyph, text, colors; never places
  → normalizer (code)   geometry from YOUR strokes; containment analysis
  → FreeSolo builder    our fine-tuned 2B model DECIDES: op + params; ZERO coordinates
  → validators          fail closed: junk output → nothing happens, never a broken page
  → renderer            deterministic seeded templates

The brain is Qwen3.5-2B, LoRA-fine-tuned by us on FreeSolo this weekend. We built a synthetic data generator that composes sketch scenes answer-first (gold output generated, noisy input derived, correct by construction) and corrupts them with hand-drawn jitter, then ran an agent-driven sweep: ~10 runs, 640 examples × 4 epochs for the champion, under $0.25 total.

Held-out test split, our 2B fine-tune vs the Gemini Flash baseline:

┌─────────────────────────────────────────────────────┬─────────────────┬──────────────────┐
│                       Metric                        │ Gemini baseline │ Our 2B fine-tune │
├─────────────────────────────────────────────────────┼─────────────────┼──────────────────┤
│ Op accuracy                                         │ 75.0%           │ 96.7%            │
├─────────────────────────────────────────────────────┼─────────────────┼──────────────────┤
│ Detail routing (word-in-box → label, colors → fill) │ 58.7%           │ 90–93.5%         │
├─────────────────────────────────────────────────────┼─────────────────┼──────────────────┤
│ Night-sky-from-rect                                 │ 25.0%           │ 100%             │
├─────────────────────────────────────────────────────┼─────────────────┼──────────────────┤
│ Hallucination                                       │ 0%              │ 0–1.7%           │
├─────────────────────────────────────────────────────┼─────────────────┼──────────────────┤
│ Abstention F1                                       │ 0.67            │ 0.97             │
└─────────────────────────────────────────────────────┴─────────────────┴──────────────────┘

Because the builder emits zero coordinates, placement drift is unrepresentable by construction. A model cannot misplace what it never places. Frontend: Next.js 15 / React 19, perfect-freehand ink, framer-motion blooms, everything on one coordinate system.

Challenges we ran into

  • Our first fine-tunes were disasters, and our eval caught them. Wave-1 adapters hallucinated commands at 48–72% (a length prior plus id-memorization overfit). Our independent 165-case test bank exposed it; data fixes (sparsity mix, non-sequential ids) cured it. Building the eval before scaling training was the best decision of the weekend.
  • Serving quirks. The serving stack ignored OpenAI response_format (its presence even broke guided decoding). We built a client-side repair-and-retry layer with a tightened grammar and a JSON walk-back parser.
  • Frame fidelity. Claude initially "improved" three plain buttons into a dark rainbow site with aurora glows. We rewrote the prompt so every visual block must trace back to a wireframe element.
  • The color chain. Getting "shade a rect purple" to come out as fill #7c3aed meant threading per-stroke color through the entire pipeline: manifest, vision prompt, builder, renderer.

Accomplishments that we're proud of

  • A 2B model we trained ourselves beats a frontier API on our task, for less than a quarter.
  • An interaction that feels genuinely new: autocomplete for a canvas, with abstention as a first-class feature.
  • Safe-by-construction output: no coordinates, no markup, no scripts from any model. A hallucination becomes nothing, never a broken page.
  • End-to-end completeness: sketch → components → layers → photos → diagrams → a downloadable working website.

What we learned

  • Evals before scale. A small, honest test bank turns "the model feels better" into "the model is 21.7 points better," and catches overfit that eyeballing never would.
  • Small models win when you shrink the problem. Perception to Gemini, geometry to code. The trainable core becomes a text-to-text decision a 2B model can dominate.
  • Post-training is data engineering. Every point of improvement came from the dataset, not knob-turning.
  • Constrained decoding lies. Never trust the sampler; validate everything, fail closed.

What's next for MaoBi

  • Real-ink hardening: preference tuning (GRPO config already staged) on the accept/reject signals the app logs today. Every tap is a gold label, a built-in data flywheel.
  • New vocabularies: flowcharts, circuits, chemistry, org charts. Each is a template pack, not a rewrite.
  • Magic wand: circle anything, say "make it three.js," and it is.
  • New surfaces: tablet/whiteboard SDK, Figma export, component-code export.

Built With

Share this project:

Updates