Mozaik Automation · project showcase · compiled 2026-09-08

A kitchen drawing goes in. A built Mozaik room comes out.

Mozaik Automation reads cabinet drawings with vision models, then drives Mozaik Enterprise on the shop's Windows PC to create the job, draw the walls, drop in the cabinets and appliances, and check the 3D result against the drawing. The designer runs it from a phone.

Scanned house floor plan rendered as a blueprint, kitchen at top left
InA scanned floor plan with an L-shaped kitchen run. This one is a CubiCasa5K sample from the vision model's training set, one of 148 kitchen plans checked into the repo.
Mozaik Room Viewer 3D render of a kitchen with oak wall cabinets, a tall cabinet, a refrigerator and a dishwasher
OutMozaik's own Room Viewer after an automation-built job (Dream 171, 3 Feb 2026): shaker wall cabinets on two walls, a tall cabinet, a French-door fridge. Built by the Win32 driver from a scripted spec, not from the plan on the left.
244
Commits
243 of them in the 40 days from 25 Jan to 5 Mar 2026
1,079
Test functions on disk
863 pytest functions in 23 files plus 216 vitest cases in 15 files
947
Mozaik controls mapped
One unattended Win32 discovery run, 58.5 minutes; 598 with automation IDs cached
0.914
Appliance F1
Fine-tuned vision-v9 on 30 golden-set plans, up from 0.461 for the off-the-shelf baseline
A1

How a job moves

Four stages, one job record, every hand-off visible on the phone

Source
server/routes.py · demo_e2e_poller.py
Statuses
11
1

Upload from the phone

The designer photographs or uploads a plan in the portal. The browser shrinks images to 2,048 px on the long side and re-encodes them as JPEG before posting the job. The server stores the image and parks the job for review.

pending → awaiting_review
"Awaiting Extraction..."

2

Read the drawing

Every cabinet and appliance is inventoried, given a wall, a left-to-right sequence and a position ratio along that wall. A QA pass under procedure extraction-qa-v1 recodes misread items and checks widths against wall length within 6 inches.

→ completed
"Extraction Complete" · "Cabinets: 4B + 7W + 2T"

3

Build in Mozaik

A watcher on the Windows PC refuses to start until validation returns zero warnings. Then it drives Mozaik Enterprise through the Win32 API: new job, Draw Walls, appliances on the Room sub-tabs, cabinets from the Products tree. Eight checkpoints post progress to the phone.

build_requested → validating → building
"[Build] Base cabinets placed: 4/4 OK"

4

Verify by code

The 3D Room Viewer is screenshotted and uploaded next to the drawing. A deterministic comparator counts what landed against what was extracted. Any single failed check fails the build. The designer approves or rejects on the phone.

verifying → build_complete | build_failed
"Verifying 3D Result..." · "Build Complete!"

Pending›Processing›Awaiting Extraction›Extraction Complete›Build Queued›Validating…›Building…›Verifying…›Build Complete/Build Failed·Failed

The eleven statuses in the server's job model, in the labels the phone shows for each.

A2

On the phone

A progressive web app at mozaik.djangoai.net is the whole interface

Source
web/src · server/routes.py
Endpoints
12 under /api/v1/jobs
MMozaik Automation Portal
Building in Mozaik...
Job 1f12560d
Single-wall96 in ceiling13 cabinets2 appliances
TallW 35 · H 96 · D 24 inBaseW 24 · H 34.5 · D 24 inWallW 36 · H 30 · D 12 in
Download JSONStart Mozaik Build
AgentCabinets: 4B + 7W + 2T. Appliances: sink, refrigerator.
You · voiceThe fridge is on the right end, not a tall cabinet.
AgentQA passed. Building kitchen in Mozaik (4B + 7W + 2T)...
Agent[Build] Base cabinets placed: 4/4 OK

Illustration assembled from the portal's own labels, status names and message formats. No screenshot of the live portal exists in the repository.

  • 11statuses, each with its own label. In-progress states spin in their own colour; terminal states get a check or a cross.
  • 12job endpoints. Submit, status, image, list, request build, submit extraction, update build status, delete, post and list messages, upload and fetch the 3D screenshot.
  • 2,048pxupload ceiling. Images are resized in the browser and re-encoded as JPEG at 85 percent; PDFs go as-is. The server rejects base64 payloads over 30 MiB.
  • 3s / 10spolling. Every 3 seconds while a job is in progress, every 10 once it is terminal, so an agent's re-submission shows up without a refresh. Messages refresh every 5 seconds.
  • 0speech servers. Voice notes use the browser's Web Speech API, continuous with interim results. The server drops an identical message from the same sender within 2 seconds.
  • 3sto the install prompt. Standalone portrait PWA with an "Install App" overlay after three seconds, hidden inside Instagram, Facebook, WhatsApp and TikTok webviews. API calls are never cached offline.
  • 3controls after a build. Approve and Reject post "User approved build" or "User rejected build" to the agent; Force 3D Comparison re-runs the code check; Force Rebuild starts over.
A3

Inside Mozaik

Driving a WinForms application nobody wrote an API for

Source
automation/fast_driver.py · demo_e2e_poller.py
Host
the shop’s Windows PC · Mozaik Enterprise

Mozaik Enterprise is a .NET WinForms application. There is no scripting interface, so the automation finds the window by title, brings it to the front, and works the way a person does: it clicks buttons and types values by position, using coordinates learned in a one-time discovery run. It walks the Products tree ("Sink Base", "Wall 24 deep", "Tall Single Oven"), drops cabinets onto the plan, and sets Width, Height and Elevation in the properties strip. When Mozaik answers with a dialog, the operator reads the title and picks the right button.

  • 1,861controls catalogued overnight by the pywinauto discovery script: 16 phases, 3 h 8 m, zero errors.
  • 947controls in 58.5 minutes when discovery was rewritten on raw Win32 plus the UI Automation COM API. The cache keeps the 598 that expose an automation ID, with window-relative rectangles: 234 on Settings, 166 on Room, 30 on Job, 24 on Order, 24 on Products.
  • 0 / 17controls found by the first smoke test. The guessed names (txtRoomName, designCanvas) were wrong. Discovery replaced them with Mozaik's real IDs: RoomNameTextBox, SchematicCanvas.
  • 6dialog types, matched by window title, each with its own button order: Place Anyway on the right, Yes on the left, OK in the centre. After 5 failed attempts the build raises AutomationStuck, saves a screenshot and exits with code 2 for a person to look at.
  • 114driver tests run on a Mac in 0.10 s with no Mozaik present, because the driver has a mock mode. 72 overnight-discovery and 39 UI-discovery tests also run without pywinauto.
Automation IDs the build touches LayoutTabControl Job / Settings / Room / Order
RoomNameTextBox · RoomTabButtonDrawWalls · Room3DViewButton
SchematicCanvas 744 × 592 px design canvas
H_WallsTextBox · H_BaseCabsTextBox · D_BaseCabsTextBox
DoorLibSelComboBox Settings › Door/Drawer Fronts
Room sub-tabs by pixel at y = 116: Sinks 527 · Fridge 631 · Range 680 · Hood 729
Window class WindowsForms10.Window.8.app.0… · title "Mozaik Enterprise (Job:<name>)"
Dialogs: "Product Won't Fit" · "Collision Warning" · "Align Cabinet on Sink" · "Join Wall"
A4

Speed

Why the driver bypasses UI Automation

Source
docs/performance-analysis.md
Measured
2026-01-28 · the shop’s Windows PC
369×Switching to the Job tab: 57.3 s through pywinauto's UI Automation lookup, 155 ms with a cached Win32 handle.

pywinauto 0.6.9 walks Mozaik's entire UI Automation tree on every element lookup. Measured on the shop machine, setting the room name took 94.1 seconds and the wall height 91.3 seconds, so filling five form fields took seven to eight minutes. Connecting alone took 19 seconds.

The fix was to stop asking Windows to find controls. Connect through win32gui, cache tab handles once, click by cached screen position and type with SendKeys. Connect dropped to effectively zero, tab switches to a few hundred milliseconds, and typing averaged 763 ms per input across 20 inputs.

The chart shows the hybrid experiment's measured numbers. The Fast Win32 Driver that followed it the same day is documented as faster still, but its per-action timings are asserted in code comments rather than logged, so they are not plotted.

Per-action time, before and after

Milliseconds, log scale. Same operations on the same machine.

pywinauto 0.6.9 (UI Automation)Win32 hybrid, cached handles
100 ms1 s10 s100 s log scale Switch to Job tab Win32 hybrid: 155 ms155 ms pywinauto: 57.3 s57.3 s Switch to Settings tab Win32 hybrid: 323 ms323 ms pywinauto: 29.3 s29.3 s Switch to Room tab Win32 hybrid: 319 ms319 ms pywinauto: 43.3 s43.3 s Type a value into a field SendKeys average over 20 inputs: 763 ms763 ms pywinauto set room name: 94.1 s94.1 s
Table view
OperationpywinautoWin32 hybrid
Connect to Mozaik19 s0.00 s
Switch to Job tab57.3 s155 ms
Switch to Settings tab29.3 s323 ms
Switch to Room tab43.3 s319 ms
Type a value (set room name vs SendKeys avg)94.1 s763 ms
A5

Reading a drawing

Extraction, then a QA pass that has to make the widths add up

Source
data/copilot/pending/1f12560d/extraction.json
Procedure
extraction-qa-v1

A real job from March 2026: a single-wall kitchen elevation, 218 inches of back wall. The first pass coded the 36-inch refrigerator as a tall cabinet and invented a freestanding range beside a built-in wall oven. The QA review found six issues, applied eight corrections, and only then did the floor-level widths sum to the wall.

floor level: 35 + 36 (fridge) + 24 + 24 + 42 + 24 + 33 = 218 in · status "OK (exact fit)"

Uppers as listed in the extraction: 35 + 36 + 24 + 27 + 39 + 25 + 30 = 216 in. B base, W wall, T tall. Widths in inches, drawn to scale along the wall.

6 → 8
issues found, corrections applied. Fridge recoded from cabinet: tall to appliance: refrigerator; the phantom range removed; the 33-inch base "right of range" recoded as a tall oven cabinet.
4B · 7W · 2T
the corrected count, 13 cabinets plus sink and refrigerator, posted to the phone in shop shorthand. Confidence after review 0.96.
± 6 in
the tolerance every wall must meet before a build starts. Position along the wall is one formula, stated identically in the vision prompt and the QA procedure.
# the same line appears in vision/extractor.py and docs/personas/extraction-qa.md
position_along_wall = (cumulative_width_before + item_width / 2) / wall_length   # rounded to 2 decimals, 0.0 to 1.0
sequence            = 1, 2, 3 …  left to right, restarting on each wall
walls               = "left" | "back" | "right" | "right-upper"      shape = "single-wall" | "L-shape" | "U-shape"

Floor-plan reader: fine-tuned vs off-the-shelf

Type-count F1 on the same 30 golden-set plans. vision-v9 is Qwen2-VL-2B-Instruct with a LoRA adapter trained in 65 minutes on one RTX 5070 Ti.

llama3.2-vision:11b, no fine-tuningvision-v9, fine-tuned
00.250.500.751.0 Appliances Baseline appliance F1 0.4610.461 vision-v9 appliance F1 0.9140.914 Cabinets Baseline cabinet F1 0.2920.292 vision-v9 cabinet F1 0.4770.477
Table view
TypeBaseline F1vision-v9 F1v9 precision / recall
Appliances0.4610.9140.898 / 0.930
Cabinets0.2920.4770.383 / 0.631
Macro0.3760.695
  • 1,442training examples for vision-v9, balanced by oversampling wall and tall cabinets, 5 epochs, LoRA rank 16, alpha 32, fp16.
  • 170annotated plans in the golden set, averaging 8.3 cabinets each, plus 148 CubiCasa5K kitchen plans and 510 flipped and rotated copies. Labels were script-validated, not human-checked.
  • 564mislabeled cabinet types fixed in a March 2026 annotation audit. Retraining lifted wall-cabinet F1 from 0 to 0.464.
  • 0.477is below the 0.70 target for cabinets, so a person still checks the cabinet list before anything is built. Appliances met their 0.80 target.
  • 16 / 12cabinet and appliance types in the spec model, with shop-rule defaults: base 34.5 in high and 24 deep, wall 30 in high and 12 deep.
A6

A shop-trained assistant

A 7B model that answers a plain-English change with a JSON edit list, trained on the shop's own GPU

Source
docs/reports/VALIDATION_REPORT_D43_V6.md
Model
mozaik-assistant:d43-v6

Type "Change all base cabinets to cherry wood with soft-close hinges" and about three and a half seconds later a small JSON block comes back: what changed, the Mozaik steps to make it happen, and a one-line explanation. Ask it about the weather and it declines and points back to cabinets.

It is a fine-tune of Qwen2.5-Coder-7B-Instruct using 4-bit QLoRA, trained in under eleven hours on the single consumer GPU in the Windows machine that also runs Mozaik, and served locally through Ollama. The adapter is 40 MB against an 8.1 GB quantised model.

// response contract every trained model must emit
{
  "spec_delta":  { "cabinets[type=base].finish": "cherry",
                   "hardware.hinges": "soft-close" },
  "actions":     [ { "type": "open_sidepanel", "path": "Materials" }, … ],
  "explanation": "Base cabinets set to cherry; soft-close hinges applied."
}
// off-topic requests
{ "error": "I only help with Mozaik cabinet design..." }
Base modelQwen/Qwen2.5-Coder-7B-Instruct
MethodQLoRA, 4-bit NF4 · LoRA r 16, α 32, dropout 0.05 · targets q, k, v, o projections
Data5,390 samples · 4,851 train, 539 validation · script-generated modification, multistep and delta requests
Training10.70 h · 912 steps · one NVIDIA RTX 5070 Ti, 16 GB, about 7.3 GB used
Artifacts40 MB adapter → Q8_0 GGUF, 8.1 GB → Ollama mozaik-assistant:d43-v6, num_ctx 4096
Held-out95.55 % next-token accuracy on the validation split, eval loss 0.129
Inference tests20 / 20 delta prompts and 20 / 20 multistep prompts returned valid JSON with both spec_delta and actions · 242 and 306 characters on average
Broader suite32 of 43 automated prompts · 17 of 21 end-to-end scenario steps · 7 of 7 manual tests · 3,537 ms average response, 71 ms fastest
Lineagemozaik-mistral v1 to v3 (Mistral-7B-Instruct-v0.3) → mozaik-assistant:d42-test → d43-v6 · 1,903-character average answers cut to 242
A7

Verified by code, not by vibes

Gates before the build, named checks after it

Source
verification/comparator.py · demo_e2e_poller.py
Rule
any failed check fails the build

Two ideas hold the pipeline honest. Before Mozaik is touched, validate_extraction() must return zero warnings: every cabinet has a wall, every position ratio is between 0 and 1, no refrigerator or dishwasher is coded as a cabinet, and the parsed counts equal the cabinet list. Otherwise the job is marked build_failed with the warning count and the reason is posted to the phone.

After the build, the 3D screenshot is analysed and compare_extraction_to_analysis() emits named pass or fail checks. Expected counts come from the extraction file; actuals come from the 3D observation. There is no rubric to argue with and no checklist to rubber-stamp: one mismatch and the verdict is FAIL. The designer can overrule it with Approve, or re-run it with Force 3D Comparison.

✓ PASS✕ FAILThe only two outcomes the comparator can produce.

Checks the comparator emits

base_cabinet_countwall_cabinet_counttall_cabinet_countappliance_sinkappliance_refrigeratorappliance_rangeno_phantom_appliancessink_bowl_matchcabinet_3_configlayout_shapecross_ref_base_countpre_build_base_count
  • 8build checkpoints post progress to the phone: create_job, draw_walls, appliances_placed, base_cabinets_placed, wall_cabinets_placed, tall_cabinets_placed, switch_3d, final_3d.
  • 2 · 3typed exit codes. 2 means AutomationStuck, a dialog that survived five dismissal attempts, screenshot saved. 3 means three or more "Product Won't Fit" warnings; the build log records each one.
  • 5swatcher cadence. The watcher and the build service poll the job API every 5 seconds; the structured-event poller every 10.
A8

Platform and delivery

One licensing server hosts the API and the phone app; Jenkins is the only road to it

Source
Jenkinsfile · deploy/ · licensing/
Probed
2026-09-08 · HTTP 200 in 0.065 s
  • livemozaik.djangoai.net answered /health with {"status":"healthy","service":"mozaik-licensing","pwa":true} over HTTP/2 when probed for this page, and served its Swagger UI at /docs.
  • 23API routes on three routers: 2 public (verify, usage), 9 admin behind an X-API-Key header compared in constant time, 12 for jobs. Plus /health.
  • 6pip packages in the production image, on python:3.11-slim. The ML stack stays on the Windows machine, deliberately.
  • 4 × 4license tiers and states. Keys look like MOZAIK-XXXX-XXXX-XXXX-XXXX, are generated from cryptographic randomness, default to a 365-day expiry, and count every successful verification. Tiers: trial, standard, professional, enterprise.
  • 32hex characters identify a machine: the first half of a SHA-256 of the BIOS UUID, Hardware UUID or /etc/machine-id. The raw hardware ID never leaves the PC.
  • 30supstream health checks from Traefik, which terminates TLS with a Let's Encrypt certificate, compresses with brotli and redirects HTTP to HTTPS before forwarding to the application container.

Jenkins job mozaik-deploy, ten stages

01Checkout
02Setup Python
03Test Backend
04Setup Node
05Test Frontend
06Build PWA
07Sync to Server
08Rebuild Container
09Health Check
10Verify API

Stages that can stop a release are marked in the wood tone: a frontend test failure, a container that does not answer /health after a 10-second wait, or a public API that returns an HTML page instead of JSON. Backend test failures are logged but do not gate the deploy. Static files are baked into the Docker image, so a Jenkins run is also the cache refresh.

A push to master is picked up by polling; the tarball hops from Jenkins to the deploy host, where docker compose rebuilds the container.

A9

The build log

Forty days of commits, one developer, a changelog that reads like a shop log

Source
git log · CHANGELOG.md
Span
2026-01-25 → 2026-03-05 (+1 on 04-26)

Commits per day

243 commits on 23 of the 40 days between 25 January and 5 March 2026. One further commit landed on 26 April.

Table view
MonthCommitsActive daysBusiest day
January (from the 25th)60728 Jan · 19
February74123 Feb · 25
March (to the 5th)10945 Mar · 67
April1126 Apr · 1
  • 61,406lines of Python and TypeScript: 10,811 in src, 31,813 in scripts, 12,403 in tests, 6,379 in the web app. Plus 39 Markdown documents, 13,892 lines, and a 541-line changelog.
  • 138 → 1,079test functions. 138 in the first commit, 461 two days later, 871 recorded in the changelog in March, 1,079 on disk today. Strict test-first development is Principle I of the project constitution.
  • 226of 244 commits carry a Claude co-author trailer: 123 Opus 4.5, 78 Opus 4.6, 25 Haiku 4.5. All 244 come from one developer.
  • 23 / 23steps passed in the first full Mozaik workflow demo on 27 January, two days in: Job, Room, Walls, 3D Preview. The first smoke test earlier that day had found nothing.

From the changelog

  • BUG-1Sink sub-tab coordinates wrong, placed Range instead of Sink. The click at x = 517 hit the Windows tab; moved to 527. Root cause recorded: coordinates never validated against discovery screenshots.
  • BUG-8"Product Won't Fit" dialogs silently dismissed. build_kitchen now returns a list of placement warnings; the build writes build_log.txt and fails with exit code 3 at three or more.
  • BUG-17Floating sink, placed as a standalone Room fixture. Sinks and hoods are cabinet-integrated; only fridges and ranges are placed as Room appliances now.
  • BUG-193D verification rubber-stamps PASS. Replaced by a data-driven checklist: expected counts from the extraction, actuals from the 3D observation, any mismatch fails.
  • Phase 11Step-based build monitoring. Eight strategic checkpoints at phase boundaries rather than per action, keeping overhead low while giving the watcher a place to abort.

General notes

What this page does not claim

  1. The repository is dormant. The last commit is dated 26 April 2026 and touched only tooling; application code was last changed on 5 March 2026, six months before this page was compiled.
  2. The cover images are not one job. The input is a CubiCasa5K training-set plan; the output is a scripted demo build (job Dream 171) in which the refrigerator and dishwasher stand mid-floor. No upload-to-3D pair from a single job is on disk, and the time from photo to 3D has never been measured.
  3. Extraction in production is human-supervised. The server stores the image and waits; a Claude Code session running the watch skill reads it and submits the spec. The fine-tuned vision-v9 model runs only when the local backend is selected, and its cabinet F1 of 0.477 is below target.
  4. At the current commit the verification package does not import. build_steps.py defines five of the eight step constants, so the checkpoint screenshot and CONTINUE/ABORT gate are inactive and only the progress messages post. The comparator's checks are unit-tested in isolation.
  5. The test suite is not fully green at HEAD. Measured on 8 September 2026: backend 731 passed, 16 failed, 24 errors of 777 collected; frontend 208 of 216. Documentation quotes older totals that disagree with each other.
  6. The assistant model is not wired into the product. mozaik-assistant:d43-v6 runs through Ollama in test scripts; nothing in the pipeline executes its action lists in Mozaik. Its 95.55 percent is next-token accuracy on held-out training data, not a measure of correct edits.
  7. Speed figures are from one machine on one day. The Fast Win32 Driver's own per-action timings are asserted in comments, not logged; only the 28 January pywinauto and hybrid measurements are plotted.