Multi Model Hackathon · SF Tech Week · Field briefing
“The camper had a blue backpack, an orange tent, and a gray dog.”
An operator said that into Patrol's console during a test. Patrol turns words like these into a search. It sweeps the ground from 40 metres up, drops down for a closer look at anything that might match, and sends back a position, a photo and a short note when it finds something. Four AI models share the work on one laptop, and none of them calls the internet.
At a glance · the console
The console is a web page the laptop serves to itself. It runs a live search with push-to-talk, or replays any recorded flight. Press play to watch a whole search, or read the annotated still below: the moment Clef-Flash confirms the blue backpack from 20 m.
01 · The problem
When someone goes missing in the backcountry, the first useful facts usually come from family: what the person was wearing, what they carried, whether the dog was with them. Nobody has trained a detector on this person's blue backpack, so any computer that helps has to work from words. Out there it usually can't send video anywhere either, because there is no cell signal.
Patrol is a prototype of a drone that works from those words by itself. It flies the search, takes a closer look at whatever might match, and only asks for a person's attention when it has something worth checking: a ground position, a photo and one or two sentences.
From 40 m a backpack is a smudge about 22 pixels across. In the air, the drone can drop down and settle a “maybe” while it is right there. After landing, every “maybe” costs another flight.
A find during the flight beats a find an hour later at a laptop. Cold, injury and avalanches don't wait for a review.
With no cell service, live video can't reach the base. An alert is about 270 bytes of text plus a 20 KB photo, small enough for a radio link. (The radio itself is not built yet.)
Patrol is decision support. A person reviews every alert. It would sit beside the tools search teams already use (thermal cameras and person detectors for spotting people, avalanche transceivers, RECCO and phone-finding for buried victims), not in place of them. Everything you see here happens in a simulator, which proves how the system behaves, not how well it would detect things on a real hillside.
02 · A flight, start to finish
The pictures below are from real flights in the simulator, with the real models deciding. The numbers are what Clef-Flash said at that moment.
Before take-off you hold the talk button and describe what the person had with them. Parakeet writes it down. Clef-Flash decides it is a description rather than a command. qwen38 turns each thing into a sentence about how it looks from the air, with a rough size, and Kokoro reads the list back. Those sentences go into every question Clef-Flash is asked for the rest of the flight. The smallest size sets how high the drone sweeps.
blue backpack · 0.5 m: “a rectangular blue fabric object with visible straps and buckles, roughly the size of a large book.”
orange tent · 2 m: “a bright orange geometric shape, likely triangular or dome-like, with visible seams and a smooth synthetic surface.”
gray dog · 0.6 m: “a small gray furry shape with four legs and a tail, appearing as a compact oval mass on the ground.”
The demo scene, from a recorded clip (Parakeet: 64 ms). The smallest size, 0.5 m, set the sweep at 36 m. The real-time flight on the map used hand-written catalogue cards instead.
The planner splits the 160 m square into 7 lanes and 28 sectors, starting from the trailhead, where the person was last seen. The drone flies the lanes at 6 m/s and takes a straight-down photo every 12 m, so each patch of ground appears in about two photos. For each photo the finder spends about 50 ms on the CPU boxing the three spots with the rarest colours. Each box becomes a 256-pixel crop, and Clef-Flash answers in about a third of a second whether it could be one of the clues.
Most crops get “continue”. When one looks like a clue (at least 10% for any clue, or 90% that it is worth a closer look), the drone notes the spot and leaves its lane. It stops 12 m short of the spot at 20 m and takes an angled zoom close-up. Then Clef-Flash answers a harder question, with the look-alikes spelled out: is this the clue, something else of a similar colour (a tarp, a rock, a log, leaves, a shadow), or nothing at all?
A confirm needs two things to agree: Clef-Flash's choice must name the clue, and that clue's own probability must reach 50%. The alert carries a ground position, worked out from the camera's position and angle, plus the close-up, the sweep crop and the confidence. In the background qwen38 writes a one-line note and what the ground team should check, and Kokoro reads it aloud. The drone doesn't wait for the note, and it moves on to the next spot.
After each find, qwen38 gets the finds so far, the trail, the stream and the sectors still unsearched, and names up to six sectors to search next with a one-line reason. Code keeps only real, unsearched sector names and moves them up the queue when the current sector ends. Meanwhile the drone climbs back to the point where it left its lane, finishes the area, and flies home. In a controlled test, reordering this way cut the time from the first find to the second by 29.5 seconds on average.
03 · Who does what
Each model does the one job it is good at, at the speed that job needs. Plain code ties them together and keeps the drone safe.
A decision model from Cloudflare. You give it a picture and a few typed questions, and it returns a probability for every answer in one pass, without writing any text. Patrol asks it whether each odd-looking spot is one of the clues, whether a close-up confirms it, and whether something you said was a description or a command.
Qwen3.8 Flash Next, a large language model that can also see images. It turns your description into what each thing looks like from above, writes the short radio note for each find, suggests where to search next, and gives a second opinion when Clef-Flash is unsure. It stays out of the fast loop; only a second opinion makes the drone wait for it.
NVIDIA's speech recogniser. It turns what you say into text, both the description before take-off and commands during the flight.
A small text-to-speech model. It reads the clue list back to you, acknowledges commands and reads every alert aloud, so your eyes can stay on the drone.
Classical computer vision on the CPU. It looks for colours that are rare in each photo and boxes the three oddest spots. That is what lets Clef-Flash look at three small crops instead of a whole 1280×960 frame, and it is the reason the drone can fly high.
The planner lays out lanes and sectors. The gate decides whether a model's answer is strong enough to act on. Safety owns the height limits, the boundary, the battery reserve and the trip home. Models advise. Code flies, and no model can override it.
A physics simulator standing in for the real hillside, the drone and its cameras. It renders each camera view in about 10 ms and keeps the answer key (where everything was planted), which is used only to score a flight afterwards.
The backpack, tent, dog and decoys were each made from one sentence by m3d, a command-line 3D tool the project's author built for this Mac. It is a local, free take on Meshy: one command runs three models and a Blender step in a row, and each model is a swappable line in its config file. Patrol called it once per prop, then converted the result into MuJoCo's format at the item's real size.
04 · How the models fit together
Clef-Flash makes every per-photo call because it answers in a third of a second. qwen38 does the few jobs that need real thought, in the background. Voice is how you talk to the drone, and code turns every answer into motion only after checking it.
The finder sees every photo. Clef-Flash sees only three small crops per photo and the close-ups. qwen38 sees only the finds. In the real-time flight that was 375 odd spots, 171 Clef-Flash looks and 6 qwen38 jobs.
Clef-Flash's probabilities pass a gate with thresholds. qwen38's plans are kept only if they name real sectors that haven't been searched. A spoken command runs only if Clef-Flash is at least 45% sure what it was.
If Clef-Flash is down or late, the spot stays unjudged, the console says FALLBACK, and the sweep carries on and comes home. Nothing is ever alerted on a fallback answer.
05 · What the drone does
No model tells the drone to go up, go down or turn. Every half second, code picks a target for the current state (the end of a sector, a hover point beside a spot, the place it left its lane, or home) and sends a velocity toward it, up to 6 m/s with at most 3 m/s² of acceleration. The models only answer questions. The state machine decides what the answers mean.
| Picture | Camera | Size | How often | Who looks at it |
|---|---|---|---|---|
| Sweep photo | pointing straight down, 34° tall | 1280×960 | every 12 m along a lane (every 6.5 m when sweeping at 21.8 m for a small clue) | the finder sees all of it; Clef-Flash sees 256 px crops of the three oddest spots |
| Close-up | zoom, aimed at the spot, framed to 4 m of ground or 3× the clue's size | 512×384 | once per step of a closer look: at 20 m, and at 12 m only if unsure | Clef-Flash; qwen38 for the note and second opinions; you, on the alert card |
| Chase view | behind and above the drone | 480×360 | every half second | only the console, for the people watching |
Steer the drone. Override the height band (8 to 70 m), the boundary (25 m outside the area) or the battery reserve (10% after the trip home). Turn a failed or late answer into an alert. Change the clue list after take-off.
It is a kinematic body in MuJoCo: no rotors, no wind, no GPS error, the heading fixed, the ground flat. The battery is a clock that counts every second in the air, hovering included: 20 minutes in every result on this page, 30 minutes from now on, a fair figure for a search drone in the cold. That is enough to test the decisions, and it is not a flight model.
06 · Fly high, look close
Height is the battery lever. The higher the drone sweeps, the more ground each photo covers, the fewer lanes it flies, and the more area one battery searches. The catch is that small things shrink to a smudge of pixels. Patrol handles that by splitting the job: from up high, it only has to notice that a spot looks odd. Deciding what the spot is happens later, close up.
Each clue card carries a rough real size, which qwen38 writes along with the description. The planner sweeps at the highest height that works for the smallest thing on the list: 40 m for a 0.55 m backpack, and 21.8 m once a 0.3 m helmet is on the list. That second case exposed a bug today. Photos were still spaced for 40 m, which left gaps at the lower height. The fix landed at 14:53 and now scales the spacing with the height (12 m between photos at 40 m, 6.5 m at 21.8 m); the held-out test passed at 15:16.
On a “maybe” the drone flies toward the spot, stops short of it at 20 m and takes an angled zoom close-up, framed to about four metres of ground (or three times the clue's size). If Clef-Flash is clearly yes or clearly no, that settles it. Only an unsure answer sends it down to 12 m for a second look. In the real-time flight all eight closer looks were settled at 20 m, in 8 seconds each on average.
07 · Each model sees only what it must
This is the routing idea at the heart of the hackathon brief, measured on the real-time flight. The CPU finder looks at every photo. Clef-Flash looks at a few hundred small crops. qwen38 is called six times.
At 2 to 5 s a call, the same 163 crops would take 5 to 14 minutes of model time (the flight took 11), and the drone would fly past each spot long before the answer came back.
Clef-Flash can say whether a backpack is in a photo but not where (asked to point, it was right 4 times in 30), and a drone needs a spot to fly to. The finder supplies the spot. In the first test at 512 px, whole-photo answers were useless above 20 m; crops kept working.
It costs 50 ms of CPU per photo and no GPU, and it turns one big question per photo into three small ones about specific places. Over 43 flights it boxed 11,064 spots, and Clef-Flash judged 5,627 of them.
08 · Talking to it
In the field the operator's hands and eyes are busy with gloves, a radio and keeping the drone in sight, and the description arrives as speech from a relative anyway. Parakeet and Kokoro run on the same laptop, so voice works with no signal. Speech only ever picks from a fixed list of commands. Code still flies.
| You say, for example | Clef-Flash routes it to | What happens |
|---|---|---|
| “Missing hiker. Blue backpack, orange tent, grey dog.” | describe | qwen38 writes the clue cards; before take-off the mission is rebuilt around them and Kokoro reads them back |
| “Start the search.” | start_search | take off and sweep |
| “Go back to the last alert.” | go_back_to_last_alert | fly back and look again (queued if it is busy with a spot) |
| “Stop searching and come home.” | return_home | straight to RETURN |
| “What have you found?” | question | a short spoken answer from the mission status (basic so far) |
A command runs only when Clef-Flash is at least 45% sure which one it was. If Clef-Flash is down, a simple keyword match takes over and the log says FALLBACK. Since 14:55 the drone acknowledges a command only after the mission has accepted it. “I can't do that now: there is no alert yet” is said aloud when it refuses one.
Parakeet transcribed it, Clef-Flash routed it as a description with 94.7% confidence, qwen38 wrote three clue cards, and the drone took off looking for them. It found and announced its first alert out loud, and then the operator told it to come home at 93 s, which it did.
Hold the button or the space bar in the console. The browser records the clip and posts it to the console server, ffmpeg turns it into 16 kHz audio, Parakeet transcribes it (20 to 90 ms), one text-only Clef-Flash call routes it, and the result shows in the log.
09 · The console
The console is a web page served from the laptop. It runs live, with push-to-talk, or replays any recorded flight. The annotated view at the top of this page shows it mid-flight; below is the same demo flight after landing, and what each panel is for.
| Panel | What it tells you |
|---|---|
| Header | state, area searched, battery, clock, and who decided: CLEF answers, FALLBACK answers (Clef-Flash failed), QWEN38 jobs, with Clef-Flash's median time |
| Looking for | the clue cards: your words, qwen38's from-above sentence, and the size that set the sweep height |
| Onboard camera | the latest sweep photo with the finder's boxes: amber when the gate said investigate, grey when judged and passed, orange “no answer” on a fallback. During a closer look it switches to the zoom close-up with the verdict |
| Decision trace | the 256 px crop Clef-Flash actually saw, its continue-or-investigate odds and one bar per clue, with the thresholds marked |
| Map | sectors, the trail, the stream and the last known point; ground already photographed; qwen38's suggested sectors; every spot looked at (grey dismissed, red alert) |
| Chase view and log | a camera behind the drone, for the audience; and every decision, note, plan and spoken line, newest first |
| Alerts | one card per find: the sweep crop beside the close-up, position, confidence and qwen38's radio note with what to check |
10 · One laptop, no internet
An M5 Max with a 40-core GPU and 128 GB of unified memory. qwen38 and Clef-Flash stay loaded as shared services, the voice models load with the console, and the simulator, finder and flight code run on the CPU. Every connection stays inside the laptop.
Clef-Flash answers in 0.3 to 0.4 s when the GPU is free. While another agent's image or 3D job ran, it took 2 to 6 s while the drone kept flying on its last command, and the power-line scene's recall fell from 5 of 6 to 3 of 6. Scored runs and the demo need the GPU to themselves.
When qwen38 is busy with a note or a plan, Clef-Flash slows a little: about 73 ms (23%) slower at the median across 25 flights, for example 321 ms idle against 395 ms busy. That is why qwen38 does only a handful of jobs per flight, in a background thread.
Nothing in the code calls out, and every model is local. The demo's preflight has a Wi-Fi-off check, but the rehearsal with Wi-Fi actually off has not been run yet.
11 · What has been proven
Every number below comes from a recorded run you can replay. No search flight so far has raised a false alert. Read them as “this behaviour works”, not as real-world detection rates: the scenes are simulated and the samples are small.
Sweep at 40 m, look closer at 20 m then 12 m. The rule for passing was written down before the test ran.
altitude testSix scenarios on items never used for tuning, plus the power-line regression. Four held-out clues at once: 4 of 4, no false alerts, within 0.05 m. The honest first try (before fixes) was 24 of 25 checks.
held-out end-to-end test reportThe six search scenarios on three seed sets with a 30-minute battery, no false alerts. The four-clue flight now covers the whole area every time (70 to 73% of the battery). The one miss: on one seed the finder never boxed the sleeping bag.
three end-to-end runs on a 30-minute battery; the 20-minute runs before them also gave 17 of 18A grey dog was planted but not described. The drone found the two things it was told about and never alerted on the dog.
demo runbook variantEvery finished search flight so far: 95 alerts, none false. Median distance from the planted item 0.05 m (simulator pose, flat ground).
all 45 finished search flightsWhen Clef-Flash was unsure at 12 m, qwen38 confirmed the real dog, helmet, sleeping bag or tent seven times and dismissed four background spots, in 1.5 to 2 s each.
every logged second opinion, matched to the answer keyThe same six flights scored two ways. Both found 14 of 16 clues. Reviewing only the sweep photos after landing raised 7 false alerts and needed 3 to 4 times the Clef-Flash calls.
live versus after-landing reportqwen38's sector plans cut the time from the first find to the second (15.5 s and 43.5 s on two seeds), for 25 to 34% more flying.
trail A/B, lockstep runs683 s at real speed, live in the console: backpack, tent and jacket found, tarp dismissed.
recorded real-time flight, score fileThe operator's sentence was transcribed, routed as a description, turned into clue cards, flown, and an alert was read aloud.
real-microphone console runWith the decision model unreachable, the drone finishes the area on FALLBACK, raises nothing, and comes home.
end-to-end scenario 4The original corridor scene passed its own end-to-end test on three seeds. In the latest run it failed once on its known vegetation false alarms (a tree that perspective puts near a wire).
corridor end-to-end runs
Figures as of 11 October 2026, 15:45: the project's status file and the recorded runs behind it, including the held-out test that passed at 15:16 and the repeats on two more seed sets that finished at 15:44.
12 · What doesn't work yet
This is a prototype built in a day. These are the known problems, roughly in order of how much they matter.
It went green at 15:16 after three fixes: photos spaced to the sweep height (photos spaced for 40 m had left gaps at 21.8 m, and the SOS letters were missed), leftover sectors flown nearest first, and trail plans refused when the battery couldn't finish them. On a 20-minute simulated battery the four-clue flight ran out before covering the area. On a more realistic 30 minutes (search drones in service fly 40 to 45 minutes and lose a quarter to a third of that in the cold) it covers everything, and 17 of 18 scenario flights pass. The one miss is a detection miss: on one seed the finder never boxed the sleeping bag, so Clef-Flash was never asked. The honest first-try number is still 24 of 25 checks.
On the tuning items Clef-Flash's close-ups score 0.9 and above. On held-out items it is often only 0.12 to 0.29, below its own 0.5 bar, and the find goes through only because qwen38's second opinion confirms it. In the passing held-out run, two of the four finds (the helmet and the sleeping bag) came that way. Without qwen38 on board they would have been dismissed.
MuJoCo scenes look like a game: flat ground, clean colours, props that came out of a 3D generator. They prove how the system behaves. The test that would say something about real hillsides is real aerial search imagery (HERIDAL, SARD, WiSARD), which has been approved; the dataset licences are being checked, and nothing is downloaded yet. The simulator stays as a switchable fallback.
Live and after-landing analysis found the same 14 of 16 clues. Live raised no false alerts against 7. A hybrid that re-checks the saved photos after landing could get both, and isn't built.
Each claim rests on a handful of flights: three seed sets per scenario, two seeds for the trail test, six flights for live versus after landing. 70 offline tests cover the logic, including one that poisons the answer key so any read of it during a flight fails.
The simulator knows the camera's exact position and the ground is flat, so alerts land within centimetres. A real drone has GPS error, slopes and wind. The alert position also comes from the high sweep photo; the close-up doesn't refine it.
The real-time flight's first note says the backpack was “28 meters north of center”; it was 28 m south. Before the fix, 18 of 78 notes had a wrong compass word. Code now writes the direction in words before qwen38 sees it (fixed at 14:37); all 10 notes since are right.
Rewording a clue after the same decoy keeps fooling Clef-Flash is not built. Answering the operator's questions is basic: it reads a one-line status, not the mission log, and its prompt still says “power-line inspection drone”.
The alert is small enough for a radio link (about 270 bytes plus a 20 KB photo), but no radio sink exists; alerts go to the console and the log.
The search uses every answer however late, and the drone keeps flying meanwhile. Under GPU contention (2 to 6 s answers) photos get further apart and gaps can appear. Scored runs need the GPU to themselves.
Nothing calls the internet, but the demo has not been rehearsed with Wi-Fi actually off.
Items sit on the ground at a random angle with no physics settling, no realism check and no contact sheet. Hunyuan3D's licence excludes the EU, the UK and South Korea, so the props would need rebuilding with an MIT-licensed engine before any use there.
It boxes only the three rarest-coloured spots per photo, so a large object in a common colour can go unnoticed. Its ranking was tuned on the tuning items.
“Go back to the last alert” with no alert yet used to get “Going back”. Since 14:55 the drone speaks only after the mission has accepted or refused.
13 · Where it came from
Patrol began at 1 am as a drone that flew along power lines and flagged broken insulators. By late morning the loop worked, but the case for it was weak: a trained detector is the honest tool for known defects, and line photos are fine to review after landing. Search and rescue needs both things this design is good at: recognising things from words, and deciding while the drone is still there.
Built by a relay of Claude Code agents, each handing over to the next through the project's handoff, status, log and decision files, with the owner steering.
14 · What comes next
| Next | Why it matters | Needs |
|---|---|---|
| The finder's blind spot | The last held-out miss was a sleeping bag the colour finder never boxed. Trying more boxes per photo, or a second finder cue, is the next lever | next |
| Wi-Fi-off rehearsal | Run the whole demo with the network actually off | ten minutes before the demo |
| Real aerial photos | Replay real search imagery (HERIDAL, SARD or WiSARD) through the finder and Clef-Flash. The only test that speaks to real terrain | approved; dataset licences being checked |
| A hybrid after-landing pass | Re-ask every saved photo after landing, but send a team only to spots no close-up has already cleared | a day of work |
| Compute budget sweep | One Clef-Flash decision every 1, 2 or 4 s: how slow can it be before finds are missed? | GPU time |
| Real flight software | PX4 in simulation, then a real drone | weeks, and hardware |
| A small trained detector | Let Clef-Flash label flights and train a tiny detector from its labels, for a cheaper first pass | later |
15 · Glossary and sources
| What | Source |
|---|---|
| The goal and the bar for done | the project brief |
| Milestones and their evidence | the project status file, where every “done” names a run or report |
| Why each choice was made | the dated decision log |
| Every flight on this page | recorded runs: each keeps a tick-by-tick mission log, a score against the answer key and its photos, and can be replayed in the console |
| How the system behaves | the search, planner, finder, geometry, voice and thinker code, which wins wherever the notes disagree |
The page is rebuilt from the project's own recorded runs. Diagrams, charts and the flight map are generated from the same data, so a new flight or a new result updates them in one step, and every number keeps a note of where it came from.