Physics World Models · Building an End-to-End AI Harvest Planner — Problem Analysis, Requirements & Architecture Design

Part 1 — Problem & Design  ·  Where the Rules Are Known but the World Is Missing

The first three posts in this series asked what happens when the rules are invisible. Demand, grid load, traffic. In each case a law was hiding inside a time series, and the work was to find it and use it to see a failure before it arrived.

This series inverts that. In physical systems the laws are not hidden. Contact, torque, gravity: none of it needs to be inferred from data. What was missing was somewhere to apply them.

The apple that has to keep its stem

An apple sold as a gift or through a premium counter has to arrive with the stem attached. Not for appearance. The stem is the seal at the top of the fruit, and when it tears out it leaves an open wound at the calyx end that turns into a decay pathway in storage.

The trade knows this. Japan wraps the stems of high-grade fruit in cotton, Spain coats them. What the trade also knows is that a machine tends to take the stem off.

There are three ways an orchard picks fruit today, and none of them is built for the farm this project had in mind.

Hand harvest protects the grade and does nothing else well. It is seasonal, it is short of people, and the window does not move when the labour does not show up.

Mass harvesters shake the tree and catch what falls. Fast, cheap per kilogram, and catastrophic for the stem: reported stem pull runs to 30 per cent, and shake-and-catch has been measured as high as 81 per cent on Red Delicious. Fine for juice. Not for a gift box.

Drone fleets such as Tevel's fly a swarm of picking arms into the canopy.¹ The grade holds. So does the price, and it is priced for an operation with hundreds of hectares and staff to run it.

Table 1. Three ways an orchard picks fruit today.

That leaves a gap with a specific shape. Equipment that protects the grade is priced for large farms. Equipment a small grower can afford does not protect the grade. Nobody was building software to close that gap with a cheap arm.

Where the stem gives way

Before going further it is worth being precise about what “keeping the stem” means, because the whole project turns on it.

Figure 1. An apple hangs from a spur via the stem. The abscission zone — a natural separation plane that weakens as the fruit ripens — sits between the two. Four points compete when a gripper pulls; which one gives way first decides what the fruit is worth.

An apple hangs from a spur, a short branch that bears fruit for several years. Between the spur and the fruit runs the stem, and where the two meet there is an abscission zone: a natural separation plane that weakens as the fruit ripens, so it can eventually fall on its own.

Four points compete when a gripper pulls. Which one gives way first decides what the fruit is worth.

Table 2. Four points compete. Which gives way first decides what the fruit is worth.

The goal is the first row. Getting there means the stem has to be bent and twisted rather than pulled, which is what a person’s wrist does without thinking about it.

What the world model had to do differently

A world model rehearses an action inside an internal model of the environment before committing to it in the real one. That is what the previous three posts used to see a shortage, a blackout and a gridlock coming.

Each of those had a world to rehearse in. A demand series with a decade of history. A grid simulator. A traffic corpus with sensor readings every hour.

This one did not.

Figure 2. The loop was already there for games and robots — after Ha & Schmidhuber (2018). For apple picking, the environment itself had to be built first. In this project: MuJoCo apple-tree environment → detected fruit position and pose → outcome model + dynamics model → harvest planner (5 modules).

There is no simulator for an apple coming off a branch. More precisely: there is no record anywhere of what happened when one was picked. Public orchard datasets label where the fruit are, how large they are, which way the stem points. None of them record the outcome of an attempt, because nobody stands in an orchard writing down whether each apple came away cleanly.

So the loop had to start a step earlier than usual. Not at learning a compressed representation of a given environment, but at the environment.

The shape that follows from it

Figure 3. Three bands, and the top one is the part most systems do not need. Model Development runs once during development to produce the labels that do not exist in public data. Harvest Planning is the module that ships. Validation re-runs the finished plan through physics with the prediction model excluded from scoring.

Three bands, and the top one is the part most systems do not need.

Model development runs once, during development, and never again on a farm. Detections are filtered, a physics scene is built for every fruit, that scene is run repeatedly to produce outcome labels, and models are trained on those labels. This band exists only because the labels did not.

Harvest planning is the module that ships. Detection coordinates go in, a plan comes out: where the base stops, which fruit in what order, how long it takes, what grade mix to expect, and what carries over to a second pass. Perception sits upstream of it and motion control downstream, both out of scope.

Validation runs the finished plan back through the physics with the prediction model excluded from scoring. That band is the subject of Part 3, and it is where the project earned most of what it learned.

What Part 2 covers

Building the environment turned out to be most of the work, and it produced two things worth writing about on their own: a physics model that had to be verified before anything could be trained on it, and a dataset with a structure that no amount of field observation could produce.

Part 2 · Building the Model covers both.