Vision System Roadmap — Sowbot / Avaota A1

crop_row_node · YOLOX-Nano Weed Detection · ROS 2

Deployment targets: UK outdoor lettuce · India young crop weeding Platform: Allwinner T527 (8× A55, Zhouyi Z2 NPU @ 2 TOPS)

---

1. Current State Assessment

What works and must be preserved

The crop_row_node architecture is sound and should not be restructured. Specifically:

• The Cherubini & Chaumette visual servoing interaction matrix is mathematically correct for monocular row-following and well-suited to the slow-speed, low-inertia dynamics of a field weeding robot.

• The ROS 2 node design — AOC conditions on separate topics, heartbeat for PERCEPTION_DEGRADED (M10), service-gated cmd_vel — is clean, safe, and should be treated as fixed architecture.

• The BSD-2 provenance chain from Agricultural-Robotics-Bonn is properly tracked. Keep it.

None of the controller logic, topic structure, or service interface needs to change at any point in this roadmap.

---

The core fragility

The entire perception stack rests on one assumption:

Crop rows are the dominant source of green pixels in the image.

For UK outdoor lettuce this assumption holds reasonably well under the diffuse overcast light that dominates British growing conditions — which is actually close to the ideal case for ExG. However it breaks down in four conditions that will be encountered regularly:

• Early crop stage — lettuce at emergence is small and sparse; weeds may be denser and larger than the crop itself

• Weed pressure — weeds are ExG-positive by definition; the pipeline cannot distinguish crop green from weed green

• Mud splash — wet UK field conditions deposit soil on lower leaves, suppressing green channel values and fragmenting contours

• Indian deployment — stronger directional sunlight and greater illumination variation cause saturation and shadow artefacts that unnormalised ExG cannot handle

The first two conditions are universal to both deployments. The last two are context-specific but both foreseeable.

---

The architectural conflict

The two subsystems are currently in direct competition:

Camera frame
  │
  ├─► ExG mask → contours → row centre    (consumes ALL green pixels)
  │
  └─► YOLOX-Nano → weed bounding boxes   (identifies a subset of green pixels)

Weeds that YOLOX correctly detects are simultaneously being used by the row-following pipeline as evidence of crop row position. Under heavy weed pressure — precisely when weed detection matters most agronomically — the row-following steers toward weeds. This must be resolved before the system is considered reliable.

---

Image topic architecture (decided)

Both crop_row_node and weed_detect_node subscribe to the same topic:

/caatinga_vision/row_nav/image_raw

Run both nodes as composable nodes in a shared container with intra-process communication. The camera image is published once, serialised once, and delivered as a shared pointer — zero copy between the two nodes on the same SBC. No code changes to either node are required; the topic name is identical in both.

# launch/vision.launch.py
from launch_ros.actions import ComposableNodeContainer
from launch_ros.descriptions import ComposableNode

container = ComposableNodeContainer(
    name='vision_container',
    namespace='',
    package='rclcpp_components',
    executable='component_container',
    composable_node_descriptions=[
        ComposableNode(
            package='sowbot_vision',
            plugin='sowbot_vision::CameraSourceNode',
            name='camera_source_node',
        ),
        ComposableNode(
            package='sowbot_vision',
            plugin='sowbot_vision::CropRowNode',
            name='crop_row_node',
        ),
        ComposableNode(
            package='sowbot_vision',
            plugin='sowbot_vision::WeedDetectNode',
            name='weed_detect_node',
        ),
    ],
)

cv_bridge.imgmsg_to_cv2() in each node still performs its own Mat allocation — at 640×480×3 this is ~1.8 MB peak simultaneous. This is not a bottleneck at these resolutions and should not be optimised until profiling demonstrates otherwise.

---

Latency budget

At 30 Hz the end-to-end budget is 33 ms from image capture to cmd_vel publish. Estimated current pipeline (unverified — profile on-device before any optimisation):

Stage	Estimated
ROS 2 intra-process delivery	< 0.5 ms
imgmsg_to_cv2 decode	1–3 ms
ExG + Otsu + dilation (640×480)	3–8 ms
findContours + centroids	1–4 ms
detect_crop_rows (polyfit × 8 windows)	< 1 ms
visual_servoing_ctl	< 1 ms
Total	~6–17 ms

There is margin. Measure this first. Every subsequent change must be benchmarked against this baseline.

---

2. Immediate — Harden What Exists

(No model training. No NPU SDK. No architectural changes.)

---

M0 · Profile on-device (Week 1)

# Measure header-stamp → publish latency on the debug image
ros2 topic delay /caatinga_vision/row_nav/debug_image

# Per-message timing
ros2 run rqt_topic rqt_topic

Record: mean latency, p95 latency, CPU core utilisation (htop), memory bandwidth (sudo sar -b 1). This is the baseline. No claim about performance is valid without it.

---

M1 · Normalised ExG (Week 1–2)

The current ExG is unnormalised. Replace the calculation in compute_exg_mask:

# BEFORE — unnormalised, saturates in bright sun
exg = 2 * g - r - b

# AFTER — normalised to [−1, 1], illumination-robust
img_f = bgr_img.astype("float32")
b, g, r = img_f[:,:,0], img_f[:,:,1], img_f[:,:,2]
total = r + g + b + 1e-6
exg = 2*(g/total) - (r/total) - (b/total)

# Scale to uint8 for Otsu (maps [−1,1] → [0,255])
exg_u8 = np.clip((exg + 1.0) * 127.5, 0, 255).astype("uint8")

Why this matters for UK lettuce: Even under consistent British overcast, the robot will operate at golden hour (early morning slots to avoid compaction on wet soil), under poly-tunnel edges, and in partial cloud transitions. Normalised ExG is stable across all of these at zero compute cost.

Why this matters for India: Strong directional sun causes channel saturation. Normalised ExG recovers correctly where unnormalised ExG collapses to zero contrast.

This is a one-function change, no retraining, no SDK. Do it first.

---

M2 · Weed exclusion mask (Week 2–3)

Before ExG processing, zero out image regions that YOLOX-Nano has already confirmed as weeds. This resolves the architectural conflict described in section 1 at zero additional inference cost.

Add a second subscriber and a temporal synchroniser to crop_row_node:

from message_filters import ApproximateTimeSynchronizer, Subscriber
from vision_msgs.msg import Detection2DArray

# Replace the single image subscriber with a synchronised pair
self._img_sub = Subscriber(self, Image, self.image_topic)
self._det_sub = Subscriber(self, Detection2DArray, "/weed_detections")
self._sync = ApproximateTimeSynchronizer(
    [self._img_sub, self._det_sub], queue_size=10, slop=0.05)
self._sync.registerCallback(self._on_image_and_detections)

Masking function — called before compute_exg_mask:

def _mask_weed_regions(self, bgr: np.ndarray,
                        detections: Detection2DArray) -> np.ndarray:
    masked = bgr.copy()
    for det in detections.detections:
        bb = det.bbox
        x1 = max(0, int(bb.center.position.x - bb.size_x / 2))
        y1 = max(0, int(bb.center.position.y - bb.size_y / 2))
        x2 = min(self.img_w, int(bb.center.position.x + bb.size_x / 2))
        y2 = min(self.img_h, int(bb.center.position.y + bb.size_y / 2))
        masked[y1:y2, x1:x2] = 0
    return masked

The 50 ms slop tolerance accommodates YOLOX-Nano running at 10–15 Hz while crop_row_node runs at 30 Hz. A false weed detection that masks a crop cluster will degrade the row centre estimate — which is exactly why M3 (YOLOX validation) must precede relying on these boxes in production.

---

M2.1 · Parameterise the dilation kernel (Week 2–3)

The 10×10 dilation kernel is hard-coded for a specific crop density. Lettuce at emergence has different spacing than lettuce at canopy closure, and Indian crop rows may have different plant separation entirely.

Move to crop_row_params.yaml:

crop_row_node:
  ros__parameters:
    # UK lettuce — adjust per crop stage
    dilation_kernel_size: 10    # px; increase at emergence, decrease at canopy
    min_contour_area: 10.0      # px²; increase to filter soil/mud splash noise
    n_scan_windows: 8
    window_width: 80

Add a merge step that groups contour centres within dilation_kernel_size pixels of each other before passing to detect_crop_rows. This makes the system tuneable across crop stages without code changes.

---

3. Short-Term — NPU Weed Detection Deployment

(Weeks 3–8)

---

M3 · Validate YOLOX-Nano on UK lettuce weeds (Week 3–5)

Before any NPU bounding boxes are trusted as exclusion masks (M2), verify the model’s performance on the actual target weed species. Common weeds in UK outdoor lettuce:

• Annual meadow grass (Poa annua)

• Fat hen (Chenopodium album)

• Chickweed (Stellaria media)

• Groundsel (Senecio vulgaris)

• Shepherd’s purse (Capsella bursa-pastoris)

• Field speedwell (Veronica persica)

Collect a minimum 300-image validation set from the target fields at multiple crop growth stages. Measure per-class precision and recall.

A false positive (crop detected as weed) that triggers the exclusion mask directly corrupts the row-centre estimate. Set a high confidence threshold (≥ 0.55) for the exclusion mask even if this reduces recall, because the cost of a false positive here is a navigation error, not just a missed detection.

For India, repeat this process locally before deployment — weed flora will differ substantially from UK species.

---

M4 · NPU inference latency benchmark (Week 4–6)

After completing the Zhouyi SDK setup (see companion deployment guide), establish actual on-device timing before committing to any inference frequency targets:

# Using the siengine runtime demo as the template
# (from AI610-SDK-r1p3-AIoT/siengine/nn-runtime-user-case-example)
./weed_detect_bench \
    --model yolox_nano_weed.bin \
    --input test_640x480.jpg \
    --iterations 200

Record: mean, p50, p95, p99 inference latency. Run with sar -b 1 in parallel to measure DDR bus utilisation during NPU inference — this quantifies memory contention with the CPU-side ExG pipeline.

Targets to confirm before proceeding to M5:

Metric	Target
YOLOX-Nano mean inference	< 80 ms (≥ 12 Hz)
CPU pipeline latency during NPU inference	< 25 ms
DDR utilisation peak	< 70%

If CPU pipeline latency degrades significantly during NPU inference, introduce a frame-skip strategy: NPU inference runs every N-th frame, crop_row_node runs every frame using the most recent available detections.

---

M5 · Composable node container integration (Week 5–6)

Formalise the shared-topic, intra-process architecture from section 1 into the production launch file. Verify with:

ros2 component list
# All three nodes should appear under /vision_container

ros2 topic hz /caatinga_vision/row_nav/image_raw
# Single publisher at 30 Hz

Confirm that intra-process communication is active by checking ROS 2 logs for "Intra-process communication enabled" on startup for each subscribing composable node.

---

4. Medium-Term — Replace ExG with NPU Segmentation

(Months 2–4)

This is the highest-impact change and the most work. ExG with weed masking (M2) is the bridge — it must be stable in production before this work begins.

---

M6 · Training data collection (Months 1–3, parallel workstream)

This is the long-lead item that gates everything in section 4. Begin immediately in parallel with M0–M5.

Target: 1,200 pixel-wise labelled images, three classes: {lettuce_row, weed, soil/background}

Capture requirements for UK outdoor lettuce:

Condition	Target images
Overcast (typical UK)	400
Bright sun / partial cloud	200
Early morning / low-angle light	150
Post-rain / wet soil with mud splash	150
Emergence stage (small, sparse plants)	150
Canopy closure stage	150

Semi-automatic labelling strategy — use existing models as labelling tools rather than labelling from scratch:

1. Run YOLOX-Nano detections → weak weed labels

2. Run normalised ExG + Otsu → weak crop/soil labels

3. Manually correct disagreements, edge cases, plant boundaries, mud-splashed leaves

This reduces manual labelling effort by approximately 60–70% compared to labelling from scratch. Tools: CVAT (self-hosted), Label Studio, or Segments.ai.

Useful pretraining datasets (European, directly applicable to UK):

• CropAndWeed Dataset — Steininger et al., 2023

• PhenoBench — Weyler et al., 2023

• WeedMap — Ramirez et al., 2020

India adaptation: A separate 400-image set captured locally is required before Indian field deployment. Weed flora, soil colour, illumination conditions, and crop appearance differ enough from UK that a UK-trained model will degrade significantly without domain adaptation. Do not skip this step.

---

M7 · Segmentation model selection and training (Month 2–4)

Target architecture: MobileNetV2 + LR-ASPP at 320×240 input.

Architecture	Input	Est. Z2 NPU latency	Verdict
MobileNetV2 + LR-ASPP	320×240	~15–25 ms	✅ Start here
STDC-Seg (STDC1)	320×240	~10–20 ms	✅ Evaluate second
ERFNet	320×240	~30–50 ms	⚠️ Marginal budget
SegNet	any	> 60 ms	❌ Too heavy

Do not use 640×480 input for the first deployed version. 320×240 reduces NPU compute and DDR transfer by 4×. The downstream detect_crop_rows polyfit is stable at half resolution — lettuce rows are regular enough that sub-pixel row-centre accuracy is not required.

All latency estimates are from comparable 2–4 TOPS embedded NPU benchmarks. Validate on T527 before committing.

Training procedure:

1. Pretrain on CropAndWeed + PhenoBench (three classes remapped to lettuce_row / weed / background)

2. Fine-tune on UK field dataset (M6)

3. Quantise to INT8 using Zhouyi Compass Optimizer (AIPUBuilder) with UK field images as calibration data — do not use ImageNet calibration

4. Acceptance criterion: INT8 mIoU must be within 3 percentage points of float model mIoU before deployment

---

M8 · Integration into crop_row_node (Month 3–4)

The segmentation output replaces exactly two function calls. Everything downstream is unchanged:

# BEFORE
mask    = compute_exg_mask(bgr)
centers = get_plant_centers(mask, self.min_area)

# AFTER — seg_mask delivered via /seg_mask topic (mono8)
# published by weed_detect_node after NPU inference
seg_mask  = self._latest_seg_mask          # 0=soil, 1=crop, 2=weed
crop_mask = (seg_mask == 1).astype(np.uint8) * 255
weed_mask = (seg_mask == 2).astype(np.uint8) * 255
centers   = get_plant_centers(crop_mask, self.min_area)

The dense weed mask replaces the YOLOX bounding box exclusion from M2 — pixel-accurate, no confidence thresholding needed for the navigation path.

NPU time-sharing strategy (Zhouyi Z2 runs models sequentially):

• Segmentation: every frame at 30 Hz (~15–25 ms), primary navigation input

• YOLOX-Nano: every 3rd frame at 10 Hz, used for precise weed localisation to drive treatment actuation (sprayer trigger / mechanical weeder position), not for navigation

At 0.15 m/s forward speed, 10 Hz YOLOX gives a weed position update every 15 mm of travel — sufficient for actuation timing on a robot at this speed.

---

5. Long-Term — Semantic Field Navigation

(Months 6–12)

---

M9 · Row-end and headland detection (Month 6)

The current node has no concept of reaching the end of a row. Add a headland class to the segmentation model trained in M7. Publish:

/aoc/conditions/row_end   (std_msgs/Bool)

The Limbic System uses this to trigger headland turning. This eliminates the need for a separate sensor or waypoint-based row-end detection.

---

M10 · Row identity and cross-row localisation (Month 7–8)

The polyfit gives row-relative position but no global field map. After segmentation is stable, add a row-counter derived from frame-to-frame tracking of the lateral phase of detected row positions. This allows the Limbic System to maintain which row the robot is in, enabling:

• Skip-row weeding patterns

• Per-row treatment records

• Return-to-row after interruption or e-stop

No additional sensors required — derived entirely from the existing segmentation output.

---

M11 · Weed pressure maps (Month 8–9)

Accumulate weed detections across the field geo-referenced by wheel odometry or RTK-GPS. Publish as:

/aoc/maps/weed_pressure   (nav_msgs/OccupancyGrid)

This produces per-row, per-bed weed density estimates — the primary agronomic data output of the system. In UK commercial lettuce production this also supports the field records required under Red Tractor and LEAF Marque standards, even for mechanical weeding operations.

---

M12 · Multi-task unified network (Month 9–12)

Replace the two-model time-sharing arrangement with a single network:

Shared MobileNetV2 encoder
  │
  ├─► Segmentation head   (lettuce / weed / soil / headland)
  └─► Detection head      (weed bounding boxes for actuation)

Single NPU inference per frame. Shared feature extraction removes approximately 60% of redundant compute. Both outputs are temporally aligned with no synchroniser required.

This is a 3–4 month model development project and should not be started before the single-task segmentation model (M7) has been validated and stable in production for at least one growing season.

---

Dependency and Risk Register

Item	Severity	Status	Mitigation
ExG corrupted by weed pressure	Critical	Active now	M2 immediately; M8 resolves permanently
YOLOX false positives corrupting row centre	High	Active when M2 deployed	Confidence threshold ≥ 0.55; validate M3 first
T527 NPU latency unverified	High	Blocks M5+	Benchmark (M4) before any Hz commitments
UK field labelling dataset — long lead time	High	Begin immediately	Semi-automatic labelling; parallel workstream from day 1
India domain shift vs UK-trained model	High	Future deployment	400-image local adaptation set; re-quantise with local calibration
DDR bus contention CPU + NPU concurrent	Medium	Measure at M4	Frame-skip if needed; time-sharing in M8
Mud splash degrading ExG in wet UK conditions	Medium	Active	M1 normalisation reduces impact; M8 resolves permanently
Composable node intra-process not activating	Low	Verify at M5	Check ROS 2 logs on startup; fall back to standard pub/sub if needed

---

Milestone Summary

#	Milestone	When	Output	Prerequisite
M0	Baseline profiling	Week 1	Measured latency budget on T527	—
M1	Normalised ExG	Week 1–2	Illumination-robust perception, one function	M0
M2	Weed exclusion mask	Week 2–3	YOLOX boxes suppress ExG false positives	—
M2.1	Parameterise dilation kernel	Week 2–3	crop_row_params.yaml tuneable per crop stage	—
M3	YOLOX UK weed validation	Week 3–5	Per-species P/R, confidence threshold set	M2 production trust
M4	NPU latency benchmark	Week 4–6	Confirmed Hz budget, DDR contention profile	M5, M7
M5	Composable node container	Week 5–6	Zero-copy image sharing, production launch	M4
M6	UK field dataset (parallel)	Month 1–3	1,200 labelled images, three classes	M7
M7	Segmentation model v1	Month 2–4	MobileNetV2+LR-ASPP, INT8, on NPU	M6, M4
M8	Unified segmentation pipeline	Month 3–4	ExG removed, seg 30 Hz, YOLOX 10 Hz	M7 stable
M9	Row-end / headland detection	Month 6	/aoc/conditions/row_end, Limbic turning	M8
M10	Row identity tracking	Month 7–8	Row counter, return-to-row	M8
M11	Weed pressure maps	Month 8–9	OccupancyGrid, per-row agronomic records	M10
M12	Multi-task unified network	Month 9–12	Single NPU inference, seg + detection	M8 one season stable