1. Result at a glance

The 2.106 ms is wall-clock throughput per original photo, computed from the evaluator's total run time divided by the number of input photos. It is not the latency of processing one photo alone: m640 processes four photos together and shares card-pass overhead between them.

2. The whole journey: from baseline to m640

This project did not start with custom kernels. We first measured the end-to-end submission, identified where time and accuracy were being lost, fixed the highest-impact causes, and only then built device kernels and chose the operating point. Each phase changed what the next phase needed to optimize.

The rest of this document unpacks the final m640 architecture and explains the mechanisms behind each phase.

3. End-to-end architecture

The model remains YOLOv4: the work was in restoring its intended behavior, selecting an efficient input/execution mode, reducing framework overhead, and replacing expensive operations with targeted kernels. The weights and YOLO architecture were not replaced with a different detector.

Where the model's pieces sit

4. Why the selected configuration is m640

m640 is a four-image mosaic using 320-pixel tiles. It is not four native 640 × 640 inferences and does not give each photo 640-pixel detail. Each source photo is resized to 320 × 320, then placed in one quadrant of a 640 × 640 canvas.

photo A ┌──────────┬──────────┐ photo B
        │ 320×320  │ 320×320  │
        ├──────────┼──────────┤
photo C │ 320×320  │ 320×320  │ photo D
        └──────────┴──────────┘
             one 640×640 model pass

The card's graph has a mostly fixed cost per pass: launching and scheduling hundreds of operations costs time even when each operation does little arithmetic. Four photos in one pass amortize much of that fixed cost. The model still processes the pixel work for all four tiles, but the per-pass overhead is shared.

Correctly recovering each photo's boxes

1. Resize each input photo to its 320 × 320 tile and place it in the 2 × 2 canvas.
2. Run the 640 × 640 YOLO graph once.
3. For each predicted box, use the grid cell that produced it to identify its tile. The implementation groups that tile's cells back into the normal per-image head layout.
4. Decode and rescale each tile's boxes in its own 320 × 320 coordinate system, then map them to the original photo dimensions.
5. Apply a small confidence penalty to boxes near internal tile edges, where an object may be cut by the mosaic boundary.

Because each prediction belongs to the cell that emitted it, tile ownership can be recovered from the head-grid location rather than guessing from the final box center. This matters when predicted boxes extend beyond a tile boundary.

Why this point on the speed/accuracy curve?

m640's practice mAP is .1116 above the scoring floor of .447. The choice gives up some accuracy versus larger single-image inputs, but keeps substantial headroom while retaining a large speed advantage.

5. How the accelerator runs it

The Wormhole n150 provides an 8 × 8 grid of 64 Tensix cores. Each core has local SRAM (L1), data-movement engines, and separate matrix and vector math units. The host CPU prepares work and orchestrates the device; it does not replace the card's neural-network computation.

• FPU: the matrix engine used for convolution/matmul work and additions.
• SFPU: the vector unit used for elementwise functions such as Mish and max operations.
• Unpacker and packer: move between L1 tile formats and registers. Tile data is distributed across cores, so layout and sharding determine whether an operation can avoid copies.
• Command queues and traces: record/replay the model work and overlap transfers with computation where dependencies permit.

Precision in the selected m640 graph

The custom kernels compute in the destination registers and the packer writes the output in the tensor's format (bf16 for m640).

6. Host pipeline: keep work moving

The reference path did not spend all its time doing neural-network arithmetic. Decode, conversion, Python dispatch and post-processing could leave the card waiting. The changes overlapped independent stages and reduced CPU scheduling overhead.

7. Accuracy recovery: restore the model's intended activation

The card port initially scored .464 mAP@50, while the same weights in an fp32 CPU reference scored .558. Increasing precision did not fix the gap: bf16 weights with HiFi4 scored .445, no better than the .464 LoFi baseline. The investigation found that the port had substituted HardMish for the YOLOv4 model's Mish activation in 72 locations.

Replacing Mish with HardMish in the fp32 reference reproduced most of the observed accuracy loss (.558 → .479, versus .464 on the card). Restoring real Mish on the card improved the 320 practice result from .464 to .5145 (about +.05 mAP), at roughly +0.7 ms for the library implementation. That accuracy fix motivated the custom Mish kernel, which made the more faithful activation much cheaper.

Precision was treated as a measured knob, not a blanket fix

• Fp32 accumulation helped at 320 (+.024 mAP for about +0.22 ms) but did not consistently help the larger or mosaic graphs; the selected m640 uses ACC=0.
• The first convolution is a targeted exception: PREC_FIRST=1 applies bf16 with HiFi4 and fp32 accumulation. It is a small portion of the graph and measured as beneficial or neutral in 13 of 14 comparisons.
• Using bf16 activations everywhere at 320 crashed the graph. Precision changes therefore remained constrained by the graph's supported layouts and types.

8. Detection heads and box processing

Return raw heads instead of hundreds of tiny card operations

The standard model's genboxes tail performs sigmoid/exp, reshapes and concatenations to turn the three detector heads into candidate boxes. It was implemented as many small operations with meaningful dispatch overhead. The raw-head path stops after the head convolutions, packs the outputs, and transfers them to the host. The host decodes candidates in compiled code and applies the same coordinate transforms and filtering logic.

At 320, the genboxes tail was about 229 operations and 1.133 ms of a 3.858 ms frame. Moving this tail off-card reduced a measured pipeline from 5.95 to 3.74 ms/frame. Those figures are from the earlier 320 pipeline, not an isolated m640 ablation.

Faster exact post-processing, then an accuracy-oriented variant

The mAP metric is AP at IoU 0.5, averaged over classes with ground truth. The post-processing changes deliberately emphasize recall and ranking. Extra candidates cost host work, so the confidence floor and NMS settings were measured rather than lowered without limit.

9. Four custom Tensix kernels

The kernels target different sources of cost: one expensive nonlinear function, repeated graph boundaries, a multi-op pooling block, and layout conversions in the neck. They were implemented as custom device operations and ported to the six selected configurations, including bf16 variants for the larger graphs.

Combined kernel result

On m640, the custom stack reduced COCO1k time from 2.416 ms/photo in the base folder to 1.672 ms/photo with kernels plus box-step v2. The comparison combines several device kernels and the host-only v2 change, so it is not a clean attribution of all 30.8% to one kernel. The paired COCO1k result was .6392 → .6439 mAP@50; the measured difference is small relative to dataset variation, not evidence that the kernels alone improve accuracy.

10. Runtime and I/O changes on the card

Remove work that does not change the tensor values

• Disable a halo-output reallocation that copied convolution outputs into fresh buffers; 20 Move operations disappeared in the tested graph.
• Skip an unnecessary input Move.
• Skip three add(x, 0) operations that only converted dtype where the following operation accepted the tensor directly.
• Copy output to a DRAM buffer and read it on command queue 1 with events, allowing the download for one pass to overlap the next trace where dependencies allow.

The combined config/I/O patch reduced the measured graph from 401 to 379 operations, cut the inter-frame idle gap from 114 to 4 µs, and saved roughly 0.12–0.23 ms/frame in the original ablation. The exact gain varies with graph and batch mode; the operation count is not a claim that every removed operation saves the same time.

Why reducing operation count matters

The 320 graph was estimated at roughly 10 GMAC, below a millisecond of idealized arithmetic on the card. In practice, many small graph operations and dispatch gaps dominated. Therefore the design prioritized fewer launches, fused dataflow, sharding-aware writes and overlapping transfers over simply increasing arithmetic precision or peak compute.

11. What drove the end-to-end improvement

The following measurements come from different stages of development. They show the direction and scale of each lever, but they are not additive: some measurements use different input sizes, datasets, timing protocols or overlapping optimizations.

Cross-checks and repeatability

• The kernels and box-step v2 were run together on all six configurations; m640 practice and COCO1k measurements are in that cross-check.
• The custom kernel stack had zero fallbacks in the six-folder runs; unsupported shapes would otherwise use stock operations.
• A broader stress suite reported zero crashes or missing outputs across 396 runs and included malformed/edge files. Its synthetic hidden-like 800-image draws are a robustness check, not the official hidden-set score.
• Accuracy comparisons use paired evaluation where possible; COCO1k bootstrap intervals are more informative than small practice-set differences, especially for rare classes.

12. Trade-offs and approaches we rejected

13. Team contributions and ownership

The final pipeline is a team integration. The documented work spans host input, model numerics, post-processing, custom device kernels and evaluation:

This attribution reflects the project notes. End-to-end latency and mAP figures are team results, not single-person results.

14. Short glossary

15. The complete YOLOv4 configuration set

The team did not tune only m640. We measured six points on the speed/accuracy frontier, then applied the custom kernels and box-step v2 across all six. The table below puts the selected setting in context and closes the journey with the alternatives it was chosen against.

All rows are the measured kernels + box-step-v2 cross-check. Practice has 200 images; COCO1k has 1,000. Timings are per source image, including batching/mosaic throughput. The suffixes encode the chosen accumulation (a0/a1) and first-layer precision (p0/p1) settings; mosaic rows process four source images together.

Why the team selected m640

• m512 wins on raw throughput, but its practice mAP is only .0317 above the .447 floor. The team chose not to risk that narrow margin for the additional speed.
• r320 has similar practice mAP to m640, but is slower because four inputs no longer share one model pass.
• r384 and r512 trade throughput for more spatial detail and higher mAP. They are valid points on the curve, but do not match the selected speed objective.
• r640 is the most accurate on practice, but the COCO1k result is almost the same as r512 while taking substantially longer.

Alternative detector architectures considered

We also checked whether replacing YOLOv4 would produce a better frontier. These are exploratory comparisons, not configurations used for the selected run. The alternative-model mAP figures below are fp32 CPU mAP@50 on COCO1k at 640 input size (YOLOv4 is shown with its card result); card timings are reported separately because the tested alternatives did not all complete the same end-to-end harness path.