Read the transcript
0:00 · Title
A short story about testing SAP systems, told by three characters: an agent, a messenger, a real system. And about a mindset, Spec-Test-Driven Development (STDD): what we test is first observed on the real system.
0:17 · Meet the cast
Stedd is the AI agent. One hat per job: plan, generate, verify, heal. MCP is the messenger: requests out to Robot Framework, answers back. Terra is the real SAP system. Two of them: Terra 2023, our target, and Terra A4H, its twin.
0:36 · The usual way
Most test automation starts from what we assume the system does: a document, a memory, a screenshot. The test is written from the assumption, without looking at the system. It runs. It turns green. But the real system was never asked: the tile does not exist, the app lives elsewhere. Green, and false.
1:00 · From TDD to STDD
You may know TDD: red, green, refactor. The test comes first, the code follows. STDD applies the same logic one brick earlier: the spec comes first. In theory, a spec is green by default: nobody tests it. STDD puts it to the test on the real system: red where it was wrong, green once observed. Only then comes the test.
1:25 · Three links
STDD is a chain of three links. Each one drives the next, and each one is checked by a script. 1 · The spec, written in human language, then confronted with the real system by an agent, before any code. 2 · The test, derived from the spec as it was observed, and sealed with the spec's fingerprint. 3 · The library, developed from what the test needs: never a workaround hidden in the test. Three scripts: agent_contract.py, check_spec_sync.py, check_conventions.py. They fail loudly.
1:59 · A real case: scenario 4
A real case: scenario 4, on Terra 2023 (ABAP Platform 2023): create a travel with a booking, in Fiori. The spec is a plain-language brief: seven lines, no code. Any form works: a goal, a user story, a BDD scenario… This one assumes two things: a Launchpad tile opens the app, and the target is the cap-sflight demo.
2:25 · Link 1 · the spec meets the real system
Link 1. The planner agent, an LLM, confronts the brief with the live system, before writing a single test. It does not guess. It sends MCP: Robot Framework drives SAP for real, the LLM only reads the answers. Assumption 1, a Launchpad tile: 62 intents checked on Terra 2023. None opens a travel app. Refuted. Assumption 2, cap-sflight: a local CAP app on port 4004. Not this target. Here, the app is RAP: /DMO/UI_TRAVEL_D_D. The real app exists anyway: 4,136 travels, and the server generates a Fiori Elements preview… behind an HTTP 500. The fix is read in the server's own source code: the address is obfuscated, +20 per character. Decoded: HTTP 200. Before writing anything: can we undo it? Discarding a draft needs If-Match. Tried, then proven: reversible. Side by side: the brief as assumed, the spec as observed on Terra 2023, line by line. No tile, another target, a required currency the screen hides, an ID that gets reassigned: none of it could be guessed. 12 deviations in all, each backed by a measurement. A human reviews them before anything is generated. The spec as observed, 12 deviations from the brief included, is written down, sealed with a sha256 and reviewed by a human. It becomes the source of truth. To remember: the spec is observed, not assumed. The surprises come before the tests, not after.
4:17 · Link 2 · the test, derived from the spec
Link 2. The generator agent turns the observed spec into a Robot Framework suite, in business vocabulary. Every step is executed on Terra 2023 first, through MCP, and written only once it works. No guessed locator. The suite carries the spec's fingerprint, its sha256: the test knows exactly which spec it was derived from. Change the spec without regenerating, and check_spec_sync.py refuses the run. Regenerated, reviewed: in phase again. A script, not a promise. To remember: nothing is written that has not run first, and the suite stays sealed to its spec.
5:02 · Link 3 · the test drives the library
Link 3. Sometimes the test needs something the library cannot do yet: a date format, a missing header… …then the gap becomes a keyword in the library, with its unit test. Never a workaround in the test: check_conventions.py watches. Six capabilities, each with its unit test: dates typed in the field's own format, If-Match, the release read over HTTP, the preview address… The library ships to PyPI, as robotframework-sapfx. The next project gets all six for free. To remember: the test drives the library, and every gap closed there helps the next project.
5:53 · An independent verifier
Stedd changes hats. The verifier reads the plan, the suite and the evidence. It runs nothing. Its first verdict on the green suite: not verified. 13 reserves, 3 of them blocking. All 13 fixed, then a counter-review: needs human, 7 more points. Fixed too. Every guard must be seen refusing. First, the reference on Terra 2023: release 758, 58 services. Then the suite is pointed at Terra A4H, on purpose: same system ID, same host name as Terra 2023. Compared: 754 ≠ 758, 38 ≠ 58, no UI5 runtime. Every guard refuses. Seen, not assumed. Second provocation: a draft planted by hand on Terra 2023. The guard spared it, the cleanup deleted it. Found, fixed, replayed. To remember: a guard that nobody has seen refuse proves nothing.
6:58 · Result
Result: 7 tests out of 7 on Terra 2023. Every travel created is deleted, and proven gone, entity by entity. Generating cost 1.67 M LLM tokens, once. Replaying costs zero tokens: plain Python, under five minutes.
7:19 · When the system changes
A real case, 1 September: the public OpenUI5 Demo Kit changed its shell overnight. Not one line of our repo moved. The campaign drops to 8 of 12. The healer's diagnosis: not a locator, the flow itself changed. The spec is marked STALE. The planner goes back to the real target, rewrites the spec on what it measures, and the suite follows: 12 of 12, sealed again. The other branch, tested blind on Terra A4H: a locator drift injected on purpose. The healer reads the live screen… …and patches one line of resources/. Tests touched: zero. Re-run: 7 of 7. STDD is a loop, not a pipeline. The healer repairs resources/, never the test. A changed flow goes back to the spec, and the suite follows it.
8:15 · Conclusion
Spec first, in human language. Test from the spec. Library from the test. Each link guarded by a script. Observed, derived, guarded, and kept alive when the system changes. Thanks for walking with Stedd, MCP and Terra.
