# Echosaw vs Evidence Copilot head-to-head

Page: https://echosaw.com/vs/evidence-copilot/

The same two files were uploaded to both products and the same seven questions (questions.json) were typed word for word, one at a time, into each product's built-in assistant: Emma in the Echosaw report page, and Evidence Copilot's evidence agent on its 7-day trial. Answers were read from the rendered page and screenshotted. No API was used to ask the questions.

## Runs
- results/2026-11-12/: original run of both products (2026-11-12, 16:00-18:05 UTC).
- results/2026-11-12-rerun/: Echosaw only, re-run on its current release (2026-11-12, 22:33-22:40 UTC), on the same two uploads, after History > New chat. No new uploads. Evidence Copilot was not re-run; its answers are the ones in results/2026-11-12/. This run still had earlier turns in its chat context (see below), so it is kept as a record only.
- results/2026-11-12-final/: Echosaw only, final re-run on its current release (2026-11-12, 23:32-23:37 UTC, chat-assistant code ec67b2a8), on the same two uploads, in a clean chat context (both threads empty before Q1). Kept for the record.
- results/2026-11-13-rerun/: Echosaw only, re-run after three Emma fixes (2026-11-13, 01:02-01:07 UTC, chat-assistant code 53a75eb6), on the same two uploads, each question in its own empty chat thread. This is the run the page uses. No new uploads; Evidence Copilot not re-run.
  Scripts: scripts/clear_thread_new_chat.py (copy, clear and check the thread, then History > New chat) and scripts/ask_r3.py (ask one question and capture the answer).
- results/2026-11-13-0ad3d4aa/: Echosaw only, re-ask of Midway Q2-Q5 on release 0ad3d4aa (2026-11-13, 01:42-01:45 UTC), same upload, same cleared-thread procedure. Raw answers and screenshots (screenshots/2026-11-13-0ad3d4aa/) only; not scored.

Each run folder has runs.json (UTC times per question and the Echosaw revision live: chat-assistant code and pipeline revision), echosaw.json (verbatim rendered answers plus the stored chat turns), and results.json (per-question verdict against the ground truth; results/2026-11-13-0ad3d4aa/ has none because it is not scored). The original folder also has evidence-copilot.json (verbatim answers), answers.md (the test log as written during the run) and network observations (URLs, status codes and times only).

## Chat-context reset before the final re-run
For organization members, Echosaw keeps one report chat thread per media, and the chat assistant sends up to the last 20 turns of that thread as history. History > New chat clears the panel but not that thread, so the 22:33 UTC re-run (results/2026-11-12-rerun/) still had the earlier runs' turns in its context. To give the final run a clean context, on 2026-11-12 at about 22:58 UTC both test threads were deleted from the gamma chat table (gamma-echosaw-chat-conversations; no other environment touched):
- Midway, media e166ecfa-ba7d-43ae-b600-0cc4b97705b3, thread media-146ca20f3e16889b9251ee432964515bafe4aee2, 29 rows.
- NASA re-upload, media 80e168fc-73d6-4c33-9b57-b576117f99f8, thread media-b4dcbb75f9b8d9eda83c66f11a754910c9e4d5e8, 15 rows.
A copy of every turn in both threads, as it was before deletion, is in results/2026-11-12-rerun/chat-threads-before-deletion.json. No other thread was changed. Both threads were checked again at 23:31 UTC (0 rows) just before the final run; its questions start at turn 0001 in each thread (results/2026-11-12-final/runs.json).

## Chat-context reset for the 2026-11-13 re-run
Each question got its own clean thread. Just before each question, the report thread (the same two threads as above, gamma chat table only) was copied, cleared, and checked at 0 rows; the report was then reloaded and History > New chat clicked. So every question is turn 0001 and its answer turn 0002, with nothing else in context. The copied rows are in results/2026-11-13-rerun/chat-threads-cleared.json (the first copy is the final run's conversation). No other thread was changed.

## Inputs
- Midway street camera: 111 s, silent. Evidence Copilot could not play or process the original 4K HEVC file, so both products got the same 1080p H.264 copy (ffmpeg -c:v libx264 -vf scale=1920:-2). Not published here (customer-provided footage, 80 MB); the screenshots and ground-truth crops show the frames used.
- NASA call: 110 s, seconds 120-230 of NASA video iss071m261621702 (public domain, https://images.nasa.gov/details/iss071m261621702), four speakers. Not published here (17 MB); cut it with ffmpeg -ss 120 -t 110.

## Ground truth
ground-truth/ground-truth.json, with the crops it cites in ground-truth/crops/. Midway: about one frame per second across the clip, high-resolution frames around each passing vehicle, then crops of every vehicle, plate area, bin and overlay, identified by eye (grille, body shape, generation); direction from consecutive frames. NASA: full listen with each change of speaker logged, names from NASA's metadata and how the speakers address each other (ground-truth/nasa_contact_sheet.jpg).

## Scripts
scripts/ holds the Chrome DevTools Protocol helpers: cdp.py (connection), ask.py (types one question into Ask Emma, waits until the answer stops changing, saves text and a screenshot), tallq.py (full-height screenshot of one answer), clicktext.py (History > New chat), and the network/report watchers used in the original run. Sign-in was done by hand; no credentials are read or stored by any script.

## Limits
One run per product per question. The first Echosaw NASA upload predates gamma speaker labels; the NASA clip was re-uploaded once after the rollout. Evidence Copilot shows no version, and its submission times were not logged (screenshot times are given instead). The Echosaw revision for each answer is matched from deploy times (CodePipeline, CodeBuild, Lambda code updates), not stamped on the answer. In the re-run, the test account is an organization member; for organization members Echosaw keeps one chat thread per media, so New chat cleared the panel but the earlier run's turns stayed in the conversation history the assistant received (see results/2026-11-12-rerun/runs.json). The final run (results/2026-11-12-final/) was asked after both threads were deleted, so it had no earlier turns in context; the 2026-11-13 re-run cleared the thread before every question.
