Release notes

Every OttoTester release, newest first. Each one starts with a sentence on what it gives you, then lists what is new and what was fixed.

If you use OttoTester at ottotester.ai, you are always on the newest release. If you run your own copy, ./ottotester update in your install folder moves it to the newest release, and ./ottotester update v1.2.3 moves it to a particular one.

v1.1.37 (November 13, 2026)

For an app linked to GitHub, the number of matching issues now counts only the issues you can import.

Fixed

On a GitHub-linked app, the issue filter’s count included issues with no label set to requirement or defect. Import issues leaves those out, so the count could promise issues you couldn’t import. It now counts only issues with one of those labels.

v1.1.36 (November 13, 2026)

Each of the AI’s browser actions now has its own time limit, tests get 3 minutes by default, and a test’s version history shows only its own changes.

New

Time limit per browser action. The Create Tests and Resume Tests settings have a new Time limit per browser action, 120 seconds by default. If one of the AI’s browser actions runs longer while it plans or writes a test, the run stops and names the setting to raise. An action never gets more time than a whole test.

Changed

Plain names for the run time limits. Page timeout is now Time limit per test, and Assertion timeout is now Wait for each check. Hover either one to see what it does.
Tests get 3 minutes by default. The default time limit per test goes from 30 to 180 seconds. Saved settings on 5, 10 or 30 seconds move to 180 when you update. Other saved time limits stay as they are.
New Test opens a dialog called New Test, and Create tests on the Proposed tab opens one called Create tests. The hint in the prompt box now says what to do on that screen.

Fixed

A test’s version history shows only its own changes. If you delete a test and later make a new one with the same name, the new test’s history no longer lists the deleted test’s changes, and you can’t restore the deleted test’s script into it.

v1.1.35 (November 12, 2026)

You now start runs from the Tests page and see the settings before any run starts, and a test run on every row of its data keeps each row’s own error, trace and report.

New

Each data row keeps its own evidence. When a test runs with All rows, each row now keeps its own error, steps, trace, screenshot and video. On the Runs page, a failed line shows its error. On Detailed test results, a Data row picker opens each row’s own report. The detailed results PDF lists each row with its own steps. Before, only the last row’s evidence was kept, and nothing said which row it was.
The Reports page’s health badge (Healthy, Attention needed, Critical) now explains itself when you hover it, using your last 5 runs.

Changed

The Tests page comes first. The side menu now reads Tests, Runs, Reports, and OttoTester opens on the Tests page.
Run Tests moved to the Tests page, next to Resume Tests. The Runs page now shows runs and doesn’t start them.
Run this Test shows its settings first. You can check or change them for this one run. Nothing you change there is saved.
Save settings replaces Save and run. In the Run Tests, Resume Tests and Create Tests settings, there is now one Run button. After you change something, a Save settings checkbox appears. Select it to keep your changes for later runs. Leave it cleared to use them for this run only.
Test data now offers One row and All rows. The old Off and Single row choices did the same thing, so both now show as One row.
The last screens, PDFs and help pages that said “Discovery” for a run that creates tests now say “Generation”.

Fixed

Uploading a CSV file with only one column to a Test Data Set works. So does a file whose lines end in a mix of styles, which happens when a file is edited in two programs.
A run template that picks specific tests no longer counts a test that was deleted, as in “2 of 1 chosen”. A run skips deleted tests, and a template whose chosen tests are all deleted won’t start a run.
Switching workspace on the Runs page now shows the new workspace’s runs, not the previous one’s.

v1.1.34 (November 11, 2026)

Pipeline snippets now fail your CI job when tests fail, times on screens and in PDFs follow your Profile time zone, and single sign-on, app scans and the Runs and Reports pages get fixes.

New

Times in your own time zone. Dates and times on screens now follow the time zone set on your Profile page. If you haven’t set one, they use your browser’s. Audit logs and run pages also name the zone, as in “2026-11-10 14:14 PST”. Before, some screens showed UTC and others your browser’s zone.
PDF reports you download or email to yourself print times in the same zone. A PDF sent automatically when a run finishes still uses UTC.
In the model lists, a model’s cost category now shows as $ for Low up to $$$$ for Premium, instead of ✦ marks. Free models still read Free.
If you run your own copy, the installer’s closing summary now ends with the command that uninstalls OttoTester.

Fixed

Copy your pipeline snippet again. The Pipeline Integration snippets now fail your CI job when tests fail. A snippet copied before this release keeps your CI green even when tests fail. Copy it again, or add --fail-with-body right after curl in your copy. The flag needs curl 7.76.0 or newer.
Single sign-on: the “Prove you own” step now shows the DNS record the check really looks for, and its value. The record’s name starts with _ottotester-sso-, followed by the connection’s ID and your domain. Before, the page showed a record name the check never looked for, and often no value. If you added that record, add the one the page shows now.
The Single sign-on settings page no longer shows a load error for every organization.
App scans no longer read the cloud’s internal address that hands out a server’s own keys, even when a page redirects there. A scan of that address stops with “OttoTester can’t scan that address.” Apps on your own network scan as before.
When an app draws its sign-in form with JavaScript, the scan no longer says the app needs no sign-in. It records that it couldn’t tell, and test planning then expects a sign-in page early.
Runs are named the same way on the Runs list, the run page and the detailed results page: the suite’s name, Test Suite Run when the suite has been deleted, or Generation or Replay. Runs from before this release that have no suite show as Test Run. Before, the Runs list called every run without a suite “AI Discovery”, and the two run pages said “Discovery run” or “replay run”. A note on the run page no longer says “coming in a follow-up”.
A new account’s Runs page now says to add a first app, instead of blaming a variant’s settings.
The Reports page’s Flaky tile now counts the same tests the Flaky tab lists, on screen and in the PDF report.
On the Tests page’s Proposed tab, the filters now sit two to a row, so their labels show in full.
Add credential no longer saves an empty login before you type anything. The new login is saved when you press Save.
Uploading a CSV file to a Test Data Set works again. Before, it could fail with “Papa.parse is not a function” after a test’s script editor had been opened.

v1.1.33 (November 10, 2026)

Tests can now rely on other tests for their data, and remove what they create when your app lets them: OttoTester plans the test that sets the data up, runs it first and hands its values on.

New

Tests that need another test first. A test can now need data that another test makes, such as “Terminate an employee” needing “Create an employee”. The planner plans these links, runs the tests in that order, and hands values on, such as the new employee’s name. If a test fails, the tests that need it are skipped, and the run says which test they were waiting for. While writing a test, OttoTester runs the tests it needs first, so it explores with real data.
Automatically plan prerequisite tests, a new setting on Create Tests templates, selected by default. With it selected, the planner plans the tests that set up data and gets twice the steps to do it. Clear it to have each test set up its own data.
The Tests page shows the links. Needs names the test that runs first, Needed by names the tests that wait for this one, and Sets up, in Details, says what the test leaves in the app.
A test that uses up its data, such as one that terminates an employee, can run with its own copy of the tests it needs, so it gets its own employee. In a run, the row of the test it needs shows each copy’s result, such as Shared: passed and For “Terminate an employee”: passed. That test still counts once.
Deleting a test that other tests need now warns you first and names them. Once it’s deleted, they don’t run until another test sets up the same thing. Create Tests can plan one.
Tests clean up the data they create. When your app lets a record be deleted, a new test that creates one, such as an employee, removes it again after it runs, whether it passed or failed. A test other tests rely on removes its data once they have all run. When OttoTester can’t confirm a removal, the run shows 1 test may have left data, and the test’s row says Check that Casey-1b534l was removed. The test keeps its own result.
With healing on, OttoTester also fixes a cleanup that couldn’t confirm. It changes only the steps that find and remove the record, and keeps the fix only once it has seen it work. For a test that passed but whose cleanup didn’t confirm, Run and Fix replaces Run this Test.
When fixing a failing test, OttoTester now takes into account the tests it needs and its setup steps.
Global Admins: a run’s drill-down on the Analytics page says how many of the planned tests got scripts, and a plan with linked tests is scored correctly.

Fixed

A model that sends one garbled tool call no longer ends the run. OttoTester tells the model and lets it try again, and stops with a plain reason only if it keeps happening.
Saved results now name why a test failed: a timeout, a failed check or a page that wouldn’t load. Before, most of these were saved as unknown. Saved totals leave out sign-in and setup steps, as the run’s own totals already did.
A fixed test no longer keeps an earlier try’s error in its saved result, and a run waits for every result to be saved before it ends.
A written test that reads a value from the test it needs no longer fails with “readData is not defined”.
When a test’s cleanup removes several records, each one gets its own time. Before, one slow search could leave the rest unchecked and marked “timed out”.
A test’s cleanup steps no longer break when OttoTester points the test at its start page.
Adding a proposed test that’s already there now says so in plain words, instead of showing a database error.
When the planner’s final plan can’t be read, OttoTester asks the planner again instead of failing the run.
When a run checks a fixed test again, tests that use New value each run get fresh values, so they don’t clash with what the first try created.

Changed

The planner’s and the test writer’s instructions now cover tests that need other tests, and cleanup steps. Prompts nobody has edited update by themselves. If you edited one on the Prompts page, your version is kept: compare it with the new default.

v1.1.32 (November 8, 2026)

Writing tests is more dependable: a slow AI answer can now be retried without losing earlier progress, a working script is no longer thrown away, and a run’s totals count only tests.

Fixed

When one AI answer takes too long while a test is being written, OttoTester now tries that step again, as long as it has retries and steps left. Before, the test and all its progress were lost.
New sign-in steps no longer fail the script check with “Cannot find name ‘process’”. Before, every new sign-in step failed it, and OttoTester spent its tries on an error that wasn’t in the script.
Resume Tests now runs a failing script once more before rewriting it, if that script passed before with the same code. A failure caused by your app being down for a moment no longer throws away a working script.
Fields that get a new value each run, set with New value each run in Test Data or picked by the planner, now get a fresh value each time OttoTester explores your app too, not only when the test runs. A name left over from an earlier try, such as a team’s, no longer gets in the way.
A run’s totals now count only tests. Before, sign-in and setup steps counted too, so “6 passed” could mean 4 tests. A run whose sign-in or setup step fails is still marked failed on the Results page, in CI and in alerts.
The Auth list in a test’s details shows each saved login once, labeled with its roles, even when a login has several. The one you pick now stays picked.
The sign-in advice now points to New Test on the Tests page, the button that builds a sign-in step.
When the planner runs out of steps, its advice names the Run budgets page only if your plan includes it. Otherwise it suggests asking for fewer tests.
Saving Run budgets on a plan that doesn’t include them now says so in plain words.
Global Admins: on the Budgets page, an empty box now shows the number it really uses.

Changed

The planner’s instructions now say a test must not rely on data another test creates. Prompts nobody has edited update by themselves. If you edited the planner’s prompt on the Prompts page, your version is kept: compare it with the new default.

v1.1.31 (November 7, 2026)

Creating tests is more dependable: runs get enough steps for every test, stop with a plain reason when they can’t go on, and Resume Tests now sits on the Tests page.

New

Resume Tests has moved from the Runs page to the Tests page, beside the tests it finishes, and shows how many it would pick up. The Runs page keeps Run Tests. To create new tests, use Create tests on the Proposed tab or New Test on the Active tab.
The Resume Tests settings now have a Max test cases field, so one run can finish several unwritten tests instead of one. Resume settings you already have keep their current number until you change it.
One saved login can now be used for several roles, for an account that is, say, an admin in one workspace and a member in another. List the roles in its Roles box, separated by commas or semicolons. Every test tagged with any of them signs in with that login, so the password lives in one place. A role name can be on only one login per variant.
Before you delete a saved login that tests or sign-in steps still use, OttoTester now tells you how many of each use it, and the button reads Delete anyway.
Global Admins: the Proposal quality report shows whether each proposed test was saved, and if not, which step left it out.

Fixed

Renaming a saved login’s role no longer cuts off the tests and sign-in steps that use it. They now stay linked to the login itself.
The planner’s step limit now applies to each test, as Run budgets describes. Before, a run that asked for several tests shared one small limit and often made no tests.
When the planner runs out of steps, the run now fails and says so, naming its limit. Before, it showed as completed with no tests.
When your app refuses the saved login while tests are being created, the run now stops and names the login, instead of going on with tests that can’t sign in. A temporary server error on your app is not treated as a wrong password.
A new sign-in step that fails is now kept and marked blocked, not deleted. Before, its tests were left linked to nothing and could not be finished.
Resume Tests no longer spends its run on a test that can’t sign in. It sets that test aside and works on one that can. Write this Test still works on the one test you chose.
The start dialog no longer says Nothing to run yet when you picked tests on the Proposed tab.
Organizations whose plan doesn’t include Run budgets no longer see that page with an error code.

Changed

Self-hosted installs: the agent step settings have new names. The planner and generator settings now count steps for each test, and the healer setting counts steps for each attempt to fix a test. MAX_PLANNER_ITERATIONS is now MAX_PLANNER_PER_TEST_ITERATIONS (default 50). MAX_AGENT_ITERATIONS is split into MAX_GENERATOR_PER_TEST_ITERATIONS (default 50) and MAX_HEALER_PER_ATTEMPT_ITERATIONS (default 10; the example settings file sets 25). The old names are no longer read: OttoTester logs a warning at startup if they are still set. Rename them in your settings file when you upgrade.
Runs may cost more. The planner’s number now applies to each test, so the same number gives a run of 3 tests 3 times as many planner steps as before. For example, an old MAX_PLANNER_ITERATIONS=10 gave a 3-test run 10 planner steps; with the new default of 50 per test, that run can use up to 150. Set lower numbers if you want to keep runs cheaper.

v1.1.29 (November 7, 2026)

You can now search an app’s issue list, and imported issues count as sources on the Proposed tab.

New

The issue list in an app’s Integrations, and Import issues on the Proposed tab, now have a search box. Type an issue number, a Jira key such as SHOP-12, or words from its title to find it. Issues you select stay selected while you search again, and one import brings in all of them.

Fixed

The import status beside each issue now keeps updating until the import finishes. Before, it stopped after about 24 seconds and showed Queued until you refreshed the list.
Sources on the Proposed tab now counts the issues your proposed tests came from. Before, it said none yet for an app whose tests all came from imported issues.
The empty upload box on the Proposed tab now appears only after you press Upload document.
The install script now shows a progress bar while it downloads OttoTester, like the one it shows when it starts OttoTester.

v1.1.28 (November 6, 2026)

Signing up with an email that already has an account now tells you so.

Fixed

If you sign up with an email that already has an OttoTester account, the sign-up form now says This email already has an account. Before, an install that verifies new emails showed “Check your inbox” and no email ever came, and one that doesn’t showed “User already exists. Use another email.” Use Sign in below the form instead.

v1.1.27 (November 6, 2026)

Scheduled runs now use the same test settings as a run you start yourself.

Fixed

A scheduled run now uses the same test settings as a run you start yourself. Before, it dropped some of them, such as the pauses, console and network error reporting, the tests you chose, and the fix and stability settings. A schedule whose suite was archived or belongs to another variant, or whose app has no tests, now fails with a message to the schedule’s owner instead of running.
Run Tests on the Runs page now lists only the suites of the variant you are on.

Changed

The screen that asks you to choose or finish choosing a plan no longer shows Account settings and Sign out links.

v1.1.26 (November 5, 2026)

Creating tests on the Proposed tab is simpler: there is one Create button, and it only counts the tests you have selected.

New

Tests → Proposed has one Create tests button, in the toolbar beside the add and refresh buttons. Select the tests you want, then press it. It says how many tests it will create, and it is greyed out until something is selected. An opened proposed test no longer has a Create button of its own.
Screens and help pages now say “select” instead of “tick”.

Fixed

Create tests now counts only the tests that are checked on screen and can still be created. A test that was archived, deleted, replaced when its document was read again, already has a test, or is hidden by a filter or search no longer counts. So one checked test reads Create test.
If none of your selected tests can be created by the time a run starts, you now see a message saying so, instead of aiInstructions-required. The message for a run with no selected tests and no Agent Prompt is now plain English too.
The GitHub card in Workspace Settings → Integrations now asks for permission to write issues only when an app is set to open an issue when a test fails. Reading issues, which proposed tests need, only takes read access.

v1.1.25 (November 5, 2026)

You can now add Excel workbooks and CSV files as reference documents, and the Sources panel on the Proposed tab is easier to read.

New

Tests → Proposed → Sources now reads Excel workbooks (.xlsx), whether you upload one, add a link to one, or import an issue that has one attached. Each sheet is read as a table under the sheet’s name, and a formula gives its last saved result. A workbook that needs a password to open, has more than 10,000 filled rows, or is too large to unpack is refused with the reason, or skipped if it came attached to an issue. Old .xls files aren’t read.
You can also upload, link or attach .csv files. They are read as plain text, so save them as UTF-8 to keep accented letters.
The Sources panel is tidier. Add a link is now Add link, and each button explains itself when you point at it. Why Import issues is greyed out now shows there too, instead of in a line under the buttons. A document’s details are labeled rows, and an uploaded file shows a file icon.

Fixed

When the list of issue trackers can’t be loaded, an app’s Integrations panel now says so, instead of saying no tracker is connected.

v1.1.23 (November 5, 2026)

The Apps page now shows an app’s details above its variants, the Proposed tab makes creating tests and importing issues easier to find, and a grade the AI grader could not read now shows as “Not graded” instead of 0.

New

Each app on the Apps page now shows, in order, About this app, an Integrations summary with one line per connection, and its Variants underneath, instead of three tabs. About shows the app’s summary, domain and who uses it without opening anything. While those are empty, it says why and offers to scan your first variant, or to add one if the app has none. Manage opens the GitHub, Slack, Pipeline and issue tracker settings in a side panel. Each variant’s own trigger rules, Slack overrides and pipeline setup are now links on that variant’s row.
On Tests → Proposed, Import issues is now a source button beside Add a link and Upload document. Until a tracker is linked, it is greyed out with a line saying which step is missing and where to do it.
Create test stays on screen on the Proposed tab, greyed out until you tick a test, and an opened proposed test now has its own Create test button.
In a test’s trace, each saved number and each number check is now a named step with the numbers in its title, so you can see what a passing check compared. Tests that already use number checks show this on their next run.
On the proposal quality report, a grade the AI grader’s reply could not be read for now shows as Not graded, not 0, and is left out of quality averages and score bands. If you see it often, pick a stronger model for the Auditor.

Fixed

The AI grader’s answers are now read correctly when it sends numbers or lists as text, and an answer that can’t be used is skipped rather than saved as a wrong score.

v1.1.22 (November 4, 2026)

You can now set how long the agent waits for one AI answer, a test can save a number and check it later, and AI cost figures say plainly that they are estimates.

New

Global Admin → AI Models → Budgets & caps has a new box for how long the agent waits for one AI answer, from 30 to 600 seconds. A saved value applies from the agent’s next AI call, with no restart. With nothing saved, the wait is now 3 minutes instead of 5. If you run your own copy and set MAX_AGENT_LLM_CALL_SECONDS, that setting still applies until you save a value on this page.
A test step can now save a number from the page, such as a count or a total, and a later step can check it went up, went down, or stayed the same. The step editor has save a number and compare a number, and the planner uses them for tests where a change would otherwise leave a normal-looking page.
AI cost figures on Analytics, in a run’s drawer and on Org Settings → Usage now say they are estimates. Your bill from your AI supplier, or from OttoCloud if you buy AI from us, is the actual cost.
If you get your AI models through OttoCloud, a model that thinks before it answers now gets the same thinking limit it gets through OpenRouter. And when OttoTester gives up waiting on a call that OttoCloud still charged for, that charge is now looked up and added to the run’s cost and your organization’s usage. The lookup is on by default. Turning it off with CALL_CHARGE_LOOKUP_ENABLED=false doesn’t stop the charge. It only leaves it out of OttoTester’s totals.

Fixed

The planner now makes sure every test has a check that would fail if the problem the test names really happened. A planned test without one gets a lower score for its checks on the plan quality report.
A run’s cost is now worked out once, so the run, Analytics and billing all show the same figure.
Through OttoCloud, every AI call is now charged to the organization it does the work for. Some background work, such as grading and duplicate checks, used to go out without saying which organization it was for.
A copy you run yourself now refuses to start if the ACCOUNT_SERVICE_OAUTH_* settings are filled in. Those settings are only for OttoTester’s hosted servers.

v1.1.21 (November 2, 2026)

OttoTester at app.ottotester.ai moves to this release, and its plans and billing now go through OttoCloud. Nothing changes in a copy you run yourself.

New

On app.ottotester.ai, you choose a plan and set up billing in OttoCloud, the same place a copy you run yourself can get its AI models.
ottotester.ai has a Download page with the one-line install command for Mac, Linux and Windows, what your computer needs, and the license terms.

v1.1.20 (November 1, 2026)

Anyone can now download OttoTester and run it on their own computer or servers, with no GitHub account.

New

The install command works for anyone. OttoTester’s images are public at ghcr.io/ottotester-ai, so installing needs no sign-in to GitHub or to any image registry, and neither does updating a copy installed this way. Each image’s page on GitHub links to OttoTester’s public repository.
A copy installed before this release keeps updating from wherever it was installed from. To move to the public download, install a new copy.
The demo app, ./ottotester demo, doesn’t work in the public download yet, because its image isn’t public.

v1.1.19 (November 1, 2026)

Global Admin has a new About page with your version, updates, license and support in one place, and a copy you run yourself keeps its license in its install folder.

New

Global Admin → About shows the version this copy runs, whether a newer release is out, the license, and where to get help. The version and Check for updates used to sit under Build at the bottom of the Global Admin home page. They have moved here unchanged. The License section links to the full license, and the Support section gives support@ottotester.ai.
If you run your own copy, your install folder now has a LICENSE file with the license of the version you run. Installing creates it, and ./ottotester start and ./ottotester update keep it matched to the running version. If it can’t be updated, the old file is kept and you see a one-line warning. Starting and updating carry on as normal. When you update to this release from an earlier one, your earlier helper runs the update and doesn’t know about the file, so run ./ottotester start or ./ottotester update once more to create it.

Fixed

On Global Admin → AI Models → Assignments, an agent’s model list now opens in full. Before, most of the list was hidden and the models lower down couldn’t be picked.
Setup’s last screen, about sharing anonymous scores, now says those scores never include your prompts, test data, scripts or plans. It used to say your prompts are never sent at all. If you picked OttoCloud for your AI models, it also says that your prompts pass through OttoCloud to run the models, and that OttoCloud doesn’t store them.

v1.1.18 (November 1, 2026)

OttoTester now has a license that says who can use it free, and the Model catalog and Scheduled jobs say what each model and job is waiting for, such as a missing AI key.

New

OttoTester is now under the Business Source License 1.1. It is free for individual developers and testers working for themselves, and for companies of 10 people or fewer with US $500,000 or less in revenue. Larger companies get one free 30-day trial, then need a license from us. The new License page in these docs sums it up and shows the full text, which also ships inside every copy.
The demo app, OrangeHRM, keeps its own license, the GPL (version 3 or later), which covers only the demo app. ./ottotester demo says so when it starts.
On Global Admin → AI Models → Model catalog, a model you can’t use now says why, such as “Needs an Anthropic key” or “Connect to OttoCloud”, instead of “Not available”. For a missing key, hover over it to see which setting the key goes in.
On Scheduled jobs, a model price or model list job that has nothing to do on your copy now shows Off and says what it is waiting for: a key for that AI service, or, for OttoCloud’s price list, connecting your copy to OttoCloud. It no longer runs every day for nothing, and it turns itself on once that is set up and OttoTester restarts. Where each organization has its own OttoCloud account, OttoCloud’s price list job stays off, because your copy has no single price list to fetch.

Fixed

On a copy whose only AI key is for OttoCloud, the Frontier filter on Model catalog now lists the top models OttoCloud offers. It used to show none.
On Global Admin → AI Models → Agent Memory, the list of recent searches no longer reloads over and over on a new copy. If it can’t load, it says so and offers Reload. Its labels now say “Recent agent searches” instead of “production”.

v1.1.17 (November 1, 2026)

If you run your own copy, Sign in and Create account now show only what works there: your email and password.

Fixed

On a copy you run yourself, Sign in and Create account no longer show Google, GitHub and Microsoft buttons. Each one failed with an error when clicked, because an installed copy has no keys for them. Signing in with email and password works as before, and so does your company’s own sign-in if an admin has set one up.
Installing, or running setup again, now leaves the Google, GitHub and Microsoft sign-in settings out of your .env file. If someone adds them back by hand, an installed copy ignores them.

v1.1.16 (October 31, 2026)

After you apply OttoCloud’s model recommendations, each agent you changed shows the model it used before and the one it uses now.

Fixed

After Apply in Show recommendations on Assignments, an agent whose new model was saved now shows Before with its old model and Now with the new one. It used to keep showing the old model as current, with a Not working badge next to “Saved and working.”
A recommendation based on grades that other copies share now reads “From 412 community shared runs”, with that recommendation’s own count, instead of “Community pick, 412 runs”.

v1.1.15 (October 30, 2026)

Reference-doc links now read pages that draw their text with JavaScript, and admins choose which sites links can reach, including sites inside their own network.

New

Adding a link under Tests → Proposed now works for pages that draw their text with JavaScript. When the downloaded page has no text, OttoTester opens it in a browser, lets it finish drawing, and saves what appears. A page that still can’t be read says so, and suggests saving it as a PDF and uploading that instead.
On Global Admin → Runtime Config, Links OttoTester can read sets which sites links can reach: public sites only, or public sites and your internal network, plus an Always allow list and a Never read list. A copy of OttoTester you install yourself starts with your internal network allowed.
A link to a file type OttoTester can’t read now says so, and suggests uploading a PDF, Word, Markdown or text file instead.

Fixed

Writing an internal address inside IPv6, such as [::ffff:127.0.0.1], no longer gets around the rules for which sites links can reach.

v1.1.14 (October 30, 2026)

You can now create an app and its first variant together, in one two-step dialog.

New

New App opens a two-step dialog: the app’s name first, then its first variant. Finish with Create & Scan or Create without scanning, and the new app opens with its variant selected.
If saving the variant fails, the dialog keeps the app and stays on the variant step, so you can fix it and try again.

v1.1.13 (October 30, 2026)

Each agent on Assignments now has its own Save that checks the model first, and self-hosted installs clean up old files on their own and can be removed with one command.

New

On Assignments, each agent has its own Save beside its model. Save checks the model first, and a model that fails the check is not saved.
With OttoCloud set up, “★ Show recommendations” on Assignments lists OttoCloud’s pick for each agent. You can apply them one agent at a time.
./ottotester uninstall (.\ottotester.ps1 uninstall on Windows) removes a self-hosted install: its services, its own database and saved files, and its images. It asks you to type delete first. A database outside the install, and images other containers use, are kept.
An install that keeps run files on disk now deletes runs older than the age limit every night. The Artifacts page shows where those files are and how much space they use.
Deleting a workspace, an organization or a data set now deletes its saved files too. A nightly cleanup also removes uploaded data set files that no row uses any more.

Fixed

Opening a run whose files were deleted shows a page that says so, instead of an error.
Applying a limit on the Artifacts page no longer makes uploaded data set files expire along with old runs, and the page shows the limit actually in place.
The installers no longer say Docker is stopped when it is missing or you don’t have access to it. They now say which one it is.
The install from a clone lists every missing tool at once, with where to get each one.

Other

OttoTester now runs on Node 24. An install from a clone needs Node 24.22.1 or newer.

v1.1.12 (October 29, 2026)

The Assignments screen is simpler: each agent shows any problem first, then its model, and the models you change are checked after you save.

New

On Assignments, each agent shows what is wrong and what to do first, with Check now beside it, then its model and the ★ Recommended pick. Temperature and other details sit under More details. The “Help me choose” slider is gone.
An agent you changed says “Not saved yet”, a note beside Save lists every unsaved agent, and each changed model is checked after you save.
The model catalog has a Free cost level, so you can list the models that cost nothing in one click.

Fixed

“Where models come from” lists only the suppliers this install is set up for, and its “Hide models you can’t use” tooltip no longer sends you to Assignments to set one up.
Setup no longer promises free models with OttoCloud, which a new account can’t use. It now says OttoCloud includes lab-tested, cost-effective models.

v1.1.11 (October 29, 2026)

Setup now picks a suggested model for each AI agent, and warns you when your OttoCloud email still needs confirming.

New

Setup sets each AI agent to the model suggested for it when your key can use that model, instead of one model for every agent. You can change them later in Global Admin, AI Models, Assignments.
A Release notes page lists what changed in each release.

Fixed

Setup warns you when your OttoCloud email isn’t confirmed yet, and lets you confirm it before going on.
A document that fails to read because OttoCloud refused to run AI now says why, for example a spending limit or an unconfirmed email.

v1.1.10 (October 28, 2026)

A new install opens straight on Create account, and setup’s question about sharing scores is clearer.

New

A new install opens on Create account instead of Sign in, so the first person can set it up right away.
Setup’s question about sharing anonymous scores now says more plainly what is never sent.

v1.1.9 (October 27, 2026)

Self-hosted installs start faster and more reliably, because the built-in embedding model now comes inside the release.

New

The built-in embedding model ships inside the release, so a new install no longer downloads it on first start.
Setup can be pointed at a different OttoCloud address, for trying an install without touching your real account.

Fixed

The sign-in and sign-up pages show every sign-in provider’s logo again.
Setup sends OttoCloud users to the right pages for their key and their billing.
The built-in embedding service recovers on its own after an interrupted download.
A fresh install no longer points at an old OttoCloud address.

v1.1.8 (October 26, 2026)

Setup asks about sharing anonymous scores last, once your provider and model are set.

New

The question about sharing anonymous scores moved to the end of setup.

v1.1.7 (October 26, 2026)

Install and update OttoTester with one command, import requirements and defects from GitHub Issues, and see coverage numbers that agree everywhere.

New

Install from the published release with one command, and update with ./ottotester update. If the new version doesn’t start, the update puts the previous one back.
./ottotester demo starts a demo app to try OttoTester on.
An older version refuses to start on data a newer version has already updated, so going back a version can’t damage your data.
Connect GitHub Issues as an issue tracker. Import issues as requirements or defects by their labels, narrow them with a visual filter, and open an issue for a failed test.
The Assignments page recommends a model for each agent, and the model catalog offers only models your install can actually use.
Mark each document you upload as requirements, defects or reference. Each part of a requirements or defects document counts toward coverage, and a person confirms the requirements OttoTester finds before they count.
After OttoTester reads a document, a popup shows what it found.
Coverage uses the same words and the same numbers on every screen, and the coverage trend ends with a live point for today.
The live run card shows which test is running.
An install can run without an email server. Invitations, password resets and notifications still work.
Setup explains the anonymous scores your install shares, and how to turn sharing off.
The Tests page shows why a test was fixed.
New run templates pause between steps and between tests unless you change them.

Fixed

The first account on a new install becomes its global admin.
A failed planning call now fails the run and says why, instead of ending it as completed.
Claude models through OpenRouter work with thinking turned on.
A fresh install has prices for newer models from the start, and setup suggests a model your key can run.
Export Tests downloads only the selected variant.
A Jira filter can no longer reach projects outside the ones the app chose.
A test is no longer saved as finished when its steps were never seen working, and editing a test’s steps replaces its old script.
Reading a document again keeps your tests connected to the proposed tests they came from.
One unreadable reply from a model no longer throws away the whole run.
Proposing tests from a document works with OpenAI models.
Two people who sign up in the same second with the same name get separate organizations.
The browser tab icon is easy to see on light and dark tab bars.

v1.1.6 (September 29, 2026)

Running your own copy gets much easier: one command to a working install, a built-in embedding model, OttoCloud as a model supplier, and storage and email beyond AWS.

New

One install command that ends on a working install, with step-by-step setup in the terminal.
A demo app ships with OttoTester, so your first run can reach a passing test.
A built-in embedding model is the default, so it needs no key of its own.
OttoCloud can supply your models. OttoTester checks your account may spend before it sends a request.
Run reports can be stored on any S3-compatible storage, not only AWS S3.
Email can go through any mail server, not only AWS SES.
Run a suite now or on a schedule from one Run Suite dialog.
Read a run’s results with a workspace API token, and copy a test plan between variants through the API.
The deduper, which spots duplicate tests, can have its own model on the Assignments page.
Advanced admin settings carry a caution note and a one-click reset to the recommended values.
OpenRouter costs are recorded as OpenRouter charged them, not estimated.

Fixed

The embedding service no longer restarts over and over on machines without a graphics card.
Adding an app works for sites that build their pages in the browser.
Filter labels in the Tests sidebar are no longer cut off.
Setup checks your Node version and offers to set up pnpm for you.

v1.1.5 (August 26, 2026)

The Tests page is reorganized around how you work, proposed tests get sharper, and you can sign in with your own Microsoft Entra.

New

Sign in to OttoTester with your organization’s own Microsoft Entra.
The Tests page groups its actions by what they act on. Run just the test you’re looking at, pick specific tests, or run and fix a failing test, all from one place.
Start a run from the Runs page and watch it there. The current run’s results show in place, with past runs folded away.
Run settings are kept in three separate lists: manual runs, scheduled runs and CI runs.
Proposed tests appear as soon as they are written and are graded after. Each document chooses which kinds of tests it suggests, and steps a browser can’t check are left out.
Edit a proposed test’s priority and type right on its card.
See when a test was proposed, created and last changed, and what it cost across every stage.
Download your tests as readable Markdown files in a zip.
Grant access by picking a person by name or email.
Models from countries you exclude no longer appear in the model catalog at all.

Fixed

A test that passes is no longer left marked as blocked.
A sign-in helper can no longer report success when nobody is signed in, and one that times out names its role and time limit.
A space pasted into a credential is caught when you enter it.
Create Tests makes as many tests as it says it will.
Documents added from the Proposed tab update on their own.
Running a test from the Tests menu shows its result.
Workspace and team members show names, not ids.
Version history shows changes from before a test’s file was renamed.
Quality reports keep what was actually spent, instead of repricing old runs at today’s rates.

v1.1.4 (August 8, 2026)

Downloaded tests and their history stay complete, even after a failed run or a server replacement.

Fixed

Each app’s test history survives the server being replaced.
Downloads and history stay complete after a failed run.

v1.1.3 (August 8, 2026)

Download an app’s tests as a runnable Playwright project, and get help from a new support page.

New

Download an application’s tests as a zip that runs as a Playwright project.
A new support page, with a FAQ in the docs.
Runs started from CI add a readable summary to the pull request.

v1.1.2 (August 5, 2026)

The trial countdown banner goes away when it should.

Fixed

The trial countdown banner no longer stays on screen for good.

v1.1.1 (August 5, 2026)

Keep out models from countries you exclude, compare models by cost band, and sign-up records your plan and consent properly.

New

Leave out models from vendors in countries you exclude, both in the catalog and when a run starts.
Cost bands are one click away, and Premium now means the priciest tenth of models.
Global admins can assign a plan to an organization.

Fixed

Sign-up records your plan and your acceptance of the terms, and refuses an invalid plan in the sign-up link.
No verification email is sent when email verification is turned off.
The Users page shows each person’s last sign-in and sign-in provider.
Models OpenRouter no longer lists, or can’t price, are dropped.
Long model replies are no longer cut short, and run details report only what was measured.

v1.1.0 (August 2, 2026)

Choosing a model gets easier with a cost and quality guide, and signing in to the app you test is far more reliable.

New

Pick a model with help: cost bands from Low to Premium, a “Help me choose” slider, and guidance on what each agent needs.
The model catalog can be filtered and sorted, shows when each model was added, and its rates are easier to read.
The Model Leaderboard plots quality against cost.
Each plan, script and fix records how long it took, not just what it cost.
Jira issues stay in sync, with a visual filter for which ones to import.
OttoTester records how it reached your app’s sign-in, step by step.

Fixed

OttoTester finds your app’s sign-in address on its own, follows redirects to it, handles Login.gov when it skips the one-time code, and picks the right role when a variant has several.
When sign-in fails, the message says plainly what to do.
A test the planner labels as signed out is no longer blocked for good.
Default test time limits are long enough for a real sign-in.
Every test written gets graded, so a run’s quality score covers all of it.
An AI call can no longer hang forever, and the default time limit per call is longer.
Model names are readable in the picker again, and the header fits narrow screens.

v1.0.9 (July 27, 2026)

A curated model picker you can filter, and runs that can reset the app you test before they start.

New

A curated model picker you can filter, with a badge for models that read images and a price you can override.
Model selection shows blended rates and a cost-to-quality slider for each agent.
A run can reset the app you test before it starts.

Fixed

A grader that replies without a score is asked again, instead of the score being lost.

v1.0.8 (July 24, 2026)

Import Jira stories as proposed tests, and see requirements coverage on the dashboard.

New

Connect Jira to a workspace, browse its issues, and import a story as proposed tests, attachments included.
Requirements coverage on the dashboard, with a list of gaps, and production defects tied to coverage.
Proposed tests now live on the Tests page, under Proposed.
Add a reference document by pasting a link to it.
Input variations: try the same steps with different kinds of input.
The public sites ask for cookie consent.

Fixed

After a new release, the app recovers on its own from an out-of-date page.
The Analytics model filter works on the quality tabs.
Proposed tests are no longer all thrown away when a document’s summary runs long.
The run log no longer shows raw cost figures.
Pages no longer jump around on phones.

v1.0.7 (July 15, 2026)

Clearer, more accurate wording on the website and the sign-in page.

v1.0.6 (July 14, 2026)

Global admins can set each plan’s AI credits and give an organization a one-time top-up.

New

On hosted OttoTester, global admins can set each plan’s AI credit allowance.
Global admins can give an organization a one-time credit top-up.

v1.0.5 (July 13, 2026)

New sign-ups get free AI credits to try OttoTester before paying.

New

On hosted OttoTester, new sign-ups get 1,500 free AI credits.

Fixed

The features page is simpler and easier to read.

v1.0.4 (July 13, 2026)

AI credits arrive on hosted OttoTester, and you can delete your own account.

New

Plans include an allowance of AI credits, with a warning when you run low. You can buy more, and bought credits last 12 months.
See AI use per person on the Org Usage tab.
Delete your own account. It is held for 30 days before it is removed.
A public features page.

Fixed

Run details show AI use in credits, not raw cost, and workspace members no longer see raw cost.
Unlimited plan limits read “Unlimited”.
Contact sales and Request a demo go somewhere useful.
The test writer no longer gives up when a test data value appears in what it clicks or checks.

v1.0.3 (July 7, 2026)

Sign up and pay on your own, keep test data in one place, see test coverage on the dashboard, and replay sign-ins instead of retyping them.

New

On hosted OttoTester, sign up on your own with a 14-day trial, manage billing, and change plan without starting over.
Test Data keeps one set per test, with columns that take a new value on each run, and a test can run against every row.
Test coverage on the variant dashboard, over time, with a coverage count on proposed tests.
Pick proposed tests when starting a generation run, with help writing the prompt.
OttoTester saves your app’s sign-in and replays it on each run, instead of filling in the form again.
A ready-made sign-in helper for Okta.
Console and network errors from your app can be captured and shown in the run, the PDF report and the run-completed email.
Analytics can filter runs by run type, call type and model, and opens a run’s details in a side panel.
Global admins can set models for each organization and each plan.
Duplicate tests are spotted by meaning, not only by name.
Run templates can slow down and pause between tests.
Global Admin has a side menu, and Models is now called AI Models.

Fixed

One-time sign-in codes are no longer reused, which caused sign-ins to fail now and then.
A working sign-in helper is no longer reset, which blocked every test that needed it.
A brief bad reply from a model no longer throws away the whole run.
A run that can’t load its tests now fails instead of reporting zero tests.
Replay runs no longer run zero tests.
Analytics is shown only to global admins with Global Admin mode on.

v1.0.2 (June 12, 2026)

OttoTester saves the sign-in address it finds, and long AI calls can be stopped.

New

After a successful sign-in, OttoTester saves the sign-in address it found.
The full OpenRouter model catalog is available.

Fixed

AI calls have a time limit, and stop when a run is cancelled.

v1.0.1 (June 11, 2026)

Sign-in is shared between planning and writing tests, scheduled jobs get their own screen, and CI can start runs.

New

Planning and test writing share one sign-in, and the Login.gov sign-in helper is chosen automatically.
A redesigned sign-in card for credentials, showing the sign-in address, with a way to stop it being used automatically.
Scheduled jobs have a catalog with details and settings, and global admins are told when a job fails.
Models get scheduled health checks and a Check now button.
Start runs from your CI pipeline through the runs API or the GitHub Action, set up from the Integrations tab.
Edit a variant’s trigger rules in a side panel.
Test Data has a catalog, a table editor and CSV upload.
Resend the verification email after signing up.

Fixed

Credentials are back, as a tab under Test Data.
Times follow your time zone setting.
Resume Incomplete Work skips tests that already pass.
The app picker switches to an app you just created.
New run templates start with sensible time limits.

v1.0.0 (June 5, 2026)

The first full release. Runs can start when GitHub reports a new deployment.

New

Start a run when GitHub reports a new deployment of your app.

Earlier releases

Releases before v1.0.0 were early previews and have no notes here.