Payments, compliance and QA in iGaming

The defect

On BetMavrik's lottery product, the game API accepted bets on a draw that had already closed. That is the one thing a lottery cannot get wrong: once the entry window shuts, no more money may be taken.

Why this is a critical, not a high

A bet taken after close is a money problem, not a display bug: the operator pays out on a result someone may already know, or faces a complaint that reaches the regulator. And it is silent. No alert, no error, and the product looks healthy until someone checks the tickets.

The part that was not mine to fix

The API belonged to a vendor, so I could not fix it. That changes the job: describe it clearly enough for someone else to act on, then check their fix instead of trusting it. I wrote a full incident report, and confirmed the fix live in a separate retest.

The step most QA work never reaches

I then turned the case into an automated test, so the build fails if the bug comes back. It runs in the product's daily checks. A vendor fix is only proven for the day you checked it: their code can change without telling you, and if this bug returns, it looks like normal traffic.

Why the automation could not see what it claimed to check

A crawler opened every game in a casino's catalogue and reported whether it loaded. But each game runs inside a window served by another company, which our code cannot see into or click. So all the crawler could really check was that the window existed. The screenshot it saved was the only real evidence, and nothing was reading it.

Checking the verdicts instead of trusting the number

I compared its verdicts against its own screenshots:

Two causes, and the window check could not see either. Some games showed a loading screen that never finished, with the window there the whole time. Others had loaded, but sat behind a start screen the crawler could not click past.

Then I checked my own fix, and it failed

I added an automatic click for the second cause. Checking every one of roughly 2,300 screenshots proved it runs on every game and still does not work: it clicks the centre of the window, while the real "continue" buttons sit near the bottom. Right coverage, wrong pixel.

The stuck loading screen is still unsolved. I built a detector that reads the text on each screenshot, tested it on all of them, and rejected it because it missed too many working games. Both gaps are written down as open.

My check was wrong too, in the other direction

The crawl runs on a test account with no money, so games that loaded fine then showed a "no funds" message over working reels. My first check read that message as an error and failed about 110 healthy games. The crawler had been right; I was wrong. Checking by hand can be wrong too.

Whose bugs these are

I built the dashboard that runs our browser test suites: a card per suite, click to run, and a coloured result at the end. Every bug below is one I shipped and then found in my own tool.

Four ways it lied

1. The badge and its own numbers judged the same run by different rules. The badge came from the test runner's result, which ignores the health-check and balance rows. The numbers underneath counted those rows as failures. So a card could show a red "1 failed" line right below a green "Passed" badge.

2. A run where nothing happened looked like a clean pass. With zero tests counted, the card showed zero of zero, in green.

3. One run could silently overwrite another's report. The daily and regression cards shared a folder and wrote to the same default file. A run started from a terminal wrote straight over the daily report.

4. The results page hid two of its own categories. It showed Total, Passed, Failed and Skipped, but there are six categories, so Warning and Flaky were invisible even while the card beside them said "1 warn". Its label read "98% passing (54 / 56)", but 54 of 56 is 96%. The percentage and the label counted different totals.

The one that actually happened

Bugs 2 and 3 combined into a real incident. An interrupted run, 34 tests, none of them run, overwrote the daily report. The daily card then showed zero of zero passed, in green, under a badge left over from an earlier run that really had passed. No error, no alert: a healthy daily check that had never run.

The fix: four groups, not two

The rule: the badge comes from the run's own result, and the numbers must agree with it. So every row goes into one of four groups instead of two:

The percentage now uses passed plus failed only, so a clean run reads 100% and matches its green badge. Each type of run writes to its own folder, so no run can overwrite another, including one started by hand.

Why a failing probe is amber and not red

The health check is a simple request that connects differently from the browser running the test; when it reports an error, that is often a bot check the real browser passes. The browser decides whether a page works, not the probe. So a probe that disagrees is information, not a verdict. Counting it as a failure would teach everyone to ignore red, which is as bad as a green that means nothing.

The symptom

A scheduled job ran the smoke tests against our live sites every day, saving reports and screenshots. It had been failing for at least seven days in a row, and nothing showed it.

What it actually was

The job needs six stored secrets: five logins and one notification channel. None of them existed. So every run failed at sign-in, every day, for a week. The tests themselves were fine; the job turned red only because there were no logins to sign in with.

Two failures, one colour

"The tests failed" and "the tests never ran" produced the same red. They are not the same event and they do not have the same fix. One is a bug in the product. The other is a week with no checks at all on sites that handle money. And a red nobody reads is not a warning at all.

The part I could not fix

Only the repository owner can add these values, so there was no code to write. The result was a clear request to the one person who could act: which six secrets, what each is for, and what stays broken until they exist.

I had to correct my own record first

An earlier note in the project said only two of the six were missing. The real number was six. The note had gone stale, and I had to re-check and correct it before I could trust my own report. That correction is why the finding is right: written facts go stale, and the worst ones are those written by someone people trust, because nobody re-checks them.

On the severity

This is Medium on purpose. Nothing in the live product was broken; what was lost was a week of checking. I would rather rate it correctly than inflate it. If everything is critical, nobody can sort the list, the same failure as a dashboard where everything is green.

Why this suite is different

Most test suites cost you time. This one spends real money: every run places real bets from a funded account. A test that checks nothing is normally wasted time; here it also spends money while reporting success.

A test that could not do what its name said

A test named "should show an error when the stake exceeds the balance" could never show it. The game has a fixed price and no stake field, so that path could never run. Instead it placed an ordinary charged bet on every run, and one path logged "PASS" without checking anything. It had a green result, a believable name, a real cost, and no value. I rewrote it to call the game API directly and check that an over-balance stake is refused with no charge.

The warning I wrote into the rewrite

The new test carries a warning about its own worst case: if the backend has no balance check, this drains the wallet rather than costing one bet. A test that probes a money limit can cost more than the bug it finds.

Two more that could not fail

A check that could never fail. The duplicate-bet test counted success messages with a method that only ever returns zero or one, then checked the count was never two. That can never be false. So duplicate-bet protection, the thing between a player and a double charge, was untested.

And it accepted almost any charge. The same test passed on any deduction below a generous limit, so two real bets went through happily. I rewrote it to count tickets before and after, and to require exactly one.

A bug in my own test code

One endpoint returns the balance in hundredths of a unit; another in whole units. A check subtracted one from the other, comparing different units, so it could pass or fail for the wrong reason. Fixed by reading both the same way. This was a bug in the test, not the product, and I record it on purpose.

The button that spent money by default

The script that ran every suite included the bulk ticket tests. So a casual "run everything", the most harmless-looking action there was, spent real money. It is now limited to the daily set and cannot spend.

Halving the cost, and what nearly broke

Set-up created five tickets per game type, across three types. Cutting that to two roughly halved the real-money cost of a regression run. The catch: the suite's "all draws closed" check used a limit tuned to the old count. Left alone, a genuinely closed draw would have turned from a clean skip into a red failure. Adjusting that limit was what stopped the saving breaking the reporting.

The symptom

I keep dated copies of all results, so any change can be compared before and after. After a routine change I did the usual check: explain every difference, or do not ship. Most explained themselves. Then 81 differences remained with no change in the inputs, 67 of them score moves on records nobody had touched. Some results appeared from nothing; others vanished. The clue: 35 of the moves were the same size, to one decimal place, in both directions.

Ruling out the simple explanations

Before hunting for a cause I ruled out the obvious ones. Every profile was exactly the same. The admin log showed no changes to any affected record. All job fields were identical. Three checks, one conclusion: something was changing the output while every input stayed still.

The cause

The scoring code built a list using a Python set, then read it back in order. The scorer cares about position: the same item is worth more in an earlier slot. But a Python set has no fixed order. It depends on a start-up setting that was not fixed anywhere, so every restart reshuffled it, and the nightly job wrote that shuffle into the database.

Two more effects made it worse than cosmetic. Records near a cut-off crossed it, so results really did appear and disappear for users between deploys. And it added constant noise to my own before-and-after comparison, the tool I use to approve changes.

Proving it, rather than believing it

A likely cause is not a proven one. I rebuilt it from clean test data and ran one pair of records under 25 different start-up settings:

The fix, and the place a text search would have missed

Read the list in sorted order, in three places, not one. The third place was found by searching the code's structure rather than its text: it saved an ordered list to the database, rewriting that field on every re-score with no real change behind it. A plain text search would have missed two of the three.

The test that had to exist

A normal test cannot catch this: the setting is fixed when the program starts, so every test in one run shares it. The regression test starts several separate runs with different settings and requires identical output. Two rules: it was proven to fail on the broken code first, and it ships with a second test confirming the example still depends on order, so the check cannot quietly stop meaning anything.

Confirmed in production, on the right measure

The measure of success mattered: two full re-scores in a row must produce identical stored results. Reading a saved copy cannot prove that. The overnight run moved exactly what the approved change predicted, same count and same pattern, so it added nothing of its own. And the clue was gone: not one difference at the tell-tale size, which had produced 97 the night before.

The symptom

One listing was dead but still being shown. The expiry check had not caught it, and it was not honestly saying "I cannot tell". It had run, decided, and got the wrong answer. It saw a page that returned success, redirected to the employer's main page, carried an error flag in the address, and had a title that was not the job. Four signs saying "gone". Verdict: keep.

The cause

The rule was: if a strong job identifier survives into the new address, the listing has only moved. Sensible. But the check for "is this part an identifier?" said yes if it contained a digit or was 12 or more characters long, a length rule meant for sites that use words as identifiers. These addresses look like /{employer}/jobs/{id}, and employer names are usually long too. So the rule saw an identifier survive and decided the job had moved. What had really survived was the company name, which always survives, because that is exactly where a site sends you when a job is gone.

The proof I care most about

Same redirect, same everything, changing only the employer name in the address:

That one table turned a vague "this seems unreliable" into a fix everyone agreed on in about thirty seconds. Then I measured how far it spread rather than guessing: roughly one in five affected listings had a name long enough to hide the problem.

The fix, and the risk it carried

Require the most specific identifier to survive: a part with digits in it is the real listing id, so require that and ignore the words. Keep the length rule only where no numeric part exists. And read the ?error flag, which had been thrown away because only the URL path was compared.

This change turns a "keep" into an "expire", and that direction is dangerous. A wrong expiry silently deletes real listings, and the original caution was deliberate. So it did not ship on a passing unit test, but after a replay against the real data:

The symptom

A user reported a link opening a site's home page instead of the job. A small thing, and I had seen one similar report days earlier. Two stories are not a bug report, so I measured: on that source, 68% of live listings were dead. I confirmed it three ways before believing my own number: a bulk check, a one-by-one recheck, and a real browser, in case the bug was in my own tool.

The cause

The check expired a listing on a few specific "not found" responses. Every other outcome fell through to "keep". That one default merged two completely different states: confirmed working and could not check.

The source answered with a bot-check page instead. That was neither "not found" nor success, so the first branch missed it and the second never ran. Verdict: keep. Forever. A bot-blocked source can never be checked, and its dead listings pile up unseen: no counter, no log line anyone read, no health check. Looking for others with the same shape, I found a second. Together, a quarter of all live listings sat in sources that had never once been checked successfully.

The finding that changed the fix

The obvious fix, detect the bot check and retry, does not work, and proving that took a real experiment. The bot check appeared for dead and live listings alike from the server, while the same addresses worked normally from a home connection. So the block was on the server's network, not on specific pages. Pretending to be a normal browser was ruled out by test, not assumption: a real browser identity was blocked in exactly the same way.

Labelling the response "could not check" makes the blind spot visible, but does not bring back the ability to check. Those were two separate problems, and I stopped treating them as one. So: fix the data at once and make it reversible, give the check three outcomes instead of two, and plan the different connection as its own piece of work. One rule written down: never expire automatically on "could not check". Being too strict there deletes real listings, and you never find out.

The programme

Michi runs an ongoing audit programme: 86 written audits covering more than 60 areas, including accessibility, security, performance, GDPR, reliability, cost and moderation quality. Each is dated, never edited afterwards, and tied to a version so it can be re-checked later.

The uncomfortable one

One audited my own test coverage, with a deliberately hard question: where is the dangerous untested part? The answer was not flattering. The backend was well tested, but the two front ends were effectively untested. It credited the backend for the right reason too: those tests check real behaviour where trust matters, such as who owns what and who may read whose data, rather than checking status codes and calling that coverage.

What happened next

The E2E suite starts its own separate backend with a throwaway database, so it never touches production data.

The two habits behind it

Finding and fixing are separate roles. Test work runs on its own, with written limits: it may read anything, run anything and write tests, but may not change product code. Its standing rule: if you find a bug while writing tests, describe it clearly and stop. That removes the strongest temptation in QA, quietly patching what you found, so that neither the bug nor the reasoning is ever written down.

Important reviews get a second, hostile pass. I re-run them with one instruction: assume the first pass was careless and prove it wrong. The most recent produced 21 corrections. That pass also caught a bug in its own tooling: two stages used different path formats, so seven corrections silently failed to apply, noticed only because the totals did not add up. The rule I wrote afterwards: a step that merges results needs a real check, not a hope, and when two verdicts disagree, keep the stricter one.