Comments
I love property-based testing, especially the way it can uncover edge cases you wouldn't have thought about. Haven't used Hypothesis yet, but I once had FsCheck (property-based testing for F#) find a case where the data structure I was writing failed when there were exactly 24 items in the list and you tried to append a 25th. That was a test case I wouldn't have thought to write on my own, but the particular number (it was always the 25th item that failed) quickly led me to find the bug. Once my property tests were running overnight and not finding any failures after thousands and thousands of random cases, I started to feel a lot more confident that I'd nailed down the bugs.
By tombert 2025-11-055:22 I had a similar thing, with F# as well actually.
We had some code that used a square root, and in some very rare circumstances, we could get a negative number, which would throw an exception. I don't think i would have even considered that possibility if FsCheck hadn't generated it.
By rmunn 2025-11-0523:23 I can no longer edit the parent comment, but I want to address one thing:
> Once my property tests were running overnight
This was happening because I was setting my test runner to say "Keep generating random data and running tests for 8 hours, no matter how many tests that is", not because the tests were slow. I was running literally millions of tests overnight (usually about 2-3 million), though later when I added some slow tests that number went down into the tens of thousands of tests overnight.
I may have given the impression that property-based testing is slow. It doesn't have to be. It can be very, very quick. But the more random tests you run, the more likely to are to uncover the really, REALLY bizarre edge cases.
That example caught my attention. What was it in your code that made length 24 special?
By bunderbunder 2025-11-0515:072 reply I love them for this, too.
Sadly I have a really hard time getting teammates to agree to using property-based testing - or letting me use it - because they take "no non-deterministic tests" as ironclad dogma without really understanding the the principle's real intent.
(I can do it to find edge cases to convert to deterministic unit tests in the privacy of my own home, of course. But not being able to commit a library of useful strategies without people calling foul in code review is obnoxious.)
> Once my property tests were running overnight and not finding any failures
Quickly addressing the time it takes to run property-based tests, especially in Python: it's extremely helpful to make sure that running subsets of unit tests is convenient and documented and all your developers are doing it. Otherwise they may revolt against property-based testing because of the time it takes to run tests. Again, especially in Python.
By NortySpock 2025-11-054:449 reply I keep thinking I have a possible use case for property -based testing, and then I am up to my armpits in trying to understand the on-the-ground problem and don't feel like I have time to learn a DSL for describing all possible inputs and outputs when I already had an existing function (the subject-under-test) that I don't understand.
So rather than try to learn to black boxes at the same time , I fall back to "several more unit tests to document more edge cases to defensibly guard against"
Is there some simple way to describe this defensive programming iteration pattern in Hypothesis? Normally we just null-check and return early and have to deal with the early-return case. How do I quickly write property tests to check that my code handles the most obvious edge cases?
In addition to what other people have said:
> [...] time to learn a DSL for describing all possible inputs and outputs when I already had an existing function [...]
You don't have to describe all possible inputs and outputs. Even just being able to describe some classes of inputs can be useful.
As a really simple example: many example-based tests have some values that are arbitrary and the test shouldn't care about them, like eg employees names when you are populating a database or whatever. Instead of just hard-coding 'foo' and 'bar', you can have hypothesis create arbitrary values there.
Just like learning how to write (unit) testable code is a skill that needs to be learned, learning how to write property-testable code is also a skill that needs practice.
What's less obvious: retro-fitting property-based tests on an exiting codebase with existing example-based tests is almost a separate skill. It's harder than writing your code with property based tests in mind.
---
Some common properties to test:
* Your code doesn't crash on random inputs (or only throws a short whitelist of allowed exceptions).
* Applying a specific functionality should be idempotent, ie doing that operation multiple times should give the same results as applying it only once.
* Order of input doesn't matter (for some functionality)
* Testing your prod implementation against a simpler implementation, that's perhaps too slow for prod or only works on a restricted subset of the real problem. The reference implementation doesn't even have to be simpler: just having a different approach is often enough.
By akshayshah 2025-11-056:17 Sibling comments have already mentioned some common strategies - but if you have half an hour to spare, the property-based testing series on the F# for Fun and Profit blog is well worth your time. The material isn’t really specific to F#.
https://fsharpforfunandprofit.com/series/property-based-test...
By sunshowers 2025-11-055:081 reply The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same.
A more complex kind of PBT is if you have two implementations of an algorithm or data structure, one that's fast but tricky and the other one slow but easy to verify. (Say, quick sort vs bubble sort.) Generate data or operations randomly and ensure the results are the same.
By bunderbunder 2025-11-0515:11 For working with legacy systems I tend to start with Approval Tests (https://approvaltests.com/). Because they don't require me to understand the SUT very well before I can get started with them, and because they help me start to understand it more quickly.
I think the easiest way is to start with general properties and general input, and tighten them up as needed. The property might just be "doesn't throw an exception", in some cases.
If you find yourself writing several edge cases manually with a common test logic, I think the @example decorator in Hypothesis is a quick way to do that: https://hypothesis.readthedocs.io/en/latest/reference/api.ht...
By disambiguation 2025-11-055:04 I've only used it once before, not as unit testing, but as stress testing for a new customer facing api. I wanted to say with confidence "this will never throw an NPE". Also the logic was so complex (and the deadline so short) the only reasonable way to test was to generate large amounts of output data and review it manually for anomalies.
By meejah 2025-11-055:15 Here are some fairly simple examples: testing port parsing https://github.com/meejah/fowl/blob/e8253467d7072cd05f21de7c...
...and https://github.com/magic-wormhole/magic-wormhole/blob/1b4732...
The simplest ones to get started with are "strings", IMO, and also gives you lots of mileage (because it'll definitely test some weird unicode). So, somewhere in your API where you take some user-entered strings -- even something "open ended" like "a name" -- you can make use of Hypothesis to try a few things. This has definitely uncovered unicode bugs for me.
Some more complex things can be made with some custom strategies. The most-Hypothesis-heavy tests I've personally worked with are from Magic Folder strategies: https://github.com/tahoe-lafs/magic-folder/blob/main/src/mag...
The only real downside is that a Hypothesis-heavy test-suite like the above can take a while to run (but you can instruct it to only produce one example per test). Obviously, one example per test won't catch everything, but is way faster when developing and Hypothesis remembers "bad" examples so if you occasionally do a longer run it'll remember things that caused errors before.
Essentially this is a good example of parametrized tests, just supercharged with generated inputs.
So if you already have parametrized tests, you're already halfway there.
By chriswarbo 2025-11-060:24 When I'm "up to my armpits in trying to understand the on-the-ground problem", I find PBT great for quickly find mistakes in the assumptions/intuitions I'm making about surrounding code and helper functions.
Whenever I find myself thinking "WTF? Surely ABC does XYZ?", and the code for ABC isn't immediately obvious, then I'll bang-out an `ABC_does_XYZ` property and see if I'm wrong. This can be much faster than trying to think up "good" examples to check, especially when I'm not familiar with the domain model, and the relevant values would be giant nested things. I'll let the computer have a go first.
By ed_blackburn 2025-11-059:161 reply I'm using sqlglot to parse hundreds of old mysql back up files to find diffs of schemas. The joys of legacy code. I've found hypothesis to be super helpful for tightening up my parser. I've identified properties (invariants) and built some strategies. I can now generate many more permutations of DDL than I'd thought of before. And I have absolutely confidence in what I'm running.
I started off TDD covered the basics. Learned what I needed to learn about the files I'm dealing with, edge cases, sqlglot and then I moved onto Hypothesis for extra confidence.
I'm curious to see if it'll help with commands for APIs. I nothing else it'll help me appreciate how liberal my API is when perhaps I don't want it to be?
By groundhogstate 2025-11-059:262 reply You might find schemathesis useful for API testing (which IIRC is built on Hypothesis). Certainly helped me find a stack of unhandled edge cases.
YMMV, but once I first set up Schemathesis on one of my MVPs, it took several hours to iron out the kinks. But thereafter, if built into CI, I've found it guards against regression quite effectively.
