"We are not only responsible for what we do, but also for what we do not do"

Shipping software means handing agency to strangers. That is the whole point. It is also why it never stays without consequence.

Some of my tools make that very concrete. sitemap-tracker builds a complete sitemap for sites that lack a usable one and tracks down dead links. c2pa-scanner checks images for provenance data so you can label your own website under the EU AI Act. console-error-scanner finds JavaScript errors and HTTP errors on websites.

Whether these are terminal tools, web apps or desktop applications does not matter here. They are helpful tools meant to find errors and support site owners and administrators. To do that, they have to analyse servers, and in doing so they generate traffic.

That is where good intentions stop being a purely technical matter. And that is where the developer has to stop thinking from a purely technical perspective, the one where patterns and elegant code are the yardstick. He has to think further and be aware that his tool will not end up in sensible hands only.

What cannot be prevented

A crawler cannot tell sensible error hunting from harmful harvesting. It sees a URL and fetches it. Whether someone is tidying up their own website or scanning a competitor’s is not in any request header.

So the starting point is a sober one: misuse cannot be prevented. Publishing a tool means giving it away. No license, no README and no well-meant paragraph titled “What it is for” changes that. Misuse is never what I intend, but “not what I intend” is an opinion, not a safeguard.

And it is not specific to my tools either. The best known example sits on almost every developer machine: curl. A tiny program, maintained for decades, built into countless scripts, build pipelines and devices, and impossible to imagine daily work without. Put it in a loop with enough parallelism and the same program becomes a hammer that takes a small server down. Still, nobody would call curl a bad tool.

The difference lies in the level of abstraction. curl is a basic tool. It makes exactly one request and leaves every further decision to the caller. If you want a brake, you pass --limit-rate. If you want to respect robots.txt, you build that yourself. That is perfectly fine, because the user of a basic tool takes control deliberately. With a finished application and a start button that shifts: the more finished the tool, the more decisions the developer makes in place of the user.

The interesting question sits one level below. Not: what will the worst user do with my tool? But: what does my tool do when nobody means any harm and simply hits start? That second question is entirely mine to answer.

Four practices I try (!) to answer it with.

1. A well-considered brake belongs in the default

A tool that touches other people’s servers needs a rate limit out of the box. Not as an option for the cautious user, but as the standard the impatient one has to switch off on purpose.

The most common confusion here: concurrency is not a rate. A semaphore caps how many requests run in parallel, not how many per second. If the target responds quickly, the tool fires the next batch just as quickly. So the well-built server takes the hardest hit. When crawling, the page count is not even known up front, because the crawler follows links down to the configured depth.

A real limiter therefore spreads all fetches evenly over time and applies to retries as well. My tools now default to 60 requests per minute, adjustable via a slider or --rate-limit, and 0 turns it off. The approach is the decisive point: off by choice, not on by choice. Anyone who wants to crawl faster makes a deliberate decision, and from then on it is theirs.

2. robots.txt properly or not at all

robots.txt deserves respect. But a “respect robots.txt” checkbox that respects nothing in practice is worse than none, because it reassures.

Rules under RFC 9309 use * and $. A naive prefix comparison never matches real-world entries. An example from the field:

Disallow: /*CR-Dokumentation.pdf$

A prefix check asks whether the path literally starts with /*CR-Dokumentation.pdf. No path does. The rule would be there and would never fire. Conflict resolution belongs to it as well: on multiple matches the longest rule wins, and Allow wins on equal length.

And when a gap remains, it belongs in the documentation. c2pa-scanner checks the pages from the sitemap, not the images, because images often live on a CDN domain with its own robots.txt. That is stated in the README. A known limit is not a flaw. A hidden one is.

3. The notice you have to confirm

In my tools a disclaimer blocks startup until it is confirmed. The consent is stored together with the wording, so you are only asked again when the text changes. For scripts and headless CI there is --accept-disclaimer.

For the record, so nobody gets the wrong idea: the disclaimer is not the safeguard. It prevents no misuse at all. What it does is make sure every user has once consciously read what the tool does, what it is for and what it is explicitly not for.

The effect lives in the defaults, the clarity in the notice. A notice without sensible defaults is cosmetics. Sensible defaults without a notice leave the thinking to the user. Only both together make a tool you publish with a clear conscience.

4. Legal texts in the reader’s language

One thing that easily slips through in tools built in a German-speaking country: the language. A disclaimer the reader cannot understand is not a disclaimer. Having it confirmed anyway is a formality without substance.

In my tools and applications the language on first start therefore follows the system environment, deliberately asymmetric: German only for a demonstrably German-speaking environment. Unknown language, empty environment or an error while reading it -> English, the language of the project documentation. In case of doubt the user gets the language they are more likely to read.

The question before the release

Open source, client project or internal tool makes no difference here. Responsibility does not hang on the licensing model. It hangs on the fact that other people and other systems come into contact with the result.

So I have made a habit of one question before every release:

What is the worst reasonable use of my tool - and what is my default against it?

Not the worst conceivable one. Nobody wins against deliberate intent. The worst reasonable one: the user who means no harm, just starts the thing and keeps the defaults. They meet exactly the defaults I set. Or the ones nobody thought about.

Voltaire allowed no excuses for responsibility in action, and even fewer for responsibility in omission. That is worth taking as a guide. With every tool we build, the question of how it can end up being used belongs in our thinking from the start.

Links: sitemap-tracker · c2pa-scanner · console-error-scanner · RFC 9309 (robots.txt)