Commit to freedom
The technology overrunning our communities isn't built for the benefit of
humanity: it's built so that billionaires can control us. Our lives don't
have to be this way. The FSF empowers users with free software,
technology built through global collaboration, mutual respect, and above all,
freedom. The FSF will always continue to resist the encroachment of nonfree
software. We're fighting to protect all computer users and build a better
future.
We can't do it alone: commit to freedom and join us today!
Join | Read more
Join
Renew
Donate
Skip to content, sitemap or skip to search.
You
are here:
Home
›
Blogs
›
Community
›
How the FSF sysadmins block botnets with reaction
Info
How the FSF sysadmins block botnets with reaction
by
Michael McMahon
Contributions
—
Published on
Jul 09, 2026 11:29 AM
I have written about the FSF facing DDoS attacks several times,
including on doing our part to clean up the internet and on
Uptime Kuma, as well as "Defending Savannah from DDoS
attacks". But I realized recently that I left out an
important tool that we use extensively. Today I will talk about
reaction. With the rise of aggressive scrapers building datasets
to train LLMs, we needed to invest a lot of resources into thwarting
the attackers. During this journey, we had several growing pains.
We noticed patterns in the scrapers that were abnormal, which gave us
material for writing regular expressions. Searching for the regular
expression then gave us a large lists of IP addresses. Looking up the
origin of those IP addresses revealed that some of the crawlers were
using botnets of residential IP addresses to scrape faster and avoid
detection.
We looked for what kinds of botnets might be generating the kind of
traffic that we were seeing, and one that we suspected was called the
"Vo1d" botnet, comprised of smart TVs running some sort of compromised
app. Recently, more information has come out about the Vo1d/Popa
botnet. Security researchers at Qurium, a digital forensics
investigation organization, intentionally ran a node and published the
data. We got confirmation that at least some of the botnet traffic
hitting GNU Savannah was originating through the Vo1d/Popa
botnet.
We placed our regular expressions in fail2ban, and found that we
were hitting the maximum rules that could be added to UFW
firewall rules on our systems which showed degradation around 65,000
rules. fail2ban is a program that can read the stream of log files and
perform actions when certain conditions are met. The actions are
typically banning an address with a firewall like UFW, which blocks or
allows connections to and from a server. We learned about ipset
and configured fail2ban to add IP addresses that it found to IP sets.
Using ipset, we kept building larger IP sets and did not find
instability with as large as five million rules.
We used fail2ban with ipset for a while but ran into limits of its
architecture with Python and SQLite. We rolled our own solutions as a
quick patch several times with BASH, awk, Perl, etc. to find the
address matching the pattern and add the rule to ipset without an
additional intermediary database. The custom scripts worked for a
moment, but it became difficult to manage when we had to run several
little scripts as more patterns appeared. Not all of the patterns
should be banned on first appearance either so we needed a solution
like fail2ban. We looked for fail2ban alternatives written recently.
We found several, but many of them were written as one-off showcases
rather than well-maintained solutions that would last. We eventually
found a promising project on Framasoft's forge Framagit called
reaction written by ppom.
It took us a bit of time to understand how to configure reaction.
Unlike fail2ban, reaction does not come with a working configuration
and you must build your own configuration with what is relevant to your
needs from example documentation. There was not an example for using
ipset so we had to build one. Overall, this approach is very good
because we now have a configuration that has everything we need and
very little that we do not need.
After we ran into scaling issues with our initial implementation, we
developed a much faster implementation where the reaction shutdown
process would export the IP sets to disk and the reaction startup
process would restore the IP sets. This allowed us to have nearly
instantaneous restarts of the service to apply new rules. We published
both of our configurations upstream to reaction's wiki so that
everyone can benefit from it. reaction's getting started documentation
now leads to the method that we proposed.
This work is not possible without your support. Because of free
software supporters just like you, we can continue to find new
patterns, block crawlers and mitigate DDoS attacks, and share our
improvements with everyone. We know not everyone is in a position to
give, but if you can, support our efforts by joining the FSF as an
associate member. An associate membership is a great show of
support we can rely on. The FSF does not compromise when it comes to
defending your freedom. Without the continued commitment of people
like you, the progress we have made is in danger, and software freedom
could be reduced to a mere wish instead of today's reality. Please
help us continue to protect software freedom and increase its global
support. Every membership this summer will help us towards our goal of
175 new members. Associate members will also be able to enjoy all the
member benefits which include merchandise discounts, a 16GB
bootable membership card, and use of our associate member
videoconferencing server.
Many sysadmins know about fail2ban, but not enough people know about
reaction. I am very grateful to ppom for the help they have
provided and for the tremendous project they have released to the
world with reaction. We have implemented other defenses as well, but
reaction is doing the majority of the automated work keeping our sites
online.
Document Actions
Share on social networks
Syndicate:
News
Events
Blogs
Jobs
GNU
The FSF is a charity with a worldwide mission to advance software freedom — learn about our history and work.
fsf.org is powered by:
Plone
Zope
Python
CiviCRM
HTML5
Link preview
How the FSF sysadmins block botnets with reaction
Commit to freedom The technology overrunning our communities isn't built for the benefit of humanity: it's built so that billionaires can control us. Our lives don't have to be this way. The FSF empowers users with free software, technology built through global collaboration, mutual respect, and above all, freedom. The FSF will always continue to resist the encroachment of nonfree software. We're fighting to protect all computer users and build a better future. We can't do it alone: commit to freedom and join us today! Join | Read more Join Renew Donate Skip to content, sitemap or skip to search. You are here: Home › Blogs › Community › How the FSF sysadmins block botnets with reaction Info How the FSF sysadmins block botnets with reaction by Michael McMahon Contributions — Published on Jul 09, 2026 11:29 AM I have written about the FSF facing DDoS attacks several times, including on doing our part to clean up the internet and on Uptime Kuma, as well as "Defending Savannah from DDoS attacks". But I realized recently that I left out an important tool that we use extensively. Today I will talk about reaction. With the rise of aggressive scrapers building datasets to train LLMs, we needed to invest a lot of resources into thwarting the attackers. During this journey, we had several growing pains. We noticed patterns in the scrapers that were abnormal, which gave us material for writing regular expressions. Searching for the regular expression then gave us a large lists of IP addresses. Looking up the origin of those IP addresses revealed that some of the crawlers were using botnets of residential IP addresses to scrape faster and avoid detection. We looked for what kinds of botnets might be generating the kind of traffic that we were seeing, and one that we suspected was called the "Vo1d" botnet, comprised of smart TVs running some sort of compromised app. Recently, more information has come out about the Vo1d/Popa botnet. Security researchers at Qurium, a digital forensics investigation organization, intentionally ran a node and published the data. We got confirmation that at least some of the botnet traffic hitting GNU Savannah was originating through the Vo1d/Popa botnet. We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules. fail2ban is a program that can read the stream of log files and perform actions when certain conditions are met. The actions are typically banning an address with a firewall like UFW, which blocks or allows connections to and from a server. We learned about ipset and configured fail2ban to add IP addresses that it found to IP sets. Using ipset, we kept building larger IP sets and did not find instability with as large as five million rules. We used fail2ban with ipset for a while but ran into limits of its architecture with Python and SQLite. We rolled our own solutions as a quick patch several times with BASH, awk, Perl, etc. to find the address matching the pattern and add the rule to ipset without an additional intermediary database. The custom scripts worked for a moment, but it became difficult to manage when we had to run several little scripts as more patterns appeared. Not all of the patterns should be banned on first appearance either so we needed a solution like fail2ban. We looked for fail2ban alternatives written recently. We found several, but many of them were written as one-off showcases rather than well-maintained solutions that would last. We eventually found a promising project on Framasoft's forge Framagit called reaction written by ppom. It took us a bit of time to understand how to configure reaction. Unlike fail2ban, reaction does not come with a working configuration and you must build your own configuration with what is relevant to your needs from example documentation. There was not an example for using ipset so we had to build one. Overall, this approach is very good because we now have a configuration that has everything we need and very little that we do not need. After we ran into scaling issues with our initial implementation, we developed a much faster implementation where the reaction shutdown process would export the IP sets to disk and the reaction startup process would restore the IP sets. This allowed us to have nearly instantaneous restarts of the service to apply new rules. We published both of our configurations upstream to reaction's wiki so that everyone can benefit from it. reaction's getting started documentation now leads to the method that we proposed. This work is not possible without your support. Because of free software supporters just like you, we can continue to find new patterns, block crawlers and mitigate DDoS attacks, and share our improvements with everyone. We know not everyone is in a position to give, but if you can, support our efforts by joining the FSF as an associate member. An associate membership is a great show of support we can rely on. The FSF does not compromise when it comes to defending your freedom. Without the continued commitment of people like you, the progress we have made is in danger, and software freedom could be reduced to a mere wish instead of today's reality. Please help us continue to protect software freedom and increase its global support. Every membership this summer will help us towards our goal of 175 new members. Associate members will also be able to enjoy all the member benefits which include merchandise discounts, a 16GB bootable membership card, and use of our associate member videoconferencing server. Many sysadmins know about fail2ban, but not enough people know about reaction. I am very grateful to ppom for the help they have provided and for the tremendous project they have released to the world with reaction. We have implemented other defenses as well, but reaction is doing the majority of the automated work keeping our sites online. Document Actions Share on social networks Syndicate: News Events Blogs Jobs GNU The FSF is a charity with a worldwide mission to advance software freedom — learn about our history and work. fsf.org is powered by: Plone Zope Python CiviCRM HTML5 Source: https://popa.live/csv/watched/git_domains.csv fsf.org · popa.live
Comments