Proposal: Stop honoring unsigned control messages (*)

[Russ Allbery]

(*) Except for alt.* and free.*, to the extent that anyone honors them.

Hi all,

I'm considering a policy change for the newsgroup lists maintained at
ftp.isc.org to only honor PGP-signed control messages except for alt.* and
free.* and wanted to run them by everyone.

Historically, control.ctl has included entries for large numbers of local,
regional, and language hierarchies that predate control message signing or
that didn't go to the trouble of creating PGP keys and setting up signing.
Since we didn't want to break anything when control message signing was
introduced, those entries were only changed if there was an abuse problem.
Many of those hierarchies are too small and obscure for anyone to have
bothered to forge control messages for them, even back in the heyday of
control message vandalism.

This has been bothering me for a while, though, since I have a rather
strong interest in making this system as automated as possible since I
have very little time to fix things manually. Vandalism would be easy to
manually repair, but it would require I go do something about it, which is
unappealing.

Possibly more relevantly, I have not seen anyone who in theory is
maintaining any of those non-PGP hierarchies issue a valid control message
in years (probably more than ten years). In practice, I don't believe
anyone is sending unsigned control messages except for alt.* and free.*
(which are intended to be a free-for-all left to each individual site to
manage), and I believe all of those legacy entries are effectively
defunct.

I am therefore proposing removing all non-PGP entries from control.ctl or,
alternately, leaving them there but commented out. I'm kind of leaning
towards the former since if anyone cares about the history for some reason
they can get it from old versions of control.ctl in the INN repository or
from <https://github.com/rra/control-archive/> (and I have no reason to
believe that the people identified with those email addresses still exist
or feel in any way responsible for those hierarchies), but I could be
convinced to leave them there commented out.

Thoughts?

[Adam H. Kerman]

Post by Russ Allbery
(*) Except for alt.* and free.*, to the extent that anyone honors them.
Hi all,
I'm considering a policy change for the newsgroup lists maintained at
ftp.isc.org to only honor PGP-signed control messages except for alt.* and
free.* and wanted to run them by everyone.

Well, I would prefer that you not do that.

chi.*, for instance, hasn't had a hierarchy administrator since Gerry
Swetsky moved away. He never sent a newgroup message to start a new
group that I recall, all groups were started before he was the
administrator. But if a group were proposed, we were supposed to
get together for an in-person meeting, probably called as Uniforum
Chicago or a successor if it's still meeting. It was pretty informal and
mostly an excuse to drink beer, if it ever happened. If people wanted a
new group, Gerry would have sent a newgroup message.

If anyone wanted to propose a newsgroup in a formerly administered regional
hierarchy and there are rules of any kind to follow, he'd have to declare
himself hierarchy administrator, or the act of sending the newgroup message
would be a default declaration of being hierarchy administrator with
respect to that one proposed newsgroup. Under such informal circumstances,
I recommend AGAINST a policy in which the hierarchy is delisted from
control.ctl without having implemented authenticated control messages.

In such a scenario, no, we don't need authenticated control messages.

Post by Russ Allbery
Historically, control.ctl has included entries for large numbers of local,
regional, and language hierarchies that predate control message signing or
that didn't go to the trouble of creating PGP keys and setting up signing.

Unless any of the massive attacks included bogus newgroup messages in
any of these hierarchies, why would they have bothered to have
implemented authenticated control messages in the past?

Regional and language hierarchies were often run on an ad hoc basis.
Please leave them that way.

I haven't reviewed the documents in years, but rone's unified
control.ctl used to list a dozen local hierarchies with a note as to
which institution or News server provider they were for. I thought once
you took over the document, you purged them as they aren't Usenet, or
you moved the list to hierarchy-notes.

Post by Russ Allbery
Since we didn't want to break anything when control message signing was
introduced, those entries were only changed if there was an abuse problem.
Many of those hierarchies are too small and obscure for anyone to have
bothered to forge control messages for them, even back in the heyday of
control message vandalism.

Well, yeah. And I would request that you continue to treat them as
"There is no problem to fix."

Post by Russ Allbery
This has been bothering me for a while, though, since I have a rather
strong interest in making this system as automated as possible since I
have very little time to fix things manually. Vandalism would be easy to
manually repair, but it would require I go do something about it, which is
unappealing.
Possibly more relevantly, I have not seen anyone who in theory is
maintaining any of those non-PGP hierarchies issue a valid control message
in years (probably more than ten years). In practice, I don't believe
anyone is sending unsigned control messages except for alt.* and free.*
(which are intended to be a free-for-all left to each individual site to
manage), and I believe all of those legacy entries are effectively
defunct.

A lot of nearly dead hierarchies may still have a bit of discussion in
the *.general or equivalent newsgroup. Let's leave the option that if
there's an actual need to propose and create a new group, that there is
no requirement to implement authenticated control messages without a
need for it.

In fact, we probably don't want to implement authenticated control
messages in such ad hoc hierarchies as no one would remember where the
key went to. Wasn't the key to ba.* misplaced for close to a decade?

Post by Russ Allbery
I am therefore proposing removing all non-PGP entries from control.ctl or,
alternately, leaving them there but commented out. I'm kind of leaning
towards the former since if anyone cares about the history for some reason
they can get it from old versions of control.ctl in the INN repository or
from <https://github.com/rra/control-archive/> (and I have no reason to
believe that the people identified with those email addresses still exist
or feel in any way responsible for those hierarchies), but I could be
convinced to leave them there commented out.
Thoughts?

Please don't.

[Russ Allbery]

Post by Adam H. Kerman
chi.*, for instance, hasn't had a hierarchy administrator since Gerry
Swetsky moved away. He never sent a newgroup message to start a new
group that I recall, all groups were started before he was the
administrator. But if a group were proposed, we were supposed to get
together for an in-person meeting, probably called as Uniforum Chicago
or a successor if it's still meeting. It was pretty informal and mostly
an excuse to drink beer, if it ever happened. If people wanted a new
group, Gerry would have sent a newgroup message.

The thing is, though is that none of this has happened. Even ten years
ago, legitimate unsigned control messages basically don't exist. So far
as I can tell, the last change to chi.* was Hipcrime sabotage that we had
to manually reverse because we still had this unauthenticated control
message policy. In fact, nearly all chi.* control messages that are
archived are abusive sabotage. Thankfully that hasn't happened since
2002, but if it happened again, it would be a giant mess and a huge pain
for me to clean up.

Post by Adam H. Kerman

Post by Russ Allbery
Historically, control.ctl has included entries for large numbers of
local, regional, and language hierarchies that predate control message
signing or that didn't go to the trouble of creating PGP keys and
setting up signing.

It turns out that I was probably wrong about this and David Lawrence
instead did tons of manual cleanup. There are a bunch of forged control
messages for chi.*, for example, from back when this was common.

Post by Adam H. Kerman
Unless any of the massive attacks included bogus newgroup messages in
any of these hierarchies, why would they have bothered to have
implemented authenticated control messages in the past?

With the above correction, I can note that this did happen, and yet they
still didn't implement authenticated control messages, unfortunately. I
suspect in most cases that's because these folks are no longer using
Usenet, and in most cases (such as with Gary Swetsky) no longer have the
email addresses that they were using to send these messages (and in some
cases may no longer be alive; it's been 30 years in many cases).

Post by Adam H. Kerman
I haven't reviewed the documents in years, but rone's unified
control.ctl used to list a dozen local hierarchies with a note as to
which institution or News server provider they were for. I thought once
you took over the document, you purged them as they aren't Usenet, or
you moved the list to hierarchy-notes.

I don't *think* I removed anything unless I could confirm that it was
defunct. But lots of these hierarchies are just unmaintained and in use
but not changing the newsgroup list.

I see that what I did for wpg.* was replace the entry with:

## WPG (Winnipeg, Manitoba, Canada)
#
# This hierarchy is still in use, but it has no active maintainer.
# Control messages for this hierarchy should not be honored without
# confirming that the sender is the new hierarchy maintainer.

I could do something similar for the others, which would avoid losing the
URL if it still works.

Post by Adam H. Kerman
Well, yeah. And I would request that you continue to treat them as
"There is no problem to fix."

The problem with doing this from my perspective is that at any point it
could turn into a giant problem for me to fix, and should that happen, the
amount of time I'd have to spend on it would be way higher than the amount
of time it would take for me to prevent this proactively now.

Post by Adam H. Kerman
A lot of nearly dead hierarchies may still have a bit of discussion in
the *.general or equivalent newsgroup. Let's leave the option that if
there's an actual need to propose and create a new group, that there is
no requirement to implement authenticated control messages without a
need for it.

I think that's what my proposal does?

Removing them from control.ctl doesn't remove the newsgroups. It just
means no changes will be honored, and the existing newsgroup list will be
kept as-is. That seems fine? If someone wants to change it, they would
have to create a PGP key and set up some software to issue the control
messages, which is a bit higher of a bar, but in practice this seems to
happen rarely and I'm sure a bunch of people here would be happy to help
if it came up.

[Thomas Hochstein]

Post by Russ Allbery
Removing them from control.ctl doesn't remove the newsgroups. It just
means no changes will be honored, and the existing newsgroup list will be
kept as-is. That seems fine? If someone wants to change it, they would
have to create a PGP key and set up some software to issue the control
messages, which is a bit higher of a bar, but in practice this seems to
happen rarely and I'm sure a bunch of people here would be happy to help
if it came up.

+1

I don't think the problem would be to set up signed control messages -
the problem would be to find someone who cares enough to send any
controls.

[Adam H. Kerman]

Post by Russ Allbery

Post by Adam H. Kerman
chi.*, for instance, hasn't had a hierarchy administrator since Gerry
Swetsky moved away. He never sent a newgroup message to start a new
group that I recall, all groups were started before he was the
administrator. But if a group were proposed, we were supposed to get
together for an in-person meeting, probably called as Uniforum Chicago
or a successor if it's still meeting. It was pretty informal and mostly
an excuse to drink beer, if it ever happened. If people wanted a new
group, Gerry would have sent a newgroup message.

The thing is, though is that none of this has happened. Even ten years
ago, legitimate unsigned control messages basically don't exist. So far
as I can tell, the last change to chi.* was Hipcrime sabotage that we had
to manually reverse because we still had this unauthenticated control
message policy. In fact, nearly all chi.* control messages that are
archived are abusive sabotage. Thankfully that hasn't happened since
2002, but if it happened again, it would be a giant mess and a huge pain
for me to clean up.

But weren't these sent as a massive denial-of-service attack and not
individually? Doesn't that allow you to thwart the attack?

Post by Russ Allbery

Post by Adam H. Kerman

Post by Russ Allbery
Historically, control.ctl has included entries for large numbers of
local, regional, and language hierarchies that predate control message
signing or that didn't go to the trouble of creating PGP keys and
setting up signing.

It turns out that I was probably wrong about this and David Lawrence
instead did tons of manual cleanup. There are a bunch of forged control
messages for chi.*, for example, from back when this was common.

Post by Adam H. Kerman
Unless any of the massive attacks included bogus newgroup messages in
any of these hierarchies, why would they have bothered to have
implemented authenticated control messages in the past?

With the above correction, I can note that this did happen, and yet they
still didn't implement authenticated control messages, unfortunately. I
suspect in most cases that's because these folks are no longer using
Usenet, and in most cases (such as with Gary Swetsky) no longer have the
email addresses that they were using to send these messages (and in some
cases may no longer be alive; it's been 30 years in many cases).

Swetsky's email address would have been recreated. CLOUT Project still
exists.

Post by Russ Allbery

Post by Adam H. Kerman
I haven't reviewed the documents in years, but rone's unified
control.ctl used to list a dozen local hierarchies with a note as to
which institution or News server provider they were for. I thought once
you took over the document, you purged them as they aren't Usenet, or
you moved the list to hierarchy-notes.

I don't *think* I removed anything unless I could confirm that it was
defunct. But lots of these hierarchies are just unmaintained and in use
but not changing the newsgroup list.
## WPG (Winnipeg, Manitoba, Canada)
#
# This hierarchy is still in use, but it has no active maintainer.
# Control messages for this hierarchy should not be honored without
# confirming that the sender is the new hierarchy maintainer.
I could do something similar for the others, which would avoid losing the
URL if it still works.

Post by Adam H. Kerman
Well, yeah. And I would request that you continue to treat them as
"There is no problem to fix."

The problem with doing this from my perspective is that at any point it
could turn into a giant problem for me to fix, and should that happen, the
amount of time I'd have to spend on it would be way higher than the amount
of time it would take for me to prevent this proactively now.

This would end archiving of control messages, then.

Post by Russ Allbery

Post by Adam H. Kerman
A lot of nearly dead hierarchies may still have a bit of discussion in
the *.general or equivalent newsgroup. Let's leave the option that if
there's an actual need to propose and create a new group, that there is
no requirement to implement authenticated control messages without a
need for it.

I think that's what my proposal does?
Removing them from control.ctl doesn't remove the newsgroups. It just
means no changes will be honored, and the existing newsgroup list will be
kept as-is. That seems fine?

I'm saying if there is a need to send a newgroup message, you would
rather complicate matters.

Post by Russ Allbery
If someone wants to change it, they would
have to create a PGP key and set up some software to issue the control
messages, which is a bit higher of a bar, but in practice this seems to
happen rarely and I'm sure a bunch of people here would be happy to help
if it came up.

How would you know it's not a troll? That's the trouble with taking
responsibiity for an unfamiliar hierarchy. I'm guessing what would
happen is you'd just end up creating a key yourself.

I don't have a good suggestion that doesn't require manual intervention
either way. If no one is maintaining a hierarchy, not even an
occassional checkgroups, maybe an exchange of emails is necessary should
someone issue a newgroup message before you archive it.

What if you delayed processing the messages for archiving to prevent a
denial of service attack?

[Russ Allbery]

Post by Adam H. Kerman

Post by Russ Allbery
The thing is, though is that none of this has happened. Even ten years
ago, legitimate unsigned control messages basically don't exist. So
far as I can tell, the last change to chi.* was Hipcrime sabotage that
we had to manually reverse because we still had this unauthenticated
control message policy. In fact, nearly all chi.* control messages
that are archived are abusive sabotage. Thankfully that hasn't
happened since 2002, but if it happened again, it would be a giant mess
and a huge pain for me to clean up.

But weren't these sent as a massive denial-of-service attack and not
individually? Doesn't that allow you to thwart the attack?

No. I mean, maybe there's some theoretical filter that could be written
that would do this, but I don't have anything like that (and malicious
people evade filters like that so it would need to be adaptive). And the
whole point for me is that I have about an hour a month to spend on this
stuff, which is definitely not enough time to write an anti-abuse filter.

Post by Adam H. Kerman

Post by Russ Allbery
Removing them from control.ctl doesn't remove the newsgroups. It just
means no changes will be honored, and the existing newsgroup list will
be kept as-is. That seems fine?

I'm saying if there is a need to send a newgroup message, you would
rather complicate matters.

This is true in that right now anyone in the world can just send a
newgroup message. So yes, the whole point of my proposal is to complicate
things, because alas I can't complicate things for the bad guys without
also complicating things for the good guys.

Both the bad guys and the good guys are nonexistent at the moment, which
is what makes this a tricky decision. My opinion is that the bad guys are
more likely to appear in the future than the good guys. (Sadly, I've
probably skewed things in that direction by even bringing this topic up.)

Post by Adam H. Kerman

Post by Russ Allbery
If someone wants to change it, they would have to create a PGP key and
set up some software to issue the control messages, which is a bit
higher of a bar, but in practice this seems to happen rarely and I'm
sure a bunch of people here would be happy to help if it came up.

How would you know it's not a troll?

It seems quite likely that other people who use the hierarchy would object
if a troll went to all the rather noisy and public work of setting up a
key and discussing it in news.admin.hierarchies and the approrpriate group
within the hierarchy (which is something we could enforce).

Post by Adam H. Kerman
I don't have a good suggestion that doesn't require manual intervention
either way. If no one is maintaining a hierarchy, not even an
occassional checkgroups, maybe an exchange of emails is necessary should
someone issue a newgroup message before you archive it.

Yes, exactly. (Well, honor it; currently everything gets archived that
isn't obviously malformed.) Requiring PGP will guarantee that exchange of
emails happens. :)

Post by Adam H. Kerman
What if you delayed processing the messages for archiving to prevent a
denial of service attack?

This still requires I go manually do something to stop an attack in
process, though. (Plus write all the code to do that.) Also, in the
past, these started with a few forged messages and then only escalated
into a denial of service attack later, plus there were other cases where
people sent one-off messages to screw with people. The Hipcrime-style
attack obviously makes the biggest mess, but the whole thing is an
attractive nuisance right now.

BTW, for full information, I did just see an unsigned but apparently valid
checkgroups message today, so apparently at least one hierarchy is sending
them (greenend.*). It is possible that they're out there and I'm not
seeing them, too, since I may not have a full control feed and definitely
don't accept all newsgroups.

[Thomas Hochstein]

Post by Russ Allbery
BTW, for full information, I did just see an unsigned but apparently valid
checkgroups message today, so apparently at least one hierarchy is sending
them (greenend.*).

A private hierachy that was not part of the last Master List I could
find - and I don't doubt that the people of
<http://www.greenend.org.uk/> would be able to sign control messages
if they cared. :)

[Russ Allbery]

Post by Thomas Hochstein

Post by Russ Allbery
BTW, for full information, I did just see an unsigned but apparently
valid checkgroups message today, so apparently at least one hierarchy
is sending them (greenend.*).

A private hierachy that was not part of the last Master List I could
find - and I don't doubt that the people of
<http://www.greenend.org.uk/> would be able to sign control messages
if they cared. :)

Yeah, I'm pretty sure this is just unintentional leakage from some private
peering and not all that relevant to this discussion.

[Richard Kettlewell]

Post by Russ Allbery

Post by Thomas Hochstein

Post by Russ Allbery
BTW, for full information, I did just see an unsigned but apparently
valid checkgroups message today, so apparently at least one hierarchy
is sending them (greenend.*).

A private hierachy that was not part of the last Master List I could
find - and I don't doubt that the people of
<http://www.greenend.org.uk/> would be able to sign control messages
if they cared. :)

Yeah, I'm pretty sure this is just unintentional leakage from some private
peering and not all that relevant to this discussion.

That’s correct.

[Julien ÉLIE]

Hi Russ,

Post by Russ Allbery
(*) Except for alt.* and free.*, to the extent that anyone honors them.
I'm considering a policy change for the newsgroup lists maintained at
ftp.isc.org to only honor PGP-signed control messages except for alt.* and
free.* and wanted to run them by everyone.

[...]

Post by Russ Allbery
Thoughts?

I have no objection.
I reckon it is the right move to do.

Post by Russ Allbery
I don't believe anyone is sending unsigned control messages except
for alt.* and free.* (which are intended to be a free-for-all left to
each individual site to manage)

Ok, though some newsgroup names are questionable (like free.biden.sucks
created today) but that's another debate!
Separate active and newsgroups files containing only alt.* and free.*
newsgroups may be provided by control-archive and put in ftp.isc.org
(just a thought, to better enhance the difference).

Post by Russ Allbery
I am therefore proposing removing all non-PGP entries from control.ctl or,
alternately, leaving them there but commented out.

I'm fine with removing non-PGP entries, including private, local,
historic and defunct hierarchies.
The main argument would be that the control.ctl file is used as a
configuration file, not as the memory of Usenet history.

Private hierarchies like bofh.* or szaf.*, and historic hierarchies like
net.* or eug.*, which have a PGP key, will remain if I understand well.

As well as reserved hierarchies (control.*, example.*, to.* ...) for
technical reasons.

[Russ Allbery]

Post by Julien ÉLIE
Ok, though some newsgroup names are questionable (like free.biden.sucks
created today) but that's another debate!

Oh, yeah, there's all sorts of nonsense in those hierarchies.

Post by Julien ÉLIE
Separate active and newsgroups files containing only alt.* and free.*
newsgroups may be provided by control-archive and put in ftp.isc.org
(just a thought, to better enhance the difference).

If I did that, I would probably generate a new list without alt.* and
free.* and leave the existing one as-is so as not to break any of the
assumptions people are making about the current list.

I personally don't think the list of all alt.* and free.* groups anyone
has ever issued a control message for has very little value, but meh, I
inherited this and I don't feel strongly enough about it to change it.

Post by Julien ÉLIE
Private hierarchies like bofh.* or szaf.*, and historic hierarchies like
net.* or eug.*, which have a PGP key, will remain if I understand well.
As well as reserved hierarchies (control.*, example.*, to.* ...) for
technical reasons.

Yes.

[Adam H. Kerman]

Post by Julien ÉLIE
Hi Russ,

Post by Russ Allbery
(*) Except for alt.* and free.*, to the extent that anyone honors them.
I'm considering a policy change for the newsgroup lists maintained at
ftp.isc.org to only honor PGP-signed control messages except for alt.* and
free.* and wanted to run them by everyone.

[...]

Post by Russ Allbery
Thoughts?

I have no objection.
I reckon it is the right move to do.

Post by Russ Allbery
I don't believe anyone is sending unsigned control messages except
for alt.* and free.* (which are intended to be a free-for-all left to
each individual site to manage)

Ok, though some newsgroup names are questionable (like free.biden.sucks
created today) but that's another debate!
Separate active and newsgroups files containing only alt.* and free.*
newsgroups may be provided by control-archive and put in ftp.isc.org
(just a thought, to better enhance the difference).

I've been thinking about this.

The issue isn't just whether Russ would allow the sample active and
newsgroups files to be updated. The more important issue is whether the
control messages would get archived. If for some reason the sample
active and newsgroups files weren't updated, it's necessary to check the
archive for bad syntax on the Newsgroups file line.

The only question might be if Russ were willing to run a second INN
server for this purpose. Leave the existing server as is. On the new
server, just process and archived signed control messages, which means
not including alt.* and free.*.

Russ's concern about a technical troll attacking the archiving INN
server but continuing to process alt.* and free.* control messages
doesn't sound like Russ could actually allow the thing to run
automatically without ever cleaning up after an attack.

Post by Julien ÉLIE

Post by Russ Allbery
I am therefore proposing removing all non-PGP entries from control.ctl or,
alternately, leaving them there but commented out.

I'm fine with removing non-PGP entries, including private, local,
historic and defunct hierarchies.
The main argument would be that the control.ctl file is used as a
configuration file, not as the memory of Usenet history.

It has less to do with nostalgia but checking to see if anyone is
attempting to revive communication in long unused newsgroups. That's why
they remained. The other purpose is to let a News administrator know the
most basic information about a hierarchy so he doesn't have to wonder
about having received a control message in a hierarchy he's unfamiliar
with.

I need to finish my list. The only hierarchies that shouldn't appear in
control.ctl are local. Again, "local" means "not Usenet" as it's local
within a network and its articles aren't intended to be distributed
beyond the network. "Regional" means the hierarchy is meant to contain
newsgroups in which topics are discussed in relation to a specific
geographical area.

Post by Julien ÉLIE
. . .

[Thomas Hochstein]

Post by Russ Allbery
I'm considering a policy change for the newsgroup lists maintained at
ftp.isc.org to only honor PGP-signed control messages except for alt.* and
free.* and wanted to run them by everyone.

No objections. control.ctl entries honoring unsigned control messages
are accidents waiting to happen.

Post by Russ Allbery
I am therefore proposing removing all non-PGP entries from control.ctl or,
alternately, leaving them there but commented out. I'm kind of leaning
towards the former since if anyone cares about the history for some reason
they can get it from old versions of control.ctl in the INN repository or
from <https://github.com/rra/control-archive/> [...]

I would prefer to keep the entries commented out (or move them to
another file ..., just to keep them around for reference), as that
makes it easier to dive down in history or check for the existence of
a (former) hierarchy, compared to checking old git commits.

But if that's more work than just dropping them, that's okay, too.

-thh

[D. Stussy]

"Russ Allbery" wrote in message news:***@hope.eyrie.org...

(*) Except for alt.* and free.*, to the extent that anyone honors them.

Hi all,

I'm considering a policy change for the newsgroup lists maintained at
ftp.isc.org to only honor PGP-signed control messages except for alt.* and
free.* and wanted to run them by everyone.

Historically, control.ctl has included entries for large numbers of local,
regional, and language hierarchies that predate control message signing or
that didn't go to the trouble of creating PGP keys and setting up signing.
Since we didn't want to break anything when control message signing was
introduced, those entries were only changed if there was an abuse problem.
Many of those hierarchies are too small and obscure for anyone to have
bothered to forge control messages for them, even back in the heyday of
control message vandalism.

This has been bothering me for a while, though, since I have a rather
strong interest in making this system as automated as possible since I
have very little time to fix things manually. Vandalism would be easy to
manually repair, but it would require I go do something about it, which is
unappealing.

Possibly more relevantly, I have not seen anyone who in theory is
maintaining any of those non-PGP hierarchies issue a valid control message
in years (probably more than ten years). In practice, I don't believe
anyone is sending unsigned control messages except for alt.* and free.*
(which are intended to be a free-for-all left to each individual site to
manage), and I believe all of those legacy entries are effectively
defunct.

I am therefore proposing removing all non-PGP entries from control.ctl or,
alternately, leaving them there but commented out. I'm kind of leaning
towards the former since if anyone cares about the history for some reason
they can get it from old versions of control.ctl in the INN repository or
from <https://github.com/rra/control-archive/> (and I have no reason to
believe that the people identified with those email addresses still exist
or feel in any way responsible for those hierarchies), but I could be
convinced to leave them there commented out.

Thoughts?
===================
Under the current scheme, invalid mailboxes (and even a NULL string) are accepted for some control messages where they shouldn't be.
Only the "drop" action (in file "control.ctl") can have the mailbox match be "*", because that is a "don't care" case.

The 'from' field in control.ctl should be changed from "*" (where that appears) to "?*@?*.??*" to make certain that a legal mailbox
is accepted. "*" by itself will match 0 characters, so the adjacent "?"s make certain that each component has at least one
character. The domain side basically needs two components with an intervening dot. Although "localhost" is an acceptable domain,
it is not useful in this context, so I intentionally suggest a syntactic pattern match that excludes it. Some patterns with
matching text may also need "*" changed to "?*" for positive actions (i.e. not drop).

What this change does:
- No NULL usernames. Must have at least 1 non-whitespace character. Although UTF-8 may be allowed in the text comment accompanying
a mailbox (usually a quoted name), only the printable ASCII set is permitted for the mailbox.
- TLDs must be at least 2 characters (and not end with a digit or dash).
- There must be at least 2 domain components ('localhost' specifically denied).

This change would then be used in combination with some exclusion rules that go after various bad mailboxes and/or mailbox syntax
such as those containing two ".." in a row, ".-", etc. I use these entries in my file "control.ctl.local":

## INVALID, BOGUS, & RESERVED MAILBOX PATTERNS (AFTER "?*@?*.??*" APPLIED)
## ------------------------------------------------------------------------
all:*@*[^-.0-9a-z]*|*@[-.]*|*@*.[-.]*|*@*-.*|*@*[^a-z.]|*@*[^a-z].:*:drop
all:*[^!-?A-~]*@*|[-.]*@*|*.[-.]*@*|*[-.]@*|*-.*@*|*fuck*@*|poster:*:drop

One could go further to deny the "example", "invalid", and "test" TLDs, and "example" SLDs, but I chose not to do that here. (I do
filter for such in my "cleanfeed.local" file).

[Russ Allbery]

Post by D. Stussy
Under the current scheme, invalid mailboxes (and even a NULL string) are
accepted for some control messages where they shouldn't be. Only the
"drop" action (in file "control.ctl") can have the mailbox match be "*",
because that is a "don't care" case.
The 'from' field in control.ctl should be changed from "*" (where that
accepted. "*" by itself will match 0 characters, so the adjacent "?"s
make certain that each component has at least one character. The domain
side basically needs two components with an intervening dot. Although
"localhost" is an acceptable domain, it is not useful in this context,
so I intentionally suggest a syntactic pattern match that excludes it.
Some patterns with matching text may also need "*" changed to "?*" for
positive actions (i.e. not drop).

I feel like this sort of tweak just makes the file harder to read and
understand without really accomplishing anything. Those control messages
are only honored for alt.* and free.* anyway, and those hierarchies are a
free-for-all (by design). I feel like there's a very long history of
people trying to "clean up" alt.* while missing the point that the way you
do that is have a managed hierarchy rather than using alt.*.

I'm also unconvinced that this will have any practical effect. The folks
issuing control messages will just switch to valid but bogus addresses,
which is more trivial to do than it is for me to constantly tweak wildcard
patterns. It's an arms race where the available resources are completely
disproportional (and not in my favor).

[D. Stussy]

Post by D. Stussy
Under the current scheme, invalid mailboxes (and even a NULL string) are
accepted for some control messages where they shouldn't be. Only the
"drop" action (in file "control.ctl") can have the mailbox match be "*",
because that is a "don't care" case.
The 'from' field in control.ctl should be changed from "*" (where that
accepted. "*" by itself will match 0 characters, so the adjacent "?"s
make certain that each component has at least one character. The domain
side basically needs two components with an intervening dot. Although
"localhost" is an acceptable domain, it is not useful in this context,
so I intentionally suggest a syntactic pattern match that excludes it.
Some patterns with matching text may also need "*" changed to "?*" for
positive actions (i.e. not drop).

I feel like this sort of tweak just makes the file harder to read and
understand without really accomplishing anything. Those control messages
are only honored for alt.* and free.* anyway, and those hierarchies are a
free-for-all (by design). I feel like there's a very long history of
people trying to "clean up" alt.* while missing the point that the way you
do that is have a managed hierarchy rather than using alt.*.

I'm also unconvinced that this will have any practical effect. The folks
issuing control messages will just switch to valid but bogus addresses,
which is more trivial to do than it is for me to constantly tweak wildcard
patterns. It's an arms race where the available resources are completely
disproportional (and not in my favor).
==========
1) I did not limit my suggestion to just alt/free.*. It applies to ALL hierarchies where the mailbox field has a wildcard and the
action is not "drop."
2) It will eliminate many of the poorly formatted fake control messages which do not use syntactically valid mailboxes.

I examined the past control message archive. Out of about 85,000 groups, only about 35,000 have valid newgroup messages. The other
50,000 fell into three categories: Bad newsgroup names, bad from mailboxes, and omitting the "For your newsgroups file:" line
followed by the group description on the next line.

What does it save: It saves e-mailing the usenet site administrator with bogus messages as the default action for newgroup,
rmgroup, and checkgroups is "mail." This way, bad from mailbox messages get dropped (as they should regardless of whether the mail
server will reject them or not).

A "free-for-all" design shouldn't accept syntactically incorrect messages. What good is having standards if they're not enforced?

[Russ Allbery]

Post by D. Stussy
1) I did not limit my suggestion to just alt/free.*. It applies to ALL
hierarchies where the mailbox field has a wildcard and the action is not
"drop."

It's irrelevant to anything other than alt.* and free.* because those are
the only ones that allow wildcard control messages, no? I think there may
be a few other minor exceptions, but nothing that I've seen in any
significant numbers in many years.

Post by D. Stussy
2) It will eliminate many of the poorly formatted fake control messages
which do not use syntactically valid mailboxes.

What fake control messages are you seeing that aren't for alt.* and
free.*?

Post by D. Stussy
I examined the past control message archive.

What goes into control.ctl is irrelevant to the control message archive.

Post by D. Stussy
Out of about 85,000 groups, only about 35,000 have valid newgroup
messages. The other 50,000 fell into three categories: Bad newsgroup
names, bad from mailboxes, and omitting the "For your newsgroups file:"
line followed by the group description on the next line.

Did you look at the dates? This is almost entirely stuff that was
archived 15 or 20 years ago.

If you're saying that I should go through the archive and delete old
invalid control messages from it, that's a whole different argument. But
nothing about control.ctl has any influence on that.

Post by D. Stussy
What does it save: It saves e-mailing the usenet site administrator
with bogus messages as the default action for newgroup, rmgroup, and
checkgroups is "mail."

How many of these do you get a week? What actual impact would this have?

[Adam H. Kerman]

Post by Russ Allbery

Post by D. Stussy
1) I did not limit my suggestion to just alt/free.*. It applies to ALL
hierarchies where the mailbox field has a wildcard and the action is not
"drop."

It's irrelevant to anything other than alt.* and free.* because those are
the only ones that allow wildcard control messages, no? I think there may
be a few other minor exceptions, but nothing that I've seen in any
significant numbers in many years.

Post by D. Stussy
2) It will eliminate many of the poorly formatted fake control messages
which do not use syntactically valid mailboxes.

What fake control messages are you seeing that aren't for alt.* and
free.*?

Post by D. Stussy
I examined the past control message archive.

What goes into control.ctl is irrelevant to the control message archive.

Waitaminit.

I thought the whole point of this that you were no longer intending to
archive control messages that weren't from hierarchies with PGP signing.

Post by Russ Allbery

Post by D. Stussy
Out of about 85,000 groups, only about 35,000 have valid newgroup
messages. The other 50,000 fell into three categories: Bad newsgroup
names, bad from mailboxes, and omitting the "For your newsgroups file:"
line followed by the group description on the next line.

Did you look at the dates? This is almost entirely stuff that was
archived 15 or 20 years ago.
If you're saying that I should go through the archive and delete old
invalid control messages from it, that's a whole different argument. But
nothing about control.ctl has any influence on that.

Please don't do that. The point of the archive it to save the invalid
along with the valid.

Post by Russ Allbery

Post by D. Stussy
. . .

[Julien ÉLIE]

Hi Adam,

Post by Adam H. Kerman
Waitaminit.
I thought the whole point of this that you were no longer intending to
archive control messages that weren't from hierarchies with PGP signing.

The subject of this thread is "stop honoring" (that is to say actually
creating and removing newsgroups in the ftp.isc.org active file), not
"stop archiving"...

[Adam H. Kerman]

Post by Julien ÉLIE
Hi Adam,

Post by Adam H. Kerman
Waitaminit.
I thought the whole point of this that you were no longer intending to
archive control messages that weren't from hierarchies with PGP signing.

The subject of this thread is "stop honoring" (that is to say actually
creating and removing newsgroups in the ftp.isc.org active file), not
"stop archiving"...

All that does is check for a newsgroups file line in proper syntax!

I just don't see how that's going to prevent a denial-of-service attack.
I understand Russ doesn't have time to deal with attacks but I'm not
following this at all.

As long as Russ intends to maintain the archives, I don't care if the
sample active and newsgroups files get updated. I was objecting to the
loss of the archives.

It'll be a loss to anyone setting up a News server that simply wanted to
use the sample active and newsgroups files but I never thought that was
a brilliant idea.

[Russ Allbery]

Post by Adam H. Kerman
Waitaminit.
I thought the whole point of this that you were no longer intending to
archive control messages that weren't from hierarchies with PGP signing.

No, this whole thread was only about the default control.ctl from INN and
ftp.isc.org and the ftp.isc.org newsgroup list.

ISC hosts the archive and I don't feel any particular need to clean it up.
People have spammed it with all sorts of crap. I added some basic
anti-binary filtering to keep from dealing with stupid copyright nonsense,
and otherwise it doesn't take up much space and I don't realy care.
(Please no one do anything that makes me have to care.)

At some point I may go clean up the archived control messages for
literally syntactically invalid groups that would never be archived today
(there are archive files for nonsense like group names containing *), but
realistlcally I'm too busy with other things and probably won't get around
to it.

[Adam H. Kerman]

Post by Russ Allbery

Post by Adam H. Kerman
Waitaminit.
I thought the whole point of this that you were no longer intending to
archive control messages that weren't from hierarchies with PGP signing.

No, this whole thread was only about the default control.ctl from INN and
ftp.isc.org and the ftp.isc.org newsgroup list.
ISC hosts the archive and I don't feel any particular need to clean it up.

There is no need to clean it up.

I am confused, though. I thought the archive was also fed by a subset of
INN processes that parse for control messages, and that a control
message is archived in the same process that the sample newsgroup and
active files are updated in.

Post by Russ Allbery
People have spammed it with all sorts of crap. I added some basic
anti-binary filtering to keep from dealing with stupid copyright nonsense,
and otherwise it doesn't take up much space and I don't realy care.
(Please no one do anything that makes me have to care.)
At some point I may go clean up the archived control messages for
literally syntactically invalid groups that would never be archived today
(there are archive files for nonsense like group names containing *), but
realistlcally I'm too busy with other things and probably won't get around
to it.

Eh. Leave the old nonsense in place.

[Russ Allbery]

Post by Adam H. Kerman
I am confused, though. I thought the archive was also fed by a subset of
INN processes that parse for control messages, and that a control
message is archived in the same process that the sample newsgroup and
active files are updated in.

The code is unrelated to INN, apart from INN providing an article feed and
the tinyleaf program that I use to process that feed. Both things are
done by the same process, yes, but the archiving is done separately from
the checks about whether to honor the message and applies only more basic
sanity checks to throw away syntactically-invalid junk and figure out what
newsgroup would supposedly be affected by the message.

The code is all in <https://github.com/rra/control-archive>.

[Adam H. Kerman]

Post by Russ Allbery

Post by Adam H. Kerman
I am confused, though. I thought the archive was also fed by a subset of
INN processes that parse for control messages, and that a control
message is archived in the same process that the sample newsgroup and
active files are updated in.

The code is unrelated to INN, apart from INN providing an article feed and
the tinyleaf program that I use to process that feed. Both things are
done by the same process, yes, but the archiving is done separately from
the checks about whether to honor the message and applies only more basic
sanity checks to throw away syntactically-invalid junk and figure out what
newsgroup would supposedly be affected by the message.
The code is all in <https://github.com/rra/control-archive>.

Ok, thanks.

[D. Stussy]

Post by D. Stussy
1) I did not limit my suggestion to just alt/free.*. It applies to ALL
hierarchies where the mailbox field has a wildcard and the action is not
"drop."

It's irrelevant to anything other than alt.* and free.* because those are
the only ones that allow wildcard control messages, no? I think there may
be a few other minor exceptions, but nothing that I've seen in any
significant numbers in many years.

Post by D. Stussy
2) It will eliminate many of the poorly formatted fake control messages
which do not use syntactically valid mailboxes.

What fake control messages are you seeing that aren't for alt.* and
free.*?

Post by D. Stussy
I examined the past control message archive.

What goes into control.ctl is irrelevant to the control message archive.

Post by D. Stussy
Out of about 85,000 groups, only about 35,000 have valid newgroup
messages. The other 50,000 fell into three categories: Bad newsgroup
names, bad from mailboxes, and omitting the "For your newsgroups file:"
line followed by the group description on the next line.

Did you look at the dates? This is almost entirely stuff that was
archived 15 or 20 years ago.
=====
I am quite aware of that. However, if it's been done before and not blocked now, it can be done again.
=====

If you're saying that I should go through the archive and delete old
invalid control messages from it, that's a whole different argument. But
nothing about control.ctl has any influence on that.
=====
I did not suggest that. I simply noted that the majority of what was archived is actually invalid. Although not stated, I also
noted that there are way many more groups that have no newgroup message at all (since the total count, including misspelled groups,
is about 300,000).

Post by D. Stussy
What does it save: It saves e-mailing the usenet site administrator
with bogus messages as the default action for newgroup, rmgroup, and
checkgroups is "mail."

How many of these do you get a week? What actual impact would this have?
=====
Preventing a (malicious) flood of these is the point. I did receive one e-mailed newgroup message for the "rtfm" hierarchy this
month, a hierarchy not in the control.ctl file. The greenend.* hierarchy checkgroups message was also emailed because there is
something wrong with its signature. I don't think that's in the control file either, but I have seen it before so it's in my
control.ctl.local file.

[Russ Allbery]

Post by D. Stussy
Preventing a (malicious) flood of these is the point.

I think this is the root of our disagreement. The way I would phrase the
differing perspectives is that you feel we should block invalid email
addresses because they were used in the past in floods of malicious
control messages. I feel that anyone who is trying to be malicious will
look at how the software is widely configured and pick a syntactically
valid email address to avoid that check (and it's trivial to do so and
there's no good way of stopping this).

Does that sound like a fair characterization?

Post by D. Stussy
I did receive one e-mailed newgroup message for the "rtfm" hierarchy
this month, a hierarchy not in the control.ctl file. The greenend.*
hierarchy checkgroups message was also emailed because there is
something wrong with its signature. I don't think that's in the control
file either, but I have seen it before so it's in my control.ctl.local
file.

Oh, good point, I forgot that we include a default action of mail for
newgroup and rmgroup. Adding a valid email address filter there makes a
lot of sense to me, given that those messages are essentially useless
unless a news administrator can contact the sender and ask what the
intention was.