In this report, we describe a series of incidents which took place during the summer of 1995, we provide a brief summary of elementary concepts in computer security, and we propose some directions the University can take to decrease the risks to its computer installations and the data they contain.
It is our hope that after reading this report, members of the University community will understand why computer security issues must be addressed, and will offer their enthusiastic cooperation for the implementation of the measures proposed in this document.
We - which usually means Paul Gill and I - looked into the ``jsmith'' account and immediately saw traces of unusual activities. A chat with the real jsmith confirmed what everyone suspected: he wasn't the villain; the real bad guy was just using his account, which was on the cracked list from McGill.
Fortunately, the intruder had left enough ``junk'' lying around in jsmith's account to give us some idea of his habits, and some clues about things to look for in other places. When we looked at the other accounts on the cracked list, we found that several had been used by the intruder, while most had not. What was most interesting, though, was that there were two patterns of activity: sometimes the intruder came in by phone; sometimes he sat at a terminal in the Hall Building. When he came by phone, he liked to hop over to a university in the Southern U.S. - let's call it ``Sussex''. When he sat at a terminal, on the other hand, he liked to visit McGill. In both cases, files were left scattered about containing credit card numbers, calling card numbers, 1-800 numbers (presumably for compromised PBX3 equipment), and pirated software.
It looked as though we had not one, but at least two intruders, and they were clearly up to no good.
We disabled most of the accounts identified by McGill as having been cracked; it would certainly not have been appropriate to let criminals run free with them, especially when the compromised accounts included those of three senior professors, as well as two accounts used for such administrative purposes as grading courses. These accounts might have had very sensitive contents. There were others, though, that were old unused accounts without special privilege. These, we decided, could be left open, but booby-trapped, so that if the intruders used them, we could see everything they were up to.
It didn't take long. As soon as ``their'' current accounts were closed, the bad guys switched to new ones. Good news: now we could follow them more closely. Bad news: only one of them chose from our collection of booby-trapped accounts. The other came in promptly as the secretary of the Graduate Studies Programme, who had not been on the cracked list, so the list was apparently incomplete. We quickly blocked the secretary's account, and within minutes, our culprit was using one of the booby-trapped accounts.
Now we watched, and waited for a chance to identify the intruders.
On one occasion, Paul saw that one of the bad guys was using a terminal in our lab in the Hall Building. Paul hurried over to the lab and confronted a rather large young man at the terminal, asking to see his identification. The man refused and hurried away.
On another occasion, seeing that the other cracker was coming in by phone and staying for an unusually long time, we tried to use internal resources to find where he was coming from. Mike Marak of the Computing Services4 Network Engineering Group contacted Wendy French of Telesis, who was able by obscure means to find that the number at the other end of the connection was in an Ontario Government office building in Toronto! We guessed that this wasn't the real origin of the call.
In August, the RCMP were brought into the case. Cpl. Bob Beaulieu of the Computer Crime Section took responsibility for the case. He asked us to send the RCMP a formal letter of complaint from a person with suitable authority, and to designate technical people to help in the investigation and prosecution of the case. Without at least this level of commitment from the University, the police couldn't proceed.
Orally, we were quick to agree. A letter would be sent, and Paul Gill and I were assigned to provide technical help. It took longer than we had hoped for a formal complaint to go out, but we managed to convince Cpl. Bob to get started anyway, because we had evidence that the intruders were involved in crimes affecting victims other than Concordia: the credit card and calling card numbers.
Cpl. Bob wanted us to be very quiet about the investigation, not informing anyone who didn't already know about it. Word spreads fast in the electronic world, and without secrecy, the bad guys might quickly come to know they were under investigation. Moreover, we were not sure where they had friends. Our traces already suggested that they had free run of Sussex's Computer Science labs, and it was possible that there might be a ``rogue'' system manager there helping them out. Even McGill wasn't known to be clean. They had had several recent intrusions, and it wasn't clear which systems were being reliably managed and secure, and which were under ``enemy'' control.
So we began our investigation in secrecy. We had two objectives: to identify the intruders, and to gather incriminating evidence. We were already pretty sure that there were at least two intruders; one on the phone, who used the IRC5 nickname ``Shrunk'', and one in person, who used several nicknames, including ``Blackrat''. We didn't yet have any indication of their real names. Cpl. Bob asked us to call his pager whenever either of them came in.
Blackrat was easy to identify: he would come into the Hall Building and spend hours there, either in H966 or in H511. We called the Mounties twice, and each time they came to make a visual identification of the suspect as he typed at a terminal. They didn't arrest him, though, because they didn't want to scare off Shrunk; and without arresting him, they couldn't attach a real name to him, so he remained ``Blackrat''.
Before we got a chance to identify Shrunk, we had some problems. First, Blackrat took a look at his ``.login'' configuration file and saw a line he didn't quite understand, so he deleted it. That was the booby-trap! Now Blackrat was using our machines and we didn't know what he was up to. Paul and I discussed our options and decided that it was too risky to put the line back: if he noticed it again he'd know we were onto him. Instead, we wrote a quick and dirty program that would capture his session from the network whenever he logged into any of his favourite sites. He could still operate unobserved on our local machines, but he usually did his dirty work at McGill anyway, so we wouldn't miss much.
As soon as that crisis was resolved, another arose. Blackrat and Shrunk met on IRC, and Blackrat gave Shrunk an account at McGill, which was ``better'' than ours because it wasn't restricted by our very tight undergraduate disk quota. Shrunk immediately disappeared from our network. We could only assume that he was now using ``his'' McGill account, which we couldn't trace at all.
At this point I decided that it was time to stop overplaying the secrecy game. I tried to call Steve Robbins, only to discover that he was no longer working at McGill. Instead, I called Anne Bennett, an analyst in Computing Services, and asked her to put me in touch with a reliable person at McGill. She gave me the number of Mike Parker, whom the Unix community knows as ``der Mouse''.
Der Mouse was very cooperative. We briefed him on our situation and told him that we'd like him to shut our villains out of McGill so that they'd come back to Concordia where we could keep track of them. We gave him all the information we had about compromised accounts at McGill. He promptly disabled the ones in his domain (CIM, the Centre for Intelligent Machines), and referred us to Jacek Slaboszewicz and Tom Levasseur in Electrical Engineering for the others. The EE guys made my day: they gave us a name for Blackrat. He was a former McGill student who had been expelled for academic reasons, but who had been known to engage in cracking activity before.
With the McGill door closed, the intruders came back ``home'' to Concordia, where we continued to amass a large volume of evidence against them. In fact, the most frustrating aspect of the project was sifting through the enormous logs to find snippets of significant, damning material. Among other things, we found that:
The Essex case was particularly disturbing. We watched an IRC session in which Shrunk was told how to break into the AIX6 operating system and obtain root7 privilege. Then we watched Shrunk do just that. And we saw Blackrat do the same. Apparently, every cracker on the Internet could get into Essex's machines as root. (We later learned that Essex had just moved their MIS system to AIX machines, so these would have been vulnerable too!)
This was too dangerous to let go. We contacted a reliable person at Essex, and soon we saw all of their machines go down for a day. When they came back up, they were no longer vulnerable. It was great fun watching the villains arguing amongst themselves on IRC as to who had ruined their cosy setup at Essex.
We eventually got lucky with Shrunk's identity. We intercepted a ``private'' IRC conversation between Shrunk and another bad guy in which Shrunk invited his friend to call him at home, and provided a phone number. This was quite a break. I called Frank Maselli of Computing Services, who was able to look up the number in a Canada-wide telephone number database and associate it with a name and address in a Toronto suburb. (Shrunk is a young offender, so we can't identify him in any detail here.)
A nice pointer, but not the hard evidence that Cpl. Bob would need. For that, we needed a live phone trace. Cpl. Bob asked us to call his pager as soon as Shrunk logged in, day or night. We tried that several times before Shrunk obliged by staying on long enough for a trace. In the movies, it takes two minutes. In this case, we needed more than an hour, because Shrunk played a mean game of phone tag.
Eventually, though, he had to stay on for a long time to download a big batch of pirated software. When I saw the connection, I called Cpl. Bob's pager. He called back within two minutes, and I told him we had a live connection. I confirmed that it was indeed Shrunk, and identified the telephone number to which he had connected. Cpl. Bob made a conference call to Bell Canada's Montreal security office to ask for a trace. After a few minutes, Bell's technical people came back with the origin of the call: it was coming from a number belonging to Unitel on rue Notre Dame Ouest. That was all they could give us. We needed to contact Unitel's security people for more details.
Unitel was more helpful. They identified the Notre Dame number as the Montreal end of a trunk line between Montreal and Toronto, and with some difficulty they picked up the trace and followed it back to the Toronto end. It was coming from a ``secure'' PBX belonging to the Government of Ontario, located in the Provincial Parliament Buildings in Queen's Park.
Cpl. Bob and I guessed that this was not the work of Premier Mike Harris! So we went to yet another conference call, this time to the technical person in charge of the PBX. He had to do some digging, but he finally came up with a number. To nobody's surprise, it was the same Toronto number that we'd seen before. That was what Cpl. Bob needed.
Over the next few days, the Mounties in Toronto monitored all calls originating from Shrunk's house. There were hundreds. Almost all went through the Ontario Government PBX, and most were to Concordia's Computer Science Department.
Cpl. Bob obtained search warrants for the homes of both Shrunk and Blackrat. On Thursday, September 7, he called me to say the warrants would be executed the following Tuesday. Unfortunately, as he was speaking, I was looking at a trace of a Blackrat session from the previous evening, in which he launched attacks against the X servers8 of Tom Levasseur at McGill and of a system administrator in the VLSI Lab at Concordia. He was capturing every keystroke they made, using an ancient security hole in the X server. It was only a matter of time until one of them would type the root password, then Blackrat would have root privileges!
I told Cpl. Bob that we couldn't reasonably wait more than another day before shutting the villains down. They had become too dangerous. He agreed, and moved the date up to Friday: the next day. He would go to Toronto to arrest Shrunk, and his colleagues would arrest Blackrat in Montreal. In the meantime, I called Tom and the local sysadmin9 to let them know that their keystrokes were being snooped, so they shouldn't type anything sensitive at their terminals until further notice.
On Friday at noon, Cpl. Bob was at the Bowmanville RCMP Detachment - the nearest one to Shrunk's house. Two of his colleagues were in the Hall Building, and a third was with me at my desk, watching for the culprits to come in. True to form, Blackrat arrived at about 12:30, sat at a terminal in H511, and began his daily routine of bad behaviour. His login tripped an alarm on my terminal, and the two Hall Building Mounties took up positions outside H511. They didn't arrest him immediately, though, because we hoped to catch the two crackers communicating together.
At about 1:00pm, Shrunk came in over the phone lines, again setting off an alarm on my terminal. I called Cpl. Bob's cellular phone, and he immediately started towards Shrunk's house. Meanwhile, I kept an eye on the trace of Shrunk's session. He had started an FTP10 session to an illegal software depot and was browsing in search of something interesting. This was a good sign. He'd be on for a while.
Unfortunately, before Cpl. Bob arrived, and before we noticed that Shrunk was logged in, Blackrat logged out and left H511. The Mounties arrested him in the hallway. One down.
Cpl. Bob was disappointed that we couldn't get the bad guys talking to each other, but those are the breaks. With the cell phone connection still open, he and his backup officer arrived at Shrunk's house at about 1:15pm, search warrant in hand, and rang the bell. The door was answered by Shrunk's mother, who called her son from upstairs. The keystrokes in Shrunk's FTP session stopped.
Cpl. Bob: Are you (Shrunk's name)?
Shrunk: Yes.
Cpl. Bob: Where is your computer?
Shrunk: Upstairs.
They go upstairs.
Cpl. Bob: DON'T TOUCH THAT KEYBOARD!
The keystrokes now resumed:
ftp> This is Bob Beaulieu
?Invalid command
ftp>
With that, our loop was closed,11and we were done. Just in time to start the Fall term.
Did I say we were done? I'm sorry, that's not quite right. We were done with the nail-biting aspect of the affair, but the hair-tearing had not yet begun.
Blackrat and Shrunk were taken to RCMP detachments in Montreal and Toronto, respectively, where they were photographed and fingerprinted. They were not charged immediately; the Crown would have to determine what charges should be laid in view of the evidence.
Shrunk, who was only 15 years old, would be tried under the Young Offenders Act. He didn't put up much of a fuss, though. His parents had been completely unaware of his activities, and were not at all pleased to find out about them in a police raid. Shrunk's father took charge of his son's case immediately, and made it very clear that the unfortunate boy would plead guilty to any charge and accept his sentence, whatever it might be. No lawyer would be needed, and no defence would be mounted. Shrunk Senior wanted to have the whole business behind him by Christmas - not an unreasonable hope, I thought, given his enthusiastically cooperative attitude.
Sadly, the justice system has a different way of approaching the matter. As this was the first case of its kind in Quebec, the Crown prosecutors were not quite sure what to do with it. The evidence was quite technical, so it was a struggle for them to understand what had actually happened, why it was wrong, and what law had been violated. The law is clear, though, and given enough time, even a lawyer can understand it. After a few months, it was decided that both Shrunk and Blackrat would be charged under Section 342.1 of the Criminal Code: ``unauthorized use of a computer.''
In April of 1996 - on time for Easter - Shrunk pleaded guilty to the one count with which he was charged. He was sentenced to six months' probation, community service, and confiscation of his computer for six months, but I suspect that his father's displeasure was a stiffer penalty than anything the courts could have imposed.
Blackrat was a tougher customer. He was quite uncooperative. In his view, if he could do something with a computer system, it must therefore be legal. So he refused to plead guilty to anything. As a result, the full fury of the law would be unleashed against him.
On the day of Blackrat's court appearance, Tom Levasseur and I arrived at the Palais de Justice at 9:00am and made our way to Room 406, where his case was to be heard. One by one, various petty criminal cases were processed. In each case, a lawyer would would appear with a calendar and ask that his client's case be deferred until such-and-such a date. The Crown would object that she would be out of town on that day, or perhaps the judge would be on vacation. After a minute or two of calendar shuffling a new date would be found, and the next case would come up. And so the session proceeded for two hours until Blackrat appeared. The judge asked if he was represented by counsel, at which his lawyer appeared and asked for a new date.
Over the next year, we would all learn just how inexorable the law can be. Anne Bennett, Tom Levasseur and I must have appeared ten times before any resolution was made. In fact, we appeared more often than the defendant himself, who seemed far less interested that we did. Each time, there was some reason to delay, but each time, it was necessary for the parties to make a full formal appearance before a judge to reschedule the hearing. The Palais de Justice apparently has no computer system for scheduling appointments. As a result, criminal lawyers seem to spend most of their time rustling the leaves of their calendars in full formal dress.
By the time Blackrat's case actually came to trial, he had finally been convinced to plead guilty to the three counts against him, but his lawyer and the Crown had been unable to agree upon an appropriate sentence. The Crown, at Corporal Bob's urging, wanted jail time - at least a suspended sentence. The defence wanted probation and community service, which would leave his client with no permanent criminal record.
The judge sided with the defence: Blackrat was sentenced to two years' probation and 180 hours' community service. Now, on April 23, 1997, we were done.
The linchpins of computer security are:
Computing resources should be available as expected, which they are not if the computer has been damaged by crackers. The resources are also unavailable if the computer is not responding properly because it is performing unauthorized tasks, which happens in denial-of-service attacks, or when unauthorized people are using the system (often due to account sharing). In an environment where students, researchers, and employees depend on the availability of the computer to do their work, its unavailability can be very costly.
Sensitive information must be kept confidential, and indeed in some cases the University has a legal responsibility to ensure that it remains so.
The integrity of data must also be protected against accidental or malicious alteration or destruction. This applies not only to sensitive or confidential University data, but to valuable user data, and to the system itself in the form of its programs, operating system, and configuration files.
As electronic communications take over an increasing proportion of business and professional communication, non-repudiation is increasingly important. This means that the sender of a message should not be able to legitimately deny that she has sent it, the receiver should not be able to legitimately deny that he has received it, and neither party should be able to alter the contents of a message so validated.
Access control is a means whereby we allow only authorized entities to use our resources, and authentication is how we determine that an entity is what it claims to be.
One of the most dangerous incidents in Michael's ``summer vacation'' was, from the affected site's point of view, the obtention of root privileges by the crackers on the machines at ``Essex''. Using the well-known12 AIX bug ``login -froot'' to get in, the crackers then had full run of the machines. They could have damaged or subverted the MIS systems, and gained access to confidential data. This could have been disastrous for that university's reputation, if publicized. Also, if the crackers had altered the data in subtle ways and had not been detected, the human cost could have been considerable. Leakage of confidential data could have led to lawsuits against that university. As it was, it lost a full day of use of those machines while they were secured from the attack, and the cost in wasted staff time of everyone there who could not work effectively must be tallied as well. The bug that was exploited in this case was in a privileged program ( login ) which came as part of the operating system of the machines. IBM has made patches (corrected versions of the faulty programs) available, but many sites have not yet applied them.
Shrunk and Blackrat obtained most of the accounts they used by cracking password files.
While using shadow password files13 can help prevent password cracking attacks, reports of password sniffing attacks (whereby the attacker ``listens on the wire'' as the passwords are transmitted unencoded across the network) are becoming very common. Until end-to-end data encryption is widely available, the only solution to these problems is to avoid using reusable passwords, at least on network segments vulnerable to sniffing. This entails carrying lists of one-time (``disposable'') passwords, or installing software which can negotiate authentication via a challenge-response mechanism (but this depends on having the software at both ends of the connection, which is not always possible), or using ``SmartCard'' technologies, which currently cost on the order of $50 per user. Because they are somewhat inconvenient, none of these solutions is likely to gain user acceptance quickly.
Shrunk was coming into our systems through the dial-in terminal server Macduf, which requires no authentication to use: anyone can call Macduf and get a connection. It was quite difficult for Michael and Cpl. Bob to find out where he was really coming from; it required not only police involvement, but it required Shrunk to stay on the line long enough for the call to be traced, and it required Michael to monitor his connections day and night.
Fortunately, Computer Science does log the provenance of incoming logins to their system. Thus, they were able to find out the origin of some of the connections to the compromised accounts on their system, and warn the other sites that there were crackers in their midst.
Of course, in the ``summer vacation'' example, Shrunk and Blackrat mostly shared accounts which they had obtained illegally, but in the case of at least one Alcor account, we believe Blackrat had been given the password by the account owner.
In an unrelated incident that took place in early 1995, a legitimate Alcor user gave her password to a companion, with whom she later broke up. She had quite forgotten about the Alcor account, though, and meanwhile the ex-companion had shared the account liberally. When we realized that unusual things were going on, the account was being used by at least five people:
- Someone who was using Alcor as a way to get around the firewall 14 of a Montreal company, which was allowing connections from Concordia, but not from elsewhere.
- Two people who did lots of newsreading, and subscribed to multiple mailing lists, apparently mostly harmless, but who meanwhile were depriving our legitimate users of hours of modem connect time. One of these people seemed to be coming from the account of a senior administrator at another Canadian university! (This later turned out to be another shared account.)
- One shady character who was very interested in Warez (pirated software), and was using our site to find and download it.
- Another shady character who discussed, in intercepted e-mail, possible break-ins at a large company in Toronto, whose reputation would have suffered had these break-ins succeeded and become known.
It took over a week of staff time to find out what was going on, warn neighbouring sites which might have been compromised, and shut the activities down.
In fact, this has been a frequent complaint of our student users, as evidenced by the traffic in the netnews group concordia.dept.comp-services.help. It seems that many people use the terminals and dial-in lines available to them for distinctly non-academic purposes, and do so to such a great extent that it becomes difficult for students with legitimate needs to get their fair share of resources.
``Elduet'', a sysadmin at one of the U.S. sites that were involved in the ``summer vacation'' incidents, turns out to have been crooked. As far as we know he was involved mainly in software piracy and in giving accounts to his co-conspirators. It is possible that he also had access to, for example, credit card numbers from his employer's client database.
Shrunk and Blackrat were able to operate with near impunity on so many computer accounts because the legitimate account owners were absent or no longer used the accounts.
One security problem that is often exploited as a prank in the Computer Science labs arises when a user types ``xhost +'' to allow the X window system to display a remote application on the local screen. Unfortunately, this leaves the screen wide open to manipulation by an attacker. Pranksters will sometimes pop up an unwanted window on the screen, or worse, they may destroy a useful window. A really hostile attacker might do what Blackrat tried to do: intercept the legitimate user's keystrokes and capture his password.
When Michael first received the list of cracked passwords from McGill, he could have simply asked the local users to change their passwords, and left matters at that. It was not at all obvious at that point that anything worse was going on than students trying out a new (to them) password cracking program, perhaps just for a lark, or to play pranks on their friends. No one could have foreseen that his extensive investigation would reveal compromised PBXs, credit card fraud, and child pornography.
For these reasons, and also because many staff members were on vacation, it took longer than necessary for the University to complete its formal complaint to the RCMP in the case of Michael's ``summer vacation''. We were fortunate that, in this case, the delay seems not to have interefered with the investigation.
A few years ago, McGill University put in place a new computer usage policy, after much uproar and consultation with its users. The result is in most ways admirable. However, the policy puts very strong constraints on what system administrators may do, because of the need to protect user privacy. As a consequence, it was in some cases difficult for the McGill system administrators to cooperate in the ``summer vacation'' investigation as fully as they would have wished.
This is what Shrunk and Blackrat did, and they were rather successful at it. Had they been really lucky, or had a system administrator used an ``easy'' password, they might even have obtained root privileges that way.
One of the ``summer vacation'' crackers took a stab at a U.S. Navy site. Had he succeeded in penetrating that site, what would Concordia's liability have been with respect to the consequent damages?
Our crackers sometimes deleted files in a user account just to make room for the files they wanted to store. What if the damage had not been noticed immediately, and what if the system administrators in Computer Science had not been diligent about doing filesystem backups? Valuable data could have been irrecoverably lost. What if that data had been someone's Ph.D. thesis?
If our crackers had been malicious instead of expedient in their file modifications, they could have subtly altered data files instead of just deleting them. What if a researcher had published findings based on the corrupted data?
Also, what if the crackers had sent out e-mail from the cracked account in a way that made it seem to have been written by the bona fide account holder? What kind of damage to a professor's reputation, for example, could such forgery have caused?
In fact, we had an incident in 1996 whereby someone posted a commercial advertisement to netnews from a cracked Alcor account, claiming to be from the real account holder. The account holder received countless ``flames''16 and complaints from all over the net, and was very embarassed. Common similar incidents which occur regularly in netnews involve posting materials purporting to be statements of sexual desires or requests for sexual favours from the account holder, who then returns to find a mailbox full of upsetting, embarrassing, and possibly quite offensive material.
A researcher working in conjunction with industry, for example, might have proprietary information on her computer account, whose disclosure might significantly affect the competitiveness of the company in question.
Our ``summer vacation'' crackers gained access to the account of a senior secretary, who in turn had access to a student information database. Information could have been altered or disclosed.
Our crackers were in fact involved in software piracy and child pornography, which is why the RCMP became interested in the case. Their activities abroad caused the U.S. Secret Service to contact Concordia about the case.
In a possibly related incident at about the same time, it was discovered that a department here at the University had given root privileges to some students, who had then become involved in Warez distribution. While the activities were shut down immediately, what might the University's liability have been had Microsoft decided to sue?
Unfortunately, the top ten problems have remained fairly constant from year to year, and reflect Concordia's situation fairly accurately in most respects; this is what we hope to change.
Some of the above problems have technical solutions; for example, programs can be installed to do better connection logging, patches for known system programming errors can be installed, and system configurations can be checked for flaws. Some of the problems will require better policies and procedures to be put into place, and many are amenable to better user education. In some cases, new services could be put into place, such as a centralized patch and security tool repository, and a central computer emergency response team for the University. Some of the problems will always remain difficult to prevent, such as user dishonesty and account sharing, but it may be possible to monitor systems in such a way as to make the detection of such abuses more likely.
Where do we start? Any decisions that are made to allocate resources to computer security (or to not allocate them!) must be made in view of the risks involved, the cost of the associated problems, and the cost of preventing the problems. For this, it is necessary to have an overall view of the importance of various parts of the Concordia Computing Facilities to the mission of the University.
On the other hand, a formal risk analysis is a large endeavour, and might be beyond the resources available to the University. IITS management will decide how to proceed in this respect, but the authors are of the opinion that it would make sense to proceed with such an exercise only if a large project were envisioned, for example converting the entire University to a centralized authentication scheme such as Kerberos, or buying expensive equipment to ensure physical security of our facilities (cooled machine rooms with uninterruptible power, video cameras in all the labs). However, we find that many effective security-enhancing projects are relatively small, and that it is less costly to implement them than to study them. In view of the decentralized nature of computing at Concordia, we have not deemed it productive at this time to consider projects which would require centralizing control of Concordia's computing facilities. We do, however, make proposals whose purpose is to improve the coordination of information flow and the sharing of expertise; with time, an improved awareness of computer security, and an improved climate of communication between computer specialists and University management about computer security issues, might well lead to projects involving some centralization of control (for example for authentication purposes). But it is, in our opinion, too soon for that now.
It is important to understand that computer security cannot be a one-off project - rather, it must be a thread that runs through everything we do. A gradual change in mentalities and procedures is what is required. It is in that spirit that we offer the following section.
However, despite those activities, much work remains to be done: not only are there problems still unresolved on equipment managed by IITS and ``large'' departments (defined for our purposes here as those having large computer installations), but we have not been successful at transferring our expertise to ``smaller'' departments, whose systems are now bearing the brunt of cracker attacks. This latter lacuna is addressed in sections 6.3 (on centralized services) and 7.1 (on O/S specific issues) below, while the first is the subject of the entire section 7 (on system and network issues).
In addition, the increasing use of the Web by the University to serve sensitive data requires careful management. This is a sufficiently important issue that we've devoted an entire section (9 on world wide web issues) to it.
Finally, we have done a poor job of educating the user community with respect to computer security issues. In view of the fact that some authors have labelled ``people problems'' (in the form of social engineering, dissatisfied insiders, and simple ignorance) as the single greatest cause of security breaches, and in view of the fact that most of the staff time the authors have spent on incident response in the past few years has been due to causes that originated with a compromised or misused user account, we attach great importance to the solution of this problem, and again have devoted a section (8 on user education) to it.
We recommend that IITS create and manage a Concordia Computer Emergency Response Team (CERT), with structure and mode of operation to be determined by IITS. Despite its name, this CERT would not only coordinate the response to computer security related incidents, but, more importantly, would facilitate the transfer of expertise required to prevent those incidents from occurring in the first place. This group would also advise on procedure and policy matters related to computer security. This is the subject of section 6.
In the sections below, we identify the specific issues we believe need to be addressed, and propose solutions. Which work would be done by members of the new CERT, which would be done by other IITS staff members or other computer analysts in the course of their regular duties, and which would require the formation of temporary working groups, or be assigned to other existing bodies, must be decided by IITS, though we make some suggestions in that direction.
In keeping with the tradition of existing similar teams throughout the world, we propose to name this permanent group the Concordia University Computer Emergency Response Team (Con-CERT). Con-CERT will apply for membership in FIRST, the Forum of Incident Response and Security Teams.
IITS participation in the upcoming FIRST18 conference and workshop, to take place in June 1998, is planned. In addition, several IITS staff members have attended network and computer security conferences, and/or participated in security-related working groups, for example within the IETF19.
Con-CERT will operate under the auspices of IITS, which will determine the details of its operational procedures, provide management and computing resources, and appoint a coordinator and additional staff as required. IITS will actively solicit the participation of other University departments.
Con-CERT will be composed primarily of technical experts experienced in the various systems in use at the University. The ``core'' membership will consist of technical people representing each major operating system in use at the University, computer networking experts, and a representative from IITS management.
The Con-CERT will also have an ``extended'' membership, which will act as a consultant to the core group, and where individual ``extended'' members will participate more directly when circumstances require their particular expertise. We expect a pool of consultants similar to the following:
The goals20 for forming an incident response team are:
A new Policy on Computing Facilities has been issued since the events of 1995, which greatly clarifies the rights and responsibilities of users and sysadmins; this has facilitated decision-making during incidents.
As of this writing, the existing Policy on Computing Facilities document is under review by John Woodrow (Director of IITS) and Bram Freedman (University Legal Counsel). Security procedures based on that policy, and other existing University policies, will be proposed by Con-CERT. One of the methods that will be used is to analyze past incidents in the light of existing policies.
Anne Bennett has already provided such an analysis, and her proposed policy amendments were discussed at the CCSA meeting of June 11, 1998.
If a situation is encountered which no existing policy can satisfactorily address, then policy changes will be suggested. Also, recommendations may be made concerning additional means of publication of the policies and procedures, for example by making them part of the Calendar, or by including them in any user education materials.
Documents whose production will be considered include:
IITS InfoNote M-002 evaluates Mac security products which address this issue.
On IITS Unix systems, privileges have been removed from all programs for which they are not strictly necessary.
Remote access controls (via tcp_wrappers) were already in use on IITS and Computer Science Unix systems; their configuration has been checked, and their use has been extended to a greater number of services (sendmail, ssh).
Stringent password selection guidelines are now enforced on all IITS Unix systems and on all Computer Science systems; this effectively negates the threat of password guessing.
End-to-end encryption software has been installed on all IITS and Computer Science Unix systems, whose use (where it is possible!) obviates the threat of data and password sniffing. Encryption is now routinely used for most system administration purposes, though its use has not yet spread to the user community. In particular, session encryption client software is not available at most sites outside Concordia.
Password shadowing has been implemented on Alcor; this effectively removes the threat of password cracking. (Password shadowing has not been implemented in Computer Science because it is not easily feasible in a heterogeneous network using NIS, but it is on the list of problems to be solved.)
The risk represented by a user's losing their Alcor ``yellow card'' (assignment of account and password) has been minimized by forcing the user to select a new password at the time of first login. The new password not only meets the anti-crack, anti-guessing guidelines, but also cannot be identical to the one printed on the yellow card. Computer Science expects to implement a similar scheme soon.
The threat represented by accounts whose owners are no longer associated with Concordia was already small, but the integration of the SIS (student information system) with the IITS computer account management system has impproved the timeliness of account removal in such cases. In Computer Science, where programmatic access to the SIS is not possible, these accounts are removed in a batch job at the end of each term.
The threat represented by idle accounts on Alcor has been diminished by the implementation of an aggressive policy of blocking inactive accounts.
Changes were made to the account management Unix client software to permit ``terminal server accounts'' to be managed; this means that we will soon be in a position to enforce authentication of all dial-in connections in IITS.
A small pilot study was done to show the feasibility of a central Unix log server; the study was successful. Computer Science is in the process of moving to a central log server this year.
User login logging has been greatly improved on Alcor; since November 1997, all user sessions, including not only regulat logins but also POP, IMAP, and X, are logged.
As mentioned previously, remote access controls (via tcp_wrappers) are used on IITS and Computer Science Unix systems.
Several tools have been developed to assist sysadmins in investigating suspicious activity; tools to audit system integrity were already in use on IITS Unix machines before the 1995 incident.
Identification to remote locations of the local (user) provenance of connections (via pidentd) was already in use on IITS and Computer Science Unix systems; its use has been extended to a greater number of hosts in various departments.
Many operating system patches have been applied, as they have become available from our vendors. This is a continuing effort.
In the case of a few very well-known Unix exploits, not only has the vulnerability been closed, but booby traps have been installed on the IITS general-purpose Unix service, Alcor, to detect attempts to gain system privileges using those vulnerabilities - several attempts have already been detected in this way.
A small pilot study was done to show the feasibility of using the DNS to store per-department and per-host contact information, using the ``RP'' (responsible person) record type.
The threat of e-mail ``relay hijacking'', whereby badguys at foreign sites use Concordia machines to send spam22 to thousands of users here and offsite, was removed in December 1997.
Programs were installed in April 1998 to monitor DNS data consistency, and report problems.
The IITS Department is very fortunate in having a locally written computer account management facility, called ``AGEM'', which interfaces with the SIS (student information system) and human resources databases to determine who is entitled to have a computer account, and which is able to track personnel and student departures from the University and issue account blocking directives in consequence.
As mentioned previously, the threat represented by accounts whose owners are no longer associated with Concordia was already small, but the integration of the SIS (student information system) with the IITS computer account management system has improved the timeliness of account removal in such cases. In Computer Science, where programmatic access to the SIS is not possible, these accounts are removed in a batch job at the end of each term.
Since all user accounts (on those systems managed by AGEM) are created by a program, the accounts are created in a consistent manner not subject to human error. Of course, having such a system at all makes it possible for IITS to manage the thousands of computer accounts which are created and deleted every year. In addition, the presence of AGEM makes it possible to monitor and block idle accounts in an automated way.
As mentioned previously, the threat represented by idle accounts on Alcor has been diminished by the implementation of an aggressive policy of blocking inactive accounts.
One criticism of AGEM is that directives are neither encrypted nor digitally signed, which provides a point of attack to network sniffers. Fortunately, encryption and digital signatures can be added quite easily with PGP, since directives are transmitted via ordinary e-mail.
The use of AGEM contributes to computer security in IITS, and the maintenance and continued use of this program is essential. Unfortunately, support of AGEM has not always received a high priority within the IITS Department.
As mentioned previously, changes were made to the account management Unix client software to permit ``terminal server accounts'' to be managed; this means that we will soon be in a position to enforce authentication of all dial-in connections. However, support for this at the level of the AGEM server has not been forthcoming.
In view of the enormous usefulness of AGEM, we must recommend that its use be expanded to other systems (in particular to the dial-ins) and other departments where possible, and that IITS recognize its importance and prioritize its support accordingly. Also, we recommend the use of PGP to encrypt and sign directives issued by the server.
We recommend setting up a group to review existing user education procedures and to recommend changes (if needed) to improve the level of computer security awareness in the Concordia user population.
The group would obtain from the O/S, network, and WWW specialists, prioritized lists of information which, it is felt, users should know, concerning computer security. The group would determine which constituencies within the University community must be addressed, and, after consulting with any existing user education groups, would recommend methods to best reach these people. If possible, methods would be recommended which cannot be repudiated, for example tests or signatures.
The group would begin by making an inventory of existing resources, and of met and unmet needs. If it were deemed likely to be helpful, the user community might be polled directly. The group would, of course, recommend the best use of resources which already exist within the University where possible, and, with the help of the O/S specialists, would help find and adapt resources in use elsewhere. Where necessary, the development of new resources (documentation, courses, computer programs) would be recommended. All methods would be considered: courses and seminars, short presentations (for example at Orientation), paper documentation (for example InfoNotes and User Guides), online documentation (such as Web pages), computer programs for training users, tests (both paper and online), videos, games, demonstrations...
Examples of topics which would be expected to be covered are:
Part of the Alcor web site has been devoted to user account security issues, and includes a checklist of ways to detect whether one's account is being used by someone else.
Also, a new program, filecheck, has been implemented on Alcor, which assists users in looking for suspicious files on their accounts.
``Secure'' (encrypting) web server software has been installed on several machines in IITS.
Web/CGI vulnerabilities have been greatly reduced on Alcor, while still permitting our users to install and run their own CGI scripts. This has been done by both running the web server in a restricted filesystem (to avoid attacks on the system and on the users' non-web data), and by ensuring that user's CGI scripts run with their privileges, not the web server's. These measures have been in place since September 1996.
However, no security measures can be effective unless they are supported by the user community. Therefore, the authors' fondest hope in issuing this report is that we will have succeeded in sensitizing the University community at large to the importance of good computer security practices in the everyday work of each and every one of us. Happy - and safe! - computing to all, and to all a good night.
As people concerned on a daily basis with computer security, however, we must reject this view as based on a fundamental misconception of the rôle of computers - especially internetworked computers such as the ones involved in our story - in the modern workplace. Authorized computer users depend on the integrity of the programs and data they use, and on the security of the channels of communication with other authorized users, whether on the same machine or on another continent. In our particular case, professors must be confident that the exams they compose are not readable (or writable!) by arbitrary intruders, that their research work is accessible only to legitimate members of their research group, and that their mail is delivered quickly and reliably to its intended recipients. Students must be confident that they will have access to computing equipment when they need it, that their work will not be corrupted, erased, or stolen, and that they will be able to use the Internet as both a source of information and a forum in which to exchange ideas. The university's administration depends on its computer systems for the full range of its activities, e.g., student record keeping, payroll, and MIS, and the law stipulates that much of this information must be kept confidential.
Our summer vacation intruders did not significantly damage any of these functions directly, nor was that the immediate purpose of their intrusion. In fact, they were not especially interested in Concordia at all. It was quite evident from the pattern of their activities that they were members of an informal community of ``hackers'' (crackers) whose objective was to gain access to as many machines as possible, with as much privilege as possible, always with the complete anonymity that comes with a ``hijacked'' account. From these accounts they could launch attacks against other sites on the Internet without fear of adverse consequences; and this is exactly what they did. From Concordia, they attacked McGill, ``Essex'' and a number of other sites. From their stolen accounts, they joined several IRC channels on which they participated actively in various forms of illegal commerce. In fact, Shrunk got access to Concordia accounts from Blackrat in exchange for access to Shrunk's pirated software repository at ``Sussex.''
The real harm done to Concordia was not the misappropriation of some otherwise idle CPU time during the least busy period of the year. It was the conversion of Concordia from a responsible, law-abiding site on the Internet into a source of anonymous and untraceable attacks against any number of other connected sites, and into an unwilling accessory to miscellaneous breaches of copyright, fraud, and obscenity laws. While the intruders were present (and undetected), Concordia was a danger to its neighbors. We had lost effective control of our site, and while that was going on, our reputation as a good network citizen was in danger. From this perspective, it makes little sense to assimilate the intrusion to a common theft. If we must use the model of theft, it would be more appropriate to consider it on a par with theft of firearms; the monetary value of the object stolen is trivial in comparison to the danger posed by the thief in possession of the loot. Perhaps it would be better to assimilate this sort of crime to trespassing. But these trespassers have not just occupied our property; they've set it up as a common bawdy house in full view of our neighbours and the police. Again, not much damage to the property has been done, but that's obviously not the real issue.
Apart from the approximately $400 of direct costs related to connection time and resource use, the most obvious cost is the investment of employee time: roughly two person-months of ``Grade 11'' analyst time ($6000) at Concordia, not to mention similar costs at McGill and the RCMP. From a legal perspective, this might be seen as a routine ``cost of doing business,'' but this presumes that we have computer security staff sitting on their hands waiting for security breaches. In the real world, we don't have the staff, so the expense is directly related to this particular breach. ``Essex'' lost the use of its computers for a day, which implies significant cost in wasted staff time.
Others at Concordia were involved as well: Telesis, Legal Counsel, the Security Department, as well as several analysts and managers in Computer Science and IITS (then Computing Services). It would not be unreasonable to suppose that their combined efforts might have cost $5000.
We are not even counting the cost of regular system security procedures, whose implementation is made necessary by precisely the Shrunk and Blackrat type of criminal.
We won't pretend that the figures are exact, but it is at least reasonable to estimate that Concordia spent more than $10,000 on this incident alone.
1 There was no footnote 1; congratulations on paying attention.
2 Thanks to Steven Winikoff, Mike Marak, Frank Maselli, and Wendy French of IITS, Paul Gill, Larry Thiel, and Chairperson Clement Lam of Computer Science, Steve Robbins, der Mouse, Tom Levasseur, and Jacek Slaboszewicz of McGill University, sysadmins at several sites whose identities have been disguised to protect their reputations, and most of all, Corporal Bob Beaulieu of the RCMP, for their assistance with incident investigations.
3 Private Branch Exchange: telephone switching equipment.
4 Computing Services is now called Instructional and Information Technology Services (IITS), after its merge in 1997 with the Audio-Visual Department; however, in this section, we will continue to use the name as it existed during the events we describe.
5 Internet Relay Chat: a popular Internet ``party line'' where people can ``chat'' with other people by typing at their keyboards.
6 AIX: a brand of Unix operating system distributed by IBM for use on their machines.
7 Root (also known as ``superuser'' or system administrator) privilege allows the user to examine and modify any file on the computer, including user files, and including the files which make up the operating system of the machine.
8 X: a ``window system'' whereby a program running on one computer can display its output on, and obtain input from, a screen, keyboard, and mouse on a different computer. The X window system is widely used on Unix machines.
9 Sysadmin: shorthand for ``system administrator'', the technical person in charge of a computer system.
10 File Transfer Protocol: a means of transmitting data files over the computer network.
11 Cpl. Beaulieu typed into Shrunk's session so that his keystrokes would appear in Michael's session log, thus making it easier to prove in court that the sessions captured at Concordia did indeed belong to the physical person who was arrested. Of course, the text ``This is Bob Beaulieu'' is not a valid command in the FTP program, which is why the program returns ``?Invalid command''.
12 The bug in question was reported in CERT Advisory CA-94:09.bin.login.vulnerability on May 23, 1994. The Computer Emergency Response Team (CERT) Advisories are widely circulated among both the good guys and the bad guys.
13 Shadow password file: a mechanism for hiding the encrypted password database on a Unix system.
14 Firewall: a computer security device designed to provide a secure barrier between a protected site and the network at large.
15 Password cracking involves guessing at a password, encrypting the guess, and comparing the encrypted guess with the real encrypted password. While this procedure is computationally expensive, it becomes much more feasible as computers get faster.
16 Flame: a very strongly worded criticism or insult.
17 SANS: you can reach the System Administration, Networking & Security conference office at sans@clark.net.
18 FIRST: Forum of Incident Response and Security Teams, an umbrella group for national and institutional CERTs
19 IETF: Internet Engineering Task Force, the body which creates technical standards for the Internet.
20 These goals are taken from a presentation given by Moira J. West-Brown of the CERT Coordination Center at Carnegie-Mellon University, at Network Security `95, Washington D. C., November 16, 1995.
21 The use of tiger teams to improve computer security awareness should not be discounted. Here's a short anecdote from 1994, about a year before the ``summer vacation'' events.
One of the authors (Anne Bennett) had just been shown the CERT advisory concerning the AIX login bug (subsequently used by Blackrat and Shrunk to gain root privilege at Essex University) by her spouse (a fellow sysadmin at another university), and neither of them could believe the seriousness of the problem. They wanted to try the exploit, but neither of them managed an AIX system. When asked whether she knew of any AIX systems managed by colleagues who wouldn't mind a small experiment, Anne thought of Carolyn Beckman, a researcher in the Concordia Biology department, who manages, on a somewhat unofficial basis, several Unix machines in that department. Anne and her spouse then tried the published exploit on Carolyn's system. Here we really must add that the ethics of doing this without asking first were highly questionable, and that Anne would definitely not do this today without first having obtained the appropriate authorization - in fact, people can be fired over such things, and user accounts have been revoked for similar reasons.
In any case, with a one-line command emitted from a machine outside the Concordia network, Anne and her spouse immediately obtained root on the target system! Extremely upset and apologetic, Anne urgently contacted Professor Beckman to explain what had happened.
At the time, the response was ``Oh, well, less qualified people are root on that system every day'' (fortunately not the crackers in this case!); in all fairness, that response was probably designed to calm Anne down - she hadn't really expected the exploit to succeed and was quite agitated over the whole thing. The system was subsequently secured by the application of a vendor patch, and Prof. Beckman has since become something of a resource person among fellow ``unofficial sysadmins''. In fact, in June 1998, while rescuing a compromised system in another department as a favour to her colleague there, she wrote:
``A long time ago you told your boy friend about the old AIX cytox and he tried to become root on it. You say it gave you heart failure, but it was the right move. I was just beginning and it taught me to pay some attention to security.''
None of Professor Beckman's systems have been compromised to date, whereas there have been several compromises in other departments where Unix systems are managed by researchers and students who are not particularly aware of or interested in computer security.
22 Spam: unwanted bulk e-mail; also used for inappropriate multiply posted netnews articles.
23 The Concordia Policy on Computing Facilities can be found on the web at
24 The World Wide Web Security FAQ, Lincoln D. Stein (lstein@cshl.org), Version 1.8.1, April 16, 1998. Copies can be obtained at
File translated from TEX by TTH, version 1.55.
Copyright, © 1998, Concordia University
Author: Anne Bennett and Michael Assels
Maintained by:
anne@alcor.concordia.ca
Last update: 1998/08/04 -- Anne Bennett