Archive for July, 2009

Hunting the Hunters

Posted in Malware, NSM on 24 July, 2009 by Alec Waters

Drive-by downloads are one of the more prevalent attacks you’ll face during your daily browsing. Some Bad Guy somewhere compromises such that it now has a hidden iframe on the first page that downloads something nasty that exploits a flaw in one of my browser’s plugins that downloads something even nastier that tells me that my computer has a zillion problems that can be fixed if only I’ll part with £££ for fake-AV-du-jour.

I’ll never get that sweater knitted at this rate…

One of the benefits of the NSM methodology is that you’re able to retrace someone’s steps. When we get a drive-by download, we can examine the evidence at our disposal and determine the initial point of compromise (usually a legitimate web page with a hidden iframe or obfuscated JavaScript on it).

At this point we can, if we choose, put on our malware-analysis hats and follow the initial hostile link. I’ll write about this more in a future post, but it usually involves de-obfuscating lots of JavaScript and working around anti-forensic traps that do silly things like redirect you to if you’ve not specified the correct referrer.

Finally, you’ll reach some hostile code. This will typically attempt to exploit a number of flaws on your machine, but the Baddies only need for one of them to be successful. A successful exploit will run a tiny bit of shellcode that normally downloads and executes something a bit bigger (fake-AV-du-jour, for example).

This secondary download will come from a server operated by the surprisingly well-organised Baddies. As well as serving up the malware, they’re keeping tabs on the progress of their malware “campaigns” in terms of the number of people exposed, the percentage of those successfully exploited, the types of successful exploit, etc etc (Dancho Danchev is the man when it comes to tracking this kind of stuff down, and sites like Wepawet and Unmask Parasites are great for spotting hidden badness on websites).

If the baddies hit you with half a dozen exploits, they’ll typically use a cookie or URL token to keep track of which one actually did the damage. And so we come to today’s story.

We were investigating a PDF-based compromise – a suspected SQL Injection attack had been used to compromise a website which was now laden with hidden, hostile, iframes leading to a malicious PDF document. We had worked our way through umpteen levels of obfuscated JavaScript etc until we had reached the point where we were able to download the second-stage payload from the Baddies’ server at will. The URL looked something like this:


The 7 in the URL denotes to the Baddies which exploit was succesful, and it really did have two e’s in exee. Because we were able to fetch this exe at will, we could observe it over the course of time to see if the Baddies kept the exploits the same but varied the payloads. This they did; over the course of a week or so we saw these files downloaded (these links are safe to click!):

The filename reported by VirusTotal varies too, indicating that the same exes were being distributed at the same time by different means.

However, one day our download was a lot smaller than usual. A quick peek inside the file reveals plain ASCII. HTML, in fact:

<?xml version=”1.0″ encoding=”iso-8859-1″?>
<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “”&gt;
<html xmlns=””&gt;
<title>MySQL Fatal Error</title>
<meta http-equiv=”Content-Type” content=”text/html; charset=windows-1251″/>
<style type=”text/css”>
body {
font-family: Verdana, Arial, Helvetica, sans-serif;
font-size: 10px;
font-style: normal;
color: #000000;
<font size=”4″>MySQL Error!</font>

<u>The Error returned was:</u>
<strong>Table ‘servtst_test.sploits’ doesn’t exist</strong>

</strong><u>Error Number:</u>

<textarea name=”” rows=”10″ cols=”52″ wrap=”virtual”>UPDATE `sploits` SET `loads`=`loads`+1 WHERE id=7</textarea><br/>


Aha! An error message! The Baddies are having trouble with their malware server! Boo hoo, no sympathy from me!

By looking at the SQL, we can see how the Baddies are keeping tabs on successful exploits:

UPDATE `sploits` SET `loads`=`loads`+1 WHERE id=7

Pretty simple stuff – the 7 in our URL denotes a specific exploit, and they’re just incrementing a counter whenever it’s successful. But it’s not working – the table isn’t there:

Table ‘servtst_test.sploits’ doesn’t exist

I wonder where it went? Now, I’m not a pentester by trade, nor am I a MySQL expert, but that SQL looks like it might be vulnerable to a SQL Injection attack. I wonder if someone out there asked for this URL:


Which might have caused it to execute the following SQL:

UPDATE `sploits` SET `loads`=`loads`+1 WHERE id=7;DROP TABLE `servtst_test.sploits`

It seems that if you live by the SQL Injection, you die by the SQL Injection. It’s just a theory, but if that’s what actually happened, it wasn’t me who did it! Anyone want to own up?

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Prevention Eventually Fails, part two – The Hot Knife of Malware

Posted in NSM on 17 July, 2009 by Alec Waters

We already know that prevention eventually fails, and that NSM can help us out when it does. We also know that defence-in-depth is a common strategy in the IT world, whereby one employs many different preventative measures designed to back each other up and to cover as many attack vectors as possible.

Consider a small office with no public-facing servers, where one of the primary defensive tasks is to defend the desktop machines from malware. To this end, the following array of defences is deployed:

  • A firewall device is at the office’s border with the Internet. As is usual, no unsolicited inbound traffic is allowed to pass through
  • Sitting behind the firewall is an IDS, on the lookout for trouble on its way in and also for compromised internal machines phoning home to Hacker HQ (this is “extrusion detection”, and you can read all about it in this book)
  • A spam firewall will scan all inbound email for viruses and spam
  • A URL filter will categorise all outbound web traffic and prevent access to anything malicious
  • Anti-virus is installed on each desktop machine, and is kept up to date
  • All machines have Microsoft patches applied automatically via WSUS
  • Users are trained not to do silly things like click on links in emails

That’s seven separate preventative measures, all employed concurrently and maintained in terms of patches, signatures, etc. In terms of coverage and thoroughness, this isn’t a bad setup.

Wouldn’t it be a gigantic shame if all seven failed at once? Surely that can’t happen…


The day starts with an infected machine showing popups for Fake-AV-du-jour, backed up by a BSOD screensaver. The support desk have already wheeled out Spybot and Ad-Aware etc. which have done a creditable job of removing the infection, and everyone’s happy.

Well, not quite everyone. We can’t consider the case closed, as we still need to know:

  • How the infection got through what we thought were comprehensive defences, and
  • What the infection did aside from the obvious popups (perhaps it also dropped a keylogger or other credential stealer?)

Fortunately, we have an NSM sensor installed behind the firewall, and we have access to all of the logs from the IDS, spam firewall, etc. Our starting point is the compromised machine, and we’re going to work backwards in time until we work out how it got like that. Working on the theory that the infection came in over the Internet somehow, we’re going to secure the following:

  • Email logs for the affected user
  • URL filter logs for hits from the compromised machine
  • Session data for all traffic concerning the compromised machine
  • Any relevant AV, firewall or IDS logs for the time frame concerned

Now we can start to put the pieces together. To cut a long story short:

  • The infection vector was email. The spam firewall failed to catch it, and it was delivered to the user – this is FAIL #1
  • The email said that the user has received an e-card, and gave a link which the user clicked, despite their training not to. Was this a blatant disregard for training? Forgetfulness? Not in this case – it turns out that twenty minutes earlier, the user had received a “legitimate” e-card email from a friend (they’d had several of these in the past without incident). When the malicious e-card email came in, the user was still in happy-e-card-land and clicked the link without giving it a second thought. FAIL #2
  • The user’s attempt to download the e-card executable was permitted, despite the presence of the URL filter. This was because the URL was “uncategorised” at the time, and the manufacturer’s default policy in this case is to allow the download. FAIL #3
  • The firewall isn’t designed to stop anything in these circumstances. As such, we’ll call this MOOT #1
  • The IDS didn’t say anything either, because it has no signature to match what is going on. FAIL #4
  • The e-card executable was downloaded and saved to the user’s machine and subsequently run. AV did nothing to intervene. FAIL #5
  • Because the user ran the executable by choice, there was no need to exploit any vulnerabilities in our fully-patched Windows victim.  MOOT #2

That’s five fails and two moots, which I reckon accounts for all seven of our defences. Oh well. Once again, the hot knife of malware scythes through the butter of prevention.

On to the second question – did the infected machine download or upload anything else? From an examination of the network session and URL filter logs, we can see that everything looks OK so far. We will assume, however, that Spybot/Ad-Aware/etc have missed something in their cleanup, and we’ll put the victim machine on the “watch list” for a week or two. If there’s any dormant undetected malware there (like a keylogger that uploads once a week or something), we’ll spot it.

Using what we learned, the following was done to shore up the defences:

  • The URL filter was changed away from its default policy in order to make it block uncategorised sites. This would have prevented the e-card download
  • The “Digital Postcards” category of websites was blocked as a matter of corporate policy. Having seen a blockpage instead of the “legitimate” e-card from a friend, the user would have hopefully remembered their training and not clicked the link in the malicious email
  • A temporary band-aid was applied to the spam filter to block all emails containing “/e-card.exe”

Of course, these are just more preventative measures, none of which are foolproof. That said, any one of them would have increased our chance of keeping the infection out in this case.

Defences will let you down from time to time, and for me at least, the “clean the malware and carry on” approach isn’t enough. If there’s a chink in the armour, I need to know where it is and what I can do about it – this is where NSM can help.

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Visualising Sguil session data with NetFlow

Posted in NSM, Sguil on 10 July, 2009 by Alec Waters

I think this is the first time I’ve explicitly mentioned Sguil, and I’m not going to talk too much about the package itself as many others have already done it for me. Basically, Sguil is a nicely integrated suite of (free) tools that will help you put NSM principles into practice. It has a wonderful investigative interface, and allows easy access to all the alert, session, and full-content data you’ve been capturing via a TAP or SPAN port.

I want to talk about Sguil’s session data capture mechanism. SANCP is used for this purpose, and it’s nicely tied into the alert and full-content data captured by Sguil. It does a jolly good job, too.

SANCP isn’t not the only tool for handling session data, though – Cisco’s NetFlow is another. What I’m going to propose might sound a little odd, but I find it useful – I hope you will, too. Here goes:

My Sguil sensors maintain two stores of session data. One is the standard set as captured by SANCP, the other is a database of NetFlow records that express the same traffic.

“Why?”, I hear you ask. Isn’t this a duplication of data? Don’t you need some fancy Cisco router to export flows? The answers to these are “sort of” and “no”, respectively. Before we get onto how we make this happen, let’s talk about why we want to bother in the first place.

  • There is no easy and cheap visualisation of SANCP data. If you’ve got a TAP on a link and you’re feeding it into a Sguil sensor, it’d be great to see a nice graph of how much traffic is passing in each direction. If you can break it down by IP address and port, so much the better. Yes, you can export the SANCP data into something like Excel to graph it, but that’s hardly “easy” or “cheap”. If we can somehow extract NetFlow data from our TAPped link, we can use any number of NetFlow analysis tools to look at what’s going on.
  • SANCP isn’t (to my knowledge) timely. If you’ve got a long running session (e.g., a huge download) it won’t get logged by SANCP until it has finished (possibly many hours later). NetFlow devices by contrast can be instructed to prematurely export flows for sessions that haven’t finished yet, giving you a more real-time view of the data.
  • SANCP logs one row of data for every session it observes, in terms of source IP/port, destination IP/port, src->dest bytes/packets, dst->src bytes/packets, etc. By storing these “bidirectional” records, you sometimes get entries like this: -> -> -> ->

    All of the rows above show sessions to or from a webserver on Look at the last row, though – somehow is listed as the source rather than the destination (you’d have to ask someone familiar with the SANCP codebase why this happens from time to time). What this means is that if I want to write some SQL to show me all of the sessions hitting my webserver, I can’t just say “WHERE dst_ip = INET_ATON(‘’) and dst_port = 80” because I would miss out on the last row. I need to alter my SQL to include rows where “src_ip = INET_ATON(‘’) and src_port = 80” as well (I guess this is why there are UNION options in Sguil’s query builder!). IMHO SANCP’s “source” and “destination” columns might be better referred to as “peer A” and “peer B”, since there’s no consistency in the way sessions are listed. NetFlow, by contrast, will store unidirectional flows with separate database rows for traffic from a->b and from b->a, thereby avoiding this confusion.

Irrespective of whether you agree with the latter points, I think the visualisation aspect of this is worth it’s weight in gold. Here’s how to do it, and actually make it work in a useful fashion.

Firstly, we need some way of getting NetFlow exports from our TAPped traffic stream. softflowd is your friend here – get it installed on your Sguil box, and run it like this:

/usr/sbin/softflowd -i eth1 -t maxlife=60 -n

Let’s look at the parameters in turn:

“-i eth1” tells softflowd to listen on eth1, which is also Sguil’s monitoring interface in this case.

“-t maxlife=60” tells softflowd to export a flow record for non-expired flows after sixty seconds. This gives us the “timeliness” that SANCP lacks.

“-n” tells softflowd where to send its NetFlow exports to, in this case a NetFlow collector running locally on the Sguil sensor on port 9996.

So now softflowd is looking at Sguil’s traffic stream, and is converting what it sees to NetFlow exports which it is sending to a collector listening on port 9996. All we need now is a NetFlow collector to receive the exports.

There are many collectors available, but the one I’m going to show here is ManageEngine’s Netflow Analyzer (NFA). I use NFA because the free version will work perfectly in this scenario, and because it has one crucial feature that makes all of this useful (we’ll get to that later on).

So, download and install the free version of NFA on your Sguil sensor, and tell it to collect flows on port 9996. In very short order, NFA should start receiving flows and drawing a nice graph. A graph which will have one obvious and fatal flaw about it:


Whoops. We’re only showing “in” traffic, and it’s actually the sum of the “in” and “out” traffic on our monitored link. When you think about it, what else was softflowd supposed to do? After all, the only traffic it sees is coming “in” to eth1, so that’s all it can report.

However, NFA has a nifty feature we can use to make sense of the data – IP Groups. Set up a new IP Group (call it whatever you like), and add the IP addresses that Sguil/Snort considers to be HOME_NET. NFA will then use these IP addresses to determine which flows should be interpreted as “in” and which as “out”. If we navigate to our IP group in NFA we now get a nice graph that looks like this:


Woohoo! Now we’re talking! Now we can use NFA’s other cool features like drag-to-zoom and breakdowns by IP address and port.

Yes, we’ve bloated up our Sguil sensor in the process. Yes, we’re doubling up on session information. But if you’ve just arrived at a customer site and have plugged your sensor into what is (for you) an unfamiliar network, what better amd quicker way to get an idea of the kind of traffic you’re monitoring?

Comments welcome. I’m sure there’s a gigantic flaw in the plan somewhere!


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

The case of the Phantom Hacker

Posted in Case Studies, NSM on 3 July, 2009 by Alec Waters

It’s another quiet day in the Secret Underground NSM Bunker. The lights on the enormous world map at the far end of the cavern are blinking green, my army of orange-boiler-suit-clad underlings are hard at work on my next diabolical scheme, and the persian cat on my lap is purring contentedly. Martini-drinking spies are nowhere to be seen.

Out of the blue, the cat’s ears prick up and she starts to hiss. Seconds later, a security alert is raised (why have an IDS when you can have a cat?). Because we’re collecting logs as well as traffic data, we can tell pretty quickly when an account is locked out due to too many failed logins. It’s usually a fat-fingered user who will shortly be calling the helpdesk, but today it’s not. Today it’s one of the domain admins. Now admins can have a fit of fat-fingeredness too, but equally someone could have tried to brute force a login into a privileged account. We need to find out which it is.

A quick call to the admin’s desk reveals that he’s not there – he’s been out of the office for the last hour, and his computer appears to be off. Furthermore, one of the admin’s colleagues has had sight of the admin’s machine for the entire time, and nobody has been near it.

So, how has a powered-down computer managed to attempt a login without anyone actually physically being near it? I can’t ask the cat, because she’s run off and is hiding behind my latest prototype doomsday device. It’s time to mine all of that information we’ve been collecting, and as logs were the trigger we’ll start with these:

  • The “account locked out” message came from a domain controller, and was preceded by the proper number of “login failure” messages.
  • All of these messages cite the admin’s machine as the computer that was used for these login attempts. We hope these messages are trustworthy, otherwise we’ve got much bigger problems (i.e., someone is messing with machine accounts and server event logs).
  • The login failed messages all say that an interactive login was attempted locally – it wasn’t a remote desktop session or other network login (SMB, etc.).

There are two ways in which a local interactive login can be attempted. You can either sit at the machine and try it, or you can use some remote-access software like VNC that only does screen-scraping rather than interacting with the foul guts of Windows. We can rule out the former, assuming we can trust what the admin’s colleague has told us. We’ve got an NSM sensor at the network’s border, so we can see if the admin’s machine is being contacted over the Internet.

A few quick queries later, we can see that there has been absolutely no Internet-bound traffic either to or from the admin’s machine since they logged out and left for lunch. This isn’t the most favourable outcome, since the alternatives are:

  • There’s a physical backdoor to our network somewhere that someone is using. A modem or a 3G phone or a rogue AP or something (wireless is not permitted at all at this location so if wireless is involved it’s not an authorised device)
  • It’s an inside job – the attacker is either inside the building right now, or has just made their getaway. This would be irritating, not least since our NSM capability only covers the network’s borders – if it’s an inside job, NSM won’t have caught it at all.

The cat is now trying to force open the covers of the doomsday device in order to set it off… Bad kitty!

The orange-clad goons are busy mobilising to secure the admin’s machine for imaging and forensic examination when we get wind of an interesting new helpdesk ticket. Apparently, someone is having trouble with their new keyboard. Their new wireless keyboard.

A penny rolls towards a precipice and gets ready to drop, and the cat heads off to her dish of cream – always a good omen.

It turns out that a few people had taken delivery of wireless keyboards that morning. The admin was one of the recipients, and had set his up, logged out of his machine, and gone to lunch. A little later, someone else had unpacked theirs. They turned on their machine, and waited for it to boot. Not being a touch typist, they dropped their head to their new toy and hit control-alt-delete, and then hit tab to skip from the pre-populated username field to the password field (bad policy! naughty policymaker!). Still with head down, they typed their password and hit return.

Looking up at the screen they saw that it was still prompting for control-alt-delete, so they repeated the process. Several times. They finally hit the keyboard’s “sleep” key to no effect before getting the old keyboard back and raising a support ticket.

Now I’m sure you’ve all worked out what was going on here (the cat certainly had). The user’s new wireless keyboard was on the same channel as the admin’s, and all his keystrokes were driving the admin’s machine, eventually switching it off. Cue much debate about the merits of wireless keyboards…

Case closed, the cat returns to my lap, the orange-clad goons get back to their diabolical tasks, and all is well in the Secret Underground NSM Bunker….

Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)

Collection is King, part two

Posted in NSM on 1 July, 2009 by Alec Waters

In part one of this post, I talked about NSM as a methodology, and the four different kinds of information that NSM collects (alert data, session data, statistical data and full-content data). All of this lot is “scraped off the wire” by some means or another and fed to a sensor for collection.

NSM is a great discipline to practice, and provides the analyst with triggers for an investigation (in the form of alerts, or anomalies in the statistical or session data) and also the means to resolve the puzzle in a timely fashion.

However, a network of routers, switches, firewalls, IDS devices, servers, workstations and applications has a lot more information to offer than just what one can scrape off the wire, namely their logs:

  • Routers, switches, firewalls and IDS will clearly have a nice subset of security-related log outputs (“I dropped a packet because an ACL told me to”, or “There are too many half-open TCP sessions”, or “I blocked access to this URL”, etc), and also many many others describing various changes in state. Cisco’s list of log outputs is here, and it’s really long.
  • Servers and workstations will also have their own logs. In the case of Windows, it’s the Event Log – documents over 10,000 different types of messages.
  • Individual applications will also maintain logs. Things like webservers, FTP servers, print servers, anti-virus servers, disk encryption management servers, etc. all create volumes of log entries.
  • Most devices contain some kind of management agent, usually SNMP or WMI based. You can poll for information like CPU/network/memory load and draw pretty graphs.

Most of the things in these logs will denote utter trivia. Knowing that favicon.ico was downloaded from your company’s website three thousand times in the last month isn’t very exciting. And it’s no surprise that Old Bob in accounts fluffed his login attempt yet again when he fired up his computer this morning. But each of these nuggets may play a part in an investigation, either as the initial trigger, or perhaps in a supporting role, providing context to data collected by an NSM sensor:

  • As a trigger. If Old Bob fluffs his login five hundred times inside of ten minutes, then something is definitely afoot. Yes, you may be able to get a whiff of a problem from your NSM sensors, but it’s certainly more convenient (and clearer to understand) if you get five hundred “failed login” messages from a server’s log.
  • Providing context. Let’s say our NSM kit tells us that something is fishy with a desktop machine at NSM will tell us exactly what this IP address has been up to network-wise, but if we’re collecting other stuff we can find out more:
    • Assuming that is a “local” Windows box and under our control, we can ask the domain controllers which user account was in use at the time of the incident, and which machine account was used by
    • We can look in the DHCP server logs to discover the hardware MAC address that was associated with it at the time of the incident (if isn’t covered by full-content capture, this is the easiest way to get this information).
    • We can look for this hardware address in the MAC tables that we’ve been pulling from our switches via SNMP, and find out which port on which switch the machine was attached to.
    • We can see if any policy enforcement devices like firewalls have taken action against

    Now we can begin to ask questions about why Dave from Marketing is apparently logged into the CTO’s laptop, which is bizarrely connected up in the broom cupboard on the sixth floor and is mercilessly scanning for anything that will listen on port 31337….

The above are just some simple examples – the point is that log collection and NSM are complementary to each other and if you can do them both you’re better off for it.

So, how do you go about collecting all this stuff? I’m not going to recommend any specific magic products, but an ideal log collecting system will be able to:

  • Receive log entries via a mechanism like syslog.
  • Reach out and take the log entries from monitored devices that aren’t able to “push” this information to the collector. By doing this, you reduce the likelihood that the Bad Guys take over your server and erase the pertinent log entries to cover their tracks.
  • Represent log entries in an abstract form to cater for messages from different vendors. An “I dropped a packet” message from a Cisco firewall will be different to one from a CheckPoint firewall, but they both mean the same thing. The log collector should present both of these to you as the same event type.
  • Sessionise events. Based upon IP addresses, ports, and the time of day, the collector should be smart enough to be able to correlate different events together and say that they were all part of the same action.
  • Allow the user to query all of the collected information with flexible filtering criteria. You should be able to ask questions like:
    • What did do between 10:00 and 10:23 on Tuesday? I want to see domain logins, denied packets, URLs requested, etc. etc.
    • How many failed logins were there last week for members of the “Domain Administrators” group?
    • Which of my access points have had failed associations in the last 24 hours? I want to see a count of failures per-device, and I want to see a graph of failures over time.
    • Use your imagination. What do you care about on your network?

Such a device will probably also offer an  “Automagical Relevant Security Evaluation” facility (“ARSE”), which can take all of these abstracted, sessionised events and correlate them together and tell you Meaningful Security Stuff (although this could well be marketing flimflam on the part of the vendor). I have only limited experience of these (very expensive) toys, but the greatest value comes from the data mining I do myself in full knowledge of how my infrastructure operates. The ARSE facility looks great during customer demos and sounds great on paper, but you can’t beat actual analysis performed by an actual analyst. I prefer to view such devices as a “Google” for my big pile o’ logs.

So why do we need to go to all these lengths to get this information? Can’t NSM deliver the goods by itself? Well, yes and no:

  • It’s much simpler to see a log message saying “Bob logged in” than it is to unpick this information from an SMB traffic stream. Having both forms of information is actually quite beneficial – they can back each other up, or they can contradict each other. NSM saying “Bob logged in” but the server saying “nobody logged in today” is an indicator in itself.
  • You might not be able to get enough information from a full content capture. For example, an ssh login won’t be decipherable from a pcap (unless you can supply the keying material), but the logs on the server will tell you who logged in.
  • You might not have enough NSM sensors to cover everywhere. Take a typical office with five workstations, a Windows domain controller, and an NSM sensor at the border with the Internet. The NSM sensor isn’t going to tell you anything about denied logins as policed by the domain controller.

The message here is that visibility is everything. Anything you can do to improve the visibility you have into the operation of your infrastructure can only be of benefit. It is not important that most of what you collect will be run-of-the-mill trivia, when each item is taken on its own. What is important is that you’ve collected a vast amount of easily-accessible jigsaw pieces, each put into the context of each other, and you’ll be really glad you’ve got it when the time comes!


Alec Waters is responsible for all things security at Dataline Software, and can be emailed at alec.waters(at)