Analyst Chat #59: Understanding the SolarWinds Incident and Recommended First Steps

Welcome to the KuppingerCole analyst chat. I'm your host. My name is Matthias. I'm an analyst and advisor at KuppingerCole analysts. My guest today is John Tolbert. He's an lead analyst with KuppingerCole working for KuppingerCole out of Seattle. Hi John. Hi Mathias. Great to have you again, in an episode of this podcast, this is the first episode for 2021. We had a short break for this podcast, but while we were away, uh, some interesting or some frightening security incidents were happening in the meantime, uh, maybe let's start off with that.

Um, John, what was happening, which really concerns us at a larger scale? Well, probably the biggest event that has happened in quite a while, uh, is the solarwinds incident. And it's been given a few different names, uh, you know, according to the malware, some Microsoft calls it slower gate fire. I discovered it, they call the malware sunburst.

Um, this week there was a new discovery related to, uh, called sunspots. Um, they all kind of fit under the rubric of the, the solar winds attack, which was first, uh, on earth in the middle of December.

But, you know, going back now, we know that, uh, it looks like the, the threat actor involves, uh, first access. So way back in September of 2019, and then a malicious version of the solar winds around management platform was delivered to up to 18,000 of their customers, uh, from roughly March in the may or June of 2020.

Uh, so there, it sat without being discovered until, uh, December of just this last year, which, you know, I think it's interesting too. We've seen a lot of metrics in the cybersecurity business around meantime to discovery, you know, being at least about six months and sometimes longer, this definitely fits into that pattern.

Uh, meantime to recovery, uh, seems to take a couple of months minimum. So at this point on January 15th, we're about one month into that two plus months, uh, containment and, uh, phase. So good time to be talking about what we've learned so far and what it means and 11 narrowly, what do we think, uh, affected organizations can do about it? Right. Exactly. I think that this is the first episode of what I expect a series of episodes around this issue. Maybe we just do what you just mentioned to start from where it originated.

Um, I've read about a, a code repository, which contained FTP passwords, which allowed for, for access to, um, some of the more critical systems there as well. And, um, as you said, who is affected, why are they effected?

Um, and what can organizations do? It's what I've read through this, this FTP passwords, um, issue being the root cause, or is there more To know? I think there's more to find out about that. Yeah. I've seen that too, that like the GitHub repository had solar winds 1, 2, 3 for a password, and they may have been alerted about that, um, you know, months before.

Um, there's also, I've seen speculation that internal accounts and solar winds are for sale on the dark web prior to that. So, you know, what the origin of that was, uh, I don't think anybody knows at this point, but yeah, I think there are a number of things obviously hard and joining passwords is never a good idea. I believe guidance was given to, um, to exempt directories in which solar winds was installed from antivirus or anti-malware scans.

So, and that this kind of brings us right back around to the whole notion of trust, uh, and why the it supply chain, uh, is at risk. Uh, and why we as customers have many of these kinds of products need to, uh, insist upon better security from the it supply chain members, right? You mentioned it rightly this is an attack, not towards an actual product, but at attack towards, um, a provider of, uh, of the, uh, in the supply chain. So what is the kind of, of influence that, um, that could occur through exploiting this, this, this breach?

What was the attack and factor that they were using or what they are Well, you know, the thing that was published this week that was interesting was sunspot, but the fact that there's malware that can sort of effectively control what gets loaded into an its suppliers build, uh, at compile time, you know, the sunspot, uh, component of this allowed the attackers to ensure that, uh, across the series of several builds that the malware was automatically included and solar winds around management platform, um, that, that means that for those companies that downloaded that between March and June or so, uh, they, they probably wound up with that, the malware on their systems, what did it do well with 18,000, um, potential companies?

I think it's hard to know upfront what all the impacts of that will be beyond the ones that have already acknowledged that like solar winds, um, Microsoft FireEye, Cisco duo, um, there were a few companies that have talked about this, uh, and, and what the impacts have been so far. Right. But In the end, we are talking about full access being possible by installing components, um, under the hood of a trusted, um, software management tool, um, bleeding to the full range of potential threats, including lateral movement, um, exfiltration of data and targeted espionage. Yes.

One of the things that we have read a couple of different accounts of is, uh, that additional accounts were added to say victims active directories, which allowed the attacker to be able to leverage, uh, privileged administrative capacities within these organizations. They also added, uh, certificates, uh, to different services. I think in the case of the administrative offices supports and DOJ in the U S uh, this allowed them access to, um, I think it was a reported 3% of their emails.

So yeah, it was, it's obviously an attempt at espionage, uh, and, and some of the longer lasting effects that victimized organizations will have to deal with are related to how their identity management systems were compromised with this addition of accounts. Uh, we also have read about how they use things like ADFS and Samuel infrastructure to move around laterally within organizations.

So that, that's another thing that, uh, the victims and, and perspective victims will have to take a look at. Right. And I understand that they are always also was malware distributed, targeted malware, um, and that Microsoft has taken measures to at least control the domain that was used as the, as the CNC domain for this malware.

Um, so at least from that perspective already measures have been taken. I assume, of course, that solar winds have taken already appropriate measures and, and tried to clean up their systems if organizations are now, uh, in the situation that they are a customer of solar winds, that they have been in the situation to be compromised, what would be the first steps to take what is really to do now? You're right.

I know Microsoft, solar winds, all the companies that have spoken out about this publicly have already taken, you know, really good measures to help remediate that, um, Microsoft sinkhole, the initial C2 domain, uh, defender can detect it. Uh, I would imagine at this point, all the major anti-malware, uh, solution providers can detect it, but it's mostly in case now of looking for, is there evidence that for those two did have the effected versions or are there other threat actions taken within their domains? So I I've seen this broken up into three stages in the attack.

Stage one is, you know, companies or organizations that downloaded it, and it may have signaled initially to that domain, but no other evidence of anything else has happened. Uh, stage two is where there was more communication, C2, communication, and it switched to using different domains. So there are multiple indicators of compromise that can be used that you can pick up through, uh, threat intelligence sources, uh, and search across your assets.

Stage three is really the, the best and most complete, where you can find signs of other kinds of threat actor actions, such as data exfiltration, uh, manipulation addition of accounts to active directory. You know, in some cases they actually added federated trust to other outside domain. So to think about authentic or send a SAML token and get access to a victim's assets that way.

So in that last case where you've got signs that, uh, you know, accounts that you're not really sure about have been added to 80, or you've got weird looking SAML tokens coming in, or other signs of data exfiltration, and that's, those are the ones that are going to need to go the extra mile to remediate, I would say on the remediation side, well, first of all, you've got a detective, uh, there are, uh, lots of IOC is available.

Now, if you are, have a sore platform, and I think it would be highly recommended at this point to be using soar, then you can load in, uh, the IOC and, and start doing some threat on even across your enterprise. If you find that, uh, things have progressed to stage three, then, you know, at that point, uh, obviously you've got to start by rebuilding your solar winds and that applies to companies in stage one or stage two as well.

And by this, I would mean, you know, from the ground up from the OSTP level up and then get rid of existing accounts, uh, that are pertaining to solar winds, uh, force multi-factor authentication for solar winds, admin users, and then, uh, place solar winds accounts under the care planning solutions, privileged access management.

Uh, I think that that's an absolute necessity for anybody running solar winds, especially those who have, um, for sure, been involved in a stage one or stage two incident beyond that, uh, for stage three, I think, you know, start with that rebuild for solar winds, but then in order to understand, you know, how much damage has been done, I think it's going to require a full IAM audit, you know, looking at your active directory as your active directory, uh, ADFS, configurations, other infrastructure that might be capable of, uh, forming SAML assertions, they're going to need to be investigated.

Um, and you know, for anything that you can't find a real purpose for, uh, it, it will have to be removed. Then I think it would also be time to start a full access reconciliation, uh, you know, looking at, you know, any governance, if you've got, uh, other accounts that may have been compromised, even for which you do have a valid business purpose, you've got to see if they have been used as part of this attack. And then lastly, uh, other standard apt and kind of incident response and mitigation plans, uh, probably need to be executed.

And, uh, as soon as possible, they already have not been started. But all that you've mentioned right now, either on the detection part requiring actually some, some, some dramatic, some massive, um, cyber hygiene, plus detective controls being in place. And for those who, where it has happened, uh, no matter in which phase they are, that you just mentioned, it really requires a, a strong set of, of, um, of skilled security staff internally or externally. So this is a must, so really having the right people, the right tools, the right knowledge and the experience in place.

So that's gonna require a lot of work and a lot of money to spend, and it will most probably come with disruption, right? Yes.

I mean, I, I don't see a way around that, especially if you are in an organization that has fallen into stage three where, um, there's good, strong evidence that the many systems have been compromised. There's really no choice other than to do many of these remediations. I know like us government agencies told people, essentially incidence build their active directory or some of their other identity management systems that may not be feasible for all organizations that may not have the labor power, uh, to do that.

Um, that's why I think at the very least, if you believe you're in that category where you have had active directory accounts added, or maybe the SAML infrastructure compromised, it's probably good to start with getting a picture of what has already happened and trying to eliminate those accounts specifically that you don't have a use for now. That doesn't mean that if an attacker is still active, they couldn't be creating other accounts, you know, kind of in a, a whack-a-mole fashion.

And I think that's really good reason why, uh, government agencies are recommending sort of a complete redo in some cases, but at least trying to get a handle on, on what you can find in terms of evidence. Uh, and then remediating that as you go by eliminating those accounts, eliminating entitlements and starting to protect the administrative users and service accounts where Pam and MFA, uh, are, are good first steps. Okay. So Basically for those that are involved in that breach, which are targets of that breach, this is really dire news. So there is lots of work to do.

Um, maybe it's even, um, the, the communication part of, to try to get in touch with those that are targets of that. Maybe not all of them actually do know that they are a target, so this will stay longer with us. And we do not yet see the end of that tunnel that we are looking into, which is, um, in, uh, still at a very early phase. And there might be more to come more to hide behind that. Yeah.

You know, I would say not dis what you see in the news about it.

I mean, there may be a tendency for some to say, well, you know, this was probably targeted at the us governments, but it doesn't really apply to me, you know, anybody that download of it that sees signs of this C2 activity, you know, they could be, uh, a target of opportunity if the information, or if the malware is still active, then who's operating, it could see this as a way to steal intellectual property, you know, so just because there are some big name actors that obviously are maybe the primary target of this doesn't mean that if you've wound up with this on your system, you couldn't be a victim as well.

So I think everyone really needs to take it seriously, uh, due to the possibility of, of losing intellectual property or, you know, other follow on effects of, uh, malware infections. Okay. Okay. So thank you, John, for being my guest today for this very, very first breaking news edition of our podcast. And I think we will need to follow up, um, after a few weeks to, to look where we are then. And if there is more to know more to do for those who are affected by this, this breach and by this, this incident.

Um, so again, thank you, John, for, for shedding some light on what has happened and giving some, some concrete advice where to start, what to do, and which are the first right measures, uh, including finally, maybe a complete rebuild of the ADA. Yeah. It's certainly not a pleasant way to begin the year, but you know, people all around the world are working on discovering exactly what happened and how best to remediate this.

So, yeah, I think, uh, findings will change recommendations may change in the month or two ahead. Uh, this is big and it's going to take time for everyone to process it, understand it, and act on it. That's a perfect summary.

So, so we, we leave it up to that here. Um, of course there's lots to read about the topics that you've mentioned about MFA, about, um, access governance about so our tools, um, if the audience is interested in that, just go to KuppingerCole dot com and download the adequate reports.

We will, um, keep up with that topic and there should be more soon in this podcast, in our blog and in our publications. So thanks again, John, thanks for taking the time.

And, um, anything final from your side, Not we'll keep an eye on this and we will, uh, reporting on what we find that we feel is relevant. Perfect. So Again, then thank you. And bye-bye, .