The following are my opinions and recommendations only. They are not intended to be critical of the fine efforts of the people currently running the Forums, in fact having experienced similar “disaster” scenarios in the past I strongly empathize with the difficulties and pressures they were in. But drawing on my experiences, I have drawn up this list of items which are intended purely as <constructive> recommendations.
Note the principles and methods described in the above link. Validation and scoping input is pretty basic and can(should have been) implemented at numerous places. vBulletin probably should be configurable to implement globally. Of course, the plugin creators (who apparently haven’t existed since this last summer a few short months when the XSS vulnerability apparently was first published approx Sept) should have implemented. Probably the vBulletin modules which accept and parse imput (eg search boxes, create post) should be implementing.
In any case, I don’t know what was behind the openSUSE forums’ emergency response to press the panic button to implement a solution that wasn’t staged so the whole website was taken down. There is no doubt that the consequences could have been serious if an actual attack was mounted.
So,
Here are my list of suggestions moving forward…
The response obviously was in the face of a threat that couldn’t be managed(else I doubt anyone would have accepted the consequences of a website down for days). I strongly suggest implementing the solution I’d posted in my Wiki for a long time now - It has the ability to do incredibly powerful search and analysis of enormous amounts of data, even in realtime. In other words, no matter the load it should not be difficult to parse real time apache logs for malformed XSS strings typical of the SEO Plugin attack, and block those IP addresses automatically. You also would be able to do ad hoc analysis quickly on whether there is a pattern of mal-formed posts and from where. http://en.opensuse.org/User:Tsu2/Install_and_Intro_Logstash-Elasticsearch-Kibana
Both before the incident and supporting future maintenance, I don’t know if there is Policy and Guideline for both Disaster Response and Normal Maintenance. If not, policies should be created. It’s normal for a business and should not be different for a community project. Included should be who to contact, possible flow diagrams for different kinds of scenarios and one should be created for just what happened while fresh in mind to improve future response. So, for instance when the software problem was discovered, was anyone in Network Administration brought in to provide input? Did anyone responsible for the software side even know that networking solutions existed that would likely have been sufficient to address the problem?
If don’t already exist, resources should be pre-allocated for at least one (maybe many) staging platforms(my software development typically averages 3). Some should be completely isolated from the Production database, some should be connected to the Production database. Taking it a step further, full or nearly full implementations can be set in place for multiple uses, eg development, recovery. These additional platforms should be easy to allocate, they just require space and typically don’t require significant resources but they do need to be secured. Of course, this requires virtualization which I don’t know if SUSE has implemented (a year plus ago, it wasn’t).
The following are recommendations which likely aren’t currently implemented. The first (5) is a guideline on XSS. As I described above, XSS filtering can be implemented at numerous places, but if vBulletin isn’t doing it there is little reason why it can’t be deployed with relatively little effort on one’s own. If filtering is global and to be applied to numerous servers, then it might be more practical to implement in a Proxy in front of the actual webservers. If the websites to be protected are few, then it can be implemented per server or application. In other words, it pays to communicate between network administration and application web admins. If co-ordination is spotty, then maybe “defense in depth” allows you to sleep easier (own your own solutions to problems).
After the XSS Prevention guide, I’m providing the links to the Apache Rewrite Module(6). As of today, my tests indicate that the old links <do not work>. I think that regardless whether a decision is made to deploy the new web root or go back to the old one, <support> for the old root is critical to avoid breaking all the links that point to old forum posts. Note that according to the Apache documentation, the mod_rewrite module is <not preferred> but I’m giving you the link anyway because in the short run implementing should only require 1 to 2 lines of code.
Repeating myself a bit if it wasn’t clear, any implementation of both XSS hardening and web root re-write can and should be implemented on a staging platform before implementing in Production. Properly done, switching over should be done transparently to all Users (although selecting a very low traffic time is highly advised).
> In any case, I don’t know what was behind the openSUSE forums’ emergency
> response to press the panic button to implement a solution that wasn’t
> staged so the whole website was taken down. There is no doubt that the
> consequences could have been serious if an actual attack was mounted.
Huh? It was an actual attack. A cracker gained entry and defaced the
forum page, and obtained copies of what he thought was the user database
with passwords. Photos of all that were published.
–
Cheers / Saludos,
Carlos E. R.
(from 12.3 x86_64 “Dartmouth” at Telcontar)
Yes, although as said “the consequences could have been serious”, i.e much more serious with longer days to resolve than it seems to have taken so far. However, it has caused enough additional work for some dedicated people.
I don’t think it’s particularly helpful to discuss and comment on technical details of attacks like this in an open forum that has already been targeted. It doesn’t make any sense, and could have been accomplished by Private Message. Perhaps “least said - soonest mended” in this case.
On 2014-01-14 00:16, consused wrote:
>
> robin_listas;2615744 Wrote:
>> Huh? It was an actual attack. A cracker gained entry and defaced the
>> forum page, and obtained copies of what he thought was the user database
>> with passwords. Photos of all that were published.
> Yes, although as said “the consequences could have been serious”, i.e
> much more serious with longer days to resolve than it seems to have
> taken so far. However, it has caused enough additional work for some
> dedicated people.
I suppose they basically reinstalled the software, a newer version,
without the vulnerable addition. The data seems to just been reused or
something, as the nntp side was running all the time and messages were
written.
> I don’t think it’s particularly helpful to discuss and comment on
> technical details of attacks like this in an open forum that has already
> been targeted. It doesn’t make any sense, and could have been
> accomplished by Private Message. Perhaps “least said - soonest mended”
> in this case.
Well, I don’t know about that. Open source is discussed in the open, all
sources are available and can be examined both by good and bad guys. I
don’t think that keeping secrets helps much, because in the end the bad
guys find about it.
I don’t know anything technical about forum software, so I can not
really disclose anything fruitful
–
Cheers / Saludos,
Carlos E. R.
(from 12.3 x86_64 “Dartmouth” at Telcontar)
tsu2 wrote:
>
> The vBulletin Plugin vulnerability appears to be likely a XSS exploit,
> if so then some basics on hardening against XSS
> http://tinyurl.com/7ky8d7
>
Is the database exposed to a mechanism where it supports requests for
data from native javascript ?
GNOME 3.10.2
openSUSE 13.1 (Bottle) (x86_64) 64-bit
Kernel Linux 3.11.6-4-desktop
On Mon, 13 Jan 2014 23:55:49 +0000, vazhavandan wrote:
> tsu2 wrote:
>>
>> The vBulletin Plugin vulnerability appears to be likely a XSS exploit,
>> if so then some basics on hardening against XSS
>> http://tinyurl.com/7ky8d7
>>
>>
> Is the database exposed to a mechanism where it supports requests for
> data from native javascript ?
I’m sure that’s been examined and determined.
The database itself is not directly accessible from the 'net.
On 2014-01-14 01:56, Jim Henderson wrote:
> On Mon, 13 Jan 2014 23:48:06 +0000, Carlos E. R. wrote:
>
>> I suppose they basically reinstalled the software, a newer version,
>> without the vulnerable addition.
>
> The latest patch level for 4.x and for 5.x both have the vulnerability.
> Being current wouldn’t have prevented it.
Oh
–
Cheers / Saludos,
Carlos E. R.
(from 12.3 x86_64 “Dartmouth” at Telcontar)
On Tue, 14 Jan 2014 01:43:08 +0000, Carlos E. R. wrote:
> On 2014-01-14 01:56, Jim Henderson wrote:
>> On Mon, 13 Jan 2014 23:48:06 +0000, Carlos E. R. wrote:
>>
>>> I suppose they basically reinstalled the software, a newer version,
>>> without the vulnerable addition.
>>
>> The latest patch level for 4.x and for 5.x both have the vulnerability.
>> Being current wouldn’t have prevented it.
>
> Oh
That’s something that’s been overlooked by a few people asking about why
the server wasn’t patched - the patch wouldn’t have made a difference,
because it was a new exploit.
Yes, overlooked by those who didn’t read between the published photos of the article by hackernews.com (referenced on openSUSE News). Here’s the quote from the original online publication:
“Another important claim by the hacker that vBulletin 5.0.5 latest version is also vulnerable to his zero-day exploit and there is no patch yet available to fix it.”
After determining the exploit vector (XSS), then it becomes obvious as I described that vBulletin does not filter XSS strings globally. As you described, vBulletin is therefor obviously potentially exploitable <if> there is code that can be manipulated and the SEO plugin apparently provided that second required piece.
So, vBulletin is not immediately exploitable by this method of attack in any obvious way with the SEO plugin removed, but until XSS filtering is enabled by vBulletin it’d be good practice that anyone who uses the software should implement that filtering on their own.
Anyone who uses vBulletin should also be wary if a different exploit vector is developed if the code remains potentially vulnerable.
I posted this because when I started Googling this vulnerability no recent results were returned except for discussions in this Forum… and so up until this thread it seems that no one has recommended any counter measures.
> I posted this because when I started Googling this vulnerability no
> recent results were returned except for discussions in this Forum… and
> so up until this thread it seems that no one has recommended any counter
> measures.
One of the challenges of discussing countermeasures is that those looking
to exploit a system tend to follow those discussions in order to learn
about the countermeasures being put in place in order to work to defeat
them.
I’m not a huge fan of security by obscurity, but there is also a concept
of making the hackers’ jobs easier by talking openly about how you plan
to thwart them.
Just like telling spammers how you detect them and ban them gives them
the opportunity to adjust their methods in order to circumvent the
detection measures.
That doesn’t mean we don’t have an ongoing discussion about how to
effectively deal with spammers. It just means we don’t have that
discussion where spammers are able to see it so all of our methods are
not known.
