We're having a problem with certain ISPs DNS cache being stuck using an old IP for Gizmodo. T-Mobile wireless users are certainly impacted, and there appear to be at least one Canadian and UK ISP that are also not respecting TTL values.

On Friday the 18th around 5pm EST we fixed the mail settings on a server. Unfortunately it it turned out several thousand old emails related to the comments system had been stuck in the queue for many months. Once the configuration was fixed they all came flooding out, confusing users with emails about very old comments.

We're working on on configuration management and provisioning tools to make this kind of thing a non issue in the future. Sorry for the confusion.

We launched a few key improvements to the Gawker Media redesign.

Key updates:

• Story stream (right side module) now has a scroll bar to simplify navigation
• Comments should be more familiar: we've basically returned the functionality from the prior site design, though there is one difference. The top thread is still available as a separate promoted thread.
• Forums are returned to prior functionality, defaulting to expanded view.
• Numerous comment related fixes: expand thread should show what you expect — all approved replies. Slightly different for Star commenters.

There are many more improvements to come including comment notifications. You should expect this *very* soon.

Old news: Gawker Media has launched a significant redesign across most of our sites (this site will migrate soon). Beginning tomorrow, we will post updates daily (at least on weekdays) about feature additions, fixes, and other items of interest.

Information about the redesign has been posted on every site, but I think this lifehacker post covers it quite well.

In the next week, I will introduce "burners" in advance of their launch. Burners (thanks Anil) are disposable commenting accounts. We will introduce them in the next few weeks.

Q: At the time of the security incident, how were passwords managed?

A: Passwords were never stored as plaintext nor were they conventionally encrypted with DES. Only hashes of the password were stored. The algorithm and use of salt has frequently been misidentified. The two password-hashing algorithms in use were traditional DES-based crypt(3) and bcrypt. The released dump did contain plain-text passwords. These were all cracked and not stored in the clear. Use of the technically-inaccurate term, "encrypted" in an early statement made about the password system contributed to confusion about how password information was stored. More »

Misc new GBOTD bugs that have already been closed:

• GawkerBook XSS
• v
• minify.php local file disclosure
• minify.php local file inclusion

Joseph Bonneau, a PhD candidate at the Security Group, University of Cambridge Computer Laboratory, contacted us to report a problem he found with non-Latin character passwords (Unicode) on Gawker Media sites:

I discovered that, after creating an account with the password 'ДДДДДДДД', I was able to successfully log in by typing '簡簡簡簡簡簡簡簡,' as well as 'ႤႤႤႤႤႤႤႤ', '©©©©©©©©'. It turns out that any string of exactly 8 characters whose unicode code point is >= 128 will be accepted. I've looked carefully at the implementation of crypt in PHP and across several platforms I tried, this is not a library problem-somehow your server is converting all of the non-ASCII characters to some fixed value prior to calling crypt() with them. It is worth noting that 'ДДДДДДДx' is not accepted when 'ДДДДДДДД' is the registered password-so a check is still being done. However, if 'ДДДДДДДx' is the registered password then '©©©©©©©x' is accepted. The most plausible explanation I can come up with is that your code is mapping all non-ASCII characters onto some canonical character (maybe �), and thus ignoring the actual character value in the hash. I'd be
curious to see exactly how/where in your stack this occurs.

The issue was in jBCrypt, a library we use for password hashing, and is outlined here. The non-technical explanation is that this issue (outlined by Joe above) affects non-Latin characters (e.g Korean word for 'password': 비밀 번호), Latin characters with accent marks, and other characters that are not in standard English usage (e.g German: Füße).

How does this affect you? It does not affect most of our users — If you are not using non-Latin characters for your password, there is nothing to do (see wikipedia for more information on the characters that are not affected — US-ASCII). If you do use characters that are non-Latin, you should reset your password to ensure it is updated to fully support these special characters.

Joe did add one more comment: "I do think users are best to avoid non-ASCII though, since it's less portable." While is it not required, I do agree with him on this point. You can still create a very secure password using the US-ASCII character set.

As a side note, you should know that we do welcome suggestions to improve our platform. Joe is one of several to do so, and the suggestions are both taken seriously and much appreciated. Send your security related comments to security at gawker dot com.

Thanks, Joe.

A couple more reported issues were addressed this weekend. One provided access to error reporting information, the other "log poisoning."

Fixed: Error Reporting.


Fixed: Log poisoning.

This collection of issues are all related. These scripts are all available under the /admin/ folder, and are therefore related. This folder is secured using HTTP auth, and always has been (contrary to the speculation that it is not secured). GBOTD #4 identifies a bit of code that shows we left admin views to APC open to anyone that was able to successfully access the /admin/ folder hierarchy. The other two issues were scripts added as conveniences and were removed. The best kind of fix.

UPDATED to include more recent issues reported under Admin. Issue #8, #9, #10 and #11 are non-issues in that the code is not in production (much of what is listed in these reports was moved to an external project nearly a year ago). The code has been removed from the project trunk as part of housekeeping.

All of the following are resolved:

• Admin APC password
• Admin Arbitrary file disclosure
• Admin ci.php
• XSS - What is XSS?
• CSRF - What is CSRF?
• XSS 2
• phpinfo()

Mike Bailey has launched the site Gawker Bug of the Day as a platform to share issues he has discovered in our code base. SInce he is providing the platform to share bugs, we will feature here the fixes, or clarification, for each issue. Consider it a status update. Many are not bugs, and we will note where appropriate the difference.

The first issue reported is here: at.js.php XSS: "Unescaped user inputs are being echoed out to the page." A fix was released on December 16th.

The following email was sent to all Gawker Media staff on Friday, December 17th, 2010.

Everyone -

As you know, this has been the Gawker tech team's most difficult week ever. This note has been too long coming, but the following is meant to communicate several things: what happened, our current activities, and our plans for moving forward. I suggest you read all of this as I am making several recommendations below, and we are implementing some changes that will affect all of you.

What Happened
Gawker Media servers and some company email accounts were compromised by hackers at some time during the last few weeks; the compromise was made public to us (and everyone else) this past weekend. In recent weeks, intruders were able to gain access to our web servers by exploiting a vulnerability in our source code, allowing them to gain access to user data and passwords. With this information, they were able to gain access to the editor wiki, some Gawker Media email accounts, and other external resources.

It is clear that the Gawker tech team did not adequately secure our platform from an attack of this nature. We were also not prepared to respond when it was necessary. These things can be attributed to several factors.

First, we never planned for such an event, and therefore had no systems, or processes in place to adequately respond. Our focus as a team (and company) has been on moving forward. This put up blinders on several fronts. As a result, numerous wrong decisions were made by me this past weekend in responding to the security breach.

Further, attention to completed work is every bit as important as attention to upcoming work. Our development efforts have been focused on new product while committing relatively little time to reviewing past work. This is often a fatal mistake in software development and was central to this vulnerability.

Finally, we have not only seen tremendous growth as a company, we have never been afraid to take an unpopular or controversial stance with regard to individuals or organizations. Let's face it: we draw the ire of many. This creates a unique set of demands to meet rapid growth as well as threats that often specifically target us. We did not establish standards and practices to handle growth and the fact that we have a target on our back.

On several fronts — technically, as well as customer support and communication — we found ourselves unprepared to handle this eventuality. The tech team should have been better prepared, committed more time to perform thorough audits, and grown our team's technical expertise to meet our specific business needs. As a result of not having done these things, we have not adhered to standards expected of us, and our response was inadequate. The remedy to this situation will not be immediate, but it will be swift as possible.

Current Activity: Regaining Control
The tech team have moved our operation to the third floor of the Gawker Media office in order to focus on the work that needs to be done. We are currently in the process of performing a complete review of what happened with an independent security firm.

Here's what we've done so far to regain control:

We have been able to establish a fairly complete timeline of intrusion activity, and have identified compromised assets within Gawker. We have re-established control of compromised systems including our Google Apps accounts. As a result, you will have to reconfigure your Google Apps access (more on this below).

In addition, we have addressed all known vulnerabilities and will continue auditing our system for security flaws, and we have made appropriate changes to administrative accounts to our web and application infrastructure. There are many people reviewing our code base, and because of this, we will also reach out to members of the technical community to harness their expertise. This process will continue as we move to an entirely new, hardened web infrastructure.

We have introduced a help desk to address commenter concerns related to the breach. This will continue to exist as long as it is needed. Scott, Greg, Jeremy, Nick and a host of interns, and many of you, have been active in the threads, and communicating as much as possible as we work through this event.

Moving Forward
We've learned many lessons from this experience, both as a tech team, as a company, and as individuals. If there's one lesson nearly all of us learned, it's that we can and must be smarter with passwords. Lifehacker is a great resource for password advice (and there are many others). I suggest you start here: http://lifehacker.com/184773/geek-to-live--choose-and-remember-great-passwords.

Efffective immediately, we have enabled SSL, a more secure method of communicating over the internet, for all users with Gawker Media accounts on Google Apps (this does not affect your personal Gmail). Those of you not using web-based Gmail will have to reconfigure your clients (this includes any desktop mail client as well as other devices). The attached document provides instructions to make this easier, and includes information to configure different devices including iPhone, Android and Blackberry phones.

Also effective immediately: If you require access to sensitive materials (legal, financial, or accounting documents) on Google Docs, you must have two-factor authentication setup on your account. No documents will be shared with personal Gmail accounts. We are also strongly encouraging all staff to setup two-factor authorization even if you do not require access to sensitive material.

We will enforce a policy that sensitive information not be posted to the editor wiki. This policy will also apply to chat communications (e.g., Campfire, AIM).

On all of our sites, we will be introducing several new features to our commenting system to acknowledge the reality that we have lost the commenters' trust and don't deserve it back. We should not be in the business of collecting and storing personal information, and our objective is to migrate our platform away from any personal data dependencies (like email & password). We will push further integration of external account verification sources using OAuth (like Facebook, Twitter, and Google) for those that want to use them, and we'll also be introducing disposable accounts. Disposable accounts are similar to the service a pre-paid phone offers to drug dealers (a disposable, untraceable communication device). Commenters seeking anonymity will be able to do so confident that when necessary they can simply toss out the account and there will be no connection to the individual. They will work like this:
- no password will be stored
- no email will be stored
- account can be used as long as you have the key code; lose or delete it, the account is abandoned.

In addition, we are establishing a public Gawker Tech & Product blog (a long time coming) from which we will communicate product information as well as product plans to our readers. You can expect to see it by early next week.

This has been a very unfortunate event in Gawker Media history, and we have learned much from it. Above all, this has been an enormous inconvenience for everyone affected, and for this I apologize. You can expect a much more responsive and proactive technology and product team for 2011. You can also expect a much more public me — if there is one critical thing that has been missing, it is a lack of consistent communication from me. That will change.

Regards,

Tom Plunkett

The Gawker Media technology and product blog has launched. You will find here information of product plans, implementation details, security information, as well as a source for general announcements meant for readers and writers of Gawker Media sites.

Thanks for coming by.

- tkp