Official Gmail Blog
News, tips and tricks from Google's Gmail team and friends.
Gmail back soon for everyone
February 28, 2011
Posted by Ben Treynor, VP Engineering and Site Reliability Czar (24x7)
Imagine the sinking feeling of logging in to your Gmail account and finding it empty. That’s what happened to 0.02% of Gmail users yesterday, and we’re very sorry. The good news is that email was never lost and we’ve restored access for many of those affected. Though it may take longer than we originally expected, we're making good progress and things should be back to normal for everyone soon.
I know what some of you are thinking: how could this happen if we have
multiple copies of your data, in multiple data centers
? Well, in some rare instances software bugs can affect several copies of the data. That’s what happened here. Some copies of mail were deleted, and we’ve been hard at work over the last 30 hours getting it back for the people affected by this issue.
To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they’re protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it’s taken us hours to get the email back instead of milliseconds.
So what caused this problem? We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version.
As always, we’ll post a detailed incident report outlining what happened to the
Apps Status Dashboard
, as well as the corrective actions we’re taking to help prevent it from occurring again. If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered.
Thanks for bearing with us as we fix this, and sorry again for the scare.
Update (3/1/11, 12:20 PM PST)
: Data for the remaining 0.012% of affected users has been successfully restored from tapes and is now being processed. We plan to begin moving data into mailboxes in 2 hours, and in the hours that follow users will regain access to their data. Accounts with more mail will take more time. Thanks for bearing with us.
Update (3/2/11, 10:55 PM PST)
: Gmail should be back to normal for the vast majority of people affected by this issue. If you are still experiencing issues, please contact us at the temporary alias we've set up for this particular issue, gmail-maintenance@google.com. Thanks again for bearing with us.
Labels
buzz
calendar
Gmail Blog
Google Apps Blog
Google Calendar
googlenew
Inbox
Inbox by Gmail
labs
mobile
Offline
reader
tasks
tip
Archive
2016
Sep
Aug
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Aug
Jul
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Jul
Jun
May
Apr
Mar
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2007
Dec
Nov
Oct
Sep
Aug
Jul
Feed
Google
on
Follow @gmail
Follow
Give us feedback in our
Product Forum
.
Get posts via email
Email:
Powered by
Google Groups
Useful Links
About Gmail
Gmail for Mobile
Gmail for Work
Follow