Comcast Voices: Online Services

Happy birthday, Internet

2009-09-02T21:32:38Z

Depending on how you count it, today is the Internet’s 40th birthday. On September 2nd, 1969 a team at UCLA transferred some data from one computer to another using nothing but an ordinary looking cable. 40 years later and the Internet, along with miles and miles of cable, is an integral part of many of our lives.

I can still remember the first time I used the web. It was the summer of 1995 and I was with my friend Glenn in one of Iona College’s computer labs. I was just hanging around, waiting for Glenn to finish up when he suggested I amuse myself by surfing the web (remember, we still used that phrase in 1995!). Not having any idea what one was supposed to do on the Internet I decided to visit the Web site of the college I was soon going to be a freshman at (Lehigh, in case you’re wondering).

A lot has changed in just those 13 years since I first used the web, let alone the 40 years since that team at UCLA set in motion the events that eventually led to ARPAnet, which in turn led to the Internet. Comcast launched our first broadband product in 1996 with only a few thousand customers, now we’re the largest residential ISP with something like 15.3 million customers. A decade ago we offered a top speed of 1.5 MBps and people wondered what they would do with all that speed. Now, thanks to DOCSIS 3.0, we have speeds up to 50Mbps, though some people still wonder what to do with all those speed. Sprinkle on all that content that is available online (including video), all those people building a myriad of applications and services on top of the Internet and 40 years later the Internet is a very different beast. I’m certain those folks in that UCLA lab had no idea what was in store for the world when they set up their equipment.

Let’s take a moment out of our busy days and thank that team at UCLA for helping to give birth to this crazy thing we call the Internet. I don’t know what I would do without it (other than look for another job, since it is very tough to blog without the Internet).

What was the first Web site you ever visited and when? Tell us in the comments.

Smart Zone followup - updated

2009-04-08T13:22:51Z

First off, we’re sorry about this weekend’s SmartZone outage. We know that email is more and more important to people these days (I have been known to check my email on four different devices simultaneously) and we understand the frustration you felt. We’re truly sorry.

Now, the details: SmartZone suffered an outage on Saturday, April 4th, starting at 7:30 am EDT and lasting until 5pm EDT. This outage was caused by a power issue in one of our datacenters that impacted some of our email servers. The outage was compounded by the fact that several databases needed to be restored from back up (that process was successful, and no data was lost which is a good thing).

Over the course of the outage, mail that was sent to SmartZone customers was stored in a number of queues across our email system. Once the entire system stabilized we were able to start delivering that queued mail. Our engineers worked around the clock (literally) to identify email that was delayed, and to get it to the proper recipients. This process was completed earlier this morning, and now all email messages queued during the outage have been delivered (one important thing to note, however, is that the email messages will show up in your inbox stamped with the date they were delivered to your inbox, not when they were sent. For example someone might have sent you an email on Saturday at noon, and it was queued over the weekend so it shows up in your inbox as a new email today. If you look at your email headers, though, you’ll see the actual send and received dates).

During the outage there was a window of time (from 11am to 2:50pm EDT on Saturday) when our email directory was in a state of flux due to the various databases being rebuilt. There were some small cases (we think a fraction of emails sent) where someone may have sent you an email during these hours and our system was unable to recognize you as a user (because servers were coming back online). In each instance Comcast sent the sender of the email a ‘user not found’ message to alert them that their email message should be resent.

I know what you’re thinking: what are you doing to sure this doesn’t happen again? Our engineers are examining every piece of our email system to make sure that we are doing everything in our power to make sure an issue like this doesn’t happen again. We are reviewing all of our systems - hardware, software and processes — to make sure that we mitigate the possibility of this occurring again. Email is a critical means of communication for us, as well as our customers, so we know that it just needs to work.

Update 4/15/09 3:24pm: Lots of folks in the comments are wondering about whether or not our datacenters have redundant power, so I thought it would make sense to add this update. Our datacenters do have fully redundant systems, including power. That said, while we didn't lose power in our data center, an element of our email platform failed (some more details can be found here) resulting in the need to restore data from backup. I hope that answers some people's questions.

SmartZone Service Restored - Update 2

2009-04-04T23:55:48Z

Around 7:30 a.m. EDT this morning, the servers for our comcast.net email system, or SmartZone, experienced some technical problems which have now been resolved. This issue impacted customers’ ability to access the SmartZone system, which provides customers with their email and address book. This issue did not affect our customers’ ability to access the Internet. It’s important to point out that no customer emails were lost in this process, however it may take several hours for customers who were impacted to receive their old email. We sincerely apologize for any inconvenience this may have caused. Our engineers have been working all day to restore this service, which was our priority.

We’ve been providing updates to customers on web forums and Twitter throughout the day. Earlier today we told customers that we expected service to be restored by mid-day, however, the process took longer than expected. We thank our customers for your patience while we sorted this out and again apologize for the inconvenience.

Update 4/5/09 3:30pm: Earlier today, some SmartZone users were having trouble connecting to their email using POP (a protocol that some email clients use). Separately, some SmartZone users experienced delays in password resets updating. Both issues have been resolved.

Update 4/6/09 5:10pm: We wanted to provide an update on the SmartZone outage on Saturday. Again, we sincerely apologize for any inconvenience our customers may have had and to let them know we're taking this very seriously. Our engineers continue to investigate and today we're working with a very small number of customers on a one-on-one basis to address specific questions.

Our priority has been to bring the servers back online, which we have, and to ensure the email system is running smoothly again, which it is. We're conducting a root cause analysis and will post the details here as soon as we are able. Thanks in advance for your patience.

Speeding is encouraged

2009-03-30T17:55:08Z

For an Internet service provider like Comcast, speed matters… because it matters to our customers. A decade ago, we were offering speeds of 1.5 Mbps. That was considered fast. I remember people asking at the time, “What are your customers going to do with all that speed?” We’ve come a long way since then and have certainly found ways to use the 1.5 Mbps and much more.

New technologies and online trends are driving the need for increasingly faster Internet speeds. We offer a solution to that: DOCSIS 3.0, also known as wideband. With wideband, we are now offering customers 50 Mbps downloads, which is about as fast as anyone is going at this point. This means you can download a typical 1.5 GB standard def movie (or the equivalent of 400 mp3 songs) in about 4 minutes. On a typical DSL connection that would literally take hours.

We have already rolled out wideband to about a dozen major cities and have plans to reach more than 30 million homes and businesses by the end of this year. We started in the Twin cities early last year, but recently announced expansion into the Bay Area. We’re also in the Greater Boston Metropolitan area, the Greater Philadelphia Metro area and New Jersey, Atlanta, Chicago, Baltimore, Seattle, Portland and Ft. Wayne. We’ll continue to launch additional markets over the next several months.

While that is great news, 50 Mbps is just the beginning of what’s possible with wideband technology. Wideband will bring an entirely new platform to create the innovative tools and applications of the future, much in the same way that broadband spurred the development of many video-rich applications that have become mainstream today. Wideband will also allow significant evolution for application development that will be available to a large number of American homes.

This platform will have the capability to deliver speeds in excess of 100 Mbps and higher in the future. That’s a far cry from the 1.5 Mbps we were offering ten years ago and I think it begs the same question that we heard when we launched broadband, “what are our customers going to do with all that speed?” I’m eager to find out.