This is the first post in a new series that highlights some of the many customers that are using Panopta’s monitoring and notification service to improve their operations.   Harish Chouhan runs Webtions, a hosting and design firm in India, and has been using Panopta’s monitoring services for the past six months.

We are a small design agency that got into the hosting business mainly to offer complete services to our clients. We never realized the amount of time we would have to spend on server maintenance, support issues & the worst issue downtime. Because of budget constraints, having a team to check servers 24/7 was not feasible and without a proper monitoring solution, we knew about downtimes mostly from our clients who called us in the middle of the night. In most situations, by the time we looked into the clients complaints of specific services on the server not working, the issue was no longer there.

After desperately trying every single monitoring solution for two months we found Panopta.  The reason we choose Panopta at the end, was because:

• They had a great website which explained what they offered in a simple non-technical way.
• The time it takes from signing up to setting up rules is lot less than I expected.  Based on my experience of trying most other companies I expected to spend few days. I was surprised to have things up and running within few minutes.
• They allowed us to monitor not only the server’s uptime, but also internal resources such as memory & CPU.
• And most importantly, the different escalation rules that can be setup ensured that even if I am not available someone else from my team will always be able to know of server issues.

Also among the many reasons we decided to finally stay with Panopta was the way their control panel was organized. Most companies I tried either had a very unorganized user interface or they had too many features but none that made my task easier.

After setting up all of our systems in Panopta, with the same team size and same hours of work every day, we have been able to better manage our servers and have less downtime related support tickets.

We’ll be highlighting other customers on a regular basis, if you’d like to share your experiences please contact support@panopta.com.

It’s that time of year again, winter’s cold is over and it’s time to do some spring/summer cleaning across many facets of your life!  If you’re responsible for keeping your infrastructure online, I recommend you do the same for your monitoring environment.

We recently did an infrastructure audit for one of our clients and found that there were definitely some areas that could use some improvement.  With a few simple changes, we were able to streamline their setup and improve overall operations.

Based on the audit, here’s our top five recommendations of things to focus on.

1. Review your notification and contact settings.

Make sure that you have the right people configured for alerts, and have their correct contact information.  As phone numbers change or people switch smartphone platforms, notification configurations need to be updated or you will end up missing alerts.  People also switch roles and responsibilities through the year – make sure that escalation paths are updated to reflect this.

If you are a managed service provider, make use of auxiliary notification schedules to send alerts directly to your customers as well as your internal staff. You should also setup time to review contact information with your customers – it’s even more likely that you’ll have missed updates there and your proactive focus could win you props.

2. Look for “extra” servers.

A company’s monitoring setup serves as a master list of its infrastructure.  In reviewing yours, you might find several servers that are no longer in active use but could still online.  Not only do these add to the complexity of your monitoring, but they also cost resources to maintain.

Clean up and shut down any infrastructure you no longer need.  If you find servers that are still in production use but could be consolidated, queue up tasks to clean things up – you’ll not only save yourself money but will cut your overall energy use as well.

3. Review your infrastructure for unmonitored servers.

After you’ve removed any unused servers, do the reverse and make sure that any servers and other devices that have been added in the last year are setup in monitoring.  Even with the best deployment checklists you can miss a step – check now to make sure that you don’t miss a problem down the road because some servers weren’t appropriately setup.

4. Setup maintenance schedules for planned downtime.

In any complex environment you’ll likely have regular maintenance work that is done, either through automatic processes or by hand.  If this is triggering alerts that go to your team that don’t require any actions, you should setup maintenance schedules to silence the alerts.

For example, the client’s database replicants had been triggering daily alerts when their offsite backups ran and temporarily delayed database replication.  This was a known issue that their operations team expected and ignored because it wasn’t a “real” problem and would resolve itself as soon as the backups completed.  To avoid this, they put in place a maintenance schedule to turn off notifications during the backup window.  If problems persist after the window timeframe, they will get notified and can take action.

Remember, alerts should only be sent when there’s an actual problem that needs to be addressed. Sending alerts that people know to ignore is the first step to missing a critical outage and suffering from real downtime!

5. Enhance your setup with new monitoring features.

We continually add features to our monitoring system in order to provide more complete monitoring and better alerting and reporting functionality. We showed the client several new tools that they were not taking advantage of.  Alerts for their core infrastructure now are sent to their operations staff through Google Chat messages as well into their internal IRC chatroom.  They also added rotating contacts to properly route after-hours emergency alerts to our on-call staff, rather than waking everyone for every problem.

Take a look through our previous blog posts and announcements in the control panel for a summary of what’s been added, and feel free to contact our Support Team if you have questions or need assistance setting up something new.

The time is now!

We’re continually working to improve Panopta’s service to provide more complete monitoring functionality for our customers.  Our development team makes a release roughly every month with new functionality, bug fixes and improvements.  Last week the latest of these releases went live, with the following main improvements:

• Export of outage data.  Details on your historical server outages can now be exported as an Excel spreadsheet for analysis and and integration into other reports.  From the Outage History section of the control panel, look for the Excel icon and link on the top right to export.
• New outage alert emails.  We have overhauled the content for outage alert emails to make the outage information clearer and easier to interpret when you’re under the gun to bring critical systems back online.  The emails now include the full outage logs as well as easy-to-use links to acknowledge the outage and escalate or delay pending alerts.  The new alert emails are being rolled out incrementally – if you’re not seeing them currently you should be within the next week or so.
• Improved handling of DNS failures.  Our HTTP checks have been updated to generate alerts when DNS resolution fails.  Previously, we would cache DNS records for as long as the TTL specified, which could result in missed alerts when your web server was functioning properly but your authoritative DNS servers were having problems.  Now DNS results are cached for at most 30 minutes.
• Parent/Child dependencies for compound services.  You can now configure parent/child dependencies for compound services as well as servers.  If you’re not currently using dependencies, they can be quite useful for avoiding a flood of outage alerts if you have problems with your core network devices.  If your router goes down, you’ll immediately see that it’s the router that’s having problems rather than having to sort through hundreds of servers that also happen to be down.
• Maintenance Schedule UI improvements.  We streamlined the process of adding lots of servers to a maintenance schedule.  You can now select servers with the previous tree-based interface, or use the new search-based interface which lets you quickly select servers by partial name matches, across any number of server groups.

Plus the usual assortment of minor bug fixes and UI improvements.

We have a long list of additional improvements to come, in particular we’ll be working on a new version of our public reports and integration API in upcoming releases.

If you encounter a situation where you’re not able to fully monitor your infrastructure using Panopta, please tell us about it.  Many of our feature enhancements are the direct result of discussions with customers – as an example, the maintenance schedule improvements for this release came directly from discussions with a major CDN provider who is using our service to monitor their global network of POPs and customer sites.

Looking to find out exactly what the Panopta agent can do?  Want to know the best suggestions for how to configure your notification?  We’ve got your answers!

We’re excited to announce the launch of the new Panopta Q&A site at answers.panopta.com.  This site is filled with information about Panopta’s features and functionality, about how to best monitor your servers and network devices as well as general information on deploying and managing large-scale online infrastructure.

The site is open for anyone to ask questions, so if you can’t find what you’re looking for, ask away!  Find a great answer?  Vote it up so others will benefit from it as well.  Get started by introducing yourself and sharing your story about how Panopta has helped your team.

We will be adding two new locations to our monitoring network in Washington, DC (173.192.213.211) and Vienna, Austria (212.69.166.79). Both locations will go live on Wednesday, February 29. Please update your firewalls accordingly if you have IP restrictions in place for our monitoring nodes. For a complete list of our monitoring network addresses, see http://www.panopta.com/about-panopta/monitoring-network/

On February 16th we will be shutting down our Philadelphia monitoring location due to changes in our datacenter provider. All checks currently being done from Philadelphia will be moved to our Newark, NJ location. If you would like to have your checks moved to a different location, please contact our Support team to make arrangements.

We’re excited to be participating in an upcoming webinar with the folks at Gigaspaces, discussing how to enhance scalability and performance of e-commerce applications.  Included in the webinar will be a review of our 2011 holiday monitoring results, including examples of sites that performed superbly and others that suffered during Black Friday and Cyber Monday.

Details on the webinar and a signup link are below.

WHAT: GigaSpaces webinar on Scalability Solutions for Peak Retail Demand
WHO: Jason Abate, Founder & CEO of Panopta and Ron Anderson, GigaSpaces Director of Architecture
WHEN: Thursday, February 9 at 12:00 pm EST
WHERE: To register go online at: https://www1.gotomeeting.com/register/834220008

A monitoring system is only as valuable as it is dependable.  If you can’t trust that you’ll get alerted anytime something is wrong with your servers then your monitoring isn’t worth whatever you’re paying for it.  Our goal at Panopta is to let you relax in confidence knowing that if your phone isn’t going off, then your infrastructure is fine.  And whenever there are problems, you know right away.

Everyone knows that Murphy’s law applies especially well to the Internet.  Things break in all sorts of different ways, from an obscure bug in code that pops up to take things down to a complete power outage at your datacenter because a truck drives into it.  You never know what will break when, so you just have to plan for the worst and make sure your system can adapt and keep running as things go downhill.

This applies to monitoring systems just like everything else.  A monitoring system has one subtle but crucial difference from the typical website or online application that further complicates our design.  Most websites have regular patterns to their traffic, with peaks during the day when most of their visitors are awake, and deep hours after hours which are great for doing maintenance - short disruptions during these times have minimal impact and can often be done without raising the ire of customers (or bosses!)

The pattern of a monitoring systems is completely different, with a flat load across any time period (well, except for a steady upward trend as we grow!)  We perform the same number of checks in the wee hours of the morning as we do in the early afternoon, and our customers rely on us to continuously check their servers regardless of the hour.

We’ve designed our infrastructure from the ground up with this in mind.  Planning for every piece of the system to eventually fail has allowed us to keep our system running without disruption for the past four years, through billions of checks and millions of outages, and lets us accommodate regular maintenance by shifting load between pools of resources.

Every piece of our infrastructure is designed with redundancy in mind.  Starting with our monitoring nodes that are distributed around the world hosted in the datacenters of dozens of different providers.  Each one of these can (and do) fail at any time.  When this happens our central infrastructure detects the failure and seamlessly moves checks to a nearby neighbor, fast enough that the overall system never misses a check.  Once the node comes back, the checks are returned and the system continues on.  The same mechanism allows for rolling upgrades of monitoring nodes when we release new functionality.

In the event of a total hardware failure of a node or a sudden spike in demand for monitoring capacity, we can completely reprovision a monitoring node from a bare OS image in about five minutes using a single Fabric command.

Our central infrastructure, which runs our outage detection, notification, reporting and control panel applications, is run out of two clusters of servers in the Dallas and Seattle datacenters of our partner SoftLayer.  Configuration and monitoring data is replicated in both directions in near-realtime between locations, making use of SoftLayer’s secure and high-speed private network.  All applications are able to run in either location, which gives us the ability to withstand a partial or complete failure of either datacenter and to shift operations around as needed for maintenance purposes.

Fortunately, SoftLayer’s been quite stable over the past four years, and the cases where we have had to fail over primary operations from one location to the other have all been due to planned maintenance that was scheduled in advance.  But we sleep well at night knowing that whenever something unexpected does come up, we’re prepared.

Of course, this raises the question of who’s monitoring the monitoring system.  For that we are covered as well, with a separate system that is continually watching all of the core components of our system to ensure that they’re all functioning as expected.  Which means that all of our customers can sleep easy at night, knowing that we’ll continue to watch their systems, every minute of the day!

We’ve had a chance to run the numbers and now have the final results of our 2011 Holiday eCommerce Availability Index. Over the course of six weeks, starting right before Thanksgiving and ending on New Years Day, we tracked the downtime of all major online retailer websites, checking every minute to see whether holiday shoppers would be able to make their purchases or would be forced to head to a competitor.

After analyzing the results, some of the losers had downtime of more than seven hours in one instance, while the winners had zero downtime throughout the entire holiday season, showing the differences of those who are prepared versus those who were not.

Overall, online retailers who experienced outages around the beginning of the holiday season tended to fix their issues. There were three times the number of downtime minutes during Thanksgiving weekend than in all of December, showing that once retailers saw their sites going down, they quickly reacted, possibly by adding more servers and enhanced infrastructure. Site performance was much more consistent in December as compared to November, with one spike on Dec 23, the last day to order with overnight shipment that would be delivered in time for Christmas.

Over 50 sites had no downtime at all and had very quick response times, including Apple, Staples, REI, and Kohls. On the flipside, over 70 sites monitored did have significant downtime, including 22 sites with outages lasting more than three hours. Some of these include OfficeDepot and GameFly.

Compared to the 2010 index, the sites that performed well continued to do so (including the big three Amazon, Target and Walmart). Some that had extensive trouble in 2010 made a big improvement (JCPenney and Express both had 100% uptime and The North Face decreased its downtime by approximately 90% .

Not all improved however, OfficeDepot had nearly eight hours of total downtime, an improvement from 2010 but still significant, while Levenger improved slightly but still had over four hours of total downtime.

Based on Forrester’s estimate of $60 billion in online sales during the holidays in the US alone, all this downtime certainly translated to lost revenue.  If we make the rough assumption that these ecommerce sites represent 50% of the total, and sales are evenly spread throughout the holiday season (very rough approximations, I admit, but enough to get an order-of-magnitude estimate) then each minute of downtime corresponds to roughly $5000 in lost sales opportunity.  Certainly some customers will wait for a site to come back and return, but regardless there is serious revenue at risk.

Aggregating all of the downtime across all 132 sites, we recorded nearly 9 days of downtime, which translates to nearly $65 million in lost sales opportunities!

Below are the full details of the measured downtime for the sites we monitored, along with their 2010 numbers.  Interested in more details?  Email info@panopta.com and we’ll send you the full dataset from the study.

How did your site do over the holidays?  What worked (or didn’t) to prepare?  Please share your ideas in the comments!

Update: We have been in contact with TigerDirect’s IT team and confirmed that one of the outages we recorded was actually due to their servers blocking our monitoring nodes rather than an actual website outage.  The numbers below and on the holiday site have been updated to reflect this.

Click on any of the headers to sort the table.

Happy New Year from all of us at Panopta. Since you’re reading this in early January, I assume you’re back at work after what was hopefully a quiet and relaxing holiday season free of too many late-night outage alerts!

Hopefully  you got a chance to check out our Holiday eCommerce Availability Index, which tracked the performance and availability of all big online shopping sites throughout the holiday season.  We’re still working on crunching the final numbers, but it looks like November, namely black Friday and Cyber Monday, was rough for many of the online retailers (some with outages of over seven hours!) but December was a bit more quiet,  with fewer disruptions.  Watch our blog for the final writeup later this week.

2011 was a great year for us at Panopta, with lots of new functionality added to our system, thanks to the great feedback that we received from our customers around the world.  Compared to the Panopta of a year ago – we have new versions of our monitoring agent, new mobile apps for the iPhone/iPad and Android, the first release of our monitoring appliance as well as lots of smaller additions for more accurate monitoring and more flexible notification.  Our monitoring network also continued to expand, with our first locations in Africa and continued growth in Asia and North America.

We are looking forward to 2012 and what it has in store, namely a number of new features currently under development to help you better manage your infrastructure.

We’ll also be ramping up the information available on our blog this year, with insights into Panopta’s architecture and technical operations, details on advanced features and capabilities about Panopta that you might not know about, and a new series of customer profiles.  So check back often! Also, if you’ve been able to solve some of your infrastructure management problems using Panopta and would like to be featured, please let me know and we’ll get an interview setup.

As always, we love to hear your feedback about the service, the blog or anything else!