Chris Lee's WebLog

Your ISP as an advocate for quality of service

Winding down from an adrenaline high after spending the better part of 50 hours troubleshooting our ISPs network for them...

The problems - technical perspective:

1. Routing decisions on the internet have a financial component - often, the lowest cost route is chosen (that's lowest cost to the carrier, not you!)
2. Internet routing is brain-dead when it comes to performance; the quality of the route is not a significant  variable in routing decisions, so long as the route is 'alive' (or still warm...)
3. There are no ubiquitous quality-of-service standards - especially across carriers.
4. The internet as a whole lacks diagnostic (and self-correcting) capabilities.

The problems - business perspective:

1. The internet is becoming an increasing critical service to businesses worldwide, yet little is invested in addressing quality concerns;
2. Far too many people - including ISPs (big and small) - have a laissez faire attitude to internet problems.  The phrase "it's just the internet" allows ISPs to delivery shoddy service - imagine if your pilot said "it's just the engine".  And ISPs aren't the only ones to blame - it saddens me that most people take this poor service for granted.
3. The internal culture of ISPs reflects the above point - I was actually told yesterday, by a major backbone ISP, that "that's just the way the internet is".  After inquiring as the escalation status of the issue, I was informed that the VP of Operations is aware of the problem.  When I asked "what is he doing about it?", the response was less than encouraging - "we have to trust in our employees evaluation of the situation" - what kind of escalation is this?  How does this help the problem get resolved?
4. ISPs internal support protocols are flawed; they are self-serving, missing the target of resolving the problem.  The image of "trained monkeys" came to mind during this exercise - they can do A, B, C, D - and if you don't gift wrap it and hand it to them on a silver platter (i.e. solve the problem for them) they can't deal with it.  What happened to "taking a problem and running with it"?   Perhaps if there was more emphasis on quality there would be fewer support incidents, reducing the number of support staff and allowing for a higher calibre of support (a viscious cycle...).

So what is the solution?  Well, you could have multiple ISPs to mitigate the problem - but this seems to me to be spending money to solve a problem that shouldn't exist in the first place.  Unfortunately, if you require a certain level of quality you are forced into this.

No, the real solution is to ensure your ISP feels the pain of poor quality every time it occurs.  If you have a capable ISP, they will be an advocate for you with downstream problems.

Make sure that you:

1. Have a well defined escalation procedure, preferably worked out as part of the initial agreement;
2. Have an SLA in place to cover the oft-neglefted dimensions of performance and incident response time, in addition to availability.
3. Work with other customers of the ISP - there is power in numbers;
4. Don't by shy about keeping quality of service front and center - it should be always be the primary goal.
5. Stay away from shoddy service providers that claim to be Tier 1; they may have backbones, but can they keep them up and running?

Additionally, it would be nice to see the internet quality of service regulated - perhaps to the point where the Internet is classified as an "essential service".  The standards and enforcement associated with this will weed out the (far too many) deadbeat ISPs.

Announcing Cheetah

Well, since no one else is up to the challenge of creating an open-source NIO servlet container, we've launched the Cheetah HTTP Engine project. 

The primary goal of this project is to provide a high-performance, scalable connector for the Tomcat servlet container; the current implementation (still pre-alpha) provides support for NIO (Java 5+).

Just finished implementing "pluggable strategies" for tuning concurrency aspects around NIO; for example, one can choose to have dedicated threads to accept connections.  This is an important aspect of Cheetah, as cross-platform support for NIO often fails to address performance semantics.  Other pluggable strategies include: single or multiple selectors; pooled or synchronous event dispatching (for those interested in low concurrency, whoever you may be...).

Tomcat & NIO

So how come Tomcat doesn't use NIO? 

At first glance, the use of NIO seems an obvious choice.  After digging further, the water becomes quite muddied...

1. Threads do use system resources.  Modern versions of Java map each thread 1:1 onto an OS level trhread (or LWP - light weight process - for Solaris).  What are the implications of this?  Two major factors: a) context switching between threads and b) per-thread stack size cuts into available JVM memory.

So how serious are these problems?  For a light to moderately loaded Tomcat server, these are not major concerns.  When you start talking about many hundred of concurrent (by concurrent I mean truly concurrent - active connections), these factors can degrade performance (at best) or cause system outages (at worst).

Consider the following scenario (this is based on a recent real-world experience):

a) A Full GC runs - say it takes several seconds;
b) Incoming connections are 'paused' as the Full GC completes;
c) Once the Full GC is complete, the incoming connections spawn new threads (assumes the existing threads are all connected - not necessarily busy);
d) The new threads each allocatememory for their stack; on a Solaris server, the default stack (for 32 bit mode) is 512k (for 64 bit mode it is 1MB).
e) If the memory to be allocated is not readily available, a Full GC will be performed to find the memory;

You can see the cyclic / cascade effect this will have - in our real-world situation, this caused the server to eventually do nothing but Full GCs.

So how do we work around this? 

a) Reduce stack size; 128k seems to work quite well with JBoss/Tomcat, though the exact size depends on the nature of your application.
b) Limit the maximum number of concurrent threads; this means that some connections cannot be established, but the server will stay running;
c) Tune garbage collection to reduce pauses; have had good luck with the concurrent mark-sweep GC with a 2GB heap. 

Note that these are only work-arounds - not happy with the fragility of the system under the noted circumstances.

The second point - context switching - is a minor concern.  This will certainly cause degraded performance, but as to whether the degradation is evident / material in delivery speed is debatable.  Certainly at extreme system load this will cause a problem - but if you routinely run your systems at/near capacity, your are asking for problems anyway.

       2.  Integration of NIO and Tomcat /Servlet API seems challenging at best.

The event-driven model of NIO seems to be at odds with the blocking model that the Servlet API and Tomcat implementation depend on.

       3.  Other vendors use forms of non-blocking IO! Weblogic, for example, has had native performance packs for years that use poll() and a small number of threads to read HTTP requests.  Recent versions (8.1) contain a NIO multiplexer as well.

Additionally, there are add-on products - such as Engine/J - that add NIO capability to Tomcat.

I can't speak for the effectiveness of Tomcat add-ons, but have had a lot of success with WebLogic in the past.

See IBM - Servlet API and NIO for a good article on how this can be accomplished.

So where does this leave us?  Certainly it would be challenging to retrofit Tomcat to use NIO - but clearly not impossible.  I believe that adding NIO support to Tomcat would make it a true contender in the enterprise arena.