Sunday, January 07, 2007
Monitoring Remote Servers: Part 1
Posted on Sunday, January 07, 2007 11:02:59 AM (Mountain Standard Time, UTC-07:00)  Comments [1] | 
Categories: .NET | Web Services

Recently I've been having some problems with the performance of my web sites. I use Montastic  (free web site monitoring service) and it is reporting multiple outages almost every day. This does not jive with the 99.99% uptime that my hosting provider claims. I should note that I've got a shared server hosting package - $10/month for 200GB, with up to 2GB in SQL Server. Not bad, but I think they overload their servers.

When I contacted them, they say that everything is fine, and that the box I'm on is not overloaded. But I'm skeptical. The main reason is that my sites should be fast.

My main site is essentially static content that only uses ASP.NET for the master page functionality. This blog is running dasBlog, which uses Xml files for storage, and caches like crazy. ArcDeveloper.net runs on CommunityServer, and does use SQL, so there's a little hit there, but it also has serious caching. Overall, things are static, or cached - essentially they should not be a lot of "work" for IIS.

This leads me to think that the server is overloaded - but I needed a way I check what's happening on a remote server that I do not have desktop access to (i.e. I cannot connect to it with Perfmon). While I'm sure there are other tools/samples etc that can do this same thing, I thought this would be an interesting little project to play with over the holidays.

Performance Data + .NET

The key to getting system  performance information in .NET is the  System.Diagnostics.PerformanceCounter class. Using this, you can basically access any performance counter on the system. Pair this with a simple web service to collect the data, a windows client to make the requests and store the data, and I had a quick and dirty server load monitor.

The rest of this post will discuss the web service and the client. A subsequent post will talk about the results, and contain the source code.

ServerLoad Web Service

It's hard to make a simpler web service than this - the GetSystemData method simply returns performance counter information in the form of a MonitorData object. This first method just returns CPU untilization, and Available Memory. While very limited, this could at least tell me if the Montasic service was wrong, of if indeed there were some issues on the server itself.

    [WebMethod]

    public MonitorData GetSystemData() {

        float aveLoad = 0;

        float totalLoad = 0;

        float aveMem = 0;

        float totalMem = 0;

        try

        {

 

            PerformanceCounter cpuCounter =

                    new PerformanceCounter("Processor", "% Processor Time", "_Total");

            PerformanceCounter theMemCounter =

                    new PerformanceCounter("Memory", "Available MBytes");

            for (int i = 0; i < 10; i++)

            {

                totalLoad = totalLoad + cpuCounter.NextValue();

                totalMem = totalMem + theMemCounter.NextValue();

                System.Threading.Thread.Sleep(100);

            }

 

            return new MonitorData(totalLoad / 10, totalMem/10);

        }

 

        catch (Exception ex) {

            throw new SoapException("Exception occured in Monitor: " + ex.Message, null, ex);

        }                 

    }

Windows Client

Instead of storing the data on the server, I chose to have it stored on the client side. The windows client simply calls the web service on a set schedule, reports the duration of the request, and the performance information in a SQL database. Since I'm usually working with ArcGIS, it was fun  to write an app that can use the native .NET data binding. In minutes I had created a database, a table, a data adapter, and bound it into a datagridview. All from within Visual Studio, and without writing a line of code. Viva DataBinding!


 

Windows Client showing CPU and Memory usage

I after running this for a few days, I was seeing very similar things to what the Montastic service was reporting. The only difference was that my monitor would report a very long request when Montastic reported an outage. Which makes sense since any request that takes longer than 5 seconds should legitimately be called an outage.


The next step was to add in more counters so I could get a better handle on what was happening on the server. Specifically I wanted more information on the ASP.NET counters.

For this, I added a second web method - GetASPData, which returns an Array of AspData objects, instead of a single MonitorData object. This is a nicer model since I can add as many counters as I want, without having to change the database schema which stores the data.



Here's he code for this web method. One change I've been thinking of making is to pass the AspData objects into the service. This would allow you to configure what performance counters you want from the client side. Maybe I'll do this before releasing the source.

    [WebMethod]

    public List<AspData> GetASPNETData()

    {

        List<MonitorCounter> counterList = new List<MonitorCounter>();       

        try

        {

            //Create Counters

            counterList.Add(new MonitorCounter("Processor", "% Processor Time", "_Total"));

            counterList.Add(new MonitorCounter("Memory", "Available MBytes"));

            counterList.Add(new MonitorCounter("ASP.NET", "Application Restarts"));

            counterList.Add(new MonitorCounter("ASP.NET", "Applications Running"));

            counterList.Add(new MonitorCounter("ASP.NET", "Worker Process Restarts"));

            counterList.Add(new MonitorCounter("ASP.NET", "Worker Processes Running"));

            counterList.Add(new MonitorCounter("ASP.NET Applications", "Compilations Total","__Total__"));

 

            //Get data every 100ms, for 1 second

            for (int i = 0; i < 10; i++)

            {

                foreach(MonitorCounter m in counterList){

                    m.CheckCounter();

                }

                System.Threading.Thread.Sleep(100);

            }

            List<AspData> aspDataList = new List<AspData>();

            foreach (MonitorCounter m in counterList)

            {

                aspDataList.Add(m.GetData());

            }

            return aspDataList;

        }

 

        catch (Exception ex)

        {

            throw new SoapException("Exception occured in Monitor: " + ex.Message, null, ex);

        }

    }


On the client side, I went through the same steps as before, but as I noted above, instead of one record per request, I now have 1 record per counter. I store the counter name so I can run queries to pull out data for specific counters. The data shown below is from my development box, so it's got a really light load.



I'm going to push this latest version up to my site later today, and capture data for a while and then write up another posting with the results and the (cleaned up) source code.
Tuesday, January 16, 2007 9:03:01 AM (Mountain Standard Time, UTC-07:00)
Have you checked the times when your site is slow, is there any pattern? My though was that you could be affected by a back-up or similar process on the server, but in that case it should be on a fixed time each day/week.

Otherwise it sounds like a typical "bad neighbor" issue if you are on a shared server.
David
Comments are closed.