A reader recently asked me to expand upon the "Keep it Simple" advice in the fundamentals section of my web site. Just like my advice, I had kept this section pretty lean...

Keep it simple

Anybody can build complex software. Creating simple software that can accomplish the same task is much more difficult. In the end though, the simpler it is, the easier it is to test, which makes it easier to maintain, and easier to extend.

To illustrate the idea, I'll talk about a project that I'm working on right now. Basically I'm helping / mentoring a team through a short term project to build a data viewer application based on Virtual Earth, Dojo, ArcGIS Server (9.2) and ASP.NET.

One of the user stories in this project is:

As an administrator I want the Data Viewer to be database agnostic so that I can easily configure it to pull point data from SQL Server, Oracle or file based data sources.

Now, this is a great idea, and certainly possible. In fact it ends up being very interesting to design (plug-ins, configuration sections and Inversion of Control oh my!).

Simple works...

In looking at the project (it's very short term) and the team (new to web development) and the current problem they are trying to solve (display data from SQL Server), my recommendation to the team is that we "keep it simple" for now, and just address the SQL Server data source.

While we could define a set of interfaces, build out a whole configuration model, and create the actual data access objects for these various data sources, in the context of this project it's extraneous. To be "successful", we need to display the current data in Virtual Earth. With tight time frames and limited resources, I believe the tradeoff between building for an unknown complex future need, and applying time and effort to the core objective, is a good one. The team will be able to get the database access tier of the system completed in a short time, and it will be very simple to understand - parameters are passed to a class, which calls a stored procedure, which returns the point data. Simple, clean, and working.

But what about the "can accomplish the same task" bit? Good question - I'd suggest that in this situation, with the current data is in SQL Server, you are doing the same task - just using a simpler design to achieve the goal.

Add Complexity when it is Needed

If / when a need arises to add data from Oracle, it will be a very simple task to create another class which mimics the SQL Server class, with the only difference being that it uses the Oracle data connection classes. Should this Data Viewer become a huge hit, and is used so extensively that it must become configurable by non-developers, it could then be refactored to add in the additional complexity required to support configuration sections and a plug-in data provider model. But for now, the much simpler solution works just fine.

So - to summarize: Make choices which trend towards keeping the code as simple as possible. Don't add extra features or design elements to handle unknown or fuzzy future requirements. Add complexity only when it's really needed. While it's fun to monkey with internal design optimizations etc, keep your eyes on the prize: software that works for the user. And keep in mind who will be maintaining the software over the long haul - while you may be a whiz at generics (for example) keep in mind that for many people they tend to make understanding the code more difficult.

That's pretty much my take on this, but I'd love to hear your thoughts on keeping software "simple". How do you decide when to add complexity? How do you define complexity?

Other thoughts on Simplicity:

Joel On Software: Simplicity

The Fourth Law of Software Design: Complexity vs. Ease of Maintenance

Various Quotes on Simplicity in Software Design

Phil Haack posted a comparison of MVC and MVP patterns with some zen on how these patterns work in the web.

Based on the outcome of the Developer Survey, I thought I' put together some posts on the common patterns referenced in some of the questions, starting with the Model View Controller (MVC) family.

There has been plenty written about MVC on the formal side of things - here are two good sources of information - Wikipedia MVC entry,  and some thoughts from Martin Fowler.

The Basics

The MVC family of patterns are about separating the presentation of data from the business logic surrounding the management of the data. There are many other patterns based on the key MVC ideas - Passive View, Supervising Controller, Presentation Model and Separated Presentation are 4 such patterns. The differences are subtle, and all share they key idea of separating the presentation logic from the business logic, from the data itself. It's worth noting that the opposite of MVC family is called "Autonomous View" in which all the logic is in the form itself - so if this is how you are developing, now you can use a fancy name for it!

Since MVC was one of the first patterns that dealt with this separation, we'll start with it, and look at the three components...

Model:

The model is the data and it's usually a class. So in GIS-land, it could be a "Parcel" class, which may have a Geometry property. The important part is that this class has no dependencies on the presentation layer, or the controller. It's just a dumb data container. Typically this class will implement one or more interfaces, which the View and Controller will utilize. Interfaces are used a lot in these patterns, so here's a link to an article that compares Abstract Classes and Interfaces that can serve as a refresher if needed.

View:

As the name suggests the View is responsible for organizing and presenting the Model. Usually this means showing it visually on a form. In order to remove dependencies between the View and the Model, the View will interact with an interface, instead of a specific view class - i.e. IParcelFeature instead of "Parcel". The view typically raises events when the user makes changes in the interface, and exposes properties allowing the Controller to "see" the model or other state of the view. The View does not make any changes to the Model itself (in some  MV* patterns does not actually have visibility of the model - the controller sets all the control values directly, and responds to all control events). Again, the events and properties of the View are defined via an interface.

Controller:

The name conveys things pretty well - this is the business logic component that controls what actually happens to the model when an event occurs in the view. There may be multiple controllers that can work with a given class, or multiple model classes that can work with a given controller. This is where the Interfaces on the Models come in. If the Controller's "contract" is based on an Interface (IParcelFeature), then we can use the Controller with any class that implements the interface. Similarly, the Controller also implements an Interface, which is it's contract with the View.

Pulling it to all together:

In MVC, the Controller gets a model and a View. It then passed the required data to the View which renders the UI. When the user interacts with the UI (i.e. clicks a button), the View raises events, which the Controller responds to by making required changes to the Model, which is then used to update the View.

Why Bother?

So - this is nice, but why separate things out like this? There are a number of reasons. First is known as Single Responsibility. This concept basically says that a class should have one purpose, and anything not connected with that purpose should be in a separate class. In regards to the subject at hand, it would tell you to avoid mixing presentation code, with business logic because they are not actually related.

Another reason for MVC is re-use. If you model your application like this, you can plug and play the components - particularly the View. So, suppose you need to create a business application that has a windows forms client for some functionality, a web client for some other functions, and a web service API. You can build out a model, a controller, and three views - one for the windows forms application, one for a web form, and a third (more limited one) for the web service.

A final (and biggest) reason is testability. If you keep the Model (data) separate from the View (User Interface) which is separate from the Controller (business logic), writing unit tests becomes much easier. If you've implemented interfaces as the contract between the components, you can test the components individually.

The Pain of Testing

Without getting too deep into unit testing, the basic idea is to have a test harness which spins up various classes in your code, and executes methods. This gets complicated because our classes have dependencies on data - stored in databases, shape files, whatever. If we want to re-run our tests frequently, we want to be sure our test data is static - otherwise the tests break, and that defeats the purpose. So - the next best option is to create "fake" static data in our tests. The brute force method would be to create specific test classes which implement the required interfaces (ISomeView, ISomeModel etc), but just contain enough logic to complete the test. Depending on the test you would have an invalid geometry, invalid PIN number or a myriad of other conditions. This would allow you to create test "Parcels" in your unit test harness, that you could then pass to the Controller to test it's methods, without actually connecting to a shape file. You could also create a class that implemented the View interface, and use that to raise events and ensure that the Controller reacted correctly. Essentially you can write unit tests for everything but the "click" event handler in the UI layer. Since that code was likely written for you by the IDE, and it's super basic, it's not a really high priority for testing.

Easing the Pain of Testing: Dynamic Mocking

Of course that's a lot of work, and developers hate extra work, so tools were created to help out. While this is beyond the scope of this post, I'm talking about "mocking" frameworks, such as NMock, RhinoMocks or TypeMock. These frameworks can create a "test dummy" class that implements a specific  interface right in the test definition. You basically "record" some values into the Mock, then pass it into a method, and it can validate behavior.

Of course you still need to fill up the dynamically mocked Interface with valid data - and the geometries are likely the scariest thing to start with. So that's where I started a few years ago.

I can haz code?

I don't have anything lying around that's easily sharable, but I did post a sample about Supervising Controller code about a year ago. The code is in VB.NET, with a Visual Studio 2005 solution, and is oriented towards showing how to create a plug-in framework, while leveraging Supervising Controller, so that's the gist of the article. It also does not have any tests, but I'll be posting more leveraging MVC patterns and unit testing in the future.

Plug-in Framework & Supervising Controller Example (Article - read this before digging into the code)

Download Sample Code

Summary

The idea here was to introduce the main idea of the MVC pattern - separation of behavior logic from presentation logic, usually to facilitate testing. The actual implementation of this idea can take many flavors, and can vary in complexity, but the underlying concept is the same.

This is the third and final part of a three part series summarizing the results of the 2008 Geospatial Developer Survey. For further information, be sure to check out read part one and part two. 

 

Use of Source Control Systems

This is where we start to see some divergence from the larger developer community.

 

Source control is as much a part of most developers lives as the language they use. It should be automatic and used by default. Subversion is good, and free. It's not trivial to setup, but with free subversion hosting services like Assembla.com, there is virtually no reason not use it. (btw - although I have an "Assembla" badge in my sidebar, I am not paid to endorse them - their service is solid and so I promote it).

Automated Build Tools

With more than half of the respondents not doing automated builds, we are seeing more of the separation from main-stream development.

All the choices on this question were .NET, but I also compiled the "Other - Please specify" results into a chart.

 

Unit Testing

There were 3 questions on unit testing, mainly because I see this as a key element raising the quality of GIS software projects. Unit testing is as much a philosophy about how you develop software as it is the actual writing of tests. Committing to doing unit testing really says you care about quality, and that you realize that manual testing, while valuable, is never as consistent as automated tests. So, with that we start with a look at what percent of projects are using unit testing.

I don't know how this stacks up to other industries, but I'd say there is room for some growth here. Everyone talks about GIS getting on the "service bus", but we need to make sure that when we get on the bus, our code does not fail!

The next question basically asked why you were not doing more unit testing.

This is actually pretty good. I'd have to question the 14% who don't think it's valuable, but I'll chock that up to them not knowing what's really involved.

I've heard the "not-enough time" thing before, but the problem is that you will have bugs. And someone will find them. Better the developer finding them sooner than a client finding them later. Also, in terms of time, on big projects, I think you actually save time.Having automated tests, helps you prevent regression. Think about all the stuff that breaks between a big ArcGIS releases - that's regression, and that's what good automated tests can catch.  

Difficulty. This is true. Writing good unit tests is hard. Using good design patterns help, and good tools can ease the pain, but even still it's not easy. Writing unit tests for ArcObjects code is very difficult, but not impossible. In these cases, I focus the tests on the most critical parts of a system - basically putting the effort where it will payoff the most. For these areas, I shoot for 100% code coverage - if they are the most complex, I want to make sure that my tests cover all possibilities. Again, difficult, but if you are writing a complex spatial rule engine, that works with relationships between multiple layers, business workflows, and user permissions, there is no realistic way to manually test it on any sort of a regular basis.

The next question was "If tests were easier to write, would you write more of them?"

So, a resounding YES on that. This is great, because as a community we can actually make this happen. 

A couple years back I threw some code out there under the umbrella of "ArcUnit" - the files referenced in those posts are now gone, but it was some demo code that showed how to unit test geometry operations, by serializing geometries and storing them in the unit test assembly. The code is now on Assembla in the ArcDeveloper project, under TestingUtilities. Seeing as there is interest in this, I'll likely write some more articles on this.

Unit Testing Frameworks

Ok - the Java people really ripped on this question. After adding an "Other", I did tabulate the results for all that as well.

This gets somewhat interesting - 60% of people said they are not using any tools, but back in the "What percent of your projects use unit testing" only 38% said "None". I guess some "homegrown" unit testing could explain a few percent, but... anyhow. Interesting that MBUnit and MSTest are so low. MSTest is baked into Visual Studio 2008 (Professional and above), and MBUnit is both free and awesome.

As for the "other", here's how that stacked up.

Apparently I could have saved myself a heap of greif if I'd just added jUnit to the list!

The final Unit Testing question went out to the edge - "What do you think of Test Driven Development"

So at least one zealot in the crowd aims pretty high! For the "No opinion" crowd, TDD is the practice of writing your unit test before the implementation. The intent of this is to essentially define the desired behavior in the test, then make the software behave in that way. No doubt this is difficult, and requires you to be a total unit testing & pattern guru. I *think* this is possible, even on complex projects like GIS, but you'd need to be working with a test friendly GIS framework (as in don't come asking me to build you an ArcGIS Server .NET ADF site using TDD!)

 

Code Refactoring

As you write code, there are times when you need to change it's structure - usually to conform to a pattern, to facilitate extensibility, or just because your function got long and messy (i.e. you're making maintenance easier!). This is called refactoring. The first question was related to how often you refactor your code.

The second question related to the use of .NET Refactoring tools. I appologize again to the non-.NET people, but I don't know of any Java Refactoring tools.

What's interesting here is that the "Not Refactoring" people are not even using the tools in Visual Studio. When we were doing a lot of desktop development, we used RefactorPro - mainly because they had good Visual Basic.NET support. Now that we're using C# and doing 99% web development, we are finding that the IDE tools / manual refactoring is sufficient.

Code Documentation Generation

This was a quick question on the use of automatic code documentation generators. They are pretty common in the larger developer community, and relatively easy to bring into the mix, and spit out some good API documentation. Worth the effort in my opinion.

 

Comments

Fifty-Two respondents left comments at the end of the survey. Most were constructive, and others... well... lets look:

WTF, no C++ on the list of languages? What language do you think the bulk of geospatial software (from GDAL to ArcGIS) is written in? C# is just a evil marketing ploy from MS to try to deflate Java's market; nobody is expected to actually _use_ C#.

Reply: Not having C++ was an oversight. As for your issues with C#, can't help you there.

Summary:

Overall, this was an interesting experience. Clearly I could do a much better job on the questions, and have a much wider range of options on the answers. I think it does show that geospatial developers are adopting main stream techniques and tools, just at a somewhat slower pace. Thoughts?

Download the results (Excel 2007 format)

Over the years I've seen a few "developer" surveys put out by various GIS organizations, and they have all been pretty bland. After talking to a few people at the Developer Summit about this, I thought I'd see if we could do a little better - thus the launch of the 2008 Geospatial Developer Survey.

Why are you doing this?

The main reason is that I'm interested in knowing more about how other people doing GIS/Geospatial development work. How many people view themselves as software engineers vs. occasional coders who hack some stuff together as needed. I want to know if people are exploring and adopting the tools, patterns and practices that are common place in main stream software development. And if not, why.

And beyond my curiosity, I'm also looking for some insight as to what topics people want to know more about. Although I have some ideas, lets hear from you. I get a fair number of requests to write more "fundamentals" type posts, and at the same time get requests for more on patterns, Inversion of Control and Unit testing. Since I don't write this blog for a living, I need to focus my efforts, so I hope that this survey will help quantify which way I should lean in future postings.

Who should take the survey?

Anyone who writes any code that is geospatial in nature. From the Google Maps Mashup dudes, to those running some geoprocessing tasks with python. If you write code that has anything to do with a map, you're the perfect person to take the survey. Be sure to pass this url around to everyone you work with who is a geospatial developer - the more data points we get, the better. Who knows - we may even be able to influence the content at some conferences...

How?

It's simple - just follow this link to SurveyMonkey, and fill out the questions. Yes there are 30 questions, but they should be pretty quick - I'd guess no more than 10 minutes.

What Next?

I'll be leaving they survey open until May 9th 2008. Then I'll crunch the numbers and do a post or two about the results.

Link (for copy pasting to all your developer buddies):

http://www.surveymonkey.com/s.aspx?sm=cugzupf5rAG_2bIu5WJ_2f7rxQ_3d_3d

UPDATE 4/8 @ Noon: After recieveing some feedback on the questions, I have added more "Not Applicable" and "Other" options.

Official Line: "The .NET ADF is a hybrid platform - mix of server side and client side development environment."

Built on ASP.NET Ajax, but still supports the ASP.NET Callback model for backward compatibility.

As a quick aside - Art Haddad was running a Mac...

Core object model is relatively the same as at 9.2

Core controls are now scriptable - they work with the MS Ajax client side libraries. The javascript is now "public" and documented. This is great - in the past we had used some of the client side javascript (not supported) and found that it changed across service packs - doh!)

Common Data Source API - standard access across the spectrum of data sources. The common data source API will expose different functionalities based on the type. You can get down the the ArcObjects if need be *I'm assuming this is the same model as at 9.2 but they were moving through stuff very quickly)

This ADF has a Visual Studio design experience that provides an option to do things without getting your fingers in javascript. That said, the javascript is public, so just like the Ajax Control Toolkit, you can just jump in and use it like the "Javascript API" (which brings up the fact there are two javascript APIs - one based on Dojo and one based on MS Ajax - choice is good).

Task Framework is now extensible in the Application Manager via web controls (dlls). This will allow you can publish new tasks, and users can then include the tasks in a published application. Nice feature if your environment is involved in the author-publish-consume workflow.

They are releasing tasks into the code gallery, will allow a faster release cycle.

More "comprehensive" class library documentation, discussions of Ajax patterns, migration from 9.2 to 9.3. Lots of work on the Resource Center to help people get rolling quickly.

Can create user controls for the task frameworks (ascx) - much simpler than creating full blown web controls in dlls.

Performance

Blending at 9.2 used a single tiling scheme. At 9.3 each resource has it's own tiling scheme, and the "blending has been massively improved"

Use of HTTPHandlers used instead of Pages so the page lifecycle is avoided, so it's faster.

Javascript is public, documented, JSON based, and object oriented.

Better activity indicator so you know that something is going on. 

Demo of 9.2 vs 9.3 performance. Definitely much faster. Eliminated lots of callbacks, httphandlers improve performance because the control tree is not re-created and destroyed.

Webcontrols

MapControl - scriptable directly. This is provided by implementing IScriptControl, and just works. The overview map had been improved a lot. The logic has been streamlined, so it's much simpler. You can enable the "intelligent" mode. At 9.2 there were 12 callbacks to get the overview. At 9.3 it's 4 and then it's just client side.

The resource manager can be managed in javascript, and has lots of configuration - i.e. layer alises, fields to show, map tips etc.

Results Viewer - bi-directional high light. The task can have different results behavior (fields, maptips etc)

Additional web controls, additional AjaxExtenders - DockExtender, HoverExpandExtender (pin window type of thing)

Supports .NET 3.5, but not required.

By final, there will be javascript intellisense for Visual Studio 2008 (you likely need to get a Visual Studio service pack as well as the ADF)

Better templates, but if you use these, be sure to send feedback to the team during the beta to get your features in the list.

To use the "Client side geometry model" - Just set the graphics layer RenderOnClient property to true. Sweet.

Automated migration for the Manager applications (author-publish-consume  three-click workflow)

There is a utility to help convert the template apps to 9.3

To do it manually - add reference to the System.Web.Extensions.dll, upgrade assembly version reference. Script manager is not required. If you want it to work with the Ajax partial postback model, you need to add a script manager.

Can use an update panel with non-Ajax controls and use triggers between the .NET ADF controls which are already Ajaxified.

Do not put the .NET ADF Controls in an update panel. This will kill your performance.

 

Questions:

Client geometry sent in JSON... GeoJSON, same JSON as REST API? It's ADF JSON - basically serialized versions of the server side .NET classes. Standard .NET design pattern

Have you worked with the controls in a javascript UI layout framework like Dojo, ExtJS, or YUI? They have done some testing with YUI components, and it's fine. All the client side code is namespaced, so there should be few issues.

Compressed / minified Javascript? By final

Thoughts:

Sounds like we can work directly with the MS Ajax Javascript API, which I personally like. The hybrid approach is very compelling for many apps which need some form of server state, but still require a very responsive interface.  Use of the ADF vs the REST/Javascript API may come down to a question of licensing.

Yesterday I focused on the REST, and Javascript API's, and today's task is to get the skinny on the re-vamped .NET ADF.

In talking to some folks, apparently the .NET ADF includes a Javascript API based on the Microsoft Ajax client library. In contrast to the relatively simple Dojo based API, apparently this one has more depth, and with that (one would assume) comes more fine grained capabilities. So, we'll see what some pointed questions can drag out.

At a high level, I think that the .NET ADF may have a tough sell moving forward. The REST stuff is clean and elegant. It may be that on paper it can't do as much, but I believe that there are creative solutions that can step around any limitations. Secondly, everyone I've talked to who has used the 9.2 .NET ADF, well, I'll be charitable and say they are "luke warm" on it. The difficulties they faced in getting good performance, and implementing seemingly simple functionality is what will drive them to try the REST API before the .NET ADF. And if the licensing story for the .NET ADF remains the same, and the REST API is gratis, well, money talks.

I hope that the 9.3 ADF does more than support the ASP.NET Ajax "extender" model (this is how the Ajax Control Toolkit works). From my work with the toolkit (which has been restricted to the ModalPopupExtender and the ListSearchExtender), it's a nice model if you really want to keep your fingers out of the client side code. The difficulty is that since the extenders are setup as part of the page lifecycle, they seem to need everything they interact with to be "runat=server". For reasons I'll skip right now, this is not always easily done. Since all this behavior is implemented in the client, theoretically you can wire up events and use the client side code however you like, but in the Control Toolkit, this is not exactly documented (please send me details if you know otherwise!).

Getting back to the .NET ADF, I hope that the Javascript libraries are designed to be directly consumed and manipulated. Personally I like the Microsoft Ajax client library, and working with server side services is just a slick developer experience. So - I have high hopes.

I've been diggin in deep this week - at 4:30pm on Friday this was my desktop...

(click to view full res -500k)

I've got the ArcDeveloper REST project up, a testing site where I'm furiously mixing the Dojo Framework with MS Ajax, and a DotNetNuke module where I'm bringing it all together...

This is just the start, but I've got Virtual Earth in a Dojo Split Container, (there will be a Dojo Tree in the lovely light blue area) with a Dojo Toolbar (icons to be changed). Not shown but also working are some Dojo dialog forms. The orange blob is a feature brought into VE via the ArcDeveloper.net REST service. I still have to wire in some point clustering, and the tree view, but given it's been a productive week.

I've been doing some work with SharpMap v0.9  (from CodePlex) for a windows forms based mapping tool, and while there are quite a few samples which show how to do basic mapping stuff, it was very difficult to find out how to locate features using an attribute query.

While SharpMap has a very obvious method for doing spatial queries -

IProvider.ExecuteIntersectionQuery(Geometry, targetDataSet)

There is no similar method for attribute queries.

After scouring the SharpMap discussions over at CodePlex, I initially resorted to a bit of a hack - I'd get the FeatureDataTable from the SharpMap.Data.FeatureDataSet, cast it to a System.Data.DataTable, and create a DataView with a filter. This worked for showing the attributes in a grid, but I had no way to get the Features because they are not stored in the DataTable.

I decided to put this on hold for a while and work on some other aspects of the application, and while debugging I was stepping through the ExecuteIntersectionQuery code in the ShapeFile provider and saw this:

What was this FilterDelegate? A little searching in the source and found a long comment section by Morten that describes how to use delegates to filter the results returned in a DataSource. You can view the highlighted source here (by the way Koders.com rocks for their indexing of open source code!)

Since this was buried, I thought I'd show a little code on how to do it. But first - if you are not familiar with Delegates, they are essentially pointers to functions.  Here are some resources: Wikipedia description, how they relate to events, and MSDN article on them.

Here's how the filter works:

As the ExecuteIntersectionQuery is looping over the dataset, it will pass the current row into the delegate and add the row to the results if the delegate returned true. Once you dig into the source code, it's actually pretty clear what's going on, but it would have been nice for this to be exposed in a more obvious manner.

Implementing A Filter 

The straight-forward way to do this is create a static method that do the evaluation and assign that method to the SharpMap.Data.Providers.ShapeFile.FilterMethod . The code snippet below shows the IsOwnerNameSet filter being used. 

 

This works pretty well if your filter criteria are static - like "is the ownername set?" But what if your criteria are changing? Since you can't pass additional arguments to the delegate, the other option is to create static member variables that the delegate will use to determine that match. This has a bit of a code-smell to me, so I ended up using Anonymous methods. In the code snippet below I am creating the anonymous method, and at the same time defining the value it is checking against. In this case I'm searching for a parcel by owner.

 

This allows you to specify the value (owner) without resorting to static members.

Hope this helps people just jumping into SharpMap

While it's main purpose is to provide an Inversion of Control container, and allow you to develop applications that are highly cohesive yet loosely coupled, a secondary benefit of using the Castle Windsor container is how it simplifies dealing with configuration files.

Typically if you need to have some external configuration, you'd put this into your own custom configuration section and write a class which either implemented IConfigurationSectionHandler or inherited from ConfigurationSection. If you needed to change the configuration, you'd need to change these handler classes. While this is unspeakably better than parsing .ini files, it's still pretty brittle - especially if you are being agile and refactoring your code to any great extent.

What is Castle Windsor?

It's a container that figures out how to create classes, along with all their dependencies, on the fly, from a configuration file. Somewhat like Configuration Section Handlers, but it's an awful lot more. For example...

<component id="VirtualEarthTileServiceManager"
                 type="ArcDeveloper.TileServer.Core.VirtualEarthTileServiceManager, ArcDeveloper.TileServer.Core">
        <parameters>
           <services>
            <dictionary>
              <item key="colorado">${ColoradoVETileService}</item>
              <item key="tahoe">${TahoeVETileService}</item>
            </dictionary>
          </services>
        </parameters>
      </component>

This little snippet of XML defines how the VirtualEarthTileServiceManager should be created. Basically the type is instantiated, and an IDictionary containing the keys and items it depends upon is passed into the constructor. The ${somename} notation refers to other classes in the same configuration - and they use similar syntax to have their dependencies injected at instantiation.

To actually get a VirtualEarthTileServiceManager, you just use this code...

IWindsorContainer container = new WindsorContainer(new XmlInterpreter());           
VirtualEarthTileServiceManager veTileManager = container.Resolve<VirtualEarthTileServiceManager>();

The container.Resolve call looks up the name (VirtualEarthTileServiceManager in this case) and creates the class, with all of it's dependencies.

Learning More

This post has just scratched the surface, so I recommend checking out some more information on Castle Windsor. There are quite a few articles on how Windsor works, but the best I found are a series of 4 written by Simone Busoli, and you can read them here: Part 1, Part 2, Part 3, and Part 4.

Since I'm also a fan of Scott Hanselman, give a listen to this podcast on Mock Objects, and this one on MonoRail and the Windsor container.

For more on the Virtual Earth Tile Service, see this previous post, or check out the ArcDeveloper project at Assembla.

If you are a .NET developer I recommend checking out DocProject. It's an open source front end for the SandCastle documentation engine. Sandcastle picks up where NDoc left off, and is a Microsoft open source project that provides services for creating MSDN style HTML and compiled help. Unfortunately it's a bit of a bear to work with (lots of Xml configuration files, batch files etc).

This is where DocProject comes in. The basic process is to add a "DocProject" or "DocSite" to your solution, and then add references to all the other sites/assemblies in the solution into the DocProject. Then during a build, DocProject actually runs Sandcastle to create the doc.

I've only just played with it a little, and it's pretty easy to get setup and running, but the actual compilation of the help was pretty slow. It was running for >15  minutes on my Core 2 Duo notebook! After digging around a little, it seemed the issue was that I have some ArcGIS Server WebService proxies in one of the assemblies, and it was cranking out all the help doc for that - which is substantially larger than the actual code I was trying to document. Once done, the output is pretty good.

Combine this with GhostDoc, and there is no excuse for not having good developer API documentation for whatever you write.

Check it out at www.codeplex.com/docproject

So I've been playing more with the AGS SOAP API, and have found a few things...

Generating Proxy Classes

While you could pass Xml back and forth, most people will opt to use some kind of proxy classes. Visual Studio will generate these for you when you add a web reference, and this is a good way to get up and running quickly.

There is one downside however - it seems that the WSDL returned by ArcGIS Server for any particular service injects the service name into itself. So if you were adding a web reference to http://myserver/arcgis/services/mytestService/MapServer?wsdl then you'd get a class that was named mytestService_mapserver. While you can change the actual service Url to point to another Map Service, you'll still have  "mytestService_mapserver" littered all over your code. To me this is ugly and a code smell if you are trying to write something which is not bound to a single service. I guess the idea is that it helps a developer figure out which web service they are talking to, and I suppose that if you needed to write an app that used multiple connections to multiple mapservices, then it may *help*, but I doubt it. Any insight on why this was done would be great.

Creating Manual Proxies: wsdl.exe

In addition to simply using Add Web Reference, you can take a more manual route and use wsdl.exe to create the proxies. The wsdl.exe tool that ships with Visual Studio 2005 and it offers a little more control. The syntax is pretty simple - here's what I used:

wsdl.exe /nologo /l:CS /n:ArcDeveloper.AGS.SOAP.Proxies /o:c:\wsdl\proxies /urlkey:MapServerUrl MapServer.wsdl

So this creates a C# proxy class for the MapServer.wsdl file, in c:\wsdl\proxies, in the ArcDeveloper.AGS.SOAP.Proxies namespace.

However, if you run this against the WSDL you download from the MapServer WSDL page, you'll get an error:

Warning: This web reference does not conform to WS-I Basic Profile v1.1.
R2028, R2029: A DESCRIPTION using the WSDL namespace and the WSDL SOAP binding namespace MUST be valid according to the XML Schemas found at http://schemas.xmlsoap.org/wsdl/2003-02-11.xsd and http://schemas.xmlsoap.org/wsdl/soap/2003-02-11.xsd.
  -  Warning: The 'name' attribute is invalid - The value '#NAME#' is invalid according to its datatype 'http://www.w3.org/2001/XMLSchema:NCName' - The '#' character, hexadecimal value 0x23, cannot be included in a name. Line 2465, position  11.
  -  Warning: The identity constraint 'http://schemas.xmlsoap.org/wsdl/:service'  validation has failed. Either a key is missing or the existing key has an empty  node. Line 2465, position 3.

For more details on the WS-I Basic Profile v1.1, see the specification at http://www.ws-i.org/Profiles/BasicProfile-1.1.html.

Writing file 'c:\wsdl\NAME.cs'.

The errors are not critical, since a file still gets written, but now the proxy class is named "NAME", which is worse than the <mapservicename>_mapservice that we got using Add Web Reference. If we look at the wsdl file, the culprit is in the last few lines of the file - we see this:

<service name="#NAME#">
  <port name="MapServerPort" binding="e:MapServerBinding">
   <soap:address location="#URL#"/>
  </port>
 </service>

I'm betting that the #NAME# and #URL# are tokens which get replaced when you get the WSDL from a live service (i.e.  http://myserver/arcgis/services/mytestService/MapServer?wsdl ). So - the fix is pretty simple - just edit your copy of the wsdl file, and assign the name to whatever you want the proxy class to be called. In my case, I set the name to "MapServiceProxy" and the Url to http://localhost/use/app.settings to remind myself to always configure the url in the app settings (the "key" is part of the wsdl command line --> /urlkey)

Note: I think you can use Add Web Reference and just point to your own local "fixed" copy of the WSDL file, and this will produce nicely named proxy classes - I did not test this since I'd already baked my own using wsdl.exe

Anyhow - I've created a set of proxy classes (VB.NET and C#), and a batch file for cooking them yourself. Grab it here: ags-proxy.zip  [UPDATE - link now works]

Just doing some research into pulling ArcGIS Server map services into Virtual Earth, and thought I'd share a some of what I'd found.

ArcGIS Server WMS Layer in Virtual Earth

Using the "Accessing WMS from Virtual Earth" article and code as a starting point, I was able to add an ArcGIS Server WMS layer pretty quickly. The only problem I had was divining the correct values for the WMS "LAYER" element - apparently the default AGS WMS service expects the LAYERS to be specified as numbers - 1, 2, 3 etc. Maybe this is standard, but I'm not a WMS guru by any means. Anyhow, the performance was good, but I will say that I've got some beefy servers - the AGS box is a Dual Quad Core box, with 8 GB RAM. The SDE box has one Quad Core. Neither have any appreciable load, so you're mileage may vary.

ArcGIS Server SOAP API

WMS is nice, but I'm going to need to do some more crafty stuff - adding things into graphics layers, drawing selections etc. This would normally mean using the ADF, but I want to be able to spread this across a web farm with out spending a zillion bucks, so the  ArcGIS Server SOAP API seemed to be the ticket.

The API itself is pretty easy to use, and the example on the AGS SOAP API page was about all I needed to get started. Where things became problematic was with the projections. Morten has a great rundown of the VE/Google Earth projection stuff which was very helpful.

The WMS requests specify a SRS which tells ArcGIS Server how to project the map so that it will match up with the Virtual Earth tiles. Going at the SOAP API, you need to do this yourself.

Armed with the Well Known Text definition of the Geographic Lat/Long system used in the VE front end, I was able to set the SpatialReference for the zoom envelope as well as the map.

wsMapService.SpatialReference sr = new wsMapService.GeographicCoordinateSystem();
sr.WKT = "GEOGCS[see Morten's post...]";
env.SpatialReference = sr;

I also found that I needed to tell the map to be projected into the same space...

mapdesc.SpatialReference = sr;

Update: The tiles are still a little off. Need to do more messing with projections...

Overall, I like the SOAP API - and until the REST API shows up at 9.3 whenever that ships, this is a pretty cool option.

Missing Tiles

Ok - with that out of the way I started to get some tiles showing up, but I also got some run-time exceptions. Running in debug mode, I found that sometimes the SOAP API would return a Url to a map image which did not exist. I added in some code which would catch this, and shoot back a "Missing Tile" image.

 

At this point the behavior is seems random - different tiles at different scales will draw. When I refreshed the page, different tiles would be drawn. My initial thought was that I was overwhelming the Server Object Containers with requests.  I had been running with just two SOC's, so I cranked it up to 10. Same results. And nothing in the logs. Next I thought that maybe it's an issue of the Url being sent back before the image is written out - so I added a delay before fetching a tile. Still the same thing. Any insights on this would be great.

Solution: esriImageReturnMimeData

Since this was bugging me, I went in an changed the return type to esriImageReturnMimeData - which would return me a byte array containing the image. A little bit more hacking and it's working just fine. Now, every tile draws every time. So - the issue seems to be AGS not writing out the files, but still sending back a url.

Adaptive Caching

Since I'm already throwing images around in the httpHandler, it was nothing to write the final images out to files, and put a quick check in up front to see if it exists before going out to AGS to create it.  Works great, but I need to add a little more logic so I don't end up with one folder containing 50,000 files.

The Code

This is horrid evil messy hacked up "research" code that likely has substantial performance issues, high-cholesterol, security holes, and lice but if you really want it, shoot me an email (Contact link above) and I'll zip it.

Got up a little early today and had some time to mess around. I had been reading about Virtual Earth 3D, and decided to see if it the control could be added to a Winforms app. I found a rough how-to from this blog, and it took a little more meddling to get things working, but it definitely does work.

Since the SDK is not actually released yet, it's going to take some stumbling around the API to do useful stuff with it, but think about what this and SQL 2008 are going to allow us to build!

Tonight added a few more projects to the ArcDeveloper Subversion repository up at Assembla (http://svn2.assembla.com/svn/arcdeveloper).

Since the goal of the ArcDeveloper.net open-source projects is to streamline unit testing for the ArcGIS developer, the logical place to start is with a methodology for passing geometries into tests.

The code that's up there handles geometry serialization / deserialization. In addition to a set of classes which handle the "work", it also includes 3 ArcMap tools:

1. Serializing selected geometries to a file,
2. Serializing a sketch to a file, and
3. Drawing serialized geometries on screen so you can be sure you captured the right thing.

It also has a simple example showing how to write tests using this methodology.

I'm not going to go into a full-blown "how to" at this point, but the general flow goes like this:

1. Use the tools in ArcMap to create serialized geometry files
2. Add these files into your Test assembly as Embedded Resources
3. Pull the geometries out of the test assembly when the tests are run.

This allows you to store all sorts of complex geometry scenarios without being reliant on particular spatial data sets residing at fixed locations - the data is embedded into the test assembly.

I will be posting a much more detailed "how to", as well as adding content into the ArcDeveloper.net wiki.

Backlog Items

These will be added into TRAC when I get time, but here's the quick list

• Enable serialization of entire features
• Add tests for the actual utility classes themselves
• Improve the messaging
• Make the Extension be a "JIT" extension so it's easily enabled/disabled
• Better exception handling (currently just MessageBoxes)

Project Status

If you are interested in following along with what's happening on the project, I'll be posting finer grained information on the "project flow" page, which like all good things has an RSS feed.

Yesterday we sent out a press release via Chris Spagnuolo's GeoScrum blog. Thus, I think it's now ok to say that my entire team and I are now with Data Transfer Solutions.

 

The team is super excited about the new opportunities this change brings (the blazing workstations, "green" servers and build out, IKEA workstations, great chairs, and a strong interest in adopting agile methods doesn't hurt either!).

On the technology front, we are still sticking with .NET and ESRI for the most part. We expect to be doing much more web work than in the past, and with the release of Visual Studio 2008 and .NET 3.5 this will be very cool. I expect we'll be working with ArcGIS Server / tile caches on the back end, as well as Virtual Earth and Open Layers for the client UI. Mix in some unit testing, and it should be lots of fun.

We're also going to be adding some new capabilities - specifically Agile Consulting and Coaching. This is still in the early stages so I'll be posting more about it as things develop, but I'm very excited about some things we have planned, and the opportunity to share our experiences, methodologies and practices with others.

I'll wrap this up by saying that I'm also very excited to work with the existing DTS development team. Although I've only met a few of them, and only very briefly, but their depth and breadth of skills are pretty amazing -- I heard that someone wrote some ArcGIS Desktop tools using Boo (scratching your head - here's a hint: Boo is a Python-like scripting language for the ECMA CLR implementations such as .NET and Mono.)  I think they will be keeping us on our toes!

I'll be posting more about starting up an office, and getting things rolling over the next few weeks. Some tasks on our current backlog - order & setup some servers, get a firewall (advice welcomed!), move out of Chris's living room into our "temp" space, finalize the build out plans for our "real" space... fun!

In the last post, I said I'd post shortly so - here's the skinny on the code that's currently in the ArcDeveloper Assembla repository.

The VSTemplates Project

Back in September I posted about using MSBuild to package up Visual Studio Item templates. I've uploaded this code into the repository, and it's ready to roll. The templates are setup for ArcDeveloper projects (i.e. an MIT license, "ArcDeveloper" etc.), but you can easily modify this and re-run the MSBuild script to create a set customized for your company / team.

At this time, there are just 3 templates - class, interface and non-COM class - with both C# and VB.NET flavors. I've got ideas for others, and will be adding more as they come up in my development.

Running MSBuild

If you have edited the templates, you'll want to run the ArcDeveloper.Templates.Build file. Here's how you do that:

1. Edit the .build file so the SolutionDir and BuildFolder are correct for your machine.
2. Open a Visual Studio Command Prompt
3. cd to the folder with the .build file
4. run msbuild arcdeveloper.templates.build

The build script will zip up the templates (as required by Visual Studio) and drop them into the BuildFolder. In order to have Visual Studio pick these up, you need to tell it where your Item Templates are. Go to Tools --> Options and select the Projects and Solutions item. Set the User Item Template location.

Now when you add an item to Visual Studio, you'll have some additional options

Get It!

If you just want the templates, you can download them from here: http://www.assembla.com/spaces/files/arcdeveloper

If you want to be able to build the templates, grab the code from Subversion: http://svn2.assembla.com/svn/arcdeveloper

I had originally setup a SourceForge project for ArcDeveloper.net, but the whole SourceForge experience is clunky and web 1.0. So when I found out about Assembla, I immediately setup a project.

Assembla is a really cool, free Subversion + TRAC + wiki + Scrum development tool / service. It's really amazing. It's totally free for projects up to 200Mb in size (which is a MOUNTAIN of source code).

ArcDeveloper Projects

A repository and some tools is nice, but a bigger question is what projects will be there. Well, my goal for ArcDeveloper is to provide a set of code resources that will make unit testing on ArcGIS projects easier. Since software quality is something that all developers must address, and since it's the same regardless of your vertical market, I think that few organizations would object to using and contributing to this effort.

Getting Access

The repository is open for all users, just point your Subversion client to http://svn2.assembla.com/svn/arcdeveloper and you can get latest on everything. If you are not familiar with Subversion, Assembla has some handy starter info here, and you can Google for mountains of information.

Project Status (TRAC)

The project is managed in TRAC - since it's a web based tool, you can access it here: http://trac2.assembla.com/arcdeveloper/wiki. There's not a whole lot in there yet, but I'll be posting about the inital project shortly.

Bug Reporting / Feature Requests / Contributing

You'll need to be a member on the project to add a TRAC ticket. The easiest way to do this is to contact me, and I'll send you an invite directly from Assembla. If you are already an Assembla member (sigh-up here - takes 30 seconds and it's free), you should be able to contact me through the project's team page. Either case, as far as I can tell, I have to invite you into a project - I don't think you can "apply" to join.

Once you are a member of the project, you will also be able to contribute code.

So that's it for now - as I said, I'll be posting about the initial code base shortly...

This week we finally completed our transition over to Subversion. As part of this transition, we moved 52 project out of a very old version of Source Gear Vault, and 13 project out of Microsoft Team System.

With this much code, we needed an automated way to clear out source control bindings. Our code base has 3 types of projects - Visual Basic 6, Visual Studio 2003 and Visual Studio 2005.

Here are the steps we followed:

1. Get all source code out of both the Vault and Team System
2. Backup the entire source tree (just in case!)
3. Search for .scc files and delete them. This removes the bindings for VB6 projects
4. Download SCC Remover tool from CodeProject.com. This will clean out the source control information in solution files (.sln), solution user option files (.suo) and the projects themselves (.csproj and .vbproj).
5. Modify code so that it can recurse down a tree (see below), and run it.
6. Open some projects in Visual Studio to ensure no bindings are present.
7. Copy entire tree to new location like c:\svnload\code
8. Create an svnimport script - I did this instead of using tortoiseSVN because I wanted to inject \src\trunk into the path, and having 65 directory trees (containing 100's of actual projects), anything manual was out of the question.
9. Run the import script - this just imports the files into Subversion - it does not create a working copy folder!
10. Get the active projects using tortoiseSVN's Checkout command.

After that, we changed out CruiseControl.net settings to run against Subversion, and our main project was back up and running.

As for why we are bailing on Team System - simple - we found that we simply do not use many of it's functions. Instead of it's work item system (or the Scrum add-in from Conchango), we use Rally. Instead of the MS Build Server, we use CruiseControl.net. We did use the SharePoint integration, but nothing beyond what you'd get with a vanilla SharePoint site. We will miss some of the features of MSTest, but NUnit and TestDriven.net are more than capable replacements.

By changing to Subversion, we can down-grade our MSDN licenses to "Professional", and save ~$5000 per year. This will allow us to purchase some other productivity tools - CodeSmith, ReSharper, TypeMock, TestDriven.net and Red Ants Profiler to name a few.  

In response to my last post on Refactoring and Unit Testing, Morten posted a comment with some very good questions...

1. Should you really have multiple test assertions in the same test? (I know is convenient to do, but also very messy from a test driven development perspective).
2. What is the code coverage of the unit tests.

Instead of just answering in comments, I thought I'd provide some more information as a separate post.

Multiple Assertions in a Test

It's true that this is a bit of a code-smell, but I view this as an acceptable trade-off between making tests easy to author (ergo you actually write and maintain them), and "best practices". As long as the asserts have good messages, what you get back is informative. Given that the class I'm focusing most of this testing on is a singleton, running multiple tests in a row is  realistic in terms of run-time usage.

Code Coverage

I had not setup code coverage when I wrote the original post. After setting that up, here's what I found:

At first glance, 73% is not bad coverage. But let's look at the class model for the SecurityManager class.

Looking at the ISecurityManager interface, we can see that the most critical method is CanUserEditObject, and it's only at 35% coverage! It's good to know that the private methods have high coverage numbers - 100%, 94% and 72%, but the actual method that is used by the client objects still has a ways to go in terms of good coverage.

When authoring the tests, my gut feel was that I had pretty good coverage - certainly shows the value of running coverage! I'm going to add more tests to up the coverage on this method, and I'll be posting more about how I created the tests later this week.

Ok, just had to share this because I think it's so darn cool. I've posted about our internal tool "ArcDAL" in the past (follow that link for details), but the quick review is that we have created a set of code-generated data access classes which wrap the geodatabase API. These classes fully support data binding, and because they are code generated, we can change the geodatabase model with limited impact to our code base.

Anyhow - while I was involved with the design and initial template generation, this is my first time actually using it in a project (This is the second major project using ArcDAL). It's just so fun to see this all come together, I thought I'd share...

So - here's how we make Object Inspectors (little UI's that "snap" into ArcMap's Attribute Editor)...

1) Create an inherited user control in the project. We created a base control to ensure consistent look and feel, so that simplifies things. Here's the new control...

2) Create an object data source from the ArcDAL List related to the feature class of interest - in this case ForestDistrictList

3) In the data source window, Set the display style for the ForestDistrictList data source to "Details"