A reader recently asked me to expand upon the "Keep it Simple" advice in the fundamentals section of my web site. Just like my advice, I had kept this section pretty lean...

Keep it simple

Anybody can build complex software. Creating simple software that can accomplish the same task is much more difficult. In the end though, the simpler it is, the easier it is to test, which makes it easier to maintain, and easier to extend.

To illustrate the idea, I'll talk about a project that I'm working on right now. Basically I'm helping / mentoring a team through a short term project to build a data viewer application based on Virtual Earth, Dojo, ArcGIS Server (9.2) and ASP.NET.

One of the user stories in this project is:

As an administrator I want the Data Viewer to be database agnostic so that I can easily configure it to pull point data from SQL Server, Oracle or file based data sources.

Now, this is a great idea, and certainly possible. In fact it ends up being very interesting to design (plug-ins, configuration sections and Inversion of Control oh my!).

Simple works...

In looking at the project (it's very short term) and the team (new to web development) and the current problem they are trying to solve (display data from SQL Server), my recommendation to the team is that we "keep it simple" for now, and just address the SQL Server data source.

While we could define a set of interfaces, build out a whole configuration model, and create the actual data access objects for these various data sources, in the context of this project it's extraneous. To be "successful", we need to display the current data in Virtual Earth. With tight time frames and limited resources, I believe the tradeoff between building for an unknown complex future need, and applying time and effort to the core objective, is a good one. The team will be able to get the database access tier of the system completed in a short time, and it will be very simple to understand - parameters are passed to a class, which calls a stored procedure, which returns the point data. Simple, clean, and working.

But what about the "can accomplish the same task" bit? Good question - I'd suggest that in this situation, with the current data is in SQL Server, you are doing the same task - just using a simpler design to achieve the goal.

Add Complexity when it is Needed

If / when a need arises to add data from Oracle, it will be a very simple task to create another class which mimics the SQL Server class, with the only difference being that it uses the Oracle data connection classes. Should this Data Viewer become a huge hit, and is used so extensively that it must become configurable by non-developers, it could then be refactored to add in the additional complexity required to support configuration sections and a plug-in data provider model. But for now, the much simpler solution works just fine.

So - to summarize: Make choices which trend towards keeping the code as simple as possible. Don't add extra features or design elements to handle unknown or fuzzy future requirements. Add complexity only when it's really needed. While it's fun to monkey with internal design optimizations etc, keep your eyes on the prize: software that works for the user. And keep in mind who will be maintaining the software over the long haul - while you may be a whiz at generics (for example) keep in mind that for many people they tend to make understanding the code more difficult.

That's pretty much my take on this, but I'd love to hear your thoughts on keeping software "simple". How do you decide when to add complexity? How do you define complexity?

Other thoughts on Simplicity:

Joel On Software: Simplicity

The Fourth Law of Software Design: Complexity vs. Ease of Maintenance

Various Quotes on Simplicity in Software Design

We just put up another "demo" site - this time working with Sanborn to provide visualization for their wildfire risk model outputs they have created for the Texas Forest Service. For information on the data, there is contact information on the "About" tab.

From a technical stand-point, we added a few new features to this viewer...

Dealing with Multiple Tile Layers

The viewer is pulling in multiple tile services, representing multiple layers in a map, as opposed to a single "fused" tile layer. This is the first time I've implemented multiple layers using the ArcGIS Server Tile Cache code, and it turns out it's currently a bit of a pain. For this site, we've got 7 tile layers, and to do that with the current tile server code, I needed to create 7 map services (mxd's), and then configure 7 tile services in the TileServer. Clearly this should be streamlined. For example, the url http://65.101.234.201/tileservice/txfirewfsi/023131001.ashx pulls a specific tile from the txfirewfsi tile service. (shown below)

However, I'm thinking that it would be nice to be able to add a layer name in the url -'parcels' in this case -   http://65.101.234.201/tileservice/txbasemap/parcels/023133.ashx. This way you could specify the fused tiles, or just a specific layer.

Adjusting Layer Transparency

This was easy, if not obvious. Adding in a Dojo.slider, and hooking up some events was pretty simple. However, it turns out that changing the .Opacity property of a VETileSourceSpecification has no effect after the layer has been added to map. Instead you need to delete the layer and re-add it with the new opacity. Since the browser is caching the tiles, this is actually very fast.

Virtual Earth 3D

This was also the first time I tried throwing things into 3D mode. It took a few quick tweaks in the javascript, and some different CSS for the markers, but for the most part - it "just worked".

There are a few issues with the dojo menus that I'm going to sort out so I'll save the Url for the 3d page until that's dialed.

So, this took about 3 days to pull together -- if you are looking under the covers, the code is VERY "demo" and has some issues - there are a few weird things with the layer list checkboxes and I have not really tested in IE7, and all bets are off with IE6. I'd love to hear thoughts & ideas though.

While at Where 2.0, Chris Spagnuolo and I recorded a podcast with the Very Spatial crew.

In this podcast we talk (at a high-level) about internals of the Global Avian Influenza mapping application we built for the Wildlife Conservation Society, and some of the complexities and how we overcame them (think 1-M-M-M database relationships, with complex query criteria and 300ms response times).

For those that know us, you won't be suprised that we also talked about Agile - how it fits in with the Where 2.0 crowd, and how we work Agile into our contracting. All -in - all, a fun experience.

The podcast went live today. Thanks Jesse & Sue!

Starting in February we have been working with the Wildlife Conservation Society to create a map viewer for their Global Avian Influenza site (www.gains.org). This updated viewer launched a few weeks ago, and I thought I'd share a little bit about the project.

About the Project

The project is pretty straightforward, with only a few top level user stories:

- Map the scientific data that has been collected as part of the GAINS program.

- Allow users to filter the displayed this data by :

country,date range,species, and influenza subtype.

Also facilitate this functionality via Url parameters.

- Functionality to be delivered as a DotNetNuke module

- It has to be fast.

Technologies:

We used Virtual Earth for the mapping canvas since WCS needed a high-quality global data set, and this platform is free for non-profits.

For the UI, we dove into the Dojo Toolkit. This ended up working really well, with the exception of the tree view. It was just a huge pain to get the Dojo Tree to work the way we wanted it to, and dropping in the YahooUI TreeView solved these issues in minutes.

On the back end, it's we used ASP.NET. Since this was going to be a heavy Ajax site, we also leveraged the ASP.NET web services which are marked up so that they "speak" json (anyone know if there is a more graceful / concise name for this??? ASP.NET Script Services?). Behind that there is a SQL 2005 tabular database, ArcGIS Server and ArcSDE.

The main UI for the map can be seen below. The data points are clustered, and symbolized based on the match to the filter criteria.

Data Filtering

The points displayed on the map meet a set of criteria, which can be chosen on this UI. Not that the design is overly beautiful, but it's a mix of Dojo and Microsoft Ajax controls - using the best parts of both. Actually applying this criteria against the database and retrieving the points was non-trivial, so I'm going to write up another post about how we got all that working.

External Data: Dynamic WMS Tile Caching

The module can be configured to pull in tile services from different WMS servers. This shows my favorite - the global poultry density map. If you are on the site, you can turn these layers on / off this from Map Data --> Manage Atlas Layers. Administrators have a UI to setup the WMS Services. Since mass poultry migrations are rare, most of this data is pretty static. Thus we dynamically cache the tiles on the WCS server so we get improved performance. We do see some offsets in the tiles at the lower zoom levels due to the variations between WGS84 and the Web Mercator projection of VE.

External Data: GeoJSON Features from ArcGIS Server 9.2

We also have the ArcDeveloper 9.2 REST API for ArcGIS Server 9.2 in the mix, pulling in flyway geometries. For species that have flyways, an extra item shows up in the identify tree. Here's a link to a map showing Calidris alpina, which has flyways. Mouse over a feature, click "View Details" in the pop-up and then expand the tree until you see "Show Flyway". This makes a JSONP call to the REST service running on a different server, grabs the geometries and drops them into the map. Since this species as multiple flyways, multiple polygons are drawn with various colors.

It's been a great project to work on, and we are continuing to work with the Wildlife Conservation Society to add additional functionality into this application.

The last two days we've been setting up a demo integrating the National Geographic MetaLens service with Virtual Earth. It's now live and can be seen at: http://www.metalens.org/mlve/.

MetaLens is a geospatial content management and archival system that National Geographic uses to secure and manage it's content. "Out of the box" it has a Flash interface, but it also has a nice REST interface. We built some server side code that requests the public assets in the current view, and then applies clustering. At this time, the site is only fetching images, but we expect to extend the demo to include audio and video as well.

Map Tools

What's a map without some tools? The Zoom To tool allows you to hop over to a location where there is some data of interest.

For Lake Tahoe, we have added in a Parcels layer that comes from an ArcDeveloper.net Tile Service.

For Vermont, we have a map from National Geographic being tiled using the same code base.

You can toggle these layers on and off using the Map Data command on the toolbar. We have also added a Base Map drop down that allows you to select which of the VE base layers to view.

What's on the Server Side

The main app is a ASP.NET with a data provider class library that actually talks back to the Metalens service. I really like this pattern because it allows us to test the provide completely independently of the web site itself. Basically we just wire up some unit tests, and then via TestDriven.net they are automagically re-run when we build that class library. Sweetness.

MetaLens

MetaLens is built on a product called Clear Path Explorer (CPX) built by a local Fort Collins company called Clear Path Labs. Their web site talks more about CPX, and it's API. From out brief experience working with it, it's pretty nice.

Tile Services

Not surprisingly we are leveraging the ArcDeveloper.net Tile Service. At this time we are just pulling in 2 services, and both use the ArcGIS Server SOAP tile provider. The only wrinkle is that one service is hosted by DTS in our Fort Collins office, and the other is on a National Geographic server.

I have to give a big shout out to my colleague Mike Juniper who did an awesome job tracking down a bunch of CSS weirdness, and put the final polish on the site. We knocked this out in about 2 days start to live. Amazing what you can accomplish these days! There are still a few strange things in IE6, but that's pretty much how it is with that browser. Works great in IE7 and FireFox.

Just a quick note about some updates to the Virtual Earth Tile Server code base. You can grab the latest code from the Subversion repository (http://svn2.assembla.com/svn/arcdeveloper/tileserver).

Note: This update will require some configuration changes, but I think they are well worth it.

WMS Support

The Tile Server now as a WMS Tile Provider. This allows you to very easily add WMS Services into your Virtual Earth applications. This is a pretty base implementation, but it works smoothly. You just need to specify the base Url (which should include the LAYERS you want in the map images) and the version of WMS to use. The WMSTileProvider takes care of the rest. I've only tested with WMS 1.1.1 at this point, but I'll be working with all flavors shortly so should there be any issues, I'll be fixing the code.

This is the configuration file in source control, and it is pulling a global bathymetry layer from mapsherpa.com. It can be a little slow at the top levels, but once it's cached, it rips.

Improved Caching Options

Caching is great, and can improve performance, but it can also eat up a lot of disk space. For example, at level 15, there are more than 1 billion tiles, so you could really fill up some disk space. Unless your data only covers a small area, the likelihood of a cache hit at the higher levels is pretty low. Thus you are storing lots of tiles which are not really helping performance. To help manage this, I've added a MaxCacheLevel property to the Tile Service. By default it will cache everything, but you can change that by just adding maxCacheLevel into the parameter list in the service configuration...

FileStorageProvider Changes

The tiles are now stored in sub-folders with the name of the service. So - instead of seeing something like:

c:\tiles\l1\

you will see

c:\tiles\someservice\l1

This allows you to  use the same FileStorageProvider across multiple services in your configuration. Not a big deal, but saves some configuration.

I needed to add a slider type zoom control into Virtual Earth, and since my project is already waist deep in Dojo, it only seemed fitting that I figure out how to use their slider control.

The Dojo framework is an open source client side javascript framework with some pretty nice UI components. You can see the slider here in the Book of Dojo.

Turns out that this is pretty simple to do:

1. Create the Dojo Slider in markup
2. After loading Virtual Earth, use map.AddControl, passing in the slider element
3. Wire up a handler that updates the slider when the map zoom changes
4. Wire up a handler that updates the map when the slider changes
5. Sip beer and congratulate your self on a job well done.

Actually the only difficult thing here was divining how to get the slider value - the Book of Dojo is a great general starting point, and the API is too sparse to help out much. There is a Dojo Feature Explorer site that helps though.

You can see a live demo here: http://www.davebouwman.net/examples/vedojoslider.htm

Download the html page

Assuming anyone finds this sort of stuff useful (comment!), I'll post more of these simplified examples using Dojo and YUI with VE.

Spent the last day or so cooking up a quick demo for our marketing guys. You can check it out here: http://65.101.234.201/mac/ 

It's pretty basic - it shows GPS points and photos from our mobile asset collection vehicle.

The demo is a small data set, but the points are in SQL 2008. They are clustered on the fly, and sent to the browser. As we add more data into this, I suspect that we will render a roads layer as tiles, and then add click handlers to the map which will allow a user to select a road segment, which will then go and get the points. We may also look at thinning out the points a bit more - partially so there is less data send to the browser, and partly because few users need to see every single photo.

The popup shows the two camera views as small thumbnails, with Next/Prev buttons to view... drum roll... the next/previous photos.

If you click on a photo in the pop-up it will display a larger set of images in a YUI panel. Again with the next/prev.

In the other SQL 2008 stuff I've done, I used stored procs to get the data, but this time I opted for dynamic sql, and it works quite nicely - also will make it easier to add more data into the mix (no need for additional sprocs)

I've been working with Dojo on another project, and I must say that the Yahoo User Interface library (aka YUI)  is refreshingly simple - if you don't need a whole ton of stuff, definitely look at YUI before resorting to Dojo or ExtJs.

Right now this site is clearly really limited but that's the nature of "demos". I'll be adding more functionality to this site (additional data sets etc), and if anything interesting comes up I'll post about it.

I've been working with Virtual Earth and SQL 2008 - mainly pulling points out of the database, clustering them, and throwing them on the map. All well and good, but I was getting some weirdness when zoomed out to the first of second zoom levels.

Specifically I was getting this error:

The specified input does not represent a valid geography instance because it exceeds a single hemisphere. Each geography instance must fit inside a single hemisphere. A common reason for this error is that a polygon has the wrong ring orientation.

The issue was that the bounding box of the map view port was more than a "hemisphere" in size, and so I was going to have to split up these queries into SQL Server. Now that sounds simple enough, but took some experimenting to work out exactly how to do this. Steve Kass (I believe of the SQL Server team @ Microsoft) has a post on the hemisphere requirement that talks about ring-direction. While we're on that sub-topic, ring-direction matters a lot here. The "left side of the line" is the "inside", and if you get it wrong you will also get this same exception.

So - my observations - it seems that your geometry has to be less than a hemisphere. The "exceeds" in the error message is a red herring.

For example the "western hemisphere" as defined below will throw an exception

However, if we tuck these values in a little bit...

All is well again.

Further experiments seem to indicate that which way you cut the globe matters as well. Despite having 1000's of points in the souther hemisphere, this query does not return any points

If I then split the southern hemisphere into two parts - East and West, I get the points again...

and

So - it would seem that the "hemisphere" they are referring to is really an EAST - WEST range. As long as your query geometry is less than 180 degrees wide, things are pretty good. Take this query, which is a patch that is ~180 wide and 90 high centered over 0,0.

This works just fine, so clearly we are not restricted by the actual Lat/Lon values just the range of values.

So as far as I can tell the rules for a valid geometry are:
1) Latitude (Y) Range must be less than or equal to 90 degrees

2) Longitude (X) Range must be less than 180 degrees

Hope this helps someone else avoid a few hours of head scratching and manual SQL querying.

I spent today doing some research on using SQL 2008 as a back end for a Virtual Earth application. Here's a rough breakdown of the steps I took to get some stuff up and running.

Get All the Parts and Pieces

First, download the November CTP of SQL Server 2008

I originally installed the Express version, but that did not include the Management Studio. Install the full version by downloading the DVD Image. You can mount the image using MagicISO Mounter

During the installation, select everything to ensure that you get the Management Studio and SQL Server Integration Services (aka "SQL Server Business Intelligence Development Studio").

Connecting from Visual Studio 2005

Download the Visual Studio 2005 patch allows it to connect to SQL 2008

This will allow you to connect to the database engine, and while the connection works, Visual Studio tries to parse the queries and complains on spatial data types. You get a message like this, and then the query proceeds.

Getting Started...

I started with John O'Brien's post and code titled "Virtual Earth and SQL 2008 Spatial - a first impression". Once I got this running, I started to experiment with my data.

Loading X,Y Data into SQL 2008 Geometry

My test data set has 250,000 records, with a Lat and Long stored as fields in the table. My first hurdle was to get the table from a SQL 2005 instance to my 2008 instance. Enter SQL Server Integration Services - Microsoft's ETL platform. This is essentially an add-in that puts more functionality into Visual Studio.

I created a new Integration Services Project in Visual Studio...

and added an ADO.NET Data Flow Source and an ADO.NET Data Flow Destination to the design canvas.

When setting up the destination, I was prompted to create the table which was handy. It's important to note that these tools do not know about the spatial data types so I could not add the geography column until the data was transferred.

Add Geometry Column and Load

I added the "Location" geography column did this right in SQL Management Studio 2008 - very simple. Then I used some SQL to convert the Latitude / Longitude columns into points in the Location column:

update  TestPoints
set location =
geography::STPointFromText('Point(' + CAST(Latitude as varchar(50)) + ' ' + cast(Longitude as varchar(50)) + ')', 4326)
from TestPoints
where Latitude is not null and longitude is not null and Latitude <= 90 and latitude >=-90

The data I was working with had a few instances where the Lat and Long were reversed, hence the where clause.

Viewing in Virtual Earth

As I mentioned earlier, I used John O'Brien's sample code to get things up and running quickly. I just had to change the connection string in his code, create some new stored procs that used my table instead of his, and change the call in his Web Service. And Voila...

Overall Impression

So this is pretty simple stuff really - show some points on the map. But most of my effort was in getting SQL 2008 installed with all the bells and whistles, and getting Visual Studio to talk to it. Beyond that things went really smoothly.

No doubt that there are many issues left - not the least of which is actually dealing with the 250,000 points. Luckily John has a post on implementing point clustering!

I will also be looking at the Vector Tiling stuff as I've got some polygons that need to be put into the map as well - but that will be another post!

Just doing some research into pulling ArcGIS Server map services into Virtual Earth, and thought I'd share a some of what I'd found.

ArcGIS Server WMS Layer in Virtual Earth

Using the "Accessing WMS from Virtual Earth" article and code as a starting point, I was able to add an ArcGIS Server WMS layer pretty quickly. The only problem I had was divining the correct values for the WMS "LAYER" element - apparently the default AGS WMS service expects the LAYERS to be specified as numbers - 1, 2, 3 etc. Maybe this is standard, but I'm not a WMS guru by any means. Anyhow, the performance was good, but I will say that I've got some beefy servers - the AGS box is a Dual Quad Core box, with 8 GB RAM. The SDE box has one Quad Core. Neither have any appreciable load, so you're mileage may vary.

ArcGIS Server SOAP API

WMS is nice, but I'm going to need to do some more crafty stuff - adding things into graphics layers, drawing selections etc. This would normally mean using the ADF, but I want to be able to spread this across a web farm with out spending a zillion bucks, so the  ArcGIS Server SOAP API seemed to be the ticket.

The API itself is pretty easy to use, and the example on the AGS SOAP API page was about all I needed to get started. Where things became problematic was with the projections. Morten has a great rundown of the VE/Google Earth projection stuff which was very helpful.

The WMS requests specify a SRS which tells ArcGIS Server how to project the map so that it will match up with the Virtual Earth tiles. Going at the SOAP API, you need to do this yourself.

Armed with the Well Known Text definition of the Geographic Lat/Long system used in the VE front end, I was able to set the SpatialReference for the zoom envelope as well as the map.

wsMapService.SpatialReference sr = new wsMapService.GeographicCoordinateSystem();
sr.WKT = "GEOGCS[see Morten's post...]";
env.SpatialReference = sr;

I also found that I needed to tell the map to be projected into the same space...

mapdesc.SpatialReference = sr;

Update: The tiles are still a little off. Need to do more messing with projections...

Overall, I like the SOAP API - and until the REST API shows up at 9.3 whenever that ships, this is a pretty cool option.

Missing Tiles

Ok - with that out of the way I started to get some tiles showing up, but I also got some run-time exceptions. Running in debug mode, I found that sometimes the SOAP API would return a Url to a map image which did not exist. I added in some code which would catch this, and shoot back a "Missing Tile" image.

At this point the behavior is seems random - different tiles at different scales will draw. When I refreshed the page, different tiles would be drawn. My initial thought was that I was overwhelming the Server Object Containers with requests.  I had been running with just two SOC's, so I cranked it up to 10. Same results. And nothing in the logs. Next I thought that maybe it's an issue of the Url being sent back before the image is written out - so I added a delay before fetching a tile. Still the same thing. Any insights on this would be great.

Solution: esriImageReturnMimeData

Since this was bugging me, I went in an changed the return type to esriImageReturnMimeData - which would return me a byte array containing the image. A little bit more hacking and it's working just fine. Now, every tile draws every time. So - the issue seems to be AGS not writing out the files, but still sending back a url.

Adaptive Caching

Since I'm already throwing images around in the httpHandler, it was nothing to write the final images out to files, and put a quick check in up front to see if it exists before going out to AGS to create it.  Works great, but I need to add a little more logic so I don't end up with one folder containing 50,000 files.

The Code

This is horrid evil messy hacked up "research" code that likely has substantial performance issues, high-cholesterol, security holes, and lice but if you really want it, shoot me an email (Contact link above) and I'll zip it.