So I'm getting ready to cook up a tile cache, and I want to use the ESRI ArcGIS Online tile scheme so I can use their tiles for the first 10 or so zoom levels. To do this, you need to have your map's spatial reference set to WGS84, which was not a problem when I was setting up the "Road Map", but I ran into a snag with the "Hybrid" (roads + parcels + orthos) - the Ortho images did not have any spatial reference defined - which means that ArcMap could not project them. Doh!

Now - I should point out that there is likely a neat-o way to do this with some Geoprocessing Python, but I'm a .NET guy, so I used my .NET hammer to solve this.

Ive done enough with images over the years to know that the spatial reference will be stored in the .aux.xml files that share the name of the raster. But for one set of images, there were no .aux.xml files. Solution: Open all the files in ArcMap and it will create these files. Sweet.

Next up, for one raster, I manually set the correct spatial reference in ArcCatalog. Easy, but there are several hundred files, so it's not going to work as a solution. Enter System.Xml...

I then opened up the .aux.xml file (in notepad) for the raster that I manually assigned a spatial reference to, and simply copied the projection information from the <SRS> element. I then cooked up a simple loop over all the files, opened them up in an XmlDocument, injected or updated the <SRS> element, saved the file and shazam! all done. The pertinent function is below.

While more over the top than may be necessary, it was quick to write and solved my problem... now to kick off the cache creation!

A few months back I was invited to talk about the "GeoWeb" at the Texas GIS Forum. Somehow I pulled a great slot - first presenter in the first session following the keynote by Dr. Bill Gail from the Virtual Earth group.  I chose to talk about "Usability in the GeoWeb" - you can download the presentation (as PDF) at SlideShare.net and view it below. The basic gist of the talk is that traditional Web Development groups are moving into our market space. In order to avoid having them "eat our lunch", we need to adopt some of their techniques to make sure we build usable, high-performance applications. Check it out.

One big hurdle for developers moving to javascript is the lack of "compile time checking". The only way to see if your code is syntactically correct is to load it into a browser and run it. But, sometimes small syntax errors can cause the javascript parser to fail, and the file will not load. This is a huge pain, and a great way to waste hours trying to find the smallest of syntax errors.

Enter Javascript Validation. There are some on-line tools for validating javascript, and Douglas Crockford's JSLint is likely the most widely known. And while these tools can be handy, copy/pasting code into a text box, validating it, making changes, re-validating etc etc is tiresome.

Luckily for all of us Predrag Tomasevic created a Visual Studio add-in that uses JSLint to validate javascript from within Visual Studio 2005/2008. It's called JSLint.VS, and the code is over at CodeProject. Get it.

The simplest way to use it is to run JSLint on a file from the context menu...

But, if you look at the options (Tools --> JSLint.VS options) you can integrate this with your build.

You can see the options that I have checked - running with minimal items checked tones things down to just checking for syntax - which end up being the most irritating bugs to catch. For example here is some code that's catching an event, and then propagating that to another method along with some customized event arguments.

onClick: function(evt){
    // stub for event propagation
    console.log("DashboardMenu::onCommandItemClick");
    this.onCommandItemClick({
        action: this._menuConfig.action,
        type: this._menuConfig.type                
    });
},

The customized arguments being passed over to onCommandItemClick are created via object notation - JSON if you will. Since I'm refactoring the code quite frequently, small little things can become problems. For example, I decided I did not need to pass "type" along anymore. Ok, I just went in an deleted that line resulting in this code...

onClick: function(evt){
    // stub for event propagation
    console.log("DashboardMenu::onCommandItemClick");
    this.onCommandItemClick({
        action: this._menuConfig.action,                       
    });
},

And as usual I refreshed FireFox, and followed along in FireBug and all was well and good. An hour or so (and a dozen or more changes) later I tried the page in IE...

Oh sweet. Without getting into dojo.require etc etc, this message is basically saying that IE could not load the class "dts.DashboardController" from the file because of some javascript syntax error. What's not really clear is that the error could exist in that file, or any other class that's dojo.require'd by the specified class. In my case, the code shown above is in dts.DashboardMenu, which is required in dts.DashboardController as shown below...

dojo.provide("dts.DashboardController");

dojo.require("dijit._Widget");
dojo.require("dijit._Templated");
dojo.require("dts.DashboardMenu"); //<-- DashboardMenu is pulled in here

dojo.declare("dts.DashboardController",
    [dijit._Widget, dijit._Templated, dijit._Container],
    { 
        //class definition here...    
    }
);

So - back to the problem - those who work with javascript and JSON every day ,and have sharp eyes might have noticed the error in the second code block - there is an extra comma in the JSON. Firefox accepts this, and works just fine. However, in IE the file fails to parse and we get that error dialog.

Turning on JSLint.VS, I get a notice about the missing comma as soon as I build... with the line number... how easy is that!

This will clearly save me a LOT of time over the long-haul - if you are using Visual Studio for your javascript development, this should be a "must-have" addin.

With the recent announcement that jQuery will ship with ASP.NET MVC and Visual Studio, the question of what javascript framework to use just got more complex.

Here's my situation:

1) I need to work with the ESRI Javascript API for the maps in my apps, and this is based on the dojo toolkit

2) I want to work with ASP.NET MVC because it's actually a sane way to built RESTful web apps on the Microsoft platform.

3) I need to send complex data between client and server as json

4) I work in an agile manner, so I need an environment that's flexible - hand coding of client-side "data classes" is a waste of time if they can be generated.

So - what to do?

jQuery is awesome, but if I already have dojo loaded into the browser for the map controls, it does not add much to the mix (besides more bytes to download!)

MSAjax and ASP.NET JSON Web Services make sending complex data across the wire super simple, but should I drag along this library just to save some effort here and there? Sending complex data is possible with dojo, but it requires coding the Json deserialization on the server, which is tedious but not that difficult.

So - what to do?  My initial thinking goes like this - for apps that use Google Maps or Virtual Earth, I'm inclined to use just jQuery. If I need to send complex data across the wire, then adding MSAjax into the mix is likely worth it. For sites using the ESRI Javascript API, I'm torn. Dojo has to be loaded, so I don't see good reason to load up jQuery as well. Again the call on MSAjax will be based on the complexity of the data services.

If you are working with this stack, I'd love to hear your thoughts on this...

Next up in my series of posts on using dojo to communicate with a Controller class is POSTing data. In this example, I used dojo.xhrPost to send Json as a form and have the MVC framework parse it into the Create(string name, string age) method on the Services controller. In this case, the data being sent back and forth is simple - a Name and Age from two text boxes.

Dojo Code

function CreateEmployeeMVCPost(){
    var responseNode = dojo.byId("responseMVCPost");
    var request = {"name":dojo.byId("username").value , "age":parseInt(dojo.byId("age").value)};     
         
    dojo.xhrPost({
        url: '<%= Page.ResolveClientUrl("~/Service/Create") %>',
        handleAs: 'json',
        timeout:3000,
        content: request,   
        //Don't set content type to Json
        contentType: "application/x-www-form-urlencoded",
        load: function(person,args){                         
            responseNode.innerHTML = "Response: " + person.Name + " " + person.Age;
        },
        error: function(error,args){ 
            alert(error);               
            responseNode.innerHtml("Error!",error);
        }
    });
}

Controller Code

ASP.NET MVC looks at the form data and matches up form elements which have the same name as the parameters of the method. So - the in-bound json {"name":"steve", "age":34} is parsed into form elements called "name" and "age", which line up with the parameters of the method (shown below). The controller just uses the inputs to create a new instance of an Employee object, which is then Json serialized and returned to the client.

This is pretty smooth and works very well for simple data. Next time I'll look at trying to POST complex objects.

Just ran across this new tool from ESRI, via the ArcObjects Blog - The API Evaluator. Basically this tool digs through your .NET assemblies and tells you all about the ESRI API usage.

ESRI created this tool so that they can get information about how developers use their APIs, without developers actually shipping the code to them. The idea of sending the results to them is that they can then focus their efforts on improving the SDK itself.

I ran it against a recent project that generates PDF files using ArcGIS Server...

From the developer perspective, you get to see how many calls your code makes into the ESRI API - this can help determine the licensing requirements as well as give you some insight into how well your code is designed - i.e. do you have calls into the API from all over your application? If so, you could refactor your code to centralize those calls.

If you are working with the ESRI APIs, check it out, and know you'll be helping improve things along the way.

My first exploration of using MVC with the dojo toolkit focused on GETting data from a Controller.

I created a simple ServiceController, and added a ListEmployees method. In MVC, this relates to a Url like  http://localhost/mvcajax/Service/ListEmployees. I added the AcceptVerbs attribute to restrict this action to GET's. Code is below:

[AcceptVerbs("GET")]
public ActionResult ListEmployees()
{
    List<Employee> emps = new List<Employee>();
    emps.Add(new Employee("Dave",37));
    emps.Add(new Employee("Kai", 2));
    return Json(emps, "text/json-comment-filtered");          
}

As you can see, this just creates two "employees", adds them to a strongly typed list, and returns them as Json. Super simple.

In the View, I wired things up using dojo.xhrGet, and then converted the json into an array of objects.

What's nice about this is that the json that's returned is automatically turned into objects.

<script type="text/javascript">
function GetEmployeesMVCGet(){
    var responseNode = dojo.byId("responseMVCGet");                       
    dojo.xhrGet({
        url: '<%= Page.ResolveClientUrl("~/Service/ListEmployees") %>',
        handleAs: 'json',
        timeout:3000,                    
        contentType: "application/json; charset=utf-8",
        load: function(person,args){                                     
            responseNode.innerHTML ="Response: " + person[1].Name + " " + person[1].Age;
        },
        error: function(error,args){ 
            alert(error);               
            responseNode.innerHtml("Error!",error);
        }
    });
}
</script>

You may notice that the url passed into xhrGet looks a little funky - well, this code was copied from the View in Visual Studio, so the <%= Page.ResolveClientUrl("~/Service/ListEmployees") %> has not been evaluated. When the page is served, this expression is evaluated, and the full url added in. This is much better than putting in the path manually, as it can be used the same way from any view.

So, this is very usable, and relatively easy to build and consume

Next up - posting Data with dojo.xhrPost...

I'm becoming more and more reliant on unit testing as a critical part of my development process, and so I'm really excited to get a chance to build some web applications (internal research sites) using the new ASP.NET MVC (model-view-controller) bits. It's not "officially" released, but it's in "Preview 5", there is a "go-live" license, and it means you can build saner web apps without all the ASP.NET Web Forms shinanigans. So lets jump in!

I'm going to be doing a number of posts related to ASP.NET MVC, and this will be the "anchor" post - I will likely be editing it over time to add in more pointers to good resources.

The Basics

I'll point you to the official www.ASP.net/mvc site to get all the gory details, but at a high-level the things I like about this style of development are pretty simple:

1. absolute control over the HTML and javascript that's sent to the browser,
2. REST concepts baked in,
3. separation of the presentation from the data from the logic (separation of concerns),
4. Framework is test-driven development friendly

In my post on the Model View patterns, I gave some background on the idea of "MVC", so I'll consider that to be covered.

Getting Started

First thing to do is grab the bits. The installer and the source code (yes, Microsoft is publishing the source code for ASP.NET MVC!) are available at codeplex.com. This builds on .NET 3.5 and you will want to use Visual Studio 2008 (I don't think it's 100% required, but it makes the experience smoother).

Once installed, you get a new project type - ASP.NET MVC Application

Immediately after this dialog you are prompted to create a test project with your testing framework of choice. I like MBUnit, so that's what I selected.

So - how's that for sweet and different? Defining your unit testing strategy UP FRONT, instead of as an after thought. Tasty.

The Code

At Preview 5, the template kicks out a starter website, that's ready to roll. It's designed to show users how to do some of the common things - membership, roles, login / logout etc.

Not surprisingly the code is laid out as Models, Views and Controllers, with folders for each. Requests are routed to Controllers which then apply logic to the Model, and then determine which View to render to the client.

This is nice in that it helps ensure consistency across MVC projects.

Looking at the default project a little closer, we see some naming conventions at work...

We see two controllers - AccountController and HomeController. Then, under Views, we see two folders - Account and Home. This is not a mistake - although not required, setting things up like this simplifies the code you need to write. In the View subfolders, we see aspx pages with names that look like they may be methods, and that's exactly the case. The HomeController has 2 methods, which relate to the 2 View aspx files in /Views/Home. Initially this looks all jumbled in terms of the Url hiearchy, but that's taken care of by routing.

Routing 101

Unlike normal ASP.NET sites, the requested Url in an MVC site does not (usually) map to specific ASPX file on disk. In fact, you usually never see a file extension at all. For example, suppose you had a Url http://foo.com/orders - in traditional ASP.NET, a request to that url would look for a "default.aspx" file in the "orders" folder, and load that. With MVC, the Url is separated from the actual content on disk. Rather, decisions on what to execute are made based on what "routes" are setup for the site.

Essentially, routing is a way to map Urls to Controllers. In the sample site, a request to "/", actually routes to the HomeController, and specifically the Index action on the HomeController.

This is all setup in the Global.asax file, and looks like this:

public static void RegisterRoutes(RouteCollection routes)
{
    routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

    routes.MapRoute(
        "Default",                                              // Route name
        "{controller}/{action}/{id}",                           // URL with parameters
        new { controller = "Home", action = "Index", id = "" }  // Parameter defaults
    );
}

The sample site is very simple, so the routing is very simple, but you can get much more creative with this stuff. What's really nice about this is that you can start to build RESTful applications - so getting the details of an order could be accessed via a url like /Orders/Details/123-098. You could also send along more information - either as another element in the url, or as a parameter, and specify the response format - much like how the ArcGIS Server 9.3 REST API works. This adds a tremendous amount of power and flexibility to the ASP.NET toolbox, allowing you to build one application which simultaneously provides a HTML based interface AND a Json interface - just by varying a parameter on the url (and a very minor amount of Controller code!)

Unit Testing

As noted earlier, unit testing is a key (but not mandatory) part of ASP.NET MVC. In the past, anyone concerned with testing usually kept their data classes (essentially the model) in a separate assembly which the web site referenced. This allowed the developer to write unit tests without meddling with classes in App_Code. But there was still all the code-behind logic which was a bear to test. This is where the Controllers come in. They do a lot of the logic that used to be in the code behinds. And they are testable. I'll be getting into this in more detail in upcoming posts, but you can test both the View output, as well as the routing logic.

Javascript: MSAjax, Dojo, jQuery oh my!

Since the developer has complete control over the markup that is emitted into the page, it's much easier to work with non-Microsoft Ajax libraries. This was somewhat complex in ASP.NET as the use of UserControls tended to cause all sorts of issues with respect to the ID of the HTML elements in the page. If you've ever looked at the HTML in a page and seen an id like "ctl1_ctl3_mydiv", that's ASP.NET trying to "help" you. This worked well for WebForms applications, but it's a total pain for Ajax apps where consistent ID's is critical. Why would we want to use another library? Consider that for sites using the ESRI Javascript API, the dojo toolkit is already loaded into the page, and it has all the Ajax plumbing we need to talk to back to the server - wouldn't it be nice to not load MSAjax if we did not have to? Later this week I'll be posting about some experiments I've been working on to see how this works.

Resources:

Tutorial Videos @ ASP.NET - good content for getting a handle on the main MVC ideas.

ASP.NET MVC Quick Start pages - high-level documentation for the framework

Here's a little something to scratch your head about...

As I mentioned in some previous posts, I'm currently working on some code that creates 100k maps based on USGS quad sheet boundaries. And all is working nicely. Except I was seeing a blue halo around some of the quad sheet boundary in the PDF...

I should note that at this point I'm creating the PDF's via unit tests - so I'm always using the same tile and varying other properties. When I zoomed in on the PDF a light went on...

Why - it's not some weird rendering "halo" artifact - it's the cyan selection box. Except I'm not manipulating the map selection. Scratched my head for a minute and then I opened up my map document...

Look at that. I had saved the document with a selection, and then when I accessed it via ArcGIS Server, the selection was still there.

Now - I guess this "makes sense" in that selections are stored in the document so they persist over time, but it kind of seems like ArcGIS Server should clear or ignore them? I mean I'd hate to think that someone was building an application that relied on a particular selection set being in a document...

Anyhow - just another thing to add to your "Publishing Maps to ArcGIS Server" checklist.

Just another note on using Dojo to communicate with ASP.NET JSON Services. In my last post on this topic, we did a simple GET request to a service, and that is all well and good, but suppose we want to POST some data?

Well, I ran into this today, and since it was a royal pain to dig the answer out of the internets, I'll share it with you (and Google) now.

Scenario

I have created a web service that cranks out 1:100,000 scale maps as PDFs. Since these can take a while to process, the service kicks up a thread and cooks the map asynchronously, and simply returns a status message to the requesting client. When the map is done, an email is sent to the user telling them that the map is ready to be downloaded.

Here's the web service signature...

We can see that the service is marked up so that it accepts JSON ([ScriptMethod] attribute) and responds with Json ( ResponseFormat = ResponseFormat.Json). Ok - moving on...

On my test page, I have a very simple form...

The HTML is below...

<form id="form1" action="" >
<ul>
    <li>
        <label for="emailaddress">Email Address:</label>
        <input type="text" id="emailaddress" value="dbouwman@edats.com"/>            
    </li>
    <li>
        <label for="tileid">Tile Identifier:</label>
        <input type="text" id="tileid" value="73"/>        
    </li>
    <li>
        <label for="service">Service:</label>
        <input type="text" id="service" value="UTM_ZN10"/>        
    </li>
    <li>
        <label for="year">Year:</label>
        <input type="text" id="year" value="2007"/>  
    </li>    
    <input id="Button2" type=button value="Submit via POST" onclick="SubmitMapRequest();" />
</ul>

As we can see, the Submit button calls the SubmitMapRequest function, which is where we actually make the xhr request. The working final Javascript is show below.

function SubmitMapRequest(){
    //debugger;
    var responseNode = dojo.byId("response");
    var maprequest = { "emailAddress":dojo.byId("emailaddress").value , 
        "serviceName":dojo.byId("service").value , 
        "mapSheetId":dojo.byId("tileid").value  , 
        "year": dojo.byId("year").value };                
    
    dojo.rawXhrPost({
        url: "./MapSheets.asmx/CreateMapSheet",
        handleAs: 'json',
        timeout:1000,
        postData: dojo.toJson(maprequest),            
        contentType: "application/json; charset=utf-8",
        load: function(data,args){                
            responseNode.innerHTML = data.d;
        },
        error: function(error,args){                
            responseNode.warn("Error!",error);
        }
    });
}

So - what's going on here...

Packing up Data as JSON

Step one is to pack up the form data into a javascript object which can then be sent to the service. It took a while to locate a sample that actually worked, so it's worth showing below...

var maprequest = { "emailAddress":dojo.byId("emailaddress").value , 
    "serviceName":dojo.byId("service").value , 
    "mapSheetId":dojo.byId("tileid").value  , 
    "year": dojo.byId("year").value };     

Since we are not using the MSAjax framework, we can't rely on just using the web service proxy to make the request. Thus we need to take a little more responsibility for constructing the json, and that's exactly what we are doing here. What's important to note is that the naming of the fields in the javascript must match the naming of the parameters in our web service method. So we send in 'emailAddress' not 'EmailAddress' or 'emailaddress'.

Posting the Data to the Web Service Part 1: dojo.xhrPost

It took a lot of digging, FireBugging and messing with Fiddler to figure this out. In the end I used rawXhrPost as it does not try to get all fancy with the content - it just POSTs the it to the Url. But for the sake of completeness and Google-indexing, here's the deal with dojo.xhrPost

My original code used dojo.xhrPost, and looked like this:

function SubmitMapRequestPOST(){
        
        var responseNode = dojo.byId("response");        
        var maprequest = { "emailAddress":dojo.byId("emailaddress").value , 
            "serviceName":dojo.byId("service").value , 
            "mapSheetId":dojo.byId("tileid").value  , 
            "year": dojo.byId("year").value };                        
        dojo.xhrPost({
            url: "./MapSheets.asmx/CreateMapSheet",
            handleAs: 'json',
            content: maprequest,            
            contentType: "application/json; charset=utf-8",
            load: function(data,args){                
                responseNode.innerHTML = data.d;
            },
            error: function(error,args){                
                responseNode.warn("Error!",error);
            }
        });
    }

Using this I would get a Json error from the service...

{"Message":"Invalid JSON primitive: emailAddress.","Stack Trace":"blah blah blah..."}

When I'd look at the POST in FireBug, it was not JSON... instead it looked like this

emailAddress=me%40foo.com&serviceName=UTM_ZN10&mapSheetId=73&year=2007

Next I tried to use dojo.toJson to force the maprequest to become Json...

        dojo.xhrPost({
            url: "./MapSheets.asmx/CreateMapSheet",
            handleAs: 'json',
            content: dojo.toJson(maprequest),            
            contentType: "application/json; charset=utf-8",
            load: function(data,args){                
                responseNode.innerHTML = data.d;
            },
            error: function(error,args){                
                responseNode.warn("Error!",error);
            }
        });

This got even more jacked up, and this is what was posted:

0=%7B&1=%22&2=e&3=m&4=a&5=i&6=l&7=A&8=d&9=d&10=r&11=e&12=s&13=s&14=%22&15=%3A&16=%22&17=m&18=e&19=%40

&20=f&21=o&22=o&23=.&24=c&25=o&26=m&27=%22&28=%2C&29=%22&30=s&31=e&32=r&33=v&34=i&35=c&36=e&37=N&38=a

&39=m&40=e&41=%22&42=%3A&43=%22&44=U&45=T&46=M&47=_&48=Z&49=N&50=1&51=0&52=%22&53=%2C&54=%22&55=m&56

=a&57=p&58=S&59=h&60=e&61=e&62=t&63=I&64=d&65=%22&66=%3A&67=%22&68=7&69=3&70=%22&71=%2C&72=%22&73=y&74

=e&75=a&76=r&77=%22&78=%3A&79=%22&80=2&81=0&82=0&83=7&84=%22&85=%7D

Ummm yeah.

Posting the Data to the Web Service Part 2: dojo.rawXhrPost

After much more Googling, I stumbled upon an example at the Project Zero forums. The scenario being discussed is similar - they were having issues posting, and they were seeing that the data was being sent as HTML FORM encoding, not Json. And the solution was to use dojo.rawXhrPost.

Using this, we see that the post actually contains the Json we created...

{"emailAddress":"me@foo.com","serviceName":"UTM_ZN10","mapSheetId":"73","year":"2007"}

and everything works smoothly.

Now one might ask - "How was I supposed to know that?" Great question. The Dojo API documentation for dojo.rawXhrPost  is obtuse at best. The Dojo Quick Start site only talks about xhrGet and xhrPost, and the Book of Dojo is the same.

However, the O'Reilly book "Dojo: The Definitive Guide" does actually cover this stuff - but it's still pretty slim.

If I have some time over the next few days I'll roll this into a sample (minus the mapsheet service stuff!)