My thoughts after Where 2008...

In my view, the "Where 2.0" movement/phenomena is about the democratization of mapping - remix this, mash in that, geocode it and get it on a map. My Map. With My Stuff and My Friends. Social graphs and all that. For many things, "close" really is good enough, and since it's "your" data, validation is less of an issue ( did you "really" like that pub? ) . Usability and performance are huge driving factors because many of these sites rely on users "wanting" to participate. This is driving the rapid evolution in terms of user experience, and poorly designed or slow sites fade quickly.

Meanwhile back in "GIScience"-land, the hard-core old guard have railed against pushpins as too simple, in-accurate or just lame. Not sure why there is so much insecurity, but maybe it's just a little envy that a bunch of javascript hackers made maps "cool". Many GIS applications have a "captive" audience (i.e. staff who have to use the site to do their job), usability has been low on the list, and performance an after though. The emphasis was placed on trying to cram desktop type work-flows and interaction models into a browser. Needless to say this has its drawbacks.

Not to use the John Hanke and Jack Dangermond session as too much of a metaphor, but these two camps seem to be coming together, or at least recognizing they both bring something valuable to the table. Sure there is still lots of misunderstanding, but things are getting better. Many of the "mash up" people I've talked to are realizing that the next step is to add spatial analysis. Not buttons & dials type analysis for the sake of it, but leveraged in context of answering a question. Sure the social graph is cool, but there are all sorts of new and interesting ways to leverage that when used in conjunction with spatial analytics. It opens lots of new scenarios. There is a realization that the scientifically backed, validated data has serious value in addition to "my data" or crowd-sourced data.

Going the other way, the GIS crowd (sadly under-represented at Where) is coming around to the idea that even their "captive" user base appreciates (or demand) a good user experience - design & performance matter. A lot more than they thought. And maybe it does make sense to dial back some of the "science" options in order to optimize performance. Some times "close" is good enough, and the performance trade off to get to "really really really close" is not always worth it.

This is a good thing and the whole industry / community will be better for it. GIScience sites/services that embrace the usability and performance of the neogeography sites will flourish if for no other reason than their users "like" using them. And if you're a social site looking to stand out, adding a shot of analysis into the sauce may be the trick.

It's been fun and I'm leaving with a head full of interesting ideas.

If you are presenting...

A parting word for anyone presenting at Where (or anything else) - Please please please read Presentation Zen (the book & site) before you show up.  I (and others I talked with) had really expected that these presentations would be on the level of TED - sadly this was the exception, not the norm. There were some big name companies up on stage with horrid slide decks - some were literally painful to view. Sure you have a cool 3 minute video that's super slick, and you likely spend 500K on, but following that with a jumble of bullets on neon-green background with heinous clip art some what spoils the effect. If you have 15 minutes in front of a crowd of thought-leaders, maybe you should spend more than 15 minutes on your slides.

Since there many other people are blogging the details of Where, I'm just going to take notes about the ideas that resonate with me...

Make your maps permalink friendly: Everyblock.com has a Uri hierarchy (RESTy?) that makes sense, and allows deep-linking and deep-search.

Need to do more than points: deal with "areas" - neighborhoods, routes etc. Especially relevant for geo-locating events/stories, where push-pins over simplify. The situation.

Roll your own maps: make the map relevant for your users. Don't show what you don't need. Control the look & feel of the cartography and make it part of the site design.

Federated Geodata: (Sean Gorman) someone other than ESRI talking about breaking data out of the silos. Not much detail on the "how", or synchronizing updates, but maybe he'll post more details.

GOOG & ESRI: Enabling data sharing / deep indexing. Good demos of ArcGIS Server analysis fronted by a consumer UI experience - geoprocessing in Google Earth. The "real" application were a change of pace - not that "my friend recommends the calamari at this place" isn't useful, but real-time forest fire modeling / evacuation planning just has a little more "bite" to it. 

Overall, an interesting morning. The John Hanke and Jack Dangermond session certainly got people talking - and that's always a good thing.  Should be interesting to see how many groups open up their ArcGIS Server 9.3 systems to Google's indexing. Discovering data that's been walled off for so long could be really interesting.

More later...

So the session was hampered by some corrupted data on the USB sticks, and was more a loosely guided tour of the stack than anything else. I had PostGIS up and running, but could not get my data (loaded into ArcSDE on PostgreSQL using the PostGIS datatype) to load into GeoServer. Tried some other data, and same problem. Unclear what the issue is.

Many of the sample SLD's would not validate (hence could not be loaded), and I could never get the WMS group layers to work. I did get some data up in Google Earth, and open layers, but overall this session confirmed for me that the open source stack learning curve is steep from the get go. I need to get into OpenLayers as the demo's were pretty cool but until I can get the back end all working smoothly it's somewhat moot.

I think that the docs for this will be up somewhere. The software installers are below.

Downloads: http://files.opengeo.org/where2/stick/software

Grass roots re-mapping efforts, focused on Europe. Crowd-sourcing. Pretty nice cartography. See the maps here: http://www.openstreetmap.org/

Note the pubs shown on the map via the Pint glass on the map. Nice.

Has features not in other maps - foot paths (dashed red lines), Pubs etc.

Starting in the US based off Tiger data, and updated.

The Back-End Story

Simple data model. Nodes, Ways (lines, or polys if node1 = nodeN), Relations

Runs on MySQL. 30,000,000 records. All changes stored. No "geo" columns, just points. Use scaled integers instead of doubles, and some saucy indexes. "SDE like" tiled indexes.

Planet.osm - rendering format in Xml. ~4Gb. Released weekly. From there you usually load it into PostGres with a Geocolumn, and then push this out via Mapserver, or GeoServer, or subset as you need.

Site runs in Ruby on Rails. OpenLayers for the slippy map. Off-line and on-line editors. All stored in Subversion

API was RESTful from day 1. http://www.openstreetmap.org/api/0.5/node/45

Tags - What is it?

Every object has keys and values associated with it. Community process for selecting keys and values. Keys and Tags must be approved, or features will not be on the map. There are 100's of tags.

Current Rendering

Mapnik (apparently easier and simpler than MapServer) c-based tool that uses a huge Xml file for styling (looks like SLD?). Runs off the Postgres version of the OSM data. Tiles are created on request, and cached. There is a background tile cacher daemon. Tile cutter is on one box, cranking like 20 million tiles a day.

Osmarender - Xml --> Xslt --> SVG. Slow to convert. Rendering is done across a network of end-user PCs. 

Tiles match the Google scheme, which makes mashups easy.

Now moving onto mobile platforms

OSM is topological from the start, so it supports routing. Currently there are some simple routing options, but until the data density is higher, it's not too useful. High-performance routing is CPU intensive, and no plans to add that as a service.

Lots of other projects going on, so check it out over at openstreetmap.org.

Ok, hopes for a Bruce Lee javascript smack down were thwarted when Andrew started off with "this is Where 2.0 101".

Started out with a quick run over the common formats - GeoRSS, KML and Microformats.

Microformats - additional HTML markup that are conventions for geolocating content. geo & adr for lat/long and address.

High-speed review of REST with "spiffy diagram"

Must do some reading on Atom - maybe have lunch with local Atom fan-boy Sean Gilles

What is Mapstraction?

Javascript mapping API wrapper covering least common denominator functionality across 9 or so map platforms. Essentially a layer of indirection that will allow you to change the back-end (i.e. Google Maps to Poly9) without breaking your application.

It can do image overlays - which is nice because you can then push very dynamic data (i.e. data that can't be tiled) into the map.

Nice tile layer schema for roll-your-own tile services

GeoRSS and geojson direct consumption.

Support Filtering by a range of things - category, date etc.

It does not break terms of service - it still uses the underlying API's, so the tile providers are cool with using this.

Still allows you to go below Mapstraction to the underlying provider. So, you'd be able to write most of the site in Mapstraction, then jump down into some VE/GM specific stuff if you need to (obviously this limits how you can change between the back-ends)

OpenSearch: Expose the search interface for a website directly into a browser search. Details at opensearch.org. OpenSearch-Geo - extensions to support location strings. Supports location name, lat/lon + radius, bbox.

This is very cool - need to push this into some of our sites.

Mapstraction is BSD license so do what you like.

Thanks to Ron Bruder who left a comment on the previous post with the link directly to the documentation for using the PostGIS geometries in the geodatabase. I did have some weirdness trying to access the link - of course you'll need to login because it's still beta documentation, but even after, that I'd see the page, and then it would jump me back to the login screen. I was able to access it by saving the link to Html and then viewing it. Moving on...

Using PostGIS Spatial Format

The ArcSDE doc says that to store data in PostGIS format, you need to create the "database" (presumably the "sde" database) using the postgis template, then create the sde user and sde schema before running the post install. Ok. Since I'd already done this, I dropped the sde database, then re-created it as per the instructions and... yeah.

So the post install failed. It did handily suggest looking at some logs buried away in the sde folder tree, and low the log did have a handy little gem:

db_table_grant_access: Grantee public does not have USAGE permission on schema sde.

Ok. Back into Navicat, grant USAGE to Public on the sde schema, and the post installer is completed. While my end goal is to use Direct Connect, to start, I'm just going to run the SDE service until I know everything else is working.

The next step is to get ArcGIS to use the PostGIS format. For this you need to edit the "dbtune file". From past experience with ArcSDE on SQL Server, I thought that "dbtune" was now a table, and it's a whole lot easier to edit a table than fiddle with the dbtune export/import mess. So, I simply updated the GEOMETRY_STORAGE parameter in the DEFAULTS section of the SDE_DBTUNE table to be "PG_GEOMETRY". So now I should be able to use ArcGIS Desktop to edit / manage data in PostGIS, while still accessing the data as native PostGIS format for say GeoServer and uDig.

Adding and Editing Data

I used the standard Import feature in ArcCatalog, and this worked smoothly. I then fired up uDig to see if it would show up a native PostGIS layer... yep... along with all the other tables in the sde schema. I think that there are likely better ways to organize your data, so this is likely not a "real" issue.

I had some issues applying a thematic renderer to the layer - for some reason it would only apply the renderer to the vectors. When I looked at the SLD xml, everything was setup as a <sld:LineSymbolizer>... since it looks fine in ArcMap (see below), and the original shapefile loads into uDig as polygons, this may be an "beta" issue. Ordinarily I'd check the beta forums right now, but I'm writing this at 38575ft en route to Where 2.0, so that will have to wait.

Loading data via ArcSDE into ArcMap works very smoothly.

And after telling ArcMap to let me edit "unversioned" data, I could edit as usual. uDig says I can't edit the layer. Being really new to this whole stack, I'm not sure if it's permissions (although I connect as the same identity in ArcMap), or a limitation (heck - I don't even know if uDig can edit vanilla PostGIS layers!)

I also wanted to try a direct connection - I stopped the ArcSDE service. Worked just fine - note the "service" name.

Summary

I've got a few other things I want to mess with, but I've got the basics working. It's been a few miss-steps but not too bad once I read the documentation. Although I've been using ArcSDE on SQL Server for a long time, it's definitely worth reading the docs when setting this up. Clearly I'd need to do more digging on optimizing the database structure, permissions and other sundry tuning things, but if you are looking to trade some time learning PostgreSQL for the $$$ involved buying Oracle or SQL Server, it's pretty compelling. I can't say much about performance as this is just one small layer (US Counties) running on my notebook. I do know people who are running some big databases on PostgreSQL + PostGIS (like millions of spatial features in a layer) and they've got it blazing.

So my company is sending me out to Where 2.0 - I've never been and expect it will be quite different from many of the other GIS conferences I've been to - more MacBooks and soul patches I'm guessing.

I'm particularly excited about the pre-conference workshops, as it will be a chance to get the low down on some technologies I have not worked with in the past.  I'm hitting the Geo-ify your Web Site session in the morning (Mapstraction) and the The Open Geo-Stack: OpenLayers, GeoServer, PostGIS session in the afternoon.

In prep for the workshops I thought I'd get PostGIS (with ArcSDE 9.3 beta) installed on my notebook - ideally I'd like to roll into the session with a little background so I can ask relevant questions!

Installing Postgresql + PostGIS + ArcSDE 9.3 (beta)

Overall, not exactly the smoothest installation I've ever done. No step was particularly difficult, there were just points where I had to do some head scratching, Googling and then "just do it". I have to give ESRI credit as the ArcSDE installer was smooth as silk, and they included Postgresql 8.2 on their media. However, having not used PostGIS before, exactly how this fits into the mix was a little unclear. I will say that if you have used ArcSDE in the past, and just want to get rolling without paying for a database - the install is really pretty easy.

As noted, I installed PostgreSql 8.2 directly from the 9.3 beta media - this is super simple. I then downloaded the PostGIS Installer for PostgreSql 8.2. Again this was pretty easy, after you locate and download it. As per the ArcSDE instructions, I then ran the ESRI post installer, authorized ArcSDE and all looked good.

I could immediately connect to the sde database as the sde user, and used ArcCatalog to load a shapefile into PostGIS. Smooth.

But how do I know that this is actually using "PostGIS"? Would this data be accessible to non-ESRI clients?

Enter uDig. This was a quick and painless install, and uDig was installed.

When I browsed to my local PostGIS server, things were not looking good.

Where was my data? Clearly something is not lining up here. When I tried to connect to ArcSDE via uDig (after I located the jar files) it could not connect. Not sure if this is related to using the beta 9.3 jar files, or something else.

Anyhow, I decided I should try loading the same data via the command line post gis tools. Here I ran into a bit of a wall. The documentation for the PostGIS tools is not bad, but certainly aimed at the unix sys admin type of person. The basic process is to convert your shapefile into a Sql file, and then run that in Postgresql. Seems simple. A quick little batch file ought to work this right out. The sql file was created just fine, but it failed to load.

Sorting this out really meant it was time to do a little learning about Postgresql. After sorting out schemas, and learning that the most likely encoding of my shapefile was "LATIN1" I was able to load my data and "see" it in uDig.

Actual loading the layer into uDig never completed. The application just sort of hung stating that the "Project command" was executing. This was just a map of all the counties in the US, so not sure what the issue was, but I'm pretty sure it was my fault.

I then tried to connect via ArcCatalog to see if I could "see" this layer... and not too surprisingly I could not.

Summary

Thus far I've simply come to realize that I have a lot to learn. I need to grok a lot more about Postgresql and PostGIS to start, and then add ArcSDE into the mix. I'm sure that if I read more of the ESRI documentation I'll find some answers, but at this point it's unclear to me if they will be supporting direct access to PostGIS data, or if they will be supporting the PostGIS spatial format for data stored in traditional ArcSDE schemas (i.e. with all the GDB_ and SDE_ tables). I also learned that I've been sheltered from the reality of "data" for many years!

I'll continue this series as I make progress getting things sorted out.

...from Jeff Atwood author of the (wildly popular) Coding Horror blog. Hopefully you all subscribe to his blog, but for those who may not, it's full of goodness, and you should check it out on a regular basis.

As for MVC - the results of the Geospatial Developer Survey show only about 50% of respondents are using patterns, and about 46% were not using any sort of Model View * patterns. This article is a great entry point to a pattern that can be applied to a very wide range of scenarios (including easing the creation of unit tests!)

Check it out at CodingHorror.com

Just a quick reminder that the Geospatial Developer Survey will be closing this Friday, May 9th.

We're at 262 respondents who are describing themselves like this:

Hey this is all .NET Stuff!

I've gotten a few comments back along these lines, so I thought I'd try to clarify this a little. Yes, on questions where you get to select a particular implementation, most of the options will be .NET, with one exception - "Other". I did not do this to slight other platforms, or indicate that .NET is somehow superior.  I live in .NET land, and I simply do not know a lot about non-.NET technologies. If you are doing automated builds with ant and maven, just pick Other and write it in (15 of you already have). I'll add these into the mix when crunching the numbers.

When will we see the results?

Survey Monkey does a great job of crunching out some simple results, and I expect to post a few interesting things during the week of the 19th.

Seems that I have not been reading my own blog on the site, and apparently the CSS was more than a little jacked up. If your read this @ the site, you may notice that it's now a fluid layout, and hopefully the font sizing makes a little more sense.

Anyhow - I'm also planning on updating my actual website (It's on the long list of "stuff" to do while out of the office and not a) hanging with my family, b) cooking or c) riding).  I've gotten quite a bit of feedback on the fundamentals section, and I've got some ideas on other topics that could be covered there. We'll see if I can get to that over the next few months.