Darb Dabney's 3d GIS – OpenSim

(With apologies to Bill Ham and ZZ Top)

Some progress, and apparently some shifting ice floes underfoot.  Some blog posts say that the SQL Server Management Studio (SSMS) 2010 Express does not feature a “Spatial Results” tab for queries.  In fact, at least this week, it does.  It does seem like full-bore database development tools are withheld  by Microsoft until the Visual Studio Premium 2010 tier, and yet I’m  presently unable to locate  an IntelliSense add-in for the OGC-compliant methods on Geometry instances.

Here they are, 43 OGC methods on geometry instances,
plus 16 OGC static geometry methods,
along with the typical MS-style augmentation
11 augmented MS-extended methods on geometry,
and 4 augmented MS-extended static geometry methods.

Another interesting topic is how to merge certain feature classes for use by both ArcSDE and SQL Spatial.  In this regard, since ArcGIS Server 9.3 it appears that one can have ArcSDE store feature classes with SQL Server 2008 GEOMETRY data type.  Here’s an ESRI documentation page on the subject.

Sometimes, it’s the simple stuff that one stumbles over (at least I do).  Here’s some of the bootstrapping stuff for editing SQL in the Visual Studio 2010 environment.  Plus, although I’m personally comfortable with programming languages like structured C, and have been trained in (but not worked professionally with) Objective C++ and Visual C++ (at 6.0), I’m not finding immediate transparency with C#; most of the Visual Studio Web Developer  tutorial examples use Visual Basic.

1) How to Start the T-SQL Editor
2) The intro videos for C# development
3)

Many interesting projects have been happening, so many that projects have been backed up while workstations grind through their days-long work flows. This creates opportunities to update systems while we wait, and so the past week has seen massive updating of Windows system stuff and deleting old development tools so that the decks were cleared for MS SQL Server 2008 R2 Management Studio. With a bit of shape2sql from SharpGIS and today we’ve had our first spatial tables loaded and fledgling spatial SQL queries made.  Oh, at the end of the day we got the third of five workstations updated to ArcGIS Desktop 10 and the full complement of seats moved over to ArcGIS 10 licensing.

What’s been holding things up has been some final processing of cartographic-grade contours, one of the key new feature classes developed for support of the ESRI Community Maps Program, where local jurisdictions grind their own cache tiles for large-scale topographic mapping.  Since it’s part of a worldwide seamless map, of course the contours and spot elevations must be metric—which meant generating new contours.  The bathymetry was done on one-meter interval, with some half-meter supplemental contours in shallower waters.  The topography was done at quarter-meter intervals up through 25 meters elevation, where we’d included all available LiDAR data, the half-meter intervals through 50 meters, and one meter up to the summits.  All non-integer meter contours were flagged as supplemental, and contour index attributes were also calculated for 2, 5, 10, 20, and 50-meter intervals.   Spot elevations were derived from VARGIS photogrammetric spot elevations that were screened down from about 75,000 points to just 440 points, using neighborhood focal statistics.

For performance’s sake, after index attributes were calculated, the contours were chopped into segments not-to-exceed 500 meters shape length.  When all the chopped segments from all the various elevation ranges had been merged to a single polyline feature class in a feature dataset in a file geodatabase, spatial indices were built for the cache tile scales about 1:1200, 1:4800, and 1:19000.  The purpose is to make the contours as fast as possible not just for web app interaction, but for rendering the large-scale cache tiles.  In the end, the contours in this one feature class have about 1.4 million polyline features that fill the file geodatabase to about 6.8 GB of data.

Meanwhile, some great brainstorming has taken place with regard to hydro-enforcement of the terrain model, so that accurate synthetic flow lines can be generated county-wide.  Based on prototype work performed by Evan K. Babb before summertime, we knew how to deal with our terrain features and manually generate hydro-enforcement features.  Now, with a funded project to do the work county-wide, we’ve needed to devise a spatial analysis technique using the approach described by Poppenga and others in 2009.  Based on some correspondence and personal communication with Poppenga, we have sketched a workflow that should automate the creation of most simple (single road-crossing) hydrologic enforcement features, especially in the absence of accurate or complete spatial features for culverts.

So with the contours completed, it should be possible to have some progress made on automation of hydro-enforcement, completing more of the data development for large-scale topographic base mapping, help my colleagues continue the ArcGIS 10 migration, and follow up with some demonstration spatial SQL queries.  It’s an exciting couple of weeks!

There’s also been a thread of thought over the past six years or so that supports the workflow of using Visual Studio to compose SQL queries, in preference to the direct SSMS (SQL Server Management Studio, Tool Formerly Known As “Enterprise Manager”) ways of composing a query, because the typed pattern matching (MS: Intellisense) pulls up all the various ST__  spatial methods that are available in SQL Server 2008 spatial, but saves one needing to look-it-up and type-it-in correctly.  Such productivity gains really help boot up the learning experience a bit faster!

Opensim has slipped to a back burner for these past few weeks, as I button up a new edition of county terrain–this one fused topography and bathymetry.
Another focus has been to compile the map data layers necessary to publish a local edition of the large-scale topographic map, using the ESRI Community Maps Program template. This (Opensim-distracting) work forms a synergy, as the terrain drives revised hillshade and topo-bathy contour lines, as well as forming the basis for refined surface flow and stream mapping, all of which show up in the large-scale topo map. Also, I’ve compiled a great deal of web research into a single feature layer: classified building footprints, many of which have use categories [commercial, medical, educational, religious, etc.] and specific organization names included, for every structure on the associated parcel.

Why generate this large-scale topographic base map? It’s purpose is to provide local control over local detail, in a cloud-based service suitable for desktop, tablet, and mobile phone users who may consume applications that mash or overlay the base map with relatively sparse point or line features. While we’ll be able to serve up the base map through our own MarinMap servers, one of the key benefits of the conforming large-scale topographic map style (as made possible by the template) is that the caches of the four largest-scale views will be rendered and contributed for inclusion in the ArcGIS Online map services, as viewed at arcgis.com and consumable by many interactive map sites. The cached map tiles are essentially raster prints of the map template (which has styles customized at four local zoom levels) that can be served extremely fast.

Those fast maps can also be efficient, for mobile phones and tablets consuming base maps with no special rendering. Along that path, I’ve rebuilt my mobile phone this past week, taking the T-Mobile G1 and rooting it, then installing various new images until I found one that was up-to-date, stable, and attractive to use. As of this weekend, I’m now running Chromatic 4.5 with SetCPU 3.0.2 overclocking. Chromatic 4.5 is a packaging of CyanogenMod 6.0.0 with the ADW launcher and many other tunes. CyanogenMod 6.0.0 is an Android Open Source Project-based build of Android version 2.2 “Froyo”. So now, when I view maps and browser content, I can pinch to zoom out, spread to zoom in, and still push around the map to pan it on the screen. As part of sharing content with mobile phones, I’ve learned how to generate QR tags to provide access to map URL or geolocation tags.

For fun today, we visited the Silicon Valley area to show a guest some of the sights. I hadn’t visited the Googleplex in a couple of years, and many of the adjacent buildings that I recall as having different companies in them are now part of the Google campus. We found one interesting sight, too–and it appears that today may have been Google Inc.’s 12th birthday. We didn’t see any partying in the mostly-empty parking lots, which wasn’t surprising for a Sunday!

Blemishes notwithstanding, nearly six months of back-burner work has reached a threshold of readiness and is outward bound to some engineering firms, flood mappers at FEMA, and interested parties within the county. A handful of known issues remain unresolved. Proper name is “tbsm45cm_20100823″, proper edition is “2010.08″.

This is the third version of the terrain. Second version was “2010.01″ and included multiple LiDAR data sets, but fewer than presently used, and was a topographic model only. First version was “2009.09″ and was mainly photogrammetry and FEMA LiDAR, and was the last version to be developed in California Coordinates. Once the massive NCALM LiDAR data sets were processed, it became easier to move everything into WGS84 UTM zone 10 north meters projection, WGS84 NAVD88 CONTUS Geoid 2003 vertical position.

The NOAA utility program VDatum, a brilliant Java-based application able to stream-process data sets of near-infinite size, brought the NCALM data to heel, and opened up decades of NOAA depth surveys to our use in integrated topographic-bathymetric surface modeling.

First-return NOAA ALACE LiDAR swaths were fused along the outer coast, as bare-earth filtered versions were not produced in 1997–2002; the benefits of LiDAR detail along the rocky coast do seem to outweigh the distracting appearance of structures near Rodeo Lagoon, Stinson Beach, and outboard Bolinas.

When ArcGIS 9.4 beta 2 reached its limit in ability to render the terrain dataset into 45cm grid over the full extent, the clipping quadrants created to resolve this problem ended up chopping a very small portion of Sonoma county that drains into Estero Americano; the full watershed remains intact in the 1-meter version of the terrain grid under analysis for county-wide hydrology. Likewise, the tighter clipping quadrants lost a few hundred meters of San Pablo Bay bathymetry just west of where Marin, Sonoma, Contra Costa, and Solano counties meet. Also, tighter clipping quadrants snipped a portion of the San Francisco Bar southerly of San Francisco’s Seal Rock that was intended to be part of the model. All of these areas exist in the 100cm grid, and will be part of drainage analysis.

Happily, we have updated the workstation to ArcGIS 10, and have been enjoying such great speed gains with Spatial Analyst that our ERDAS use has been noticably reduced. Finally, Spatial Analyst is often showing performance nearly on par with ERDAS. Thank goodness that the Raster Calculator survived the transition to version 10 ArcGIS!

Painfully, the existence of unutilized bathymetric data sets for upper broad-channel Corte Madera Creek and Bolinas Lagoon have been revealed this week. Hey, there’s already something to look forward to for the next build!

The new terrain is getting some immediate use in support of an effort to participate in ESRI’s Community Maps Program for large-scale topographic mapping. The Program provides a template geodatabase with 36 vector feature classes and two raster, into which local agencies may pour their data. Once tucked into a conforming schema, a template multi-scale map document is provided with 120 layers—30 at each of four large scales that correspond to Google Maps and Bing Maps projection and cache tiling schema. The difference is that the template document makes use of ESRI tools to allow much more local detail to be packed into a map designed with notably more sophisticated cartography than either Google or Bing maps now have. The Community Maps Program concept is that local agencies may publish their local detailed content in a fairly uniform style, while retaining a world-wide seamless context for their surrounding area.

Qualitatively, the effect is that, when viewing the ArcGIS.com topo map alongside either Google or Bing maps (on two monitors, with comparison made at the same scale), the ArcGIS.com map looks to be a larger scale. It isn’t, and I’ve measured the size of features to convince myself, but my mind insists that I’m zoomed farther in on the ArcGIS.com map for some reason. My guess is that it is a perceptual effect of the much greater amount of information that is cleanly displayed in the ArcGIS.com map versus the much sparser Google and Bing content at these large zoom levels. Try it out—it’s like a carto version of an optical illusion!

The 120 layers in the template large-scale topographic base map from the ESRI Community Maps Program are arranged to provide four precise cartographic designs for Google/Bing map cache levels 16 through 19, which correspond to these display scales
1:15000–1:6001 (level 16, a.k.a. ~9k)
1:6000–1:3501 (level 17, a.k.a. ~4.5k)
1:3000–1:1501 (level 18, a.k.a. ~2k)
1:1500–1:501 (level 19, a.k.a. ~1k)
One of the most attractive areas currently online is Toronto, ON where at levels 18 and 19, individual building outlines are graced with street addresses.

Anyhow, the new tbsm45cm model will serve County of Marin’s effort at large-scale topographic mapping several ways. First, it has made possible a very detailed hillshade that helps emphasize the grading around each hillside structure in the county. Second, it helps us to create the required metric topographic contours. These are necessary to meet world-wide mapping standards, and throughout this weekend, contours are being generated from a related (smoothed version) of the terrain on 50cm vertical interval. Needless to say, most of these won’t get used in the map renderings, but the ESRI cartographers have shared a very clever indexing scheme that will help us use this single set of metric contours to support the requirements for all four of our topographic map scales.

This edition of the county-wide terrain model has been upgraded several ways in terms of input data sets, extent on land, and integration of bathymetric soundings to eliminate coastline clipping.

The LiDAR data set from NCALM / GeoEarthScope has been reprocessed to use inidividual ground-classified returns rather than the 50cm gridded surface; this was made possible by NOAA VDatum, the Java application that has industrial-strength powers to position XYZ files.

Also, NOAA Airborne LiDAR for Assessment of Coastal Erosion (ALACE) data, which are first-return only from 1998–2002, were included where available along the Pacific ocean coastline and edges of some estuaries.

An Airborne 1 LiDAR data set was graciously provided by Marin Municipal Water District where it was available in the lower Lagunitas Creek drainage, and this was used where available and not overlapping with the higher-density NCALM data at Point Reyes Station.

More of the FEMA LiDAR data from Dewberry was included along the eastern, urban floodplain areas. In this terrain model, all FEMA points below 25 meters elevation NAVD88, OR points landing on areas with slope less than or equal to 11 percent, were included. In the 2010.01 terrain, only points below 50 feet NAVD88 were used.

NOAA depth surveys were included with very little filtering, for all nearby soundings since 1931—knowing that tidal bars are dynamic, but including all data as a starting point.

The California Seafloor Mapping Project’s phenomenal multibeam sonar work was incorporated from 2-meter grids offshore to California’s 3-nautical-mile limit, and 1-meter grids within west San Francisco Bay.

The earlier photogrammetric breaklines from VARGIS/Infotech were classified into ridge and road types, and only the road types were retained as hard breakline constraints. Ridge lines and water lines were retained as soft breakline constraints. This has mitigated some of the effects of ridge lines artifacts that derive from inconsistencies between the VARGIS photogrammetry work breaklines and contours, and between VARGIS breaklines or contours and overlapping LiDAR data sets.

Gridding the terrain dataset into a dsm, once again we are flirting with the limits of ArcGIS stability. The processing workstation is imaged with Windows Server 2003 to remove limits on output file size, and to permit reliable killing of wayward processes. (At this time, we’re not certain whether this constitutes a “top kill”, or a “static kill”, but in any case the process ends up terminated ;^)

The 2010.07 edition is a topographic-bathymetric surface model, and its prime use case is the generation of accurate synthetic drainage networks. Together, these features motivate a larger modeled extent than the 2010.01 edition. Along the northern area, the extent was grown to include all watersheds that touch or drain Marin County areas, with some clipping of the easternmost portions of the Petaluma River watershed. Offshore, the legal extent of Marin County guided the clipping area in San Pablo Bay; data extent from CSMP guided eastern limits at Richmond Channel, San Francisco’s North Beach and Presidio shorelines, and westerly from Seal Rock to the 3-nautical-mile limit, then northerly to include all of Bodega Head as imaged by NCALM data.

The extra area has made it infeasible to generate a single 40-cm grid as before, but we have cut the area into quadrants and have generated four tiles as 45-cm gridding. More news soon.

As I learned here, it is even easier than ever for Windows users to browse the Marin dsm40cm terrain.
All those keyboard shortcuts one may have learned to use Google Earth now work in Google Maps with the “Earth” button.
It’s a gas, and all the 40-cm detail is there.

I’ve tested it using Chrome browser on Ubuntu, and the Earth button does not show up.
But using Chrome on Windows XP, it’s there and it works pretty well. No need for composing a web site with the Earth plug-in. It’s just there in the default Google Maps on systems where the plug in would work.

For many of the world’s users of browser-based interactive mapping, integrated 3D viewing has arrived.
Check it out. Link provides an initial point near Civic Center, viewed in Google Earth through Google Maps.

Users who access Google Maps using Windows or Mac should be able to seamlessly launch the Google Earth plugin within Maps.
Just click the Earth button (should be up where the different views are in the upper-right part of the map) if it exists.
If it doesn’t exist, then you are using Linux or a smartphone or are in some other environment that doesn’t support the Earth browser plug-in.

My favorite Google Earth navigation shortcuts:

(Mouse control)
- Drag Mouse regular button to push globe around.
- Release while moving to let globe continue to move with intertia.

Mouse scroll one way or PageUp = pan forward fast
‘+’ = pan forward slowly
Mouse scroll other way or PageDown = pan reverse fast
‘-’ = pan reverse slowly
‘r’ (lowercase R) = return to vertical view, North upward: “get me out of this mess and back to Map view”

(keys used by themselves)
‘w’ or up-arrow-key = spin globe forward
‘a’ or left-arrow-key = spin globe left
‘d’ or right-arrow-key = spin globe right
‘s’ or down-arrow-key = spin globe back

(Ctrl- variants)
(two non-opposing directions at once are allowed)
Ctrl – ‘w’ or up-arrow-key = tilt view upward toward the zenith
Ctrl – ‘a’ or left-arrow-key = pan view left
Ctrl – ‘d’ or right-arrow-key = pan view right
Ctrl – ‘s’ or down-arrow-key = tilt view downward toward nadir

(Shift- variants)
(two non-opposing directions at once are allowed)
Ctrl – ‘w’ or up-arrow-key = orbit point-of-view downward around center view
Ctrl – ‘a’ or left-arrow-key = orbit point-of-view rightward around point in center view
Ctrl – ‘d’ or right-arrow-key = orbit point-of-view leftward around point in center view
Ctrl – ‘s’ or down-arrow-key = orbit point-of-view upward around center view

WILD and FAST
Use Mouse alternate button to free-form navigate
- where you “right” click on the map, a target appears for as long as you keep the alternate button down.
relative to the target:
- drag the cursor up to zoom out
- drag the cursor down to zoom in
- drag the cursor right to rotate view counter-clockwise
- drag the cursor left to rotate view clockwise.

And if everything is just too much, there’s always ‘r’ (lowercase-R) to set everything straight.

Enjoy!

It’s not fully 3D immersive, but hey, 2-1/2D ain’t half bad. The “dsm40cm” model of Marin County has been published as the county’s default terrain on Google Earth. It’s a great pleasure to work with folks who are not troubled by a county representing its surface on a 40cm single-precision float grid that weighs in at 77 GB. In terms of data bulk, that is about the same as the entire 30-meter version of the US National Elevation Dataset.

What one gets when piling that much detail into a single county of around 520 square miles of land area is every building pad, driveway, and crown of road paving that were resolved. The dsm40cm model was derived from an ESRI Terrain Dataset that incorporates our best available topographic contours (1:4800 scale 10-foot; 1:2400 scale 2-foot,) photogrammetric break and water lines, FEMA LiDAR and NCALM (GeoEarthScope) LiDAR data sets. The Terrain Dataset currently comprises 40 GB of vector GIS data.

When the finely detailed surface grids were first developed, we broke the county up into 20 work areas to maintain ArcGIS 9.3.1 in a stable and productive state, and 30cm posting interval grids were generated that covered the entire county–at least during development. When necessary, these grid tiles were mosaicked with ERDAS Imagine into a single seamless grid. The 40cm version was produced directly as a single seamless grid using ArcGIS 9.4 beta 1, on a workstation imaged with Windows Server 2003. The WGS84 UTM, NAVD88-Geoid 2003 result was provided to the Google Earth team earlier this year.

As with all GIS data sets, it seems, the more detailed it is, the more rapidly it may need updating. In the works for the next year or so are several improvements to the dsm40cm model. First: the photogrammetric break lines will be segregated into steeper sets that tend to run along ridges, and shallower slopes that tend to delineate road cuts and building pads. The ridge set will be used as soft constraints to resolve some artifacts where they rise above some contours.
Second: incorporate new LiDAR data as it becomes available. Some data has already been provided for the lowest part of Lagunitas creek, and it appears that Prof. Ellen Hines of San Francisco State University’s Department of Geography and Human Environmental Studies has been funded by USGS to gather LiDAR county-wide this year.

So there will be revisions, but an exciting aspect is to see data flows being brought into existence that support different levels of mirror world development.
Publishing the dsm40cm model in Google Earth is an important (and beautiful) threshold to cross. Making use of the dsm40cm model in county operations such as creek and watershed delineation will be the practical benefit that drives the work in the first place. And before too many more weeks, there may be entirely new approaches to publishing the data in an immersive environment (neither Second Life nor Opensim) to share.

Kent Woodlands building pad and driveway, in the shadow of Mt. Tam

Next week I plan to be on a panel at the Virtual Edge Summit 2010 in Santa Clara on Monday 22 Feb. Inspired by the style of a presentation by State of California GIO Mike Byrne at last month’s BAAMA.org educational session, I will try to prepare an Ignite-inspired talk for Virtual Edge. Since the virtual environment space almost demands it, I will use a film rather than a slide stack to structure my speaking.

The film is mostly cut, and a draft is uploaded in HD – for your perusal. If you want to hear my words try to keep pace with the film, consider attending Virtual Edge Summit, either in person or through one of its virtual channels. Hope to see you there!

If you have the bandwidth, please view the film in HD–I spent most of the past weekend cutting original FRAPS takes at full resolution so that the presentation could stand up to 1080p HD.

Many thanks to Singularity U, director Matt Rutherford, and to Randall Hand who brought it to my attention After chatting at SLCC 2009 this past summer, I appreciate the immediacy of this lecture. OpenSim is discussed around minute 37 (video is available at 720p HD, and is just over 51 minutes long.)
Discussion of augmented reality, and mirror world creation in Second Life and virtual world simulators, just after minute 44.

It’s hard for me to listen to the entire talk just one time and retain the best explanations – but clear and current they are. In a virtual environment, immersed in near-infinite possibilities, Rosedale may no longer be guiding the Second Life ship, but I believe he remains the compass needle

I don’t have much to say about these regions that hasn’t been written already, and my views have been less aesthetic than Shenlei’s.
But in the interest of boosting the bandwidth by which I can share OpenSim, I’ve invested in a much newer Adobe Premiere Elements than I’d been using for the past five or so years. It’s a gas to have it multi-thread while rendering, and I have direct-to-FLV write. Trying to share as much of the motion and fidelity via YouTube as I possibly can, I’ve crafted a video resolution that is a multiple of my Hippo / SL viewer screen. The FRAPS video direct to AVI (sorry, it’s Win XP) is 1600 x 1140 @ 10 fps. Yup, those are video frames. In the interest of surviving an upload, I’ve rendered them highly time-compressed, with output at 1515 x 1080 @ 15 fps. As of tonight there’s no sound, no intertitles, just the rushes.
oops, if I read the YouTube Instructions for best formats, I should have trimmed the width to 1440, which is a multiple of 16.
Also, I have more direct upload options now with Premiere 8 than I had with my (recently demised) copy Premiere 1.0. Go Figure ;^)
While the Windows box grinds out the video print, I’m over here on Ubuntu blogging in a tab of 64-bit Chrome 4.0.249.43 and it is fine & fast.

For these videos, I visited ScienceSim Geography22_44 region and set the view to wide angle, then sat up at about 500 meters and watched the regions rez their terrain. For some folks, it will rank right up there with watching lead-based paint dry. For geography folks I’m hoping that these few minutes of sped-up video will convey, by dogged repetition, the primacy of regions in the provision of virtual environment simulators.

By the way, I’ve got a task: I need to find a better buzz word for the GIS community. I’ve been advised by some serious and well-intentioned (not to mention well-informed) folk that terms like “virtual” and “immersive” are actually boring to GIS’ers. So I’ll need to think about how to convey the concepts of “Mirror World”, “Multiuser Virtual Environment”, “Immersive Connected Experience”, “Third-Person Virtual World”, and related concepts into a catchy moniker. Hopefully, one that is not presently trademarked, either!

I’m trying to remain serious about this, but some of the options are treacherous. Geography in Social Media has a possibly awkward acronym; maybe it can be saved in recognizable form as “GIS for Social Media” or “Geography for Social Media”: GFSM
The term “3D Map with Me” is terse, slightly ambiguous

Here is the video chopped as it was when uploaded with 1515 x 1080 resolution. Problem with that is that by not preserving dimensions at a multiple of 16, and saving my viewer’s aspect ratio rather than the (standard since 16mm film) 4:3 aspect, my upload is clobbered into something perhaps suitable for a smartphone. So please consider this the Smartphone Version of last night’s rushes:

Then, once again with feeling, or at least with a little more rest, there is what I hope to be an HD-friendly moving vision of OpenSim, as it appears on the ScienceSim Geography regions. Yes! After it ripened on the YouTube servers for a few hours, I now see all the higher-res versions available. At 1440 x 1080, this is pretty close to what I see on my screen with a live Hippo viewer.

And after a day’s cogitation: anyone care to comment on the term: “Social Immersive Media GIS” as a moniker? Oops — I used “immersive” 8^(