Darb Dabney's 3d GIS – OpenSim

Blemishes notwithstanding, nearly six months of back-burner work has reached a threshold of readiness and is outward bound to some engineering firms, flood mappers at FEMA, and interested parties within the county. A handful of known issues remain unresolved. Proper name is “tbsm45cm_20100823″, proper edition is “2010.08″.

This is the third version of the terrain. Second version was “2010.01″ and included multiple LiDAR data sets, but fewer than presently used, and was a topographic model only. First version was “2009.09″ and was mainly photogrammetry and FEMA LiDAR, and was the last version to be developed in California Coordinates. Once the massive NCALM LiDAR data sets were processed, it became easier to move everything into WGS84 UTM zone 10 north meters projection, WGS84 NAVD88 CONTUS Geoid 2003 vertical position.

The NOAA utility program VDatum, a brilliant Java-based application able to stream-process data sets of near-infinite size, brought the NCALM data to heel, and opened up decades of NOAA depth surveys to our use in integrated topographic-bathymetric surface modeling.

First-return NOAA ALACE LiDAR swaths were fused along the outer coast, as bare-earth filtered versions were not produced in 1997–2002; the benefits of LiDAR detail along the rocky coast do seem to outweigh the distracting appearance of structures near Rodeo Lagoon, Stinson Beach, and outboard Bolinas.

When ArcGIS 9.4 beta 2 reached its limit in ability to render the terrain dataset into 45cm grid over the full extent, the clipping quadrants created to resolve this problem ended up chopping a very small portion of Sonoma county that drains into Estero Americano; the full watershed remains intact in the 1-meter version of the terrain grid under analysis for county-wide hydrology. Likewise, the tighter clipping quadrants lost a few hundred meters of San Pablo Bay bathymetry just west of where Marin, Sonoma, Contra Costa, and Solano counties meet. Also, tighter clipping quadrants snipped a portion of the San Francisco Bar southerly of San Francisco’s Seal Rock that was intended to be part of the model. All of these areas exist in the 100cm grid, and will be part of drainage analysis.

Happily, we have updated the workstation to ArcGIS 10, and have been enjoying such great speed gains with Spatial Analyst that our ERDAS use has been noticably reduced. Finally, Spatial Analyst is often showing performance nearly on par with ERDAS. Thank goodness that the Raster Calculator survived the transition to version 10 ArcGIS!

Painfully, the existence of unutilized bathymetric data sets for upper broad-channel Corte Madera Creek and Bolinas Lagoon have been revealed this week. Hey, there’s already something to look forward to for the next build!

The new terrain is getting some immediate use in support of an effort to participate in ESRI’s Community Maps Program for large-scale topographic mapping. The Program provides a template geodatabase with 36 vector feature classes and two raster, into which local agencies may pour their data. Once tucked into a conforming schema, a template multi-scale map document is provided with 120 layers—30 at each of four large scales that correspond to Google Maps and Bing Maps projection and cache tiling schema. The difference is that the template document makes use of ESRI tools to allow much more local detail to be packed into a map designed with notably more sophisticated cartography than either Google or Bing maps now have. The Community Maps Program concept is that local agencies may publish their local detailed content in a fairly uniform style, while retaining a world-wide seamless context for their surrounding area.

Qualitatively, the effect is that, when viewing the ArcGIS.com topo map alongside either Google or Bing maps (on two monitors, with comparison made at the same scale), the ArcGIS.com map looks to be a larger scale. It isn’t, and I’ve measured the size of features to convince myself, but my mind insists that I’m zoomed farther in on the ArcGIS.com map for some reason. My guess is that it is a perceptual effect of the much greater amount of information that is cleanly displayed in the ArcGIS.com map versus the much sparser Google and Bing content at these large zoom levels. Try it out—it’s like a carto version of an optical illusion!

The 120 layers in the template large-scale topographic base map from the ESRI Community Maps Program are arranged to provide four precise cartographic designs for Google/Bing map cache levels 16 through 19, which correspond to these display scales
1:15000–1:6001 (level 16, a.k.a. ~9k)
1:6000–1:3501 (level 17, a.k.a. ~4.5k)
1:3000–1:1501 (level 18, a.k.a. ~2k)
1:1500–1:501 (level 19, a.k.a. ~1k)
One of the most attractive areas currently online is Toronto, ON where at levels 18 and 19, individual building outlines are graced with street addresses.

Anyhow, the new tbsm45cm model will serve County of Marin’s effort at large-scale topographic mapping several ways. First, it has made possible a very detailed hillshade that helps emphasize the grading around each hillside structure in the county. Second, it helps us to create the required metric topographic contours. These are necessary to meet world-wide mapping standards, and throughout this weekend, contours are being generated from a related (smoothed version) of the terrain on 50cm vertical interval. Needless to say, most of these won’t get used in the map renderings, but the ESRI cartographers have shared a very clever indexing scheme that will help us use this single set of metric contours to support the requirements for all four of our topographic map scales.

This edition of the county-wide terrain model has been upgraded several ways in terms of input data sets, extent on land, and integration of bathymetric soundings to eliminate coastline clipping.

The LiDAR data set from NCALM / GeoEarthScope has been reprocessed to use inidividual ground-classified returns rather than the 50cm gridded surface; this was made possible by NOAA VDatum, the Java application that has industrial-strength powers to position XYZ files.

Also, NOAA Airborne LiDAR for Assessment of Coastal Erosion (ALACE) data, which are first-return only from 1998–2002, were included where available along the Pacific ocean coastline and edges of some estuaries.

An Airborne 1 LiDAR data set was graciously provided by Marin Municipal Water District where it was available in the lower Lagunitas Creek drainage, and this was used where available and not overlapping with the higher-density NCALM data at Point Reyes Station.

More of the FEMA LiDAR data from Dewberry was included along the eastern, urban floodplain areas. In this terrain model, all FEMA points below 25 meters elevation NAVD88, OR points landing on areas with slope less than or equal to 11 percent, were included. In the 2010.01 terrain, only points below 50 feet NAVD88 were used.

NOAA depth surveys were included with very little filtering, for all nearby soundings since 1931—knowing that tidal bars are dynamic, but including all data as a starting point.

The California Seafloor Mapping Project’s phenomenal multibeam sonar work was incorporated from 2-meter grids offshore to California’s 3-nautical-mile limit, and 1-meter grids within west San Francisco Bay.

The earlier photogrammetric breaklines from VARGIS/Infotech were classified into ridge and road types, and only the road types were retained as hard breakline constraints. Ridge lines and water lines were retained as soft breakline constraints. This has mitigated some of the effects of ridge lines artifacts that derive from inconsistencies between the VARGIS photogrammetry work breaklines and contours, and between VARGIS breaklines or contours and overlapping LiDAR data sets.

Gridding the terrain dataset into a dsm, once again we are flirting with the limits of ArcGIS stability. The processing workstation is imaged with Windows Server 2003 to remove limits on output file size, and to permit reliable killing of wayward processes. (At this time, we’re not certain whether this constitutes a “top kill”, or a “static kill”, but in any case the process ends up terminated ;^)

The 2010.07 edition is a topographic-bathymetric surface model, and its prime use case is the generation of accurate synthetic drainage networks. Together, these features motivate a larger modeled extent than the 2010.01 edition. Along the northern area, the extent was grown to include all watersheds that touch or drain Marin County areas, with some clipping of the easternmost portions of the Petaluma River watershed. Offshore, the legal extent of Marin County guided the clipping area in San Pablo Bay; data extent from CSMP guided eastern limits at Richmond Channel, San Francisco’s North Beach and Presidio shorelines, and westerly from Seal Rock to the 3-nautical-mile limit, then northerly to include all of Bodega Head as imaged by NCALM data.

The extra area has made it infeasible to generate a single 40-cm grid as before, but we have cut the area into quadrants and have generated four tiles as 45-cm gridding. More news soon.