Archive for the 'Large-scale Topo Map' Category

Apr 05 2011

Cache on the Barrel – Community Maps Work

The CPU cycles are burning as I set up some test cache building for our local edition of ESRI Community Maps – Large-Scale Topographic Base Map.  It’s rather a treat to see a bunch of execution threads and trying my best to keep them all busy!

As I work through glitches here and there, I’m catching up on some of the good information that is online to help folks in my situation, like Advanced Map Caching from the 2010 ESRI Developer Conference.


No responses yet

Jan 07 2011

Terrain progress linked to Community Basemap Large-Scale Topo

The ESRI Community Basemap program’s new ArcGIS 10 template for Large-Scale Topographic Mapping is somewhat streamlined and cleaner when compared to its ArcGIS 9.3.1. predecessor.  But the template map document still has the ability to bring my workstation to its knees when doing stuff like printing.  I find myself unable to print 11×17 or export a B-size PDF without having incomplete results.  Mercifully, I can print and export to A-size so people are starting to get a taste of what is to come when we start building cache tiles.  Everyone who has seen the base maps was quite pleased, and some were almost gushing over them.

Preparing input features to pour into the template geodatabase that  are good enough to be worthy of 1:1200 scale mapping is every bit the challenge it might seem.  Each feature class that is making its way into the template seems to require a unique bit of spatial analysis to get pulled together, and usually this involves combining our best available input data in ways that we haven’t before.  This week’s challenge has been water polygon and water line feature classes.  For utility, we’re choosing to split out the water polygon from its water line.  For lakes and ponds, they will coincide, but for our tidal coast, we’re conveying extra information by setting the polygon to represent Mean Low Water, and the water line to represent Mean High Water.  This helps to convey the widely varying slopes that Marin County has at the tidal shores, and provide some hint as to the width of the public access way below the high tide line.

These tidal shores have been compiled from the 25cm-interval topo contours that were generated for the Community Basemap, and in NAVD88 we used nominal 50cm contour for Mean Low Water, and 175cm contour for Mean High Water.  Between these elevations one often finds various “water” polygons depending on the application; in our base map we’ll have a purposeful gap.

Along the Sonoma borderlands I struggled for a few hours hand-tracing ponds using NAIP 2009 near-infrared grayscale—the one where standing water is basically black.  I was a bit sloppy and found myself tracing at 1:4000 on screen but it was a fairly thorough job through the adjacent shared watersheds along Estero Americano, Stemple Creek, San Antonio Creek, and Petaluma River.  I was just about fed up with drawing lines around standing water when I checked in with MarinMap member Bill Voigt of San Rafael for other reasons.  He provided an inspired suggestion that I use the water lines from our 2004 photogrammetry.  Of course!  I had used them in the terrain, but they were not hard constraints and the sparse contouring in west county lands tended to wash over stock ponds in the contours.

Empowered by Voigt’s suggestion, I found that the 2004 photogrammetric water line work, which was only available within Marin County proper, had very high fidelity with 2009 NAIP summer ponds, and it was vastly easier to select the pond-circumscribing lines, sometimes 300 segments at a time, copy and paste them into the Inland_Waters_li features, Merge them all at once into a single compound line, then Explode the multi-parts into discrete ponds.  With that approach I was able to harvest hundreds of Marin stock ponds, reservoirs, and vernal pools, and have geometry that appears smooth and accurate at better than 1:2400.  There are also quite a few meandering creek features in the west county that are well represented horizontally in the water break lines, but my impression of them was that they took a lot of upgrading to serve as 3D guides in the terrain, and construction rules broke the pond lines wherever a drainage reached its edge.  So there were a great many segments that needed to be merged to create closed loops that could build pond polygons.

By the end of this week, we had over 1400 water bodies identified in the watersheds that touch Marin county, and this exercise that was initially motivated by improved cartography may augment our set of candidate sites for wetland inventory.  These  water features are set into a regional water layer from The National Map that is valid at 1:24k and includes a refined coastline from Big Sur to north of Point Mendocino, and inland to the Sierra crest, from San Luis reservoir up to Mendocino Lake.

We refined Marin and adjacent shorelines (as both MHW and MLW boundaries) from Sears Point, up to and down from Petaluma, along San Pablo, San Rafael, Richardson and San Francisco Bays, the Golden Gate, Gulf of Farallones, Drakes Bay, outer Point Reyes, Tomales and Bodega Bays, and Bodega Head using 25cm interval topographic contours.  All Farallon Islands were traced near 1:2400 scale as visible in NAIP 2009 near infrared band.

Next week should find the last couple of administrative layers (Parks and Open Space) and refinements to road centerlines to reflect docks and proper arterial status, and then it will be time to start making test tiles.  With any luck, we’ll take some CPU cycles from production server to give us the best shot at getting good tiles fast.  With 120 layers, several of which use representations, and many of which use complex Maplex rules for labeling, the Community Basemap Large-Scale Topographic template for ArcGIS 10 is by far the most demanding map document I’ve ever manipulated.  Everyone is looking forward to getting the tiles generated for testing, and sending them off to ESRI for review while sending samples to MarinMap members for their comments as well.

No responses yet

Oct 20 2010

Oh say, can you SQL (Spatial)?

Many interesting projects have been happening, so many that projects have been backed up while workstations grind through their days-long work flows. This creates opportunities to update systems while we wait, and so the past week has seen massive updating of Windows system stuff and deleting old development tools so that the decks were cleared for MS SQL Server 2008 R2 Management Studio. With a bit of shape2sql from SharpGIS and today we’ve had our first spatial tables loaded and fledgling spatial SQL queries made.  Oh, at the end of the day we got the third of five workstations updated to ArcGIS Desktop 10 and the full complement of seats moved over to ArcGIS 10 licensing.

What’s been holding things up has been some final processing of cartographic-grade contours, one of the key new feature classes developed for support of the ESRI Community Maps Program, where local jurisdictions grind their own cache tiles for large-scale topographic mapping.  Since it’s part of a worldwide seamless map, of course the contours and spot elevations must be metric—which meant generating new contours.  The bathymetry was done on one-meter interval, with some half-meter supplemental contours in shallower waters.  The topography was done at quarter-meter intervals up through 25 meters elevation, where we’d included all available LiDAR data, the half-meter intervals through 50 meters, and one meter up to the summits.  All non-integer meter contours were flagged as supplemental, and contour index attributes were also calculated for 2, 5, 10, 20, and 50-meter intervals.   Spot elevations were derived from VARGIS photogrammetric spot elevations that were screened down from about 75,000 points to just 440 points, using neighborhood focal statistics.

For performance’s sake, after index attributes were calculated, the contours were chopped into segments not-to-exceed 500 meters shape length.  When all the chopped segments from all the various elevation ranges had been merged to a single polyline feature class in a feature dataset in a file geodatabase, spatial indices were built for the cache tile scales about 1:1200, 1:4800, and 1:19000.  The purpose is to make the contours as fast as possible not just for web app interaction, but for rendering the large-scale cache tiles.  In the end, the contours in this one feature class have about 1.4 million polyline features that fill the file geodatabase to about 6.8 GB of data.

Meanwhile, some great brainstorming has taken place with regard to hydro-enforcement of the terrain model, so that accurate synthetic flow lines can be generated county-wide.  Based on prototype work performed by Evan K. Babb before summertime, we knew how to deal with our terrain features and manually generate hydro-enforcement features.  Now, with a funded project to do the work county-wide, we’ve needed to devise a spatial analysis technique using the approach described by Poppenga and others in 2009.  Based on some correspondence and personal communication with Poppenga, we have sketched a workflow that should automate the creation of most simple (single road-crossing) hydrologic enforcement features, especially in the absence of accurate or complete spatial features for culverts.

So with the contours completed, it should be possible to have some progress made on automation of hydro-enforcement, completing more of the data development for large-scale topographic base mapping, help my colleagues continue the ArcGIS 10 migration, and follow up with some demonstration spatial SQL queries.  It’s an exciting couple of weeks!

There’s also been a thread of thought over the past six years or so that supports the workflow of using Visual Studio to compose SQL queries, in preference to the direct SSMS (SQL Server Management Studio, Tool Formerly Known As “Enterprise Manager”) ways of composing a query, because the typed pattern matching (MS: Intellisense) pulls up all the various ST__  spatial methods that are available in SQL Server 2008 spatial, but saves one needing to look-it-up and type-it-in correctly.  Such productivity gains really help boot up the learning experience a bit faster!

No responses yet

Aug 27 2010

Marin County Topo-Bathy Surface “tbsm45cm” Released as 2010.08

Blemishes notwithstanding, nearly six months of back-burner work has reached a threshold of readiness and is outward bound to some engineering firms, flood mappers at FEMA, and interested parties within the county. A handful of known issues remain unresolved. Proper name is “tbsm45cm_20100823”, proper edition is “2010.08”.

This is the third version of the terrain. Second version was “2010.01” and included multiple LiDAR data sets, but fewer than presently used, and was a topographic model only. First version was “2009.09” and was mainly photogrammetry and FEMA LiDAR, and was the last version to be developed in California Coordinates. Once the massive NCALM LiDAR data sets were processed, it became easier to move everything into WGS84 UTM zone 10 north meters projection, WGS84 NAVD88 CONTUS Geoid 2003 vertical position.

The NOAA utility program VDatum, a brilliant Java-based application able to stream-process data sets of near-infinite size, brought the NCALM data to heel, and opened up decades of NOAA depth surveys to our use in integrated topographic-bathymetric surface modeling.

First-return NOAA ALACE LiDAR swaths were fused along the outer coast, as bare-earth filtered versions were not produced in 1997–2002; the benefits of LiDAR detail along the rocky coast do seem to outweigh the distracting appearance of structures near Rodeo Lagoon, Stinson Beach, and outboard Bolinas.

When ArcGIS 9.4 beta 2 reached its limit in ability to render the terrain dataset into 45cm grid over the full extent, the clipping quadrants created to resolve this problem ended up chopping a very small portion of Sonoma county that drains into Estero Americano; the full watershed remains intact in the 1-meter version of the terrain grid under analysis for county-wide hydrology. Likewise, the tighter clipping quadrants lost a few hundred meters of San Pablo Bay bathymetry just west of where Marin, Sonoma, Contra Costa, and Solano counties meet. Also, tighter clipping quadrants snipped a portion of the San Francisco Bar southerly of San Francisco’s Seal Rock that was intended to be part of the model. All of these areas exist in the 100cm grid, and will be part of drainage analysis.

Happily, we have updated the workstation to ArcGIS 10, and have been enjoying such great speed gains with Spatial Analyst that our ERDAS use has been noticably reduced. Finally, Spatial Analyst is often showing performance nearly on par with ERDAS. Thank goodness that the Raster Calculator survived the transition to version 10 ArcGIS!

Painfully, the existence of unutilized bathymetric data sets for upper broad-channel Corte Madera Creek and Bolinas Lagoon have been revealed this week. Hey, there’s already something to look forward to for the next build!

The new terrain is getting some immediate use in support of an effort to participate in ESRI’s Community Maps Program for large-scale topographic mapping. The Program provides a template geodatabase with 36 vector feature classes and two raster, into which local agencies may pour their data. Once tucked into a conforming schema, a template multi-scale map document is provided with 120 layers—30 at each of four large scales that correspond to Google Maps and Bing Maps projection and cache tiling schema. The difference is that the template document makes use of ESRI tools to allow much more local detail to be packed into a map designed with notably more sophisticated cartography than either Google or Bing maps now have. The Community Maps Program concept is that local agencies may publish their local detailed content in a fairly uniform style, while retaining a world-wide seamless context for their surrounding area.

Qualitatively, the effect is that, when viewing the topo map alongside either Google or Bing maps (on two monitors, with comparison made at the same scale), the map looks to be a larger scale. It isn’t, and I’ve measured the size of features to convince myself, but my mind insists that I’m zoomed farther in on the map for some reason. My guess is that it is a perceptual effect of the much greater amount of information that is cleanly displayed in the map versus the much sparser Google and Bing content at these large zoom levels. Try it out—it’s like a carto version of an optical illusion!

The 120 layers in the template large-scale topographic base map from the ESRI Community Maps Program are arranged to provide four precise cartographic designs for Google/Bing map cache levels 16 through 19, which correspond to these display scales
1:15000–1:6001 (level 16, a.k.a. ~9k)
1:6000–1:3501 (level 17, a.k.a. ~4.5k)
1:3000–1:1501 (level 18, a.k.a. ~2k)
1:1500–1:501 (level 19, a.k.a. ~1k)
One of the most attractive areas currently online is Toronto, ON where at levels 18 and 19, individual building outlines are graced with street addresses.

Anyhow, the new tbsm45cm model will serve County of Marin’s effort at large-scale topographic mapping several ways. First, it has made possible a very detailed hillshade that helps emphasize the grading around each hillside structure in the county. Second, it helps us to create the required metric topographic contours. These are necessary to meet world-wide mapping standards, and throughout this weekend, contours are being generated from a related (smoothed version) of the terrain on 50cm vertical interval. Needless to say, most of these won’t get used in the map renderings, but the ESRI cartographers have shared a very clever indexing scheme that will help us use this single set of metric contours to support the requirements for all four of our topographic map scales.

No responses yet