Jan 07 2011

Terrain progress linked to Community Basemap Large-Scale Topo

The ESRI Community Basemap program’s new ArcGIS 10 template for Large-Scale Topographic Mapping is somewhat streamlined and cleaner when compared to its ArcGIS 9.3.1. predecessor.  But the template map document still has the ability to bring my workstation to its knees when doing stuff like printing.  I find myself unable to print 11×17 or export a B-size PDF without having incomplete results.  Mercifully, I can print and export to A-size so people are starting to get a taste of what is to come when we start building cache tiles.  Everyone who has seen the base maps was quite pleased, and some were almost gushing over them.

Preparing input features to pour into the template geodatabase that  are good enough to be worthy of 1:1200 scale mapping is every bit the challenge it might seem.  Each feature class that is making its way into the template seems to require a unique bit of spatial analysis to get pulled together, and usually this involves combining our best available input data in ways that we haven’t before.  This week’s challenge has been water polygon and water line feature classes.  For utility, we’re choosing to split out the water polygon from its water line.  For lakes and ponds, they will coincide, but for our tidal coast, we’re conveying extra information by setting the polygon to represent Mean Low Water, and the water line to represent Mean High Water.  This helps to convey the widely varying slopes that Marin County has at the tidal shores, and provide some hint as to the width of the public access way below the high tide line.

These tidal shores have been compiled from the 25cm-interval topo contours that were generated for the Community Basemap, and in NAVD88 we used nominal 50cm contour for Mean Low Water, and 175cm contour for Mean High Water.  Between these elevations one often finds various “water” polygons depending on the application; in our base map we’ll have a purposeful gap.

Along the Sonoma borderlands I struggled for a few hours hand-tracing ponds using NAIP 2009 near-infrared grayscale—the one where standing water is basically black.  I was a bit sloppy and found myself tracing at 1:4000 on screen but it was a fairly thorough job through the adjacent shared watersheds along Estero Americano, Stemple Creek, San Antonio Creek, and Petaluma River.  I was just about fed up with drawing lines around standing water when I checked in with MarinMap member Bill Voigt of San Rafael for other reasons.  He provided an inspired suggestion that I use the water lines from our 2004 photogrammetry.  Of course!  I had used them in the terrain, but they were not hard constraints and the sparse contouring in west county lands tended to wash over stock ponds in the contours.

Empowered by Voigt’s suggestion, I found that the 2004 photogrammetric water line work, which was only available within Marin County proper, had very high fidelity with 2009 NAIP summer ponds, and it was vastly easier to select the pond-circumscribing lines, sometimes 300 segments at a time, copy and paste them into the Inland_Waters_li features, Merge them all at once into a single compound line, then Explode the multi-parts into discrete ponds.  With that approach I was able to harvest hundreds of Marin stock ponds, reservoirs, and vernal pools, and have geometry that appears smooth and accurate at better than 1:2400.  There are also quite a few meandering creek features in the west county that are well represented horizontally in the water break lines, but my impression of them was that they took a lot of upgrading to serve as 3D guides in the terrain, and construction rules broke the pond lines wherever a drainage reached its edge.  So there were a great many segments that needed to be merged to create closed loops that could build pond polygons.

By the end of this week, we had over 1400 water bodies identified in the watersheds that touch Marin county, and this exercise that was initially motivated by improved cartography may augment our set of candidate sites for wetland inventory.  These  water features are set into a regional water layer from The National Map that is valid at 1:24k and includes a refined coastline from Big Sur to north of Point Mendocino, and inland to the Sierra crest, from San Luis reservoir up to Mendocino Lake.

We refined Marin and adjacent shorelines (as both MHW and MLW boundaries) from Sears Point, up to and down from Petaluma, along San Pablo, San Rafael, Richardson and San Francisco Bays, the Golden Gate, Gulf of Farallones, Drakes Bay, outer Point Reyes, Tomales and Bodega Bays, and Bodega Head using 25cm interval topographic contours.  All Farallon Islands were traced near 1:2400 scale as visible in NAIP 2009 near infrared band.

Next week should find the last couple of administrative layers (Parks and Open Space) and refinements to road centerlines to reflect docks and proper arterial status, and then it will be time to start making test tiles.  With any luck, we’ll take some CPU cycles from production server to give us the best shot at getting good tiles fast.  With 120 layers, several of which use representations, and many of which use complex Maplex rules for labeling, the Community Basemap Large-Scale Topographic template for ArcGIS 10 is by far the most demanding map document I’ve ever manipulated.  Everyone is looking forward to getting the tiles generated for testing, and sending them off to ESRI for review while sending samples to MarinMap members for their comments as well.

No responses yet

Mar 12 2010

Sharing Terrain With the World – Google Earth style

It’s not fully 3D immersive, but hey, 2-1/2D ain’t half bad. The “dsm40cm” model of Marin County has been published as the county’s default terrain on Google Earth. It’s a great pleasure to work with folks who are not troubled by a county representing its surface on a 40cm single-precision float grid that weighs in at 77 GB. In terms of data bulk, that is about the same as the entire 30-meter version of the US National Elevation Dataset.

What one gets when piling that much detail into a single county of around 520 square miles of land area is every building pad, driveway, and crown of road paving that were resolved. The dsm40cm model was derived from an ESRI Terrain Dataset that incorporates our best available topographic contours (1:4800 scale 10-foot; 1:2400 scale 2-foot,) photogrammetric break and water lines, FEMA LiDAR and NCALM (GeoEarthScope) LiDAR data sets. The Terrain Dataset currently comprises 40 GB of vector GIS data.

When the finely detailed surface grids were first developed, we broke the county up into 20 work areas to maintain ArcGIS 9.3.1 in a stable and productive state, and 30cm posting interval grids were generated that covered the entire county–at least during development. When necessary, these grid tiles were mosaicked with ERDAS Imagine into a single seamless grid. The 40cm version was produced directly as a single seamless grid using ArcGIS 9.4 beta 1, on a workstation imaged with Windows Server 2003. The WGS84 UTM, NAVD88-Geoid 2003 result was provided to the Google Earth team earlier this year.

As with all GIS data sets, it seems, the more detailed it is, the more rapidly it may need updating. In the works for the next year or so are several improvements to the dsm40cm model. First: the photogrammetric break lines will be segregated into steeper sets that tend to run along ridges, and shallower slopes that tend to delineate road cuts and building pads. The ridge set will be used as soft constraints to resolve some artifacts where they rise above some contours.
Second: incorporate new LiDAR data as it becomes available. Some data has already been provided for the lowest part of Lagunitas creek, and it appears that Prof. Ellen Hines of San Francisco State University’s Department of Geography and Human Environmental Studies has been funded by USGS to gather LiDAR county-wide this year.

So there will be revisions, but an exciting aspect is to see data flows being brought into existence that support different levels of mirror world development.
Publishing the dsm40cm model in Google Earth is an important (and beautiful) threshold to cross. Making use of the dsm40cm model in county operations such as creek and watershed delineation will be the practical benefit that drives the work in the first place. And before too many more weeks, there may be entirely new approaches to publishing the data in an immersive environment (neither Second Life nor Opensim) to share.

Building pad in Kent Woodlands shows driveway-level detail

Kent Woodlands building pad and driveway, in the shadow of Mt. Tam

No responses yet

Jul 06 2009

OpenSim Terrain notes, and Darb has Process Credit history!

I’d read about this, but never before experienced the agony first-hand.  Extracting funds from SL, the wait for funds to arrive at PayPal was a bit slow.  In fact, in the time it took funds to go from Linden to PayPal, a bamboo shoot in my back yard could have grown taller than me (that’s my RL not SL height!), and would have been over 2 meters tall.  Anyway, Process Credits are quite lacking in symmetry with how quickly credit charges can flow into the Linden realm.

During this week of waiting my random prims have been cleared out from Amida and nary a trace of Berkurodam BART Station remains besides a video in Gualala.  The video screen was actually entombed by a neighbor, who may not like it but did not send any message.

Anyway–for me this week is all about generating maps and graphics while keeping up with work.  I’ve generated a 50cm terrain grid for parts of my county where perhaps 150,000 people live.  With computational process improvements I should be able to make production stable enough to generate a 25cm grid.  The point is to model terrain slope and aspect within urban parcels.  OpenSim can pack 64 terrain megaprim sculpties over each region to refine terrain more than the built-in 1-meter postings, and display 10cm orthoimagery at full resolution.

Last year, I used first-return LiDAR data of the UC Berkeley campus to generate a 25cm grid for 10cm imagery.  Now, I’m working with bare-earth LiDAR data from FEMA, topographic contours (densified to 1.5m vertex spacing), and most importantly, photogrammetric terrain and water break lines.

Throwing all those data into the mix, the data are built into an ESRI Terrain Dataset, from which I generate TIN and GRID models at various reolution and extent.  The ESRI ArcGIS 3D Analyst Terrain-to-TIN generator breaks down after about 10 mega-faces (so would I…)  And the ArcGIS Terrain-to-GRID generator seems to drift into Windows-unconsciousness after about 1.0 giga-cells.  So for the grid, I break it down and do the pieces, then merge the tiles using ERDAS Imagine, because the ESRI ArcGIS raster mosaic function does not produce output grids much over 10 GB.  As annoying as learning these ArcGIS limits can be, it is very satisfying (and instructive) to see huge swaths of seamless terrain with great detail once it all comes together.  Thanks to the break lines, many driveways and most home building site cuts and fills are resolved.  And it will be a lot of terrain by OpenSim standards–enough to calibrate terrain for over 20,000 contiguous regions–not that I ever expect to build it all at 1:1 scale!

No responses yet

Mar 25 2009

Terrain Tenacity, fresh ortho pixels

Terrain has been in the mix for me quite a bit these past four weeks.  I’ve worked on pushing ESRI ArcGIS 3D Analyst to its limits of masspoint digestibility, trying hard to bring everything into focus at the same time that everything is sinking down to NAVD88 datum.  An abundant set of waterlines and terrain breaklines have helped to make possible some terrain models that appear to be as good as any one is likely to get from photogrammetric data.  As with LiDAR source, I’m working toward a 30cm gridding interval to sample any reasonable-looking TIN models.

One fascinating aspect of the terrain model is where it ends.  There appears to be a new 1:1200 or 1:4800 shoreline that can be sussed out of some combination of 2.5-foot elevation waterlines, 2-foot elevation contours, and related artifacts.  In fact, it’s a great patchwork of artifacts that must be stringed together.  In the tidal flat areas, there is also plenty of need for validation with multiple photos (hopefully shot at times of lower tides).

Adding to the data bulk there’s a new ortho in town, 30cm natural color flown just about two years ago.  There’s hope of extracting it from the grip of California HARN coordinates after it is all mosaicked.

No responses yet

Feb 16 2009

Terrain on tap – OpenSim on deck

There’s some progress on a couple of project fronts.  I’ve started assembling some USGS terrain for a 1:10 scale Level 1 build that could involve more OpenSim regions than I’ve ever stood up on one machine before.  Snapshot of progress is here, with a goal of 304 OpenSim regions for the model.  I expect that the OpenSim server will be re-imaged and a new build attempted in the next couple of weeks.

Vicinity of Colorado Springs, CO - with a full mile of terrain elevation subtracted

Vicinity of Colorado Springs, CO - with a full mile of terrain elevation subtracted

The site design for the Marin Civic Center build in Second Life Stanford region is also moving along with its target 1/8 region (two SL acres) based on RL terrain and building at 1:1 scale Level 3 build.  Progress sketch below:

Context model data of terrain for Civic Center Administration Building

Context model data of terrain for Civic Center Administration Building

A bit more can be reviewed by looking at the PDF of the same map here.cc_topo_20090211

Finally, a sky tag has been added to the space above the build. It is visible as a streak to anyone who visits SLURL.com by a mouse roll.  The Stanford region now has a large “MARIN” visible in its upper reach, squarely in the middle of the ancient Second Life Outlands.

Stanford vicinity from SLURL.com on 15 Feb 2009

Stanford vicinity from SLURL.com on 15 Feb 2009

One response so far

May 03 2008

Terrain Megaprim Sculpties – HOWTO

Published by under SL In General

Today I would like to share the inside production notes (it’s quite low tech for the most part) on making terrain sculpties. I have included a full region’s worth of working raw terrain, and a set of four megaprim sculpties that should help to clarify some of my mutterings in earlier posts. Stuff like precisely which values go into the sculptie gradient maps (shown in a spreadsheet), what it looks like when one takes textures that are 960×960 and add a 64-pixel-extent collar around them, and how to actually configure a region to load and display the 250-meter square at Military Grid Reference System / US National Grid 10SEG_6550_9200, with all the necessary bumpmap and texture files, and instructions for both OpenSim console-side and SL client-side actions.

obdam_40h_php This is how I have created the 40 regions using MGRS / US National Grid naming convention, particularly if the terrain has been scaled to 1.024:1 so that exactly 250×250 meters of RL terrain are loaded into each OpenSim region.

obdf_2_7_f32 This is a single region’s raw float terrain file. The original input digital elevation model had been gridded to have postings every 30cm in X and Y. The nominal scale for OpenSim terrain is 1 meter in X and Y. For large grids of real-world regions, there is reason to scale things up slightly so that there are exactly 16 regions per square kilometer. When one does this, as I have here, the samples are 977 mm and the real-world scale is 1.024:1 or a couple of percent larger than life.

sculpt_gradients_132 is the magic for the sculpties, all one needs to do is take the precise spreadsheet values and create three 8-bit grayscale images from them, using a raster program of your choice, to fit 132×132 size for use as a starting point. Then in the middle 130×130 area of the Z-value image (third or blue channel), insert your 8-bit rescaled values of terrain surface. After that, stack the X, Y, and Z grayscales together as Red, Green, and Blue channels to make a single RGB that will be your UV bumpmap.

Working Example single terrain megaprim bumpmap and texture
This is a single sculptie bumpmap+texture set intended to be placed on a megaprim named ‘nw’ that is sized with ‘edit-scale nw 132 132 164’ on the OpenSim console.

Full 1.024:1-scale region with f32 terrain and four megaprim terrain sculpties This is the real deal, one of the 40 regions in the Open Berkurodam sim and I think it’s an interesting part of its steeper area. The link is to a 10MB zip file that is named for the 250-meter square MGRS/US National Grid region that it represents: 10SEG_6550_9200, the grid point at its southwesterly corner. This archive contains a single-float terrain “obdf_2_7.f32” raw file ready for loading into an OpenSim region; the file name results from a raster dicing script and this is the second region down and seventh region over from the northwesterly corner of the 40-region sim. The archive also contains four pairs of bumpmap+texture Targa files; their file name results from the same dicing script but the indices are higher because these are quarter-region areas.

10SEG_6550_9200_xml Region configuration file for the following example (I neglected to include it above)

To try out the full region set, take an available OpenSim region, and load in the terrain from the OpenSim console with the following sequence

change-region 10SEG_6550_9200
terrain load obdf_2_7.f32
terrain bake

Next, seed the region with four prims. I tend to fly into the middle of the region or teleport and turn left 90 degrees so I am facing northerly, drop four cubes a few meters apart, then name them by their quadrant: ‘nw’, ‘ne’, ‘sw’, and ‘se’, ensuring that the Prim’s new names have stuck by checking at least one. Then I fly to the outer edge of the region, or just over into the next southerly one. This is not strictly necessary, but it feels like the right thing to do, sort of like walking a safe distance away after having set four large underground charges. That’s because the next step involves super-sizing. In my lab, the OpenSim server is an Ubuntu Linux box running Mono 1.2.6, but my SL client is on Windows XP, and I switch between machines on a KVM switch going from the OpenSim console to SL client, and it is so much easier to see the megaprims if you aren’t inside them after they have been inflated. On the console:

edit-scale nw 132 132 164
edit-scale ne 132 132 164
edit-scale sw 132 132 164
edit-scale se 132 132 164

The Z value is the one I use in the steepest part of the sim, and this makes some really big cubes. I tend to pull them apart far enough to tell them apart.  Be careful here–I’ve hung a sim that was running fine for a week by planting the seed prims not close to the center and then dragging the centroid of the megaprim over into the next region. Sometimes when handling these megaprims it’s handy to ensure that you have the SL client’s draw distance maxed out to 512 meters, and also to zoom the view out from default one notch with “CTL-8” to trade of field of view with distance from the prims you are handling. So when you’ve got one of the megaprims selected for editing in the SL client, move them into position by keying locations into the Object tab while being very careful not to touch the values in the Size category (thus saving yourself a visit back on the OpenSim console to reinflate the megaprim) When using but four terrain sculptie megaprims per region, their positions in X and Y are always the same, and the Z position will depend on how you’ve rescaled your floating-point terrain to fit into the 8-bit unsigned approximation. For the example megaprims that I have posted, use these:

Prim ‘nw’ XYZ = 64, 192, 188
Prim ‘ne’ XYZ = 192, 192, 188
Prim ‘sw’ XYZ = 64, 64, 189.5 (tweaked for amphitheater vs. region terrain)
Prim ‘se’ XYZ = 192, 64, 188

Once I can see that the prims have snapped to fully cover the region and are all nearly the same height, I change their Building Block type to Sculpted and see four really large apples that may be somewhat subterranean. If I hadn’t already done so, I use File > Bulk Upload to get all the bumpmap and texture Targa files into inventory. When you have four megaprim sculpties, you should choose the following for Object/Sculpt Texture and Texture/Texture:

Prim ‘nw’ Sculpt = ‘ob40_03_13_z3.tga’ ; Texture = ‘ob40e_03_13.tga’
Prim ‘ne’ Sculpt = ‘ob40_03_14_z3.tga’ ; Texture = ‘ob40e_03_14.tga’
Prim ‘sw’ Sculpt = ‘ob40_04_13_z3.tga’ ; Texture = ‘ob40e_04_13.tga’
Prim ‘se’ Sculpt = ‘ob40_04_14_z3.tga’ ; Texture = ‘ob403_04_14.tga’

For the sort of appearance that looks best at first, I have kept the background color in the texture to all 255’s and set the Full Bright to checked. That setting does not do well when it’s night in the sim, but it overcomes some sort of fade that is visible in the texture around the edges of the sculptie when Full Bright is not set. More improvements for the future!

Deep thanks to Adam Zaius for pointing out that I hadn’t really made clear these details in the blog. Enjoy!

No responses yet

Apr 02 2008

Terrain Sculpties – OpenSim does Google Earth

Published by under OpenSim

The past four days have been a tremendous blur of internalizing NURBS into my mind, at least the SL sculptie variant of them. Now I’ve been aware for several months of how NASA used sculpted prims to represent detailed Mars craters (as published by Ireton), and I’ve certainly followed the beautiful work for David Rumsey done both by Telemorphic in 2003 (3D plots of historic Lake Tahoe area) and more recent historic Yosemite by Nathan Babcock). But there was something confusing and ultimately mysterious about using sculpties for terrain.

Not so much any more. Through several helpful blog and forum posts, and a score of hours spent in experimentation, I feel that I’ve brought the sculptie to heel for my terrain rendering purposes. Mostly, especially for OpenSim, it’s simply to display draped orthoimagery over an already precisely customized region terrain.

What I’ve learned is that for 1:1 mapping, where regional terrain is not available at more detail than the 10-meter postings from seamless.usgs.gov, then one can configure precise sculpted megaprims, only four to a region, and drape imagery quite effectively. The result is real-life imagery draped in the style of Google Earth, but coming out of a free OpenSim server into a free Second Life client, for a dozen or more regions on one server core. When using the technique that I’ve worked out, having only four scuplties to seamlessly cover the region means that the terrain sculpties will rez fully sixteen times (16X) faster than will either the David Rumsey or NASA educational islands.

There’s no special magic here: the region terrain is far superior as a way to represent real life terrain, as it can hold 64K of single-precision floating point values. A sculptie, by comparision, holds a mere 900 usable values that must be compressed to an 8-bit signed integer, for any one of the 900 points’ X, Y, or Z values that are practical to use to guide facets in a terrain “diamond”. This “diamond” is a way of describing what the terrain sculptie looks like after defining the outermost ring of UV values to wrap around to a single point safely below terrain surface, so as not to interfere with the 30×30 values useful to describe terrain in a way that cleanly tiles to cover multiple regions. The vertical scale of the terrain described this way is adjusted with the Z-dimension of the sculptie spheroid, which must be tuned using back-end OpenSim command “edit-scale” if one is manipulating a megaprim.

Anyway, when I get a chance to demo this for some Berkeley terrain, I’ll be sure to post a dramatic screen shot. As it is, the 12-region sim looks so realistic right now that almost any shot would be immediately recognizable by someone familiar with the site, so the good ones will wait for project time. Meanwhile, this one is intended to prove validity of sculpted terrain megaprims for draping orthoimagery.

Watch Out Google Earth

The tools that I used were: ArcGIS to reproject the seamless terrain and orthoimagery into a rotated local grid variation of WGS84-UTM; ERDAS Imagine to perform mosaicking, dicing, image stretch/rescaling, and layer stacking to build precise UV maps from real life terrain; and OpenOffice.org spreadsheet to calculate precise gradient values for the X and Y components of the UV maps. On the back end was OpenSim 0.5 using region definitions in the 0.4 style, and the terrain build was performed using the standard Second Life 1.19.0.(5) client running on Windows XP with a Radeon X1300 Pro.

A couple of days later, I revisited the sim and made a couple of updates to sculpties with the new 1.19.1.(4) Second Life client, and the orthoimage colors look different depending on sun angle–thanks to Windlight.  It’s not a bad thing, and gives one a reason to look up and appreciate the beautiful sky!   Back on Agni (standard Second Life Grid) by comparison, all the prims seem far more intensely colored and somehow more detailed with the new client.

No responses yet