This is the first of what should be a set of posts that detail a server build process for the San Francisco Enterprise Geographic Information Systems Program (SFGIS) Standard Geospatial Server (SGeoS). In fact, the build work has been ongoing for several weeks and is concluding here, with OpenSim.
The motivation for including OpenSim in the platform was a desire to provide support for legacy .NET applications that may exist in various departments. In the interest of creating a Microsoft-neutral build that is framed with Open Source components, it was natural to bundle the Mono framework into the SGeoS design. And while individual department applications are their own business and not part of the standard build, OpenSim serves as an excellent demonstration of the utility of the Mono framework as included on the server. That , together with my perspective that immersive 3D clearly should be associated with geospatial servers, is why OpenSim is included in the Standard Geospatial Server.
OpenSim is not trivial by any means, and yet it is not such a resource hog that it would be infeasible to bundle it. What’s more, it is an opportunity to distribute immersive 3D technology packaged with other geospatial capabilities.
Since the build descriptions are being transcribed from a build document that is approaching 80 pages on Google Docs, it seems prudent to break it up into individual modules. And since WordPress here is configured to show older posts below newer ones—I’ll start down at the end modules and post new build descriptions for earlier modules in later days.
The original notion for SGeoS was to have modular build chapters that could provide a unit of capability. That way, only selected modules need be configured. After discussions with VMware engineers, I became intrigued by the notion of making a single server image that could run everything, all at once, and then disable unneeded featured in an actual deployment. So the build document was initially structured with module-like chapters, but in fact the server builds them all—so it’s worth viewing the build document in sequence.
The modules will probably end up numbering about 10, including packaging for production and possibly default-disabling of most items. If one watches too closely, it might seem like I’m making a countdown to completion. But this will end with a stub for deployment packaging, work back through an OpenSim build, and end up with imaging an install of CentOS 6.5 onto a new VM guest system.
Thanks to the astronauts aboard the International Space Station, their time-lapse photography at very high ISO that helps to share some of what their eyes may well see, and of course Michael Koenig for his care and smoothing of the HD video, with some loungy score, too.
Take five (minutes) and watch it on HD in a darkened room. You might find yourself pausing, reviewing, and spending 20 minutes enjoying.
I was fascinated by an orange wiggle, that turned out to be the astoundingly well-lit India-Pakistan border, around 1000 km long.
Meanwhile, I’m forming some plans for next semester’s course, and have realized that it may well be possible to offer students training in multi-user virtual environments without hacking one of the lab workstations to image it as an Opensim server. Thanks to the incessant business analytics of Maria Korolov over the past few years, it was possible for me to quickly get caught up in the new and improved options for cloud hosting of Opensim regions.
Right away it became clear that the business model of Kitely was quite compatible with my modest but area-expansive needs for real-life terrain simulations. I’ve found it quite easy to get set up with a single region, and that’s a really big start. I was able to use the latest beta Second Life 3.2 viewer to connect to the latest Opensim 0.7.2 stable release, tweak terrain and set up a few flexi-prims to test the weather. Nice work technically, and a very nice pricing scheme for my sort of use. I’m also very sympathetic to Ilan Tochner’s philosophy of “just keep building new regions”—it’s a consistent theme with cloud solutions, and refreshing to see it in connection with Opensim.
Each year for the past 40 or so years, the American Geophysical Union has met in San Francisco around December for what is now the Fall Meeting. I’ve been an AGU member for about 28 years, and for a time was attending each and every Fall meeting—but these days it’s about once every three years.
This was one of those years, and it was a great pleasure today running into a good handful of friends and former school colleagues from years past! Also, it was much fun to present a poster that summarized an analysis of synthetic flow lines built on the integrated topographic-bathymetric surface model. Basically, with a very detailed 3D surface grid that runs continuously from mountaintop to offshore out to the 3-nautical-mile legal boundary of California counties, it is possible to draw streams as they would have flowed when sea level was lower, like 7000 years ago.
Much of the interesting topography from those streams got clobbered by sea level rise. As the Ice Age retreated and continental glaciers melted out, the waves from the Pacific Ocean pounded the coast back to where it is today, and planed off much of where the streams once ran.
With ArcHydro-style drainage analysis on our terrain model that has fused detailed multibeam bathymetry from the California Seafloor Mapping Project, it is possible to identify extremely subtle signatures in the portion of the offshore platform that is Santa Cruz mudstone formation, a harder Miocene formation that expresses bedding in its surface.
With the analysis, when synthetic drainage paths are symbolized to emphasize flow lines with greater catchment area one can observe suggestions of right-lateral offset. In California, this is a signature pattern for tectonic offset of drainages that cross strike-slip faults with right-lateral offset. Because the formation where the analysis has detected possible offset is older (Miocene is more than 5 million years old, but the offset is perhaps only in the last 1 million years), this result should not cause much excitement with regard to modern seismic hazard. It could however prove helpful to those who would decode the geologic structure of the Point Reyes peninsula.
I don’t have much to say about these regions that hasn’t been written already, and my views have been less aesthetic than Shenlei’s.
But in the interest of boosting the bandwidth by which I can share OpenSim, I’ve invested in a much newer Adobe Premiere Elements than I’d been using for the past five or so years. It’s a gas to have it multi-thread while rendering, and I have direct-to-FLV write. Trying to share as much of the motion and fidelity via YouTube as I possibly can, I’ve crafted a video resolution that is a multiple of my Hippo / SL viewer screen. The FRAPS video direct to AVI (sorry, it’s Win XP) is 1600 x 1140 @ 10 fps. Yup, those are video frames. In the interest of surviving an upload, I’ve rendered them highly time-compressed, with output at 1515 x 1080 @ 15 fps. As of tonight there’s no sound, no intertitles, just the rushes. oops, if I read the YouTube Instructions for best formats, I should have trimmed the width to 1440, which is a multiple of 16.
Also, I have more direct upload options now with Premiere 8 than I had with my (recently demised) copy Premiere 1.0. Go Figure ;^)
While the Windows box grinds out the video print, I’m over here on Ubuntu blogging in a tab of 64-bit Chrome 18.104.22.168 and it is fine & fast.
For these videos, I visited ScienceSim Geography22_44 region and set the view to wide angle, then sat up at about 500 meters and watched the regions rez their terrain. For some folks, it will rank right up there with watching lead-based paint dry. For geography folks I’m hoping that these few minutes of sped-up video will convey, by dogged repetition, the primacy of regions in the provision of virtual environment simulators.
By the way, I’ve got a task: I need to find a better buzz word for the GIS community. I’ve been advised by some serious and well-intentioned (not to mention well-informed) folk that terms like “virtual” and “immersive” are actually boring to GIS’ers. So I’ll need to think about how to convey the concepts of “Mirror World”, “Multiuser Virtual Environment”, “Immersive Connected Experience”, “Third-Person Virtual World”, and related concepts into a catchy moniker. Hopefully, one that is not presently trademarked, either!
I’m trying to remain serious about this, but some of the options are treacherous. Geography in Social Media has a possibly awkward acronym; maybe it can be saved in recognizable form as “GIS for Social Media” or “Geography for Social Media”: GFSM
The term “3D Map with Me” is terse, slightly ambiguous
Here is the video chopped as it was when uploaded with 1515 x 1080 resolution. Problem with that is that by not preserving dimensions at a multiple of 16, and saving my viewer’s aspect ratio rather than the (standard since 16mm film) 4:3 aspect, my upload is clobbered into something perhaps suitable for a smartphone. So please consider this the Smartphone Version of last night’s rushes:
Then, once again with feeling, or at least with a little more rest, there is what I hope to be an HD-friendly moving vision of OpenSim, as it appears on the ScienceSim Geography regions. Yes! After it ripened on the YouTube servers for a few hours, I now see all the higher-res versions available. At 1440 x 1080, this is pretty close to what I see on my screen with a live Hippo viewer.
And after a day’s cogitation: anyone care to comment on the term: “Social Immersive Media GIS” as a moniker? Oops — I used “immersive” 8^(
I’d read about this, but never before experienced the agony first-hand. Extracting funds from SL, the wait for funds to arrive at PayPal was a bit slow. In fact, in the time it took funds to go from Linden to PayPal, a bamboo shoot in my back yard could have grown taller than me (that’s my RL not SL height!), and would have been over 2 meters tall. Anyway, Process Credits are quite lacking in symmetry with how quickly credit charges can flow into the Linden realm.
During this week of waiting my random prims have been cleared out from Amida and nary a trace of Berkurodam BART Station remains besides a video in Gualala. The video screen was actually entombed by a neighbor, who may not like it but did not send any message.
Anyway–for me this week is all about generating maps and graphics while keeping up with work. I’ve generated a 50cm terrain grid for parts of my county where perhaps 150,000 people live. With computational process improvements I should be able to make production stable enough to generate a 25cm grid. The point is to model terrain slope and aspect within urban parcels. OpenSim can pack 64 terrain megaprim sculpties over each region to refine terrain more than the built-in 1-meter postings, and display 10cm orthoimagery at full resolution.
Last year, I used first-return LiDAR data of the UC Berkeley campus to generate a 25cm grid for 10cm imagery. Now, I’m working with bare-earth LiDAR data from FEMA, topographic contours (densified to 1.5m vertex spacing), and most importantly, photogrammetric terrain and water break lines.
Throwing all those data into the mix, the data are built into an ESRI Terrain Dataset, from which I generate TIN and GRID models at various reolution and extent. The ESRI ArcGIS 3D Analyst Terrain-to-TIN generator breaks down after about 10 mega-faces (so would I…) And the ArcGIS Terrain-to-GRID generator seems to drift into Windows-unconsciousness after about 1.0 giga-cells. So for the grid, I break it down and do the pieces, then merge the tiles using ERDAS Imagine, because the ESRI ArcGIS raster mosaic function does not produce output grids much over 10 GB. As annoying as learning these ArcGIS limits can be, it is very satisfying (and instructive) to see huge swaths of seamless terrain with great detail once it all comes together. Thanks to the break lines, many driveways and most home building site cuts and fills are resolved. And it will be a lot of terrain by OpenSim standards–enough to calibrate terrain for over 20,000 contiguous regions–not that I ever expect to build it all at 1:1 scale!
Sometime, it just isn’t worth it. Such is my new view of tier, in the context of what matters to me with immersive 3D and GIS. For about six months I’ve continued my hold on some land in the classic Stanford sim of Second Life, without quite being able to work out the boundary changes to just barely squeeze in a 1:1 scale model of a single large building. Even if I had been able to get the parcel into the shape that I needed, I still would not be able to model the structure’s dome with a prim that naturally had the large radius required. Not everyone is trying to model a Frank Lloyd Wright public building; perhaps the land can be better used by someone else with an architectural focus.
I’m scaling back ownership this week to the tier-free 512 square meter level in Second Life. I’m also building up a freshly configured Ubuntu 9.04 Jaunty Jackalope 32-bit server (dual 3.4 GHz Xeon – 4 GB, HP DL360 G4) to do some more serious sort of work with OpenSim. In the past five months I’ve developed some terrain data that can handily provide 1-meter postings over more than 500 square miles. With that much to publish, I really need much, much more than 1/8 of a sim, even a suberbly cool sim like Stanford.
View of beautiful Stanford sim with pond features
The orange area is available at L$20/square meter
So if anyone reading this has use for a great 7520 (< 1/8 sim) mainland location in Second Life with over 40 meters of terrain sculptability, it’s available for L$20/square meter. Discount available for OpenSim community members or known GIS people. With the world’s economy as challenged as it seems to be, I’ve decided that it’s time to focus on where things matter most, and for me now that’s OpenSim more exclusively.
As my patience with Second Life wanes, and I wait for more architectural input for my next SL build project, I have a dark OpenSim server with no fixed IP. I’m having stability issues with the Linux SL client, but have upgraded the workstation to Ubuntu 9.04 Jaunty Jackalope. Google Earth client there is more stable, the NVidia drivers install themselves (sans Envy), and everything Ubuntu-wise seems to be getting incrementally better by the quarter.
I’m grinding some large images that have taught me that one very special difference between Windows XP variants and Windows Server 2003 is the latter’s ability to open files on the high side of 80 GB. I’d never quite realized it before but the moderately massive mosaics that I have created in years past (edging toward 250 GB single files) actually depended on Server 2003 to get created. Once the destination file exists, then XP can take it from there, and in all cases Windows Explorer can copy the monster files. But in that tenuous moment when a mosaic first grabs its space on disk for a huge output—one can’t seem to do that with XP.
So while I’m enjoying Google Earth on Ubuntu, there is something cool that I go back to Windows for, and that’s the new Google Earth browser plug-in. Since I’m gaining a bit of facility with the keyboard shortcuts in the full-stop Earth client, these all carry over to the plugin. My first test page has been stood up here and I’ve been deep into four continents with it so far. I understand that the plugin is only available for Windows and Mac systems at this time. If you can, Enjoy!
Also, as I get even faster with my keyboard navigation of G-Earth, I’ve actually seen some artifacts that are quite familiar from OpenSim. While zipping about between the Gulf of Yakutat and Canada’s Mount Logan, at certain viewing elevations I can accelerate the point of view forward quite fast. Doing so in this very mountainous terrain, I saw blocks of terrain standing up along what look like sim edges, resolving in a few seconds as more (sculpty?) bumpmap arrives. This is the same sort of artifact I’ve seen with terrain sculpties and sometimes, with region crossings in OpenSim. Also, I’ve found a couple of wild terrain grid errors in G-Earth. In one, a quarry dug hundreds of feet below sea level, right next to the sea, is displayed as positive elevation (absolute-value terrain, anyone?). In another, a boundary between US and Canadian terrain has a glacier flowing uphill onto a plateau. Go figure. Blame Canada! ;^)
There’s some progress on a couple of project fronts. I’ve started assembling some USGS terrain for a 1:10 scale Level 1 build that could involve more OpenSim regions than I’ve ever stood up on one machine before. Snapshot of progress is here, with a goal of 304 OpenSim regions for the model. I expect that the OpenSim server will be re-imaged and a new build attempted in the next couple of weeks.
Vicinity of Colorado Springs, CO - with a full mile of terrain elevation subtracted
The site design for the Marin Civic Center build in Second Life Stanford region is also moving along with its target 1/8 region (two SL acres) based on RL terrain and building at 1:1 scale Level 3 build. Progress sketch below:
Context model data of terrain for Civic Center Administration Building
A bit more can be reviewed by looking at the PDF of the same map here.cc_topo_20090211
Finally, a sky tag has been added to the space above the build. It is visible as a streak to anyone who visits SLURL.com by a mouse roll. The Stanford region now has a large “MARIN” visible in its upper reach, squarely in the middle of the ancient Second Life Outlands.
Today the (San Francisco) Bay Area Automated Mapping Association hosted a wonderful URISA Certified Workshop given by Tim Case, describing Best Practices and Project Implementation Methods for 3D Geospatial work. The all-day event provided a very broad and even-handed overview of many 3D technologies that hold promise for the near future.
With this presentation as an extra boost for my focus on a new build, I’m gearing up with even more enthusiasm for a new build in the Agni grid Mainland. I’ve also tuned the Berkeley parcel for sale. Its price amounts to about US$382.00, and that price is set to help cover purchase costs for the next build’s likely parcel. The tuning involved reducing the parcel size by 64 square meters, so that the three Gualala parcels total 4608 square meters, or exactly the maximum amount allowed for Linden Lab’s US$25/month tier rate. With that size, it would be possible for an interested party to purcahse the Berkeley BART station and maintain it for $300/year in tier (the Linden land property tax).
Also, based on today’s Geospatial tag, I’ve noted just this morning two mentions of the Berkeley BART build. The New World Notes item by Wagner James Au 2009 01 19 was wonderful to find after our in-world messages last month. For clarification, while true at the time of that conversation, no longer do I work for City of Berkeley. The TidalBlog item by Peter Miller mentions interesting developments in the overlap between simulators and geospatial models, as well as some shots from his visit to the Berkeley BART model. Thanks to both authors for their posts!
ABSTRACT: When creating sims based on real-world terrain, there’s usually a lot more extent and detail available than is typical when one must hand craft every region landscape. When compared with commercial multi-user virtual environment (MUVE) hosting, OpenSim has extremely low marginal cost for adding more simulator regions. Whenever a real-world region of interest has free 1/3 arc-second DEM data available for download from the USGS seamless server, then with a bit of GIS help from someone at a local university or interested government office, the free DEM data can be converted to a fine 1/10 (1:10) scale sim of terrain in OpenSim. This case study takes an idea posted to a blog, and turns it into a sim of 54-regions in OpenSim based on real-world terrain available at a 10-meter posting interval. To run all the regions, one first obtains OpenSim, then downloads the 54 terrain tiles, the 54 region definition XML files, and a loader script available from this post. The region definition files will likely need to be regenerated with the supplied PHP script to point to your machine’s IP address.
The Case Study tour video is nearly 10 minutes, and if it doesn’t embed in your browser is here at http://www.youtube.com/v/EfBvEtTvLYU
Two days ago I was intrigued by Brian White’s posting of a Tutorial for real-world terrain in OpenSim and I followed several of his well-documented steps. Then I read all the way to the end of his posting and realized that he wasn’t fully satisfied with the results. I’ve helped a couple of folks with generating specific terrains before, and Orcas looked like an intriguing bit of topography–sort of a miniature version of how I imagine Greenland might look in, say, a couple decades…
So without an intent to cross-talk Brian’s posting, I’ll offer my own parallel perspective on how I’d approach making a recognizable, maybe even fun-to-have-around model of the Orcas Island in OpenSim. Starting from where he did, I’d like to emphasize the profound opportunities for following this approach, as it would work virtually anywhere in the continental USA (CONUS) where the 1/3 arc second seamless DEM is pretty much complete. In other words, what I’ll show here could be applied in an hour or so’s effort to produce an OpenSim version of Yosemite Valley, for example.
First, choose thy locale. The USGS seamless server is a worldwide resource, but the greatest detail available seems to be in the “lower 48” states. A digital elevation model (DEM) with postings on 1/3 arc second interval has samples about every 10 meters in Y (north-south) and more closely spaced than that in X (east-west) everywhere off the equator. The 1/9 arc second data that Brian mentioned was actually not available for Orcas, although some urban areas may have it at this time. In this case, Orcas Island offers a nominal 10-meter DEM downloadable from seamless.usgs.gov server.
Second, define thy scale for the sim. If one will be hosting on a commercial sim, even like Linden’s Open Space sims in Second Life’s Agni grid, then there are very strong reasons to scale things down and perhaps pack everything into a single region or four. With OpenSim, unless you’re going to have tons of prims and game-grade dynamics running, it may be possible to balance the number of regions you set up with the terrain quality that you desire. For the Orcas, I downloaded the islands (using the default format which seemed to be an ESRI GRID) and explored them in ESRI ArcGIS. With that I had the sense that I wanted more or less 24 km in X, and 16 km in Y, to cover the Orcas. Since OpenSim allows me to load floating-point terrain samples on a 1 meter posting interval, I knew that my available (1/3 arc second) DEM would support a 1/10-scale sim, analogous to a 1:10 scale map. That way, I would not discard too much of the terrain information on its way into OpenSim.
Third, do the math to tile the regions. It’s not too hard, but it’s fairly unforgiving. After a bit more ArcGIS exploration, I found a way to crop the DEM to exactly 23.04 km in X, and 15.36 km in Y. Sounds wierd, or compulsively precise? Not really. Since each of my terrain samples was planned to be on a 10-meter grid interval in real-world, and my sim scale is 1/10, that means that every region would cover a 2560-meter square, or 2.56 km on a side. The dimensions that I chose were simply six regions north-south and nine regions east-west, for a total of 54 regions of 1:10 scale sim.
Fourth, get your DEM gridded. Sure, it comes from USGS that way, but the seamless server works world-wide, and that requires a geographic (Longitude / Latitude) way of defining its coordinates. To do this rigorously, I used Geographic Information Systems (GIS) software to re-project the DEM from geographic, the way it downloaded, into a projected grid [World Geodetic System 1984 (WGS84) datum, Universal Transverse Mercator (UTM) zone 10 north projection, with distances measured in meters]. The UTM grid system is defined world-wide, with the proviso that at any given spot, only one of the sixty (60) grids in the system would be the best choice. For the Orcas, it’s zone 10. For the reprojection process, I used a remote sensing package, Leica ERDAS Imagine that provides precise control over both the reprojection and the resulting data’s sampling interval. I used ERDAS Imagine to grid the UTM-projected DEM into a Cartesian grid of 10-meter interval in both X and Y, and cropped the projected data to 2304 by 1536 samples.
Fifth, dice the DEM. This is necessary for multi-region sims where your grid has more than (and exact multiples of) 256 samples in X or Y. ERDAS Imagine dices, so i used it. If you’re trying this in Photoshop, be sure to keep the pixels in 32-bit (single precision) floating point values, and don’t go to integer grayscale or the flatter terrain will get stair-step artifacts. The naming convention used in ERDAS Imagine starts at the top row and increases downward for Y, and also puts Y first then X in the tile name so that the south-westernmost of the 54 regions is called “orca_6_1”
Sixth, reflect each diced DEM tile, this reverses the direction of samples in Y dimension. What starts out as the top, northernmost row is moved into the bottom-most position, and vice-versa. This does not change the size of the tile, nor any of the gridded elevation values themselves. Another way to describe this is to flip the image around the X axis. ERDAS Imagine has a geometric correction function that allows one to specify this as an affine mapping, which I did. I’d expect that most of the world uses GIMP or Photoshop to do this, too. Although it would appear that this will scramble the terrain, it is necessary to do this to adapt the data to the sequence in which OpenSim reads the terrain file when loading it into the region.