Archive for June, 2008

Jun 27 2008

I am not a number, I am a free avatar! – OpenSim post 0.5.8 svn_5234 ODE

Published by under OpenSim

With many thanks to the globe-spanning OpenSim community, I have seen the error of my sourcing ways and replaced a version of Open Dynamics Engine that I had compiled on my Ubuntu x86_64 system, on 20080220, leaving the product of that effort, libode.so, up at /usr/local/lib which seemed well and good at the time. As things have evolved, ODE was brought into opensim-libs svn at some point after the last change log entry at 20080328. I was happy to find that ODE was included in the OpenSim svn, and turned on the physics engine in my OpenSim.ini configuration file.

Where I went wrong was not clearing out the library I’d compiled back in February, because that directory was getting sourced before my latest version in each of the nine OpenSim svn builds I’ve made since April. Last night, Dahlia helped me get focused on fixing the problem that I was having and she demonstrably was not. Teravus cleared up that my issue was squarely that of ODE and nothing but ODE, and Nebadon pointed me to exactly the updated documentation that I needed to build my own new ODE for the first time since I’d migrated the test server to Ubuntu 8.04 from 7.10.

In the process, I’ve learned to expect more efficient memory usage than I’d been understaning. I had written to Nebadon that my 40 regions were using 1.4 GB of memory, but once I had to start killing processes after every sphere I’d rez, I saw that the processes that I was killing were all less than 900 MB memory. And now, with ODE, MySQL region and asset storage, and 40 regions, I get OpenSim console statistics showing 194 MB “Allocated to OpenSim”.

Here is a video of what the old ODE (20080220) did with a really big cube (25-meter sides dropped from 500 meters)

After compiling ODE from the OpenSim-libs/unmanaged trunk, and adding a few build tools to my server, I got a fresh libode.so that was 3.7 MB rather than the 3.1 MB that was distributed with svn 5234. I did not tweak any compile flags as this was from OpenSim folks’ favored setup. After ODE was working, I slept well and awoke very optimistic about my prospects for getting a demonstration public-facing by end of July. I also have enjoyed testing the performance of ODE, particularly after Dahlia shared some time on her Dev server at OSGrid.org.

With ODE running, certain physical prim movements can consume plenty of CPU cycles, and even with two avs and some complex mesh interactions, there can be striking lag. So with my own regions working once again, I wanted to see how that looked, and I was inspired by the 1967 TV series The Prisoner to create an oblate spheroid (5 m by 5 m by 4 m) as a physical prim to roll after Ruth. The first of my efforts was shot here, and as you can see, I used Rover’s Dutch cousin, as orange was much easier to see than the original Welsh white Rover.

Finally, to impart a sense of drama, I textured Rover with a menacing City of Berkeley sewer manhole cover, and finished the celebration of ODE with a long take including several region crossings and a pleasant sunset hue from Windlight. As I hope is clear with this video, OpenSim has really come a long way in the past couple of months!

No responses yet

Jun 24 2008

Progress of a sort OpenSim 0.5.8_5195 on mono / MySQL

Published by under OpenSim

I’ve been trying to be patient and catch a good wave in OpenSim’s evolution, so that I can configure a demo of terrain prims for late July. Much good seems to be afoot, and yet I am trying to find the right point amidst all this progress to grab an SVN and build on it for a few weeks. Tonight I had some luck with getting my OpenSim.ini configuration together better than before.

For all OpenSim back end storage now I’ve gotten MySQL 5.0.51a-3ubuntu5.1 working fine. I had a glitch in my configurations where I worked with a configuration setting that referred to Catalog / schema (what MS SQL folks might know only as a “database”) but interpreted as schema certain tables within that one schema known as users, userfriends, terrain, prims, land and such. For a few nights I seemed sleepily unaware that I had confused the table ‘opensim’ with the catalog ‘opensim’—my bad for naming them all the same without being sure enough what I was doing. The symptom was simply that
../cli OpenSim.exe
refused to load storage, which is to say that the sim wouldn’t start up, and refused to go more than the first few lines into a startup sequence that for my 40 regions is typically many hundreds of status lines. Anyhow, that’s all happy now and every storage invocation in my OpenSim.ini config is using the local MySQL instance through loopback. Bottom line for me was this—the OpenSim.ini and mysql_connection.ini config files are connecting to a catalog, even through in places they contain references to certain tables.

On the physics side, I’ve got ODE turned on for all 40 regions, and the startup is still very fast at less than 30 seconds with no prims. The 1-meter gridded terrain loads into 40 regions in about 15 seconds [hardware: 2x Core2 Duo E6550 overclocked to 3.4 GHz with 3.9 GB RAM]. Everything seems to be going my way until…

I try to create prims right now. Tonight at svn 5195, I can create cube prims, have them be physical and drop to the ground. When either I turn them into spheres, or create them as spheres, in regions near (where I first rez) or far (at the northeast end of my 40 region sim), the whole sim hangs brutally. I’ve never had the sim hang this way before, where it’s a ‘kill -9’ to get its attention and make it go away.

With the persistence of MySQL running, if I quickly restart the sim without taking time to command the previous “../cli OpenSim.exe” into oblivion, then I can’t log in because the second sim still thinks that I’m already logged in, from the first hung sim’s session. I flailed around a bit this evening learning this and creating new users along the way. At one point, I was so discombobulated between hung mono and typing anything *.exe on a command line that I restarted Ubuntu to wash my hands of the matter. Now wasn’t that silly? I certainly think that it was. No amount of C# code under mono is going to turn Ubuntu into the sort of bounce first, ask questions later mode favored by some Windows server administrators, including myself in impatient moments.

But whatever happens tonight at svn 5195, it appears to have an evil effect on mono when meshing a single spherical prim, or a box that’s become a sphere. This is using SL Windows client 1.19.1.4 or SL Linux client 1.19.1.4 on x86_64 Ubuntu 8.04 Hardy Heron.

Thanks to Nebadon who helped me get a pulse on my memory usage. Interesting experience tonight was that now that I’ve started using MySQL and making sure to kill off zombie ../cli processes, my memory use is down toward 780 MB, rather than the 1.4 GB I’d mentioned on IRC. At that size, there does not seem to be too much memory use penalty under Linux+mono versus native Windows servers running OpenSim. Selfishly, I’m glad of that because I’ve already switched my opensim test servers off of Windows and onto Ubuntu, and I’d prefer not to come across too many reasons to want to go back!

No responses yet

Jun 17 2008

Reflective DEM has been gridded

Published by under OpenSim

After mucking around a bit, I was able to use free tools to browse the contents of the classified LiDAR, then used ArcGIS 9.2 tools from 3D Analyst and ERDAS Imagine to get where I wanted to go with this surface. First, I needed to know how many returns there were, and what each of the classes meant. The LAStools info function helped there. Then I used ArcGIS 9.2 3D Analyst “LAS to multipoint” conversion tool, but selected only the first return. Multipoint was an annoying format because it did not seem to fit anywhere in the cool new ESRI “terrain” feature data type. In the end, I gave up on ESRI terrain and went straight to the classic TIN. For maximum overlap, I did not filter out any specific angle from nadir, taking whatever was sent along from the contractor to Alameda County.

Of course, I had to negotiate the treacherous 3D Analyst menu items that were necessary. Getting multipoint into a TIN required creation of a TIN (obvious, but with blank result) and then the non-obvious choice of “Edit TIN” which effectively accepted the multipoint data that were imported from LAS and allowed me to specify the delunay method of choice. Once canned as a TIN, it was a familiar step to specify a raster gridding. I haven’t found a way to reproject the TIN, so I was still in NAD83 California coordinate US Survey feet, and an assumed NAVD88-Geoid 2003 CONTUS-feet vertical while I tried several grid resolutions. In the end, I was happy with 1 foot gridding.

Then, raster on disk, I was able to reproject to WGS84 UTM zone 10 north meters, and chose bilinear resampling on a 25 cm grid posting interval. Once in my favored projection, I rescaled the Z values to NAVD88-Geoid 2003 CONTUS-meters, and began to examine the need for a bit of grayscale morphological processing. I’ve been a great fan of mathematical morphology for over 20 years, so it was a pleasure to craft a kernel or 3 to compensate for some artifacts. Because the TIN-to-grid was so highly oversampled, I was able to use a combination of a tall, narrow 7×3 kernel for morphological CLOSE, followed by a 3×3 DILATE, and a diamond-shaped 5×5 ERODE to finish off the task. In case this morphological stuff sounds like odd stuff to do, these operators are variations on focal max and focal min convolutions. The results are rather important for my application, as shown in the following images.

First is the reflective DEM surface, and the same with the Open Berkurodam 40-region overlay.

This is the gridded reflective DEM here it is with the OB40 image overlay

Next are more detailed images, near the Greek Theater, showing why I ran the morphological filtering and also how I was able to mostly conserve building footprint areas while inflating trees. The main artifact attenuated was interlace-type effects at the end of overlapping LiDAR scans. The long axis of the morphologcial CLOSE kernel was perpendicular to these artifacts.

Here is the reprojected and rescaled reflective DEM, unfiltered Same area, but with morphological filtering as sketched above

Here is the morpho-filtered reflective DEM, with the 10 cm natural color imagery overlaid.

overlay of natural color image on morpho-filtered reflective DEM

Next up, I’ll need to figure out how to best use this 25 cm surface. It really seems a shame to use it in the way that I have thus far with terrain megaprims–where using four megaprims per region I have effectively downsampled the terrain to 4.26-meter grid postings. That wasn’t so bad for the bare earth model. Here I’ve got something over 290 times denser with 0.25-meter grid surface samples.

But to use many more than 160 megaprims for the entire 40-region model, I really must automate the placement of the (auto-generated) sculpties. For that, I’ll need to ask around the OpenSim community for advice!

No responses yet

Jun 13 2008

Classified LiDAR data have been viewed

Published by under SL In General

The classified LiDAR data that I hope will provide some inflated structure and tree surfaces for draping the orthophoto have been reviewed. I find the data beautifully detailed, and fascinating to see with GeoCUE Point View LE. I’m working with the UC Berkeley Geospatial Imaging and Informatics Facility UCB GIIF, also known as the Maggi “Kelly Lab” when proximal to Mulford Hall.

Right now my goal is to interpolate the first return surface in a way that I can grid and filter most appropriately to inflate buildings and trees. In principle, I should be able to use the first return LiDAR point cloud to create a NURB surface that would be expressible as an OpenSim/SL sculptie. But I’m going to take a more cautious approach and try to get the whole thing gridded in a consistent way so that I can reasonably expect to cover the entire 40-region sim with good inflated surfaces rather than the bare earth that has been a fine demonstration, but a bit flat for draping the orthophoto.

I’m going to throw out a lot of images and let them speak somewhat for how the classified (into ground, structure, low veg, med. veg, tall veg) LiDAR point clouds look.

Here’s the plain elevation image and the classified view of same

elevation view of classified LiDAR Classified LiDAR of UC Berkelye vicinity
this is how the classified image looks with intensity shading. It gives a first impression like a photo
classified LiDAR near UC Berkeley more detailed view of UC Berkeley area

Some perspective views also help to show what information will be available for gridding. For these I’ve displayed with vertical exaggeration of 1.5X
Northeasterly perspective view of LiDAR Easterly view of UCB campus in classified LiDAR

No responses yet

Jun 06 2008

Still At OpenSim 0.5.7_5002 ODE and 40 regions

Published by under SL In General

This experience was on 3 June but I’m only writing about it now. Much the same as on Monday where the regions start up like gangbusters, terrain loads in a snap, and everything is navigable with no prims. When I get myself to the most interesting terrain, at UCB’s Greek Theater, I rez a cube and it sits on the ground. When I carelessly resize it to 10 meters in all dimensions, part of it sits below ground. After all, it is not physical yet. Then when I set it to be physical, either as a cube or after making it a sphere, the whole sim crashes. Looking at Mantis I had the sense that some aspects of this issue have been worked on very recently and resolved. So far for me, no joy.

I also have a challenge with getting region and asset storage working on MySQL rather than SQLite. People need persistence for any difficult build, and when things get large that’s not the time one wants to run up against the limits of the storage technology. But I’m flummoxed by the necessary OpenSim.ini config. I’ve seen this work on other sims at earlier revisions, so I know that I’m close. But I can’t get OpenSim to connect, although I have no problem getting to the catalog with MySQL-administrator and I do see some tables get created if I leave SQLite for region storage and MySQL for asset. But when I try to use MySQL with all storage, OpenSim complains that it can’t find a responsive instance of MySQL. Suspicions are pointing toward my mixed use of localhost loopback 127.0.0.1 and local network address 10.x.x.x among OpenSim and MySQL installs. Even though I run standalone, I need OpenSim to respond to the local network address to access OpenSim from other machines in the lab, and I thought that I had MySQL set up to do the same. Apparently some connections must use loopback and that may be creating inconsistencies that keep me from launching OpenSim in a non-SQLite setup

No responses yet

Jun 02 2008

Finding Limits – OpenSim 0.5.7_4952 can be crashed

Published by under OpenSim

For the Linux SL client on my HP keyboard, (Alt- + Windows- = Alt- ) as the SL client works in Windows.

Yesterday evening I added a YouTube embed to a post, and it showed up today with a toxic URL in it. That edit was made from Windows, so tonight I’m running Ad-Aware full scan, which takes awhile. So to keep at it, I took the test server (E6550-3.4 GHz/4GB) and ran it with 40 regions standalone, real UC Berkeley terrain, and ODE; then to be testing I installed the latest now-Beta SL client 1_19_1_4 and went for it!

Things are getting ever smoother with the Second Life client for Linux. I first fired it up and went to Agni, and saw that the 1:25 scale Berkurodam model rezzed much more slowly than it did when I last tried the Windows client a couple of nights ago. I say that because I saw the ellipsoids of the sculpties, as ellipsoids, for many seconds.

Then I quit and launched with “./secondlife -loginuri host:9000” and saw the terrain rezzing like never before. One of the wild things about OpenSim is that if you try something that you’ve done before eons ago, like three weeks, things can be different in some really good and unexpected ways. Like the speed with with terrain rezzed once I set my draw distance out to 512 meters and flew to a NEly corner of a sim. Wow, I’ve never seen so many regions filling in at once, and nary a delay for the little texture patches that follow along. It made me think that network speed limits some of the experience, even when its a local 100-Mb wire.

Anyway, I was able to saunter in flight all about the 40 regions and be fairly impressed. Then I stopped by the Greek theater site, rezzed a 10-meter cube and threw it up 1 kilometer into the air. It landed with much less bounce than I saw on the default sinc-shaped islands last night, but still looked as slippery as an ice cube while it wiggled its way into the very lowest spot of the stage area. I tried to make a machinima of the experience using the SL client feature, but I did not take time to lower my resolution from 1600×1200 for the video, and I never could find the AVI file that I expected to have made. Still, although at this point I was getting the CPUs up toward 70% at times, as soon as I cooled off and stared at the Ubuntu System Monitor, things got quiet fast, like 5% on each core.

Everything still seemed to be just ducky, until I found one more cool thing. You see, I’d been grasping about for the proper keyboard shortcuts to gain camera control on the SL Linux client. Like in Photoshop or the Windows SL client, I tend to use the keys around the space bar, Alt-, Ctl- and the arrows quite a bit. So I’ve been frustrated with the Linux client because the same Ctl-Alt combination that I want to use to spin the camera around usually does something nasty to the Gnome window when dragging the mouse. But no more. I stumbled on (what surely must be documented somewhere) the solution–the dreaded Windoze key on my HP keyboard works with the SL Linux client just the way that I expect the Ctl- key to work.

For the Linux SL client on my HP keyboard, (Alt- + Windows- = Alt- ) as the SL client works in Windows.

Once I got that grokked, I was doing some very mobile camera work for a couple of minutes, and then I tried to rez a physical sphere to see how it dropped. But it didn’t. Although my SL client was quite happy—I’d managed to hang OpenSim with this stacktrace, and now although I restart OpenSim, I can’t log in.

Native stacktrace:

../cli [0x51bb67]
../cli [0x43dacd]
/lib/libpthread.so.0 [0x7f13d3fcd7d0]
/usr/local/lib/libode.so(_Z27gim_trimesh_update_verticesP11GIM_TRIMESH+0x205) [0x7f13d06b2c75]
/usr/local/lib/libode.so(_Z18gim_trimesh_updateP11GIM_TRIMESH+0x18) [0x7f13d06b2d58]
/usr/local/lib/libode.so(_ZN9dxTriMesh11computeAABBEv+0xcc) [0x7f13d06a2a1c]
/usr/local/lib/libode.so(_ZN11dxHashSpace10cleanGeomsEv+0x34) [0x7f13d0670714]
/usr/local/lib/libode.so(_ZN11dxHashSpace10cleanGeomsEv+0x5f) [0x7f13d067073f]
/usr/local/lib/libode.so(_ZN11dxHashSpace8collide2EPvP6dxGeomPFvS0_S2_S2_E+0x39) [0x7f13d0670639]
[0x4173f5ed]

No responses yet

Jun 01 2008

Getting Physical with OpenSim 0.5.7_4952 – ODE with 40-region Standalone

Published by under OpenSim

After learning how the terrain sculpties could be handled by Meshmerizer if running a physics engine like Open Dynamics Engine (ODE), I have taken a couple of weeks to proceed slowly, cautiously as I bulk up the demands on the hardware. After all, my original notion of doing large swaths of real-life terrain on a single standalone sim was based on loading that terrain into the regions then using only Basic Physics to reduce the load.

But in the months since I first started loading real terrain (starting with Mt. Tamalpais in 200710), truly phenomenal, awesome progress has been made in how efficiently OpenSim runs for me on Ubuntu/Mono. Sure, at new year 2008 my OpenSim test environment upgraded from a P3-800/1.5 GB Coppermine system to an E6550-3.4 GHz/4 GB system. But what was limiting last Fall was the chatter among the various regions, so that I could add more: 49, 81, 100 regions–but then the CPU load with no client logged in would hum up toward 70%, and running a physics engine would be a challenge with many fewer regions.

These days, that seems like a stone-age experience. The rate at which regions now load on startup is incomparably faster (even on the old Coppermine), and the chatter is almost nil–no clients logged in looks truly quiescent at 1% to 2% CPU. All this has emboldened my interest in trying ODE again. And that experience likewise is so much better. Time was, there was reason to visit the ODE site and build one’s own, and even then stuff could get strange. I was inspired by the videos that Nebadon posted showing many hundreds of blocks falling. But I experienced things like tripping over what felt like a singularity that shot my unfortunate avatar hundreds of meters into the air, bouncing like some tire that fell off of a jet after takeoff. That was then.

Now I see Ruth’s legs bend a little bit under the effects of gravity, but I do have 40 regions humming along in standalone with quiet-state CPU load of 2% to 5%. So I have plenty of reason to expect that I’ll be able to do this with the 40-region model, using terrain sculpties that are physical as long as they don’t tilt over against the terrain surface like they do in the SL Agni grid.

Video demo of Ruth narrowly avoiding getting squished by a 10-meter cube
If you’ve got the embed blocked, the link should be http://www.youtube.com/v/Jz9234jYbkw

My next goal for Open Berkurodam is to generate a new surface. I may have a good copy of what is called categorized or classified LiDAR data, where individual points in a cloud are tagged with an estimate of whether they are from bare earth, tree crown, rooftop, and such. This sort of LiDAR data should support the sort of grid that is not just bare-earth terrain, but actually has the proper size, shape, and height bump for every tree and building. This would be ideal for draping with the orthophotography, because within the limits of parallax that have been corrected in the orthoimagery, every building should sort of take shape on its own.

I don’t hold any fantasy that things will look properly immersive right on the classified LiDAR grid, but I have a sense that there will be enough detail to guide a reasonably accurate build with just a bit of training on the part of the builder to recognize their way past registration artifacts–where the bump from the LiDAR surface doesn’t align with the roof part of the orthoimage.

To bring this to a presentable stage, I hope to somehow have a live version of the 40-region UC Berkeley and vicinity 1.024:1 model with classified LiDAR surface on physical sculptie megaprims, on a public server by mid-July.

No responses yet