Wow it has been a lot of effort to get the 40 regions built out with their LiDAR surface sculpties. Someday when the process gets automated I’ll look back and likely feel like a fool for not loading the BLOBs in MySQL directly, and coding the XML to load these directly. But I built them by hand, using 64-prim linked sets that covered each region. Region-wide linking works fine when the Admin settings are used to “go to God”. Also there’s a new SL client 18.104.22.168456 that works OK and helped a wee bit with the build.
My priority is to create good graphics for large-format display, but I’ve posted a rush of the first end-to end plod by the Ruth named “UC08 Visitor2”. At the moment, I’m still a bit shaky about just how solid the sim will be for demonstration purposes. None of he LiDAR surface sculpties are physical; I’ve been able to turn on ODE. I can’t bear the time to let the sculpties get meshed so there’s none of the cool walking on them, but I have tried boosting the detail settings to 64, in hopes that perhaps the sim will put out 4096 points per sculptie [my bump-maps to define each one are 16K so there’s plenty of info behind the detail boost.
With many thanks to the globe-spanning OpenSim community, I have seen the error of my sourcing ways and replaced a version of Open Dynamics Engine that I had compiled on my Ubuntu x86_64 system, on 20080220, leaving the product of that effort, libode.so, up at /usr/local/lib which seemed well and good at the time. As things have evolved, ODE was brought into opensim-libs svn at some point after the last change log entry at 20080328. I was happy to find that ODE was included in the OpenSim svn, and turned on the physics engine in my OpenSim.ini configuration file.
Where I went wrong was not clearing out the library I’d compiled back in February, because that directory was getting sourced before my latest version in each of the nine OpenSim svn builds I’ve made since April. Last night, Dahlia helped me get focused on fixing the problem that I was having and she demonstrably was not. Teravus cleared up that my issue was squarely that of ODE and nothing but ODE, and Nebadon pointed me to exactly the updated documentation that I needed to build my own new ODE for the first time since I’d migrated the test server to Ubuntu 8.04 from 7.10.
In the process, I’ve learned to expect more efficient memory usage than I’d been understaning. I had written to Nebadon that my 40 regions were using 1.4 GB of memory, but once I had to start killing processes after every sphere I’d rez, I saw that the processes that I was killing were all less than 900 MB memory. And now, with ODE, MySQL region and asset storage, and 40 regions, I get OpenSim console statistics showing 194 MB “Allocated to OpenSim”.
After compiling ODE from the OpenSim-libs/unmanaged trunk, and adding a few build tools to my server, I got a fresh libode.so that was 3.7 MB rather than the 3.1 MB that was distributed with svn 5234. I did not tweak any compile flags as this was from OpenSim folks’ favored setup. After ODE was working, I slept well and awoke very optimistic about my prospects for getting a demonstration public-facing by end of July. I also have enjoyed testing the performance of ODE, particularly after Dahlia shared some time on her Dev server at OSGrid.org.
Finally, to impart a sense of drama, I textured Rover with a menacing City of Berkeley sewer manhole cover, and finished the celebration of ODE with a long take including several region crossings and a pleasant sunset hue from Windlight. As I hope is clear with this video, OpenSim has really come a long way in the past couple of months!
I’ve been trying to be patient and catch a good wave in OpenSim’s evolution, so that I can configure a demo of terrain prims for late July. Much good seems to be afoot, and yet I am trying to find the right point amidst all this progress to grab an SVN and build on it for a few weeks. Tonight I had some luck with getting my OpenSim.ini configuration together better than before.
For all OpenSim back end storage now I’ve gotten MySQL 5.0.51a-3ubuntu5.1 working fine. I had a glitch in my configurations where I worked with a configuration setting that referred to Catalog / schema (what MS SQL folks might know only as a “database”) but interpreted as schema certain tables within that one schema known as users, userfriends, terrain, prims, land and such. For a few nights I seemed sleepily unaware that I had confused the table ‘opensim’ with the catalog ‘opensim’—my bad for naming them all the same without being sure enough what I was doing. The symptom was simply that ../cli OpenSim.exe
refused to load storage, which is to say that the sim wouldn’t start up, and refused to go more than the first few lines into a startup sequence that for my 40 regions is typically many hundreds of status lines. Anyhow, that’s all happy now and every storage invocation in my OpenSim.ini config is using the local MySQL instance through loopback. Bottom line for me was this—the OpenSim.ini and mysql_connection.ini config files are connecting to a catalog, even through in places they contain references to certain tables.
On the physics side, I’ve got ODE turned on for all 40 regions, and the startup is still very fast at less than 30 seconds with no prims. The 1-meter gridded terrain loads into 40 regions in about 15 seconds [hardware: 2x Core2 Duo E6550 overclocked to 3.4 GHz with 3.9 GB RAM]. Everything seems to be going my way until…
I try to create prims right now. Tonight at svn 5195, I can create cube prims, have them be physical and drop to the ground. When either I turn them into spheres, or create them as spheres, in regions near (where I first rez) or far (at the northeast end of my 40 region sim), the whole sim hangs brutally. I’ve never had the sim hang this way before, where it’s a ‘kill -9’ to get its attention and make it go away.
With the persistence of MySQL running, if I quickly restart the sim without taking time to command the previous “../cli OpenSim.exe” into oblivion, then I can’t log in because the second sim still thinks that I’m already logged in, from the first hung sim’s session. I flailed around a bit this evening learning this and creating new users along the way. At one point, I was so discombobulated between hung mono and typing anything *.exe on a command line that I restarted Ubuntu to wash my hands of the matter. Now wasn’t that silly? I certainly think that it was. No amount of C# code under mono is going to turn Ubuntu into the sort of bounce first, ask questions later mode favored by some Windows server administrators, including myself in impatient moments.
But whatever happens tonight at svn 5195, it appears to have an evil effect on mono when meshing a single spherical prim, or a box that’s become a sphere. This is using SL Windows client 22.214.171.124 or SL Linux client 126.96.36.199 on x86_64 Ubuntu 8.04 Hardy Heron.
Thanks to Nebadon who helped me get a pulse on my memory usage. Interesting experience tonight was that now that I’ve started using MySQL and making sure to kill off zombie ../cli processes, my memory use is down toward 780 MB, rather than the 1.4 GB I’d mentioned on IRC. At that size, there does not seem to be too much memory use penalty under Linux+mono versus native Windows servers running OpenSim. Selfishly, I’m glad of that because I’ve already switched my opensim test servers off of Windows and onto Ubuntu, and I’d prefer not to come across too many reasons to want to go back!
This experience was on 3 June but I’m only writing about it now. Much the same as on Monday where the regions start up like gangbusters, terrain loads in a snap, and everything is navigable with no prims. When I get myself to the most interesting terrain, at UCB’s Greek Theater, I rez a cube and it sits on the ground. When I carelessly resize it to 10 meters in all dimensions, part of it sits below ground. After all, it is not physical yet. Then when I set it to be physical, either as a cube or after making it a sphere, the whole sim crashes. Looking at Mantis I had the sense that some aspects of this issue have been worked on very recently and resolved. So far for me, no joy.
I also have a challenge with getting region and asset storage working on MySQL rather than SQLite. People need persistence for any difficult build, and when things get large that’s not the time one wants to run up against the limits of the storage technology. But I’m flummoxed by the necessary OpenSim.ini config. I’ve seen this work on other sims at earlier revisions, so I know that I’m close. But I can’t get OpenSim to connect, although I have no problem getting to the catalog with MySQL-administrator and I do see some tables get created if I leave SQLite for region storage and MySQL for asset. But when I try to use MySQL with all storage, OpenSim complains that it can’t find a responsive instance of MySQL. Suspicions are pointing toward my mixed use of localhost loopback 127.0.0.1 and local network address 10.x.x.x among OpenSim and MySQL installs. Even though I run standalone, I need OpenSim to respond to the local network address to access OpenSim from other machines in the lab, and I thought that I had MySQL set up to do the same. Apparently some connections must use loopback and that may be creating inconsistencies that keep me from launching OpenSim in a non-SQLite setup
After learning how the terrain sculpties could be handled by Meshmerizer if running a physics engine like Open Dynamics Engine (ODE), I have taken a couple of weeks to proceed slowly, cautiously as I bulk up the demands on the hardware. After all, my original notion of doing large swaths of real-life terrain on a single standalone sim was based on loading that terrain into the regions then using only Basic Physics to reduce the load.
But in the months since I first started loading real terrain (starting with Mt. Tamalpais in 200710), truly phenomenal, awesome progress has been made in how efficiently OpenSim runs for me on Ubuntu/Mono. Sure, at new year 2008 my OpenSim test environment upgraded from a P3-800/1.5 GB Coppermine system to an E6550-3.4 GHz/4 GB system. But what was limiting last Fall was the chatter among the various regions, so that I could add more: 49, 81, 100 regions–but then the CPU load with no client logged in would hum up toward 70%, and running a physics engine would be a challenge with many fewer regions.
These days, that seems like a stone-age experience. The rate at which regions now load on startup is incomparably faster (even on the old Coppermine), and the chatter is almost nil–no clients logged in looks truly quiescent at 1% to 2% CPU. All this has emboldened my interest in trying ODE again. And that experience likewise is so much better. Time was, there was reason to visit the ODE site and build one’s own, and even then stuff could get strange. I was inspired by the videos that Nebadon posted showing many hundreds of blocks falling. But I experienced things like tripping over what felt like a singularity that shot my unfortunate avatar hundreds of meters into the air, bouncing like some tire that fell off of a jet after takeoff. That was then.
Now I see Ruth’s legs bend a little bit under the effects of gravity, but I do have 40 regions humming along in standalone with quiet-state CPU load of 2% to 5%. So I have plenty of reason to expect that I’ll be able to do this with the 40-region model, using terrain sculpties that are physical as long as they don’t tilt over against the terrain surface like they do in the SL Agni grid.
My next goal for Open Berkurodam is to generate a new surface. I may have a good copy of what is called categorized or classified LiDAR data, where individual points in a cloud are tagged with an estimate of whether they are from bare earth, tree crown, rooftop, and such. This sort of LiDAR data should support the sort of grid that is not just bare-earth terrain, but actually has the proper size, shape, and height bump for every tree and building. This would be ideal for draping with the orthophotography, because within the limits of parallax that have been corrected in the orthoimagery, every building should sort of take shape on its own.
I don’t hold any fantasy that things will look properly immersive right on the classified LiDAR grid, but I have a sense that there will be enough detail to guide a reasonably accurate build with just a bit of training on the part of the builder to recognize their way past registration artifacts–where the bump from the LiDAR surface doesn’t align with the roof part of the orthoimage.
To bring this to a presentable stage, I hope to somehow have a live version of the 40-region UC Berkeley and vicinity 1.024:1 model with classified LiDAR surface on physical sculptie megaprims, on a public server by mid-July.