www.quadsoftware.com
FAQ  FAQ   Search  Search   Memberlist  Memberlist   Usergroups  Usergroups
Register  ::  Log in Log in to check your private messages


Post new topic  Reply to topic
 Need advice... « View previous topic :: View next topic » 
Author Message
keinmann
PostPosted: Thu Dec 03, 2009 6:55 pm    Post subject: Need advice... Reply with quote



Joined: 14 Sep 2009
Posts: 15

Hello there,

I am the point in engine development where real terrain is an absolute must. I was just exporting to Collada -> Maya -> .fbx -> XNA/Engine. This was a HORRIBLE "hack". It took several hours just to load the .dae file into Maya, not to mention all the other time it took to make it into .fbx, where the XNA framework can handle it. The terrain is so vital because I'm working on flight physics, and it is impossible to analyze the code in action without a "ground" to orient yourself. That's why I spent the time to do that ugly method I mentioned.

Now things have progressed, greatly, and it's time to get REAL terrain. The old "hack" way doesn't have any respect for texture layers, shaders, or anything. However, I'm unsure of which WAY to go with this. Confused

As far as I know, Collada would be a poor choice for representing the terrain as it looks in Grome and is supposed to be. Writing an extension for XNA's ContentPipeline to support Collada would be a wasted effort I think. Only two examples exist (which hardly work), they are extremely limited, and they ONLY work on old versions (VS '05 and XNA 1.0). So I don't see any sense in wasting time in this direction.

I really DON'T want to be stuck using the SDK to write a plugin in C++. C# is where I shine. I'm quite sure I CAN do it (I've done a lot of difficult things in C++), but I'm painfully slow and don't particularly enjoy it, to say the least. Rolling Eyes

But, I need my Grome terrain/worlds, one way or another. And they have to be correct. I'm just totally confused about which way I need to be going. I "just" need my terrain geometry to render, the correct UV coordinates (so slopes won't be smeared), and the capability to render the textures in layers as they are supposed to be. Any advice and information you can give me about this will be GREATLY appreciated. Is there an existing export format that I can parse in C# and reconstruct my terrain correctly? Any special tips tricks? Or am I obligated to suicide inspiring frustration with C++? Crying or Very sad
Back to top
View user's profile Send private message
ALicu
PostPosted: Fri Dec 04, 2009 9:13 am    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Hi,

First, you are right, COLLADA is not suitable to contain complete terrain data. It is a generic mesh exchange format and should be used in case you want to bring Grome data in general modelers (3DS Max, Maya etc).

Next I will try to give you several variants you can use for real-time terrain rendering formats:

1. First, if you don't want to use the C++ SDK, indeed you need to use one of the existing formats. You have multiple variants here:

1.1 Text exporter format. This is a simple to parse text format which contain everything you need: heightmap, texture mapping, normals, instances of objects etc. You can find an example here:
http://www.quadsoftware.com/storage/exportsamples/TextExpSample.rar

You should be able to write a XNA content pipeline plugin to support this or to have a converter to one of your own binary formats (text formats tend to be bigger and get longer to parse).

The exporter also saves the layers images. It doesn't contain information about detail and water surfaces though. You do have the entire plugin C++ source code inside the Grome SDK.

1.2 Export heightmap as raw using the generic large data sets exporter. Read the raw in XNA, while for texturing you will need to reassemble the layers or you can use one big baked texture (but the resolution would suffer unless you implement a trick using shaders, similar with mega texturing approach with streaming from disk).

If you use multiple layers of texture (not baking) you will need to implement the terrain texture splatting (multiple layers). You should know the shading operations of each Grome layer (or at least the ones you need to support). You can find the operations in the SDK documentation in advance section. You will need to create HLSL code to apply the necessary shading.

Also you can create and use some sort of standard for the layers, for example: limit to 8 layers, first 2 are vertical map layers, the next 4 are masked texture layers while the next is detail layer and on top of that is the lightmap layer. Then you can create one single shader to combine all those since there are not multiple combination possibles, only one configuration.

2. Create your own exporter to your own binary format. Writting the C++ plugin is not that hard and you have plenty of documentation and examples in Grome SDK. This may be a better solution than 1. since you will not depend on other format and text can be slow to parse and read.

3. Use Graphite library. This is of course C++ but you can create a wrapper around it to expose it as managed code. It renders the code with DX9 (DX10 is in construction) so you should be able to access it in XNA.

Graphite is an optimized outdoor library, created specifically to be Grome reference renderer and can be integrated into any external engine. It has all you need, export of data from Grome in native binary format, rendering of terrain (all Grome features supported and more), water, vegetation layers etc. It supports both ascii and unicode character set, 32 and 64bit platforms.

The license is free in binary form for non-commercial usage. For commercial use please contact us (you can contact me directly at licu at quadsoftware.com) but the licensing price is not very indie oriented.

More information about Graphite here:
http://www.quadsoftware.com/index.php?m=section&sec=product&subsec=graphite

Let me know if I can provide more information. I am more than happy to assist you in your task, no matter which variant you choose.

Regards,
Adrian L.
Back to top
View user's profile Send private message
keinmann
PostPosted: Fri Dec 04, 2009 2:58 pm    Post subject: Reply with quote



Joined: 14 Sep 2009
Posts: 15

Well, thanks again Alicu. I absolutely marvel at the support here! I can't thank you enough, but I hope telling everyone I know how wonderful Grome and the community are will do! Wink

Well, I got over my "mental block". I've got to do this one way or the other, so I may as well make it as painless as possible. I think some sleep did the trick!

I opened up the SDK and looked at in in VS, and it wasn't as bad as I thought. You're right, the documentation is excellent, and the code is very well structured and organized. There is even a strict naming convention! Don't see that very often in "3rd party" code! haha! Wink My biggest concern though is time, time, time. I started C#/.NET quite a while back, and it has basically made my use of C++, personally, obsolete; since I never really targeted anything besides Windows. And Mono has opened up new opportunities to me anyway. Needless to say, the level of comfort I have with C# just isn't there, so I'm naturally quite a bit slower with C++, even though I understand the nuances of even the dreaded pointers (use them in "unsafe" C# sometimes too).

Before I drag you into this and waste any of your time, I'm going to fool around a bit and try to familiarize myself with everything. I first want to see if I can find an ideal, pure .NET way of doing this. I exported last night to every available format so I can have some "targets" to experiment with. I'll need to analyze what sort of output I get from different options/parameters as well. Then I need to look into the structure of those files to get an idea of how they can be parsed and turned into meaningful objects. I tend to agree the text may not be an ideal solution, and a bit slow. However, if it comes down to it, I'm a wizard with mutli-threading in C# and I can get it done fast. But I'd rather avoid "ugly" solutions where I can.

I have another idea for this as well. I think making each terrain zone much smaller than I have been could be advantageous. It would help my rendering engine cull more unnecessary geometry (faster), and possibly be helpful in the shading department. It may chop things up a bit more, but I definitely see the potential for a performance gain. This world is going to be absolutely massive, and this could even make it more modular in construction for future expansion. Not to mention, making changes to existing terrain should be less painful in smaller bits. If you see any pitfalls to this idea, please point them out.

Also, I don't see any reason that I actually can't call .NET code in a plugin? I know you can use IJW and some marshaling tricks for native <-> managed interop. I've interoped with native code a lot in C#, but never the other way around. But others have done this with great success. Is there anything that would prevent this from being a possibility? This could really be great. I could simply use the Win32 API to put up a simple UI in Grome, and then marshal all the data over to one of my C# libraries, which could make short work of the terrain, and actually build it into an instance of one of my engine classes (UTerrain : Scene, IObject). I don't think it could get more "plug n' play" than that? That could then be serialized into binary, then encrypted to create a new file format. In turn, I would be able to use EITHER the ContentPipeline OR a more DirectX-style access to the assets. Again, if you see any pitfalls here, please point them out. To me, this sounds like an "All in One" solution which will tighten the workflow.

Your mention of Graphite's flexibility has peaked my interest. I'm taking special care in designing this engine. It's "core" is not dependent on any single platform; and it can run standalone or integrated with other APIs and engines. It's currently fully operable on DX and XNA. I'm also going to support OGL, and possibly OGRE. One of the primary focuses is the ability to work with Visual3D.NET, which I'm planning on employing for our actual simulation. And I'm hoping Mono will allow me to go beyond the boundaries of Windows (I just need other machines to test on at this point). I'm terrified of being "chained" to any single platform. What if MS cancels XNA and reverts to MDX? What if V3D changes hands and becomes a totally exclusive product? Anything can happen in the computer world, so that's why I'm staying as broad as possible. Supporting Graphite sounds like an excellent venue to explore, and I will definitely contact you sometime soon for some questions and answers. I'm even hoping that when I can hire more help I will be able to work towards a full C++ API for this engine as well, which can open up DX10 & 11 as well. But yes, I need more help to explore that! Smile

I'm tempted to knock out this Grome -> DX/XNA project, and make my implementation public for other users. That would open up the wonders of Grome to a LOT more users. I'll just have to think about what drawbacks that may have. We're quite concerned about security after deployment, since I've seen some nasty things happen to other developers (such as assets being ripped, stolen, and sold online). Just maybe. Wink

Well, sorry for being so verbose, heheh. I'll report on what I find/learn asap. And, I'm going to email you today, Alicu. I think I just might have something that can help YOU guys this time. And don't worry, I don't want money. Smile
Back to top
View user's profile Send private message
ALicu
PostPosted: Fri Dec 04, 2009 3:49 pm    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Hi, i am glad I can help you.

Quote:
I opened up the SDK and looked at in in VS, and it wasn't as bad as I thought. You're right, the documentation is excellent, and the code is very well structured and organized. There is even a strict naming convention! Don't see that very often in "3rd party" code! haha! Wink My biggest concern though is time, time, time. I started C#/.NET quite a while back, and it has basically made my use of C++, personally, obsolete; since I never really targeted anything besides Windows. And Mono has opened up new opportunities to me anyway. Needless to say, the level of comfort I have with C# just isn't there, so I'm naturally quite a bit slower with C++, even though I understand the nuances of even the dreaded pointers (use them in "unsafe" C# sometimes too).


Yes, the code is pretty well documented and I am more than happy to provide you with support in case you are stuck.

Quote:
Before I drag you into this and waste any of your time, I'm going to fool around a bit and try to familiarize myself with everything. I first want to see if I can find an ideal, pure .NET way of doing this. I exported last night to every available format so I can have some "targets" to experiment with. I'll need to analyze what sort of output I get from different options/parameters as well. Then I need to look into the structure of those files to get an idea of how they can be parsed and turned into meaningful objects. I tend to agree the text may not be an ideal solution, and a bit slow. However, if it comes down to it, I'm a wizard with mutli-threading in C# and I can get it done fast. But I'd rather avoid "ugly" solutions where I can.


Yes, this seems like a good idea. I've suggested text exporter since it is the easiest to parse and use. You can make a preprocessor of it and transform it into a binary format.

Quote:
I have another idea for this as well. I think making each terrain zone much smaller than I have been could be advantageous. It would help my rendering engine cull more unnecessary geometry (faster), and possibly be helpful in the shading department. It may chop things up a bit more, but I definitely see the potential for a performance gain. This world is going to be absolutely massive, and this could even make it more modular in construction for future expansion. Not to mention, making changes to existing terrain should be less painful in smaller bits. If you see any pitfalls to this idea, please point them out.


One thing to care of this is to avoid too many rendering calls. DX9 (as opposed with OpengGL and DX10) spends a lot of time per rendering call. Otherwise this is a good idea and Grome should not cause too much problem if you use smaller and more rendering zones (maybe editing will slow from user point of view as he/she needs to deal with multiple selections but that should not be too critical).



Quote:
Also, I don't see any reason that I actually can't call .NET code in a plugin? I know you can use IJW and some marshaling tricks for native <-> managed interop. I've interoped with native code a lot in C#, but never the other way around. But others have done this with great success. Is there anything that would prevent this from being a possibility? This could really be great. I could simply use the Win32 API to put up a simple UI in Grome, and then marshal all the data over to one of my C# libraries, which could make short work of the terrain, and actually build it into an instance of one of my engine classes (UTerrain : Scene, IObject). I don't think it could get more "plug n' play" than that? That could then be serialized into binary, then encrypted to create a new file format. In turn, I would be able to use EITHER the ContentPipeline OR a more DirectX-style access to the assets. Again, if you see any pitfalls here, please point them out. To me, this sounds like an "All in One" solution which will tighten the workflow.


I see no reason why you cannot access managed code inside the plugins. In case you have any issues I would try to help you but I've personally done few work on managed code.

Quote:
Your mention of Graphite's flexibility has peaked my interest. I'm taking special care in designing this engine. It's "core" is not dependent on any single platform; and it can run standalone or integrated with other APIs and engines. It's currently fully operable on DX and XNA. I'm also going to support OGL, and possibly OGRE. One of the primary focuses is the ability to work with Visual3D.NET, which I'm planning on employing for our actual simulation. And I'm hoping Mono will allow me to go beyond the boundaries of Windows (I just need other machines to test on at this point). I'm terrified of being "chained" to any single platform. What if MS cancels XNA and reverts to MDX? What if V3D changes hands and becomes a totally exclusive product? Anything can happen in the computer world, so that's why I'm staying as broad as possible. Supporting Graphite sounds like an excellent venue to explore, and I will definitely contact you sometime soon for some questions and answers. I'm even hoping that when I can hire more help I will be able to work towards a full C++ API for this engine as well, which can open up DX10 & 11 as well. But yes, I need more help to explore that! Smile


Yes you need to keep your code as independent on external libs as possible, especially those that are not backed up by long time companies. Also, you need to look for full source code licensing (as Graphite offers) since you then have the code at your disposal.

Ok, so my suggestion is to start up with text format and see how it works for you. Formats are not that big task actually. The bigger task would be the actual rendering of Grome terrain with all the features on and with good framerate.

Looking forward for your email.

Regards,
Adrian L.
Back to top
View user's profile Send private message
keinmann
PostPosted: Fri Dec 04, 2009 4:32 pm    Post subject: Reply with quote



Joined: 14 Sep 2009
Posts: 15

Thanks again!

Yes, I know what you mean about making too many rendering calls. This is just an unfounded theory of mine that there will be some performance gain by increasing the effectiveness of the "chunked" rendering system. Only REAL diagnostic analysis will prove or disprove this. The way I will probably do this is just by drawing each triangle within the view frustrum of the camera. Partially contained triangles get fully drawn, as it will be to slow to check every single vertice. Those "chunks" which aren't in the vicinity of the camera are totally ignored, and don't even need to be checked. I'm using a visibility distance of 10km, because it's in open desert, with no clouds, and an aircraft pilot MUST be able to see the earth at those altitudes. That will be cut down quite a bit for soldiers and vehicles. I can probably find the mathmatical ratio between view distance and terrain zone size which will make the best/most efficient use of the GPU power. Theoretical, once again, but I'll think about it.

Since I'm going as far as cracking out the SDK for some ol C++, I may as well broaden my horizons a bit. I think my file format should be a type of XML which is serialized to binary and maybe even compressed. And I'll put a checkbox for the option of the C# library converting it to a managed class instance for DX and XNA. That way, I don't exclude this format from being useful in C++, since the Win32 API can easily parse XML/Markup. That will also make the plugin and file type extensible and lightweight. It'll be a bit tricky for me, but I can do it. But I will still take your advice and see how the provided formats work.

One question I have now is one of efficiency. Would it be more efficient to embed textures and HLSL inside the file, or should they simply be external, like they are to Collada and other formats? I think .FBX actually embeds materials and textures into itself. I think it would be a bit more convenient to have one single file which holds everything, but of course, that may be an inefficient and ridiculous notion.
Back to top
View user's profile Send private message
ALicu
PostPosted: Fri Dec 04, 2009 4:44 pm    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Quote:
One question I have now is one of efficiency. Would it be more efficient to embed textures and HLSL inside the file, or should they simply be external, like they are to Collada and other formats? I think .FBX actually embeds materials and textures into itself. I think it would be a bit more convenient to have one single file which holds everything, but of course, that may be an inefficient and ridiculous notion.


Actually many engines use some sort of packaging for data (Unreal3, Quake engines etc). The packages some times are simple zip archives but they can also be custom formats to offer more advantages (e.g. encryption of data).

Grome file subsystem, for example, can use both packed and unpacked (normal) folders. So it can read the folders as they are on disk, with individual files, or it can have them packed (in custom format). The later variant is not used in the public build but it may be useful in the future. It is basically a file system inside the native operating system file system. The idea is that you can use the loose (unpack) variant when working internally (since it is easier to update files and see what's inside the folder) and then when giving the final build pack all assets (encrypt and archive them) into one chunk of data. Quake for example has .pak files (which are essentially zip archives). If the engine finds the files individually on disk, it uses those. If not tries to find them in a package (with the same path inside the package as they were individual on disk).

You can start with individual files and later develop the package system but you need to have all this in mind when designing the resource system.

Other than security and less data on disk, packaged resources don't have a performance impact (actually they may be slower to read from packages).

Regards,
Adrian L.
Back to top
View user's profile Send private message
keinmann
PostPosted: Fri Dec 04, 2009 5:25 pm    Post subject: Reply with quote



Joined: 14 Sep 2009
Posts: 15

Yes, I'm familiar with resource archives. Smile

I've modded IL-2 quite a bit which uses .SFS archives. I used 3D Gamestudio for a little while, which uses .WRS. And of course, I've seen them on Unity and Unreal, and even some independent implementations. I've got one in planning, but of course, I need to be sure of everything I need to put into it and what it needs to do first. Wink

You're right, it can be a bit slower, but you've got some fairly beefy security and your deployment size can be significantly reduced. Like everything in computing, there's a tradeoff.

I'm just thinking in terms of this single terrain (*.whatever) file holding its own embedded pixel data and HLSL. Naturally, that one file itself will be bigger than just one single file of another type, but it seems to me it just might be more convenient and clean. For instance, instead of having a folder in my solution with the terrain file, the textures, and shaders, it could be just one single file holding ALL of that data. That may or may not be a good idea.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Based on a template by Dustin Baccetti
Powered by phpBB © 2001, 2005 phpBB Group