www.quadsoftware.com
FAQ  FAQ   Search  Search   Memberlist  Memberlist   Usergroups  Usergroups
Register  ::  Log in Log in to check your private messages


Post new topic  Reply to topic
 Memory leak with Grome SDK and ui while using color layer « View previous topic :: View next topic » 
Author Message
MPujol
PostPosted: Fri May 14, 2010 2:15 pm    Post subject: Memory leak with Grome SDK and ui while using color layer Reply with quote



Joined: 15 Mar 2010
Posts: 8

Hello,
I wrote a plugin to import large terrain (currently 10km x 10km with a precision at 8 meters).
It imports a list of tiles to create. For each tile I create a new zone, then I set the heightmap and after I want to set a texture. Before creating the next tile, I swap the zone.
I encounter many problems of memory when I add a color layer. When I add textures it seems that it is not deleted from memory after swap or delete of the tile and/or color layer. I have the problem when i create tile from the sdk or the ui.
If I made a new project, add 10 zone and for each add the same texture the memory increase (normal reaction Very Happy) but when I delete my 10 tiles the memory decrease only few MB. The only way to free the memory is to create a new Project.

Could you explain me wich is wrong in my code and my approach? Thanks.
Regards

Mathieu
Code:

printToConsole("Project files opened");
      PlugInGeoMapXmlReader::TileData lTile;

      csdk::iTerrainEd * lTerrainEditor = (csdk::iTerrainEd*)g_sdk_root->GetInterface(C_TERRAINED_INTERFACE_NAME);
      
      //get heigh layer 0
      csdk::iTerrainHeightmap * lHeightLayer =(csdk::iTerrainHeightmap *)lTerrainEditor->GetLayer(csdk::iTerrainHeightmap::TypeString(),0);

      //get color layer 0
      csdk::iTerrainMaterialLayer * lColorLayer =(csdk::iTerrainMaterialLayer *)lTerrainEditor->GetLayer(csdk::iTerrainMaterialLayer::TypeString(),0);
      if(!lColorLayer)
      {
         //create it if not exist
         lColorLayer =(csdk::iTerrainMaterialLayer *)lTerrainEditor->AddLayer(csdk::iTerrainMaterialLayer::TypeString(),NULL,-1);
      }

      //while there is tile to compute in import file
      for(lTile = lXmlReader.getNextTile();
         !lTile.mTileName.isEmpty();
         lTile = lXmlReader.getNextTile())
      {
         printToConsole(lTile.mTileName);
         
         //create a new zone
         csdk::iTerrainZone * lZone[1];
         csdk::t_char wStr[2048];
         mbstowcs(wStr,lTile.mTileName.toAscii().data(),2048);
         lTerrainEditor->AddTerrainZones(wStr,//name
            1,1,//create one zone
            lTile.mTileResolution-1,//number of quad in the zone
            lTile.mTileSize/(float)(lTile.mTileResolution-1),//resolution of a quad in meter
            lTile.mTilePos,//position in the world
            lZone);

         //add heigh data
         lHeightLayer->SetData(lZone[0],lTile.mHeightData);

         //add a texture
         mbstowcs(wStr,lTile.mTextureFile.toAscii().data(),2048);
         //create a texture node
         csdk::iTexture* tex = (csdk::iTexture*)g_sdk_root->NewNode(C_NODE_FACTORY_TEXTURE, NULL, NULL, NULL);
         //set the file
         tex->SetSourceImage(wStr,NULL);
         //load the texture
         tex->Load();
         
         //Texture
         csdk::sTextureChannel texture_channel;
         texture_channel.id = C_TEX_CHANNEL_TEXTURE;
         texture_channel.mapping = NULL;
         texture_channel.source = tex;

         //Materiel
         csdk::sMaterialDefinition material_definition;
         material_definition.name = M_SZ("Simple color");
         material_definition.channels = &texture_channel;
         material_definition.channels_no = 1;

         //add the texture to the color layer for this zone
         lColorLayer->Assign(lZone[0],&material_definition);

         //swap the new zone
         lZone[0]->SwapData();
      }
Back to top
View user's profile Send private message
ALicu
PostPosted: Fri May 14, 2010 3:18 pm    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Hi,

After looking a little over your code I think you should close the interface to the texture after assigning it. So you should call tex->CloseInterface() after calling Assign to the color layer. Grome will make a copy of the texture/image and I think that your temporary texture remains created in memory.

I will test this more on Monday once I am back at the office. In the mean time please let me know if this fixes your issue.

Also, is this memory leak issue happening when you are doing it from the editor UI, too?

Regards,
Adrian L.
Back to top
View user's profile Send private message
MPujol
PostPosted: Fri May 14, 2010 4:53 pm    Post subject: Reply with quote



Joined: 15 Mar 2010
Posts: 8

I have tried to close the texture interface but it does not change my problem.
For example i have 4 tiles (128 x 128 quad) with 4 textures 2048*2048.
I start Grome it take 50MB / 50MB (Ram/virtual memory)
I execute my plugin it take 132MB / 219MB
I swap my 4 tiles it take 83MB / 120MB
I unswap tiles 132MB / 219MB
I delete tiles and the color layer 129MB / 166MB.
The problem is that I could reproduce it without using my plugin juste by creating tiles with the ui and set the texture manualy.
Maybe the problem is the undo framework which keep data in memory ?
Back to top
View user's profile Send private message
ALicu
PostPosted: Sat May 15, 2010 7:53 am    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Hi,

Undo system is using a scratch file on disk to keep data so it should not keep significant data in ram.

One thing to notice here is that task manager should not be trusted in memory consumption since the operating system will still keep memory pages around (as a cache optimization) even if they are marked as free (they are completly remove from process space when other applications request them). Complications are made by the video driver too as it may keep AGP memory occupied until needed for other things.

In Grome we have our own memory tracking system which we run in debug mode and this sys doesn't report any leaks.

I will check this again. One way to check it is by creating another 4 new zones and see if the memory increases or the already reserved memory is used.

I will investigate this and let you know my findings.

Regards,
Adrian
Back to top
View user's profile Send private message
MPujol
PostPosted: Mon May 17, 2010 7:42 am    Post subject: Reply with quote



Joined: 15 Mar 2010
Posts: 8

I call that memory leak but maybe it's not at the programming signification.
I have test if I create 4 tiles + texture and delete them, when I create four new tiles + texture the memory always increase.
And when I run my plugin wich create one hundred tiles which are swapped, Grome take 2Go of Ram and when I unswap few tiles, some textures are missing and geometry is corrupted, while thos problem does't appears whit small textures.
Back to top
View user's profile Send private message
ALicu
PostPosted: Mon May 17, 2010 8:05 am    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Hi,

After some tests and talks with a programmer working at the swap system, he confirmed me that for delete, the undo system still keeps some data in memory. So in case of very many delete and recreation of zones, you eventually end up out of memory. The programmers had some problems with this situation and couldn't use scratch disk. This is a very rare situation since delete and recreation is not a task you do it regularly for many times.

The undo system should not create memory leaks as it is using the scratch disk (I've tested this, by swapping / unswapping many times and the memory reported by task manager settled to a fixed memory consumption). Just complete delete of zones will keep some zone data (like layer structure and heightmap) until the undo stack is full (in which case the old records are eventually deleted).

So, in your case, the plugin should not create memory leaks. Can you please send me a compiled version of the plugin to test it here (you can use my email: licu at quadsoftware.com). The plugin can be in binary form, we will debug it by tracing the calls to the Grome SDK.

Best Regards,
Adrian
Back to top
View user's profile Send private message
MPujol
PostPosted: Mon May 17, 2010 12:43 pm    Post subject: Reply with quote



Joined: 15 Mar 2010
Posts: 8

Thank for your usefull help.
I think I have isolated the problem.
It is when I call the méthode SetSourceImage() of iTexture then the memory increase.
If I comment this line my memory is still stable.
So I think I don't use correctly the mecanisme of image loading.
Without this line, calling my plugin don't increase memory occupation, if I uncoment it the memory is growing fast.
I could send you an example but it take 80MB so if you could explain me how to load many texture maybe it could resolve my problem.

Code:

csdk::iTexture* tex = (csdk::iTexture*)g_sdk_root->NewNode(C_NODE_FACTORY_TEXTURE, NULL, NULL, NULL);
//set the file (if I uncomment this line, the memory use by grome grow)
//tex->SetSourceImage(wStr,lStorage);
tex->CloseInterface();
Back to top
View user's profile Send private message
ALicu
PostPosted: Mon May 17, 2010 1:28 pm    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Oh, I see.

Basically there were two unrelated issues.

1. First the plugins issue.

After you assign the texture layer, because you are painting on its texture (because it is a "simple color" layer and not a "masked tiled" layer), Grome will make a copy of its image and place it in the project folder. So you should do call SetSourceImage. Your image is then cloned inside the project folder and after assignment of the layer you should call:

g_sdk_root->DelNode(tex);

So you delete your temporary loaded texture and its loaded image data (the copy of the image is already made after assignment). Otherwise you will keep in memory two copies of the image (the original image you used when creating the layer and the editing layer image).

So just call DelNode at the end of your code (don't need to call CloseInterface on texture anymore since this is done by DelNode).

NOTE: to be noted that you are creating a Simple Color layer, that is a layer without mask. In Grome there are 2 main types of layers: simple color and masked textures. On the first one, the layer is visible on the entire terrain and you paint with the color brush in the RGB channels. The layer image is kept in the scene folder. The later type is a layer on which you paint the mask (a greyscale image which indicate the coverage of the layer and which is kept in the scene folder). The mask indicate where a tiled external image is visible on the terrain. The tiled image is not copied inside the scene project, it is left external, on disk, in the libraries and shared between all the layers since you don't paint on it making it unique per terrain zone (you paint on his mask). In your plugin you are creating simple color images which are cloned inside the project folder.


2. The second issue.

This you've discovered while trying to replicate the bug from Grome U.I. This is the issue with the undo system leaving some deleted zones data in memory.

These two are actually two separated issues, not related.

I hope this fixes your issue and somewhat clarifies the cause of this. Sorry for the lengthy post but I wanted to explain the cause of it.

Best Regards,
Adrian L.
Back to top
View user's profile Send private message
MPujol
PostPosted: Mon May 17, 2010 2:09 pm    Post subject: Reply with quote



Joined: 15 Mar 2010
Posts: 8

Thanks you for your help, your solution fixes my problem. Now I can load more than hundred texture without any memory overflow.
I have an other question. When my plugin run, it could take lot of time, during this time, Windows say that grome is crashed. Is there a mechanisme to do an update from the plugin to refresh the ui. It's just for the user experience.
Regards
Back to top
View user's profile Send private message
ALicu
PostPosted: Mon May 17, 2010 2:14 pm    Post subject: Reply with quote



Joined: 12 Feb 2007
Posts: 1326

Hi,

We do plan to add a windows update function in the SDK (or even a cancel button). For now there is no way to update the UI (except the progress bar).

Regards,
Adrian L.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Based on a template by Dustin Baccetti
Powered by phpBB © 2001, 2005 phpBB Group