[REL|SA] Map Cleaner

[steve-m]

Update: I've uploaded a new version of the Map Cleaner, should be fully functional. It's recommended to use the default settings, so just select your SA directory and press start. Check out the readme for full documentation and hints. Stripped mission script and water file included.

 

 

[ Get SA Map Cleaner 0.4 Beta ]

 

There's also a test version of the upcoming 0.5 release available.

 

- - - original post - - -

 

illspirit and I were working on cleaning the SA map the last days, mainly for total conversions such as MI, but weren't very successful yet. When deleting too much, the game either crashes immediately after everything was loaded, or it enters an infinite loop of doom.

 

It's no problem to get the (visual) map clear. All you need to do is remove all ipl entries from the gta.dat and delete all binary ipls in gta3.img. It's even possible to remove the interior ide entries and delete the complete gta_int.img (!), but as soon as you start removing some of the city map ide files, the problems begin.

 

As far as we know, you can remove all of the \LA\ ide files, and very few of the others, without a crash. illy tried to delete single object defs in one ide that wasn't working, and eventually got it running, but only with half the defs still in, and it didn't seem to matter which half was deleted.

 

So then we went on with deleting unused files from the img, since that was a problem for VC. I even coded a tool for that. But still not more successful than before.

 

So that's why we are publishing this, and we hope someone might still have any idea we overlooked, or can do some more testing and maybe get it working by deleting just some more col files or anything. I've uploaded my tool and a stripped water file and mission script, all in the following package. Take a look at the readme before using the tool! And remember to have a complete backup copy of SanAn, YOU WILL NEED IT!!!

 

Download

 

 

Edit: And on a side note, we know that there must be some data that define how items are grouped and replaced by LOD (parent objects and such). And I'm quite sure that's the cause of the above mentioned problems. But atm we have no idea where these are hidden. Possibly binary and text ipls, since they have a new parameter (which is -1 for LODs and non-child objects), and there could be some other data hidden in the binary ipls now. Or, it has something to do with the 64 nodes*.dat files, which might divide the map into a 8x8 (or deeper) grid, containing links and parameters to the item instances, which might be used by the engine for some sort of quadtree sorting. But that's only a wild guess, since I haven't really looked at those files yet.

Edited October 24, 2007 by steve-m

[Opius]

Perhaps there's a minimum number of IDE entries that need to be there? Have you tried dummy IDEs, with the same ammount of entries as the originals, but all the entries point to a single DFF and TXD?

[steve-m]

No. Do you want to try?

[Opius]

I'm at school, fool

[Gforce]

i've only just noticed this thread, i'll give this a try later.

 

nice work so far steve.

[Inky]

It's no problem to get the (visual) map clear. All you need to do is remove all ipl entries from the gta.dat and delete all binary ipls in gta3.img. It's even possible to remove the interior ide entries and delete the complete gta_int.img (!), but as soon as you start removing some of the city map ide files, the problems begin.

So from what I understand, you can make all of SanAndreas disappear right? But when you delete say, LAe.ipl, the game crashes? I know this is pretty simple, but did you delete the lines in gta.dat too?

[steve-m]

So from what I understand, you can make all of SanAndreas disappear right? But when you delete say, LAe.ipl, the game crashes? I know this is pretty simple, but did you delete the lines in gta.dat too?

I can make the map disappear by deleting all the IPL stuff. The problem are the IDE files. And I'm not going to answer that last question...

 

BTW, here is a screen I made earlier, showing the side of the parametric water and seabed outside of the 6000² map area.

 

[JernejL]

so you tried removing the _streaming ipls and paths too?

maybe it is the cols, exe associates cols to ides not ides to cols..

maybe they are ifp animations for objects..

 

[steve-m]

Yes. That's mentioned in that readme.

[JernejL]

 

Yes. That's mentioned in that readme.

sorry about that, i havent checked yet what is in the package

[Opius]

Sorta on-topic: Steve, does that mean that the supposedly infinite water and seabed is controlled in some way by the map files?

 

Come Friday and I'll have two weeks off, I'll look into these things.

[flitcroft101]

I don't know anything about coding or map modding(or anything else lol) couldnt you add MI next to sna andreas, theres alot of water or is that impossible?

[JernejL]

 

## Load IDEs first, then the models and after that the IPLs#SPLASH loadsc2## Object types#IDE DATA\MAPS\generic\dynamic.IDEIDE DATA\MAPS\generic\dynamic2.IDEIDE DATA\MAPS\leveldes\seabed.IDE# interiors# vehicle upgrade modelsIDE DATA\MAPS\veh_mods\veh_mods.IDE# txd parents# have to load map.zon before any of the IPLsIPL DATA\INFO.ZON## Scene information#SPLASH loadsc2# interiors# level design##  Interiors#SPLASH loadsc4

 

 

it works.. you can also remove the splashes but seabed, veh_mods, info zone and dynamic objects must stay there.

 

[aad]

so when i use that as a gtasa.dat my whole city is gone except the ide you still have putted in the gtasa.dat Delfi?

now lets hope that someone will release a dff converter as soon as possible.

[GT-1]

Is that when using the stuff that Steve gave or not? Crashes at the very start of the loading bar for me.

My fault, tried it with the wrong SA files. After some funky stuff on the loading bar it looks to have work.

[JernejL]

so when i use that as a gtasa.dat my whole city is gone except the ide you still have putted in the gtasa.dat Delfi?

now lets hope that someone will release a dff converter as soon as possible.

no, i didn't add anything, just removed as much as i could at which state the game still ran.

[DexX]

Delfi and GT-1 speak the truth. aside from some uber-weird effects while the loading screen progresses, it works. funky audio from the hydra as well, no engine sounds. but thats a bug i've been getting anyway even during normal gameplay.

 

so all thats left (hopefully) is the radar and maybe the occlusion?

[JernejL]

another vierd thing happened with my ide file - i did remove all paths but still seen one normal boat created ingame also are airport locations hard-coded?? because they are still in stripped version.

 

[Fro]

You guys never cease to amaze me. I'll try this in a minute.

[AleXXX]

This tools didn't work in my SA version.

[GT-1]

Err... next stop Myriad. Although there seems to be plenty of other things to figure out yet.

 

[aad]

Im sure myriad will be the first total conversion for gta sa all the people working on Myriad already know a lot of things and im sure steve-m already discovered a lot of secrets like formats and colfile stuff hopefully some stuff will get released soon. and amazing screen btw GT-1.

[Opius]

Delfi and GT-1 speak the truth. aside from some uber-weird effects while the loading screen progresses, it works. funky audio from the hydra as well, no engine sounds. but thats a bug i've been getting anyway even during normal gameplay.

EXACT same symptoms, except my sound works normally on an unstripped install.

 

I also found another interesting, probably memory related problem.

For sh*ts and giggles, I drove my Hydra directly down into the water, killing myself. When I respawned, there were these strange lines off the distance. Whenever I looked at their origin, the game crashed. Brought up the contrast since you couldn't really see them in the original.

[illspirit]

Hmmmm... I don't get it. The game didn't work for me if we leave the rest of the generic ide's and levelxre.ide in. And we kinda need that stuff if we're to add, well, anything. Procobj.ide is needed for col's to have plants on them, and vegepart.ide is needed for any trees. Multiobj.ide has stuff like beach towels that spawn with peds in beach areas. And levelxre.ide has like all the stuff for mini games.

[ThAtSgOtTaHuRt]

If you have ever played the mission where you fly out to Liberty City - you will clearly see there is almost an UNLIMITED amount of space within the surrounding ocean. Even not during that mission; I have started flying out to sea and stayed in a strait line for over 5 minutes without hitting the edge of the world. At first I thought maybe you actually hit the edge of the map and they somehow replay the water, sky, and cloud textures\objects on a loop. But amazingly I turned around and it took EXACTLY the same amount of time to fly back - proving that all five minutes of flying I was actually flying over NEW land\sea - not just a loop. A simple test would be to place an object WAY outside main-land SA and fly to that object - to actually see if it is true. If you can modify IPL and IDE limits that SA will allow. (like the VCLimitAdjuster did with VC) without it crashing - you may be able to keep SA while adding myriad (or other maps). The space is there but having too many objects is the only problem.

Edited June 24, 2005 by ThAtSgOtTaHuRt

[Fro]

If you have ever played the mission where you fly out to Liberty City - you will clearly see there is almost an UNLIMITED amount of space within the surrounding ocean. Even not during that mission; I have started flying out to sea and stayed in a strait line for over 5 minutes without hitting the edge of the world. At first I thought maybe you actually hit the edge of the map and they somehow replay the water, sky, and cloud textures\objects on a loop. But amazingly I turned around and it took EXACTLY the same amount of time to fly back - proving that all five minutes of flying I was actually flying over NEW land\sea - not just a loop. A simple test would be to place an object WAY outside main-land SA and fly to that object - to actually see if it is true. If you can modify IPL and IDE limits that SA will allow. (like the VCLimitAdjuster did with VC) without it crashing - you may be able to keep SA while adding myriad (or other maps). The space is there but having too many objects is the only problem.

Makes sense, and seems probable.

[burpicane]

If you have ever played the mission where you fly out to Liberty City - you will clearly see there is almost an UNLIMITED amount of space within the surrounding ocean. Even not during that mission; I have started flying out to sea and stayed in a strait line for over 5 minutes without hitting the edge of the world. At first I thought maybe you actually hit the edge of the map and they somehow replay the water, sky, and cloud textures\objects on a loop. But amazingly I turned around and it took EXACTLY the same amount of time to fly back - proving that all five minutes of flying I was actually flying over NEW land\sea - not just a loop. A simple test would be to place an object WAY outside main-land SA and fly to that object - to actually see if it is true. If you can modify IPL and IDE limits that SA will allow. (like the VCLimitAdjuster did with VC) without it crashing - you may be able to keep SA while adding myriad (or other maps). The space is there but having too many objects is the only problem.

tools like vc limit adjuster are much more harder to make because sa has securom 7 protection

[illspirit]

The problem with adding another city to San An isn't physical, or, ummm, virtual space. In theory, objects can be placed as far away as there are 32bit integers large enough to store the location with. The real problem is there's only so much memory for storing the object definitions and locations. I'm not sure just how it works, but there looks like the game can only handle 256 ipl's, and within those, there's only so many items that can be in any given area.

[DexX]

 

Hmmmm... I don't get it. The game didn't work for me if we leave the rest of the generic ide's and levelxre.ide in. And we kinda need that stuff if we're to add, well, anything. Procobj.ide is needed for col's to have plants on them, and vegepart.ide is needed for any trees. Multiobj.ide has stuff like beach towels that spawn with peds in beach areas. And levelxre.ide has like all the stuff for mini games.

working off the stripped installation, i added these lines back into my gta.dat and it still loaded (with the same graphical glitches as before)..

 

IDE DATA\MAPS\generic\vegepart.IDEIDE DATA\MAPS\generic\multiobj.IDEIDE DATA\MAPS\generic\procobj.IDE

 

so they arent a problem.

 

i also added these lines...

 

IDE DATA\MAPS\leveldes\levelmap.IDEIDE DATA\MAPS\leveldes\levelxre.IDE

 

and the game still loads fine. even if i comment out teh levelmap.IDE, it loads ok

[ThAtSgOtTaHuRt]

as I said:

 

If you can modify IPL and IDE limits that SA will allow. (like the VCLimitAdjuster did with VC) without it crashing - you may be able to keep SA while adding myriad (or other maps). The space is there but having too many objects is the only problem.

 

illspirit:

 

The problem with adding another city to San An isn't physical, or, ummm, virtual space. In theory, objects can be placed as far away as there are 32bit integers large enough to store the location with. The real problem is there's only so much memory for storing the object definitions and locations. I'm not sure just how it works, but there looks like the game can only handle 256 ipl's, and within those, there's only so many items that can be in any given area.

 

 

 

I totally didnt think of it as being a memory issue although you are 100% correct and only so much tweaking can be made. Well with Vice City you were able to increase the object instances (which are stated in IPL files) ALOT - adding a thousand+ more than original (at least I did). But the actuall objects (located in IDE files) could only be increased so much before having a problem. So having replecas (other instances - IPL) of SA's objects would be very easy to do - but actually adding in your own map objects (IDEs) would\may not be possible with the entirety of SA still there. I dont really know all the facts, so I will just have to wait and see what info develops and how far GTASA mapping can really go.