SCM size limits

[spaceeinstein]

Something is wrong. When I put this in a blank SCM, this mod worked fine. But when I put this in my modded SCM (with Steed, chop shop, invincibility, first person camera, fast heli take off, and heli magnets), the game crashes when I tried to activate this. Why?

[PatrickW]

I've noticed the same thing while working on my Mission Master Mod. After adding a considerable amount of code, the game just crashes some seconds after CJ is spawned next to his bike. Even without activation of the newly added code.

 

From further experimenting I deduced that it must be the size of the main part that seems to be limited. Once I removed some of the original threads from the main part ( that were no longer needed due to my mod) the game started to work again without crashes.

I haven't yet found the exact size of the main-part at which it crashes, but I think the limit is somewhere around 202 KB.

 

This seems to be a very tight limit, which is easily reached when adding a few mod-threads. And I sure do hope that in the near future someone will be able to figure-out the mechanism behind this limitation, and will be able to build some kind of limit-adjuster to increase this limit.

 

 

[spaceeinstein]

Oh my god. That is a crappy limit. Now my future All In One Mod will be crappier than Vice City's.

[Opius]

I found simply merging the SANGA and SACS threads into one was enough to stop the crashing.

[PatrickW]

Opius,

 

What's the size of the main-part of your code ( reported by SA Builder) ?

 

 

PS.

Could you please split of these last couple of posts ( about the crashes) into a seperate topic in mission coding discussion forum?

As I think this topic will need a lot more discussion and I don't want to pollute your showroom topic with this off-topic subject.

[Opius]

First four segments: 209,947

Largest mission: 68,439

Largest external script: 35,122

 

I'm assuming the main part is the first four segments. (even though there's more than four now)

[PatrickW]

Yeah

That first four Segment number is what I was talking about.

I've had crashes with that number at 203 KB, and succesfully ran a script with it at 201 KB...

Therefor my assumption that there's a limit 202KB.

 

I'm at a foreign computer right now, without SA builder, so can anybody check that size of an unmodded scm file ?

 

[Un3462]

limits for san andreas:

main limit (first segments): 200000 bytes.

missions: 69000 bytes each.

[Demarest]

Forgive me for being the SA newb, but couldn't you just farm code out to external SCMs? I haven't yet looked into how all that works, so somebody slap me with an education if I'm just totally out of bounds here.

[PatrickW]

Forgive me for being the SA newb, but couldn't you just farm code out to external SCMs? I haven't yet looked into how all that works, so somebody slap me with an education if I'm just totally out of bounds here.

% Slap: Syntax Error-Education not found

 

In theory, that should be possible, but as far as I know it is still unknown what kind of things you can or can't do in external script (e.g. things like start missions, are all opcodes allowed ?).

Furthermore, there may be limits on the maximum number of extenal script, or the maximum number that can run simultaneous. Has anyone tried adding their own external script yet ?

 

 

[Demarest]

Well consider me on the edge of my seat. While I'd like to be SA modding just to be able to help more, part of me is glad I can't so that by time I can, some of this stuff will be sorted. But when I first peeked at PS2 code, the external scripts mystified me. I definitely want to be party to whatever is found out

 

Didn't we used to have a juggernaut around here to tackle such things?

[Un3462]

the idea seems to be that you use main for things that run continuously, missions for, well, missions, and the external scripts for 'on-demand' things. someone needs to test whether the status of external threads is saved in the savegame.. because i don't think it is. if it is saved though, then we can use external threads for pretty much anything. otherwise you can only really use them for events and such.

[jonc]

I just did a test and it appears external threads are not saved. After loading, the counter no longer increments. It does keep incrementing after the save operation though.

 

As a side note, GTA apparently has a bug with loading save files with very few global variables. Before adding the large dummy array, the count variable was corrupted when loaded.

 

 

int count = 0;int dummy[1000];start thread main {Player player;Actor playerActor;// Some standard initializationplayer=CreatePlayer(0,2488.56,-1666.87,12.88);MakePlayerVisible(player);playerActor=MakeActorFromPlayer(player);LoadExternalThread(counter);while (!IsExternalThreadLoaded(counter)) { Wait(0); }StartThread(counter);while (true) { if (count == 10) { 	count = 100; 	TogglePlayerControllable(player, false); 	ShowSaveScreen(); 	while (!IsSaveDone()) { Wait(0); } 	TogglePlayerControllable(player, true); } Text1numberLowpriority('HJ_IS',count,500,1); Wait(500);}}external thread counter {while (true) { count = count + 1; Wait(2000);}}

 

[PatrickW]

That 's a shame. It surely limits the number of things external threads can be used for.

 

@CyQ: Do you have any idea what limits the Main-part to 200,000 bytes ? If so, we might be able to craft a tool that expands this limit to a more usefull size ( or make it configurable).

[Demarest]

When you call an external script, can you pass it variables like you can when creating a thread? If so, you might be able to get away with whenever a save routine is run, get all the variables from the SCM, kill the SCM, save, then restart the SCM tossing it back the vars? Might be code-efficient until somebody (hopefully) can raise the limit.

[Un3462]

jon; thanks for testing that. the need for dummy vars is very strange though.. i'll look into that.

 

patrick; two things create the limit; first the actual memory layout. there's a 200000 byte array for main code, followed by a 69000 byte array for the currently loaded mission (i always wondered why they didn't just dynamically allocate memory for the scripts, but they seem to be doing that for the external threads now). second, the calls that load main.scm simply read 200000 bytes from offset 0. no more, no less.

 

with some complicated and careful exe editing, we could shrink the mission code array and add some bytes to the main code array, but the gain would be minimal and (imo) not worth it.

 

a better idea would be to optimize main, like we did with gta3/vc. when you actually take a close look, rockstar's code is pretty horrible and there are size/speed improvements to be made pretty much everywhere. you can move repeated code blocks to subs, change 0.0 floats to 0 ints, optimize loops, combine nested ifs, etc, etc. we could either do this manually, and release an 'optimized scm' (copyright issues aside), or jon could implement various optimizations in his compiler, if he's willing (probably the best option, but since he's still working on the decompiler, we'd have to be patient ).

 

demarest; it all depends on what is actually happening. it might be that the thread descriptor is actually saved to scm, but it simply stops running because the external isn't loaded. in that case it might restart automatically, with variables intact, when you reload the external. before jon's test i thought it might keep running even if the external isn't loaded, but that would probably result in a crash, and jon didn't experience that, so some mechanism is either killing or pausing the thread. if the thread is really killed, we could possibly implement something like you propose, but i think it would get much too complicated to be really useful.

[StEaLtH-SnIpEr]

going from what CyQ said and everoyng else here i understand that there is a main block limit, im a newb at modding the scm, but cant some mods be put somewhere else outside the first four segments, and clear up the scm file like CyQ said to make this size much smaller?

[Y_Less]

If you cant run missions from external scripts, it shouldn't be too hard to say, run a MAIN thread to test the status of say a variable from the external script (e.g. $missiontorun (guess where I'm going with this)), and if the variable is greater than 0, the conditions for that mission (as determined by the external script) have been reached and the MAIN thread will run the mission (using start_mission $missiontorun (which does work btw)).

[jonc]

I extended the test to see if the thread state is intact and it is not. It isn't loaded when the game is loaded. If you load it, it does not resume execution. And if you start it, the variables and programcounter are reset.

 

... 	TogglePlayerControllable(player, true); 	if (!IsExternalThreadLoaded(counter)) {   LoadExternalThread(counter);   while (!IsExternalThreadLoaded(counter)) { Wait(0); }   StartThread(counter); 	}...external thread counter {int localCount = 0;while (true) { localCount = localCount + 1; count = localCount; Wait(2000);}}

 

 

I am pretty sure, though, that external threads don't have any restrictions on what opcodes they can run.

 

It would be possible to restart externals after a save, but the author has to be very careful that their code can be arbitrarily stopped and started without destroying the game state. One way to make this easier would be like this:

 

bool g_WantToSave;int g_SaveCriticalSection;function DoSave {g_WantToSave = true;while (g_SaveCriticalSection > 0) { Wait(0); }TogglePlayerControllable(player, false);ShowSaveScreen();while (!IsSaveDone()) { Wait(0); }TogglePlayerControllable(player, true);// Restart threads if loadedif (!IsExternalThreadLoaded(dostuff)) {LoadExternalThread(dostuff);while (!IsExternalThreadLoaded(dostuff)) { Wait(0); }StartThread(dostuff);}g_WantToSave = false;}function EnterSaveCriticalSection() {while (g_WantToSave) Wait(0);g_SaveCriticalSection += 1;}function LeaveSaveCriticalSection() {g_SaveCriticalSection -= 1;}int x = 0, y = 0, z = 0;external thread dostuff {while (true) {EnterSaveCriticalSection();int temp = x;x = y;y = z;z = temp + 1;LeaveSaveCriticalSection();// ...}}

 

That would ensure a save never took place while x, y, and z were being swapped around which could result in the z = temp + 1 statement never executing and the game state being destroyed. But it also has the possibility of a deadlock if the same thread enters the critical section it is already inside, and isn't exactly ideal. This code would also need to be able to reenter the while having skipped all the "// ..." in the case where the game was loaded.

 

 

And finally, yeah, CyQ, do I plan to make the compiler optimizing to some degree. Right now it is pretty much the opposite of optimizing (in a few particular cases). It is far more efficient with variables than Rockstar though so it should be able to cut the number of globals needed by quite a bit. Especially if we can make the optimization that mission variables aren't saved between mission runs, and since only one mission runs at a time any globals they use are fair game to be shared.

 

CyQ: Are you saying that if given an integer constant parameter their parameter read code automatically casts it to a float when needed?

[Y_Less]

Unfortunately, simply cutting and sharing all mission variables won't work, as globals are the only way to pass information between missions and MAIN, which is frequently done, unless all variables used in this way are pre-declared in MAIN.

[jonc]

Y_Less: Point uses fully scoped variables so it knows precisely which variables belong to a particular mission/thread and which belong as true globals. Even in decompiled SCM files since Rockstar never reused variables anywhere so it can scope variables down to a single mission/thread in most cases and in some cases even a particular function.

[Y_Less]

Fair enough then.

[Un3462]

shame about those externals.

 

your example, however, would be safe anyway; all code inbetween waits is executed consecutively, it's a simple cooperative multitasking model. in fact, the external in your example would freeze the game, since it doesn't have any waits at all.

 

as for parameters, they're simply loaded into a dword array. so 04:00 and 06:00-00-00-00 both become 00000000, and look no different to an opcode. in fact, you could use partype 01 instead of 06 everywhere and things would work (though it's not very friendly towards a disassembler/decompiler).

 

the scm format allows for many other simple and neat optimizations, here are two more i just thought of;

- multiple passes to fit some of the label addresses into 04/05 partypes.

- you can put 32-bit constants that are used multiple times in the global var space, and refer to them by 02:xx-xx instead of 01/06:yy-yy-yy-yy. works for label addresses, big ints, floats (would be a good idea for the -100.0 float, probably).

[jonc]

Haha, yes, Waits. I forgot about those. And good point about the cooperativeness. For some reason I was thinking SA could interrupt whenever it needed.

 

Good ideas for the optimizations. All of them would be fairly easy to add to Point except multiple passes for label int sizes. It just leaves 4 byte blanks in the file as it writes and then fills in all the addresses in a second pass.

 

I'm not sure if you can consider this an optimization, but right now Point takes some functions (those that access local variables) from the R* main segment that are used by multiple missions and pushes them into the mission segment. As long as the mission they go into wasn't in danger of going over its limit, you can call this an optimization. Otherwise its a curse.

[Un3462]

sounds cool. if i understand you correctly, that would increase the total size of the scm, though.. but as long as we get more room for main code, it's ok, i suppose.

 

i've been looking into the external threads. haven't been able to find a size limit yet, but the total number of externals seems to be limited to 82. i've also located the flag that marks a thread as external, and the savefile writing function does indeed filter those threads out.

[Demarest]

What are the chances of hacking the EXE to NOT filter those? And would this be an answer to the size limitation?

 

Anybody else notice how R* seemed to intentionally up the bar for modders? You guys should be on their payroll

[jonc]

Yeah, CyQ, you are correct.

 

Do you mean 82 loaded threads, or 82 total?

 

Demarest: Their payroll? Nah, I'd rather be on my current employer's payroll.

 

I think a hacked EXE though wouldn't be able to load the threads back in though since the loading code no doubt expects them to be in the main segment of memory.

[Hammer83]

Total number of threads is 96, including externals.

[Un3462]

jon; the latter, unfortunately. and the number of running threads is limited as well, as hammer83 notes.

 

demarest; it's like jon says. you'd load up the savegame with running external threads, sa would try to start executing code from the external threads, and crash because they aren't loaded into memory. next question would be: can we make the savegame loading code also load the necessary externals? answer: waaay too complicated.

 

but i'm pretty confident that we can optimize a generous number of kbs out of main.

[Hammer83]

The basketball glitch (where all basketballs disappear) occurs when external script BASKETB (15) terminates (there is an odd end_thread in the middle of the script which is triggered at some point in the game). The only way to restore the basketballs is thus to reinitialize the external thread 15. But the code which does it at the beginning

 

0929: init_external_script_with_object_trigger  15 (BASKETB) object #BSKBALL_LAX priority  100  70.0 -1 0929: init_external_script_with_object_trigger  15 (BASKETB) object #BSKBALLHUB_LAX01 priority  100  70.0 -1 0929: init_external_script_with_object_trigger  15 (BASKETB) object #VGSXREFBBALLNET priority  100  70.0 -1 0929: init_external_script_with_object_trigger  15 (BASKETB) object #VGSXREFBBALLNET2 priority  100  70.0 -1 

 

 

never executes again. The way I was able to restore it in my save game was to load the save game, then patch the part of main.scm in memory that loops continuously (somewhere between labels MAIN_392 through MAIN_641) so that these four commands execute, and then patch back the original code, taking precaution of checking main thread's position along the way to make sure it is in the right place.

 

It works and I guess can be adapted to initialize any external script that executes only once.