[SCM] Problem when replacing global variables with arrays.

[goodidea82]

I'm working on a refactoring tool that replaces global SCM variables with SCM arrays in order to reduce the number of global variables used in main.scm (SA). The technique has been described here and here.

Unfortunately I'm stuck with a problem (I hope it is just one problem). It seems that some global variables cannot be replaced with array expressions. I have reduced the problem so that the tool replaces only one global variable with an array while still showing the bug. The bug happens when the global variable:
$Video_Game //($708)
is replaced with
&0($15010,1i)
, where $15010 points to some custom memory. I'm sure that $15010 is correctly initialized. Of course, all occurrences of $Video_Game are replaced.

 

 

 

The following are probably not relevant details since I'm sure that I correctly initialize, save, and restore the extra memory. Nevertheless, here is the relevant initilization code:

 

 

//This File was generated by WriteReadArrayTest.py developed by GoodIdea82.0002: goto @ExtraVarsSection   //ExtraVars memory is defined from here to the definition of the label somewhere below. This is used by ExtraVarSaveLoadHander.asi by GoodIdea82.hex 11 22 33 44   //This code marks this memory for saving and loading by ExtraVarSaveLoadHandler.asi.endhex //Extra bytes for the case that the following memory is not aligned 00 00 00 00 end:bank0  //globalVarId=15010, sizeInDWORDS=1, sizeOfElement=4 hex  00 00 00 00 endhex//The following are extra bytes. When the memory is not aligned to 4, i.e. 31/16=1, these bytes prevent that wrong memory is accessed. Long strings are 16bytes long. 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00 end:ExtraVarsSection....//This File was generated by WriteReadArrayTest.py developed by GoodIdea82.//sizeInDWORDs=1, sizeOfElements=4  $15010 = @bank0  $15010 /= 4

 

 

I have created an asi plugin that saves and loads the memory between hex 11 22 33 44 end and the label :ExtraVarsSection . For example, when saving my log file says:

 

...getJumpOffset(0) = ea9cgetJumpOffset(ea9c) = 10f20getJumpOffset(10f20) = 11154getJumpOffset(11154) = 11a08getJumpOffset(11a08) = 11a14getJumpOffset(11a14) = 11a24getJumpOffset(11a24) = 11a47Found custom memory marker. Saving the following memory region from index 11a24 to 11a4711a20: 3e 2 0 0 2 0 1 47 1a 1 0 11 22 33 44 0 11a30: ff ff ff ff 0 0 0 0 0 0 0 0 0 0 0 0 11a40: 0 0 0 0 0 0 0 a4 3 9 4d 41 49 4e 0 0 ....

 

The first part jumps the jump-statements in main.scm until it reaches my memory marker 11 22 33 44. After that you can see the normal beginning of the main code a4 3 9 4d 41 49 4e //03A4: name_thread 'MAIN'. ff ff ff ff is the value of the array expression &0($15010,1i). I think this is strange.

 

When loading the game my log file says:

 

....Reading 35 BYTEs from extraVars\saveVars7.bin------------------- 2 0 1 47 1a 1 0 11 22 33 44 0 ff ff ff ff 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0-------------------==========================Resulting memory in main.scm after loading file extraVars\saveVars7.bingetJumpOffset(0) = ea9cgetJumpOffset(ea9c) = 10f20getJumpOffset(10f20) = 11154getJumpOffset(11154) = 11a08getJumpOffset(11a08) = 11a14getJumpOffset(11a14) = 11a24Found custom memory marker11a20: 3e 2 0 0 2 0 1 47 1a 1 0 11 22 33 44 0 11a30: ff ff ff ff 0 0 0 0 0 0 0 0 0 0 0 0 11a40: 0 0 0 0 0 0 0 a4 3 9 4d 41 49...

 

 

Hence, I know for sure that my save/load plugin works correctly.

 

 

 

I start a new game and see no bug. Then I save the game, shut-down, restart, and load the savefile. I'm immediately the in-game video game "They crawled from Uranus". This is just an example of a bug, but potetially there are many such bugs so the actual reason must be understood and solved.

 

The variable $Video_Game //($708) is used with opcode 0871 here :

...:Case_Video_Game0871: init_jump_table $Video_Game total_jumps 8 default_jump 0 @End_Case_Video_Game jumps 0 @MS_Game_TheyCrawledFromUranus 1 @MS_Game_Duality 2 @MS_Game_GoGoSpaceMonkey 3 @MS_Game_LetsGetReadyToBumble 4 @MS_Game_TrackBetting 5 @MS_Game_Pool 6 @MS_Game_Lowrider 0872: jump_table_jumps 7 @MS_Game_BeefyBaron -1 @End_Case_Video_Game -1 @End_Case_Video_Game -1 @End_Case_Video_Game -1 @End_Case_Video_Game -1 @End_Case_Video_Game -1 @End_Case_Video_Game -1 @End_Case_Video_Game -1 @End_Case_Video_Game :MS_Game_TheyCrawledFromUranus0004: $ONMISSION = 1 0417: start_mission 3  // Video Game: They Crawled From Uranus0002: jump @End_Case_Video_Game ...

Currently I belife the reason is that some opcodes do not work correctly, when global variables are replaced by array expressions. Could someone check how 0871 works and let me know if my assumption is correct?

If the assumption is correct, then the next question is how to find out which opcodes are compatible with the array-trick and which are not.

 

(I'm testing this on the original game US 1.0 no-cd with modified main.scm (as described), FLA 2.1 where only the SCM limits were increased, SCRLog, cleo 4.1 (same results with cleo 4.3))

Edited December 18, 2015 by goodidea82

[goodidea82]

(With permission from Link.)

Yep, I saw that thread, but I looked into 0871 and there was nothing avoiding it from being used like this.

Ah, and timer opcodes (e.g. 014E) and others that "attach a variable" to a internal game state won't work with indices further than 16 bits.

The follow-up question is basically what I wrote in the forum. Do you have an idea how to find all opcodes where global variables cannot be replaced with arrays (or address memory outside the global variables space)? I suppose that not the value or the variable is used but the pointer (or index) of the variable.

The issues with 0871 is strange. When I'm back at my modding computer I'd like to send you my files, if that's ok for you.

Sure, you can.

And yes, it seems easy as pie to get which opcodes have that restriction. Any opcode that uses this function to get the var

00464700 ; uint16_t __thiscall CRunningScript__CollectGlobalVariableParameter(CRunningScript *this)

Just check xrefs.(X button with the cursor over 00464700).

Ok, too curious not to look. Those are the opcodes:
014E
014F
0151
03C3
03C4
04F7
059C
065D (NO OP)
065E (NO OP)
065F (NO OP)
0868 (NO OP)
0890

Edited December 27, 2015 by goodidea82

[fastman92]

(With permission from Link.)

 

Yep, I saw that thread, but I looked into 0871 and there was nothing avoiding it from being used like this.

 

Ah, and timer opcodes (e.g. 014E) and others that "attach a variable" to a internal game state won't work with indices further than 16 bits.

 

The follow-up question is basically what I wrote in the forum. Do you have an idea how to find all opcodes where global variables cannot be replaced with arrays (or address memory outside the global variables space)? I suppose that not the value or the variable is used but the pointer (or index) of the variable.

 

The issues with 0871 is strange. When I'm back at my modding computer I'd like to send you my files, if that's ok for you.

Sure, you can.

 

And yes, it seems easy as pie to get which opcodes have that restriction. Any opcode that uses this function to get the var

 

00464700 ; uint16_t __thiscall CRunningScript__CollectGlobalVariableParameter(CRunningScript *this)

Just check xrefs.(X button with the cursor over 00464700).

 

Ok, too curious not to look. Those are the opcodes:

014E

014F

0151

03C3

03C4

04F7

059C

065D (NO OP)

065E (NO OP)

065F (NO OP)

0868 (NO OP)

0890

 

 

CRunningScript__CollectGlobalVariableParameter

Could be improved to support array.

[goodidea82]

If the improvement would not have any disadvantages, e.g. incompatibility with cleo, then this would be the preferred way (otherwise I have to create a filter, so that variables used with these opcodes do not get replace). Deji has warned us about changing the addressing space of global variables (__int16 to __int32), so I don't know if this would apply here too or not.

[goodidea82]

I found the reason for the problem (replacing $Videa_Game with an array) and solved it. (At least I belief so after testing the solution.)

The solution was given by Link some time ago in this post. As Link pointed out:

 

But here's a thing: It seems scripts will run one tick (until the first wait of each of 'em) at the end of GenericLoad.

If what you actually want to do is run before this script tick, do this:

typedef function_hooker<0x618F05, void(void)> ldngb_hook;make_function_hook<ldngb_hook>([](ldngb_hook::func_type CTheScripts_Process){CallOnLoad(GetSlot());return CTheScripts_Process();});

 

Back then I found another bug that solve my issue there and I didn't know that what Link described would be an issue here.

I have implemented a filter in my refactoring tool so that variables are not replaced with arrays which are used with the opcodes 014E, 014F, 0151, 03C3, 03C4, 04F7, 059C, 065D (NO OP), 065E (NO OP), 065F (NO OP), 0868 (NO OP), 0890
However, patching CRunningScript__CollectGlobalVariableParameter would be the preferable solution. Hope someone will do it.

What all that means is that the hardest part of step 1 is perhaps accomplished (will do more testing). I will start doing experiments for step 2. I will do the whole process with SA first. Only when the whole transformation work for SA, I will continue with VC.
One definite show-stopper is still this.

Edited January 16, 2016 by goodidea82