Compiled Path Nodes

[steve-m]

Anyone up for a new riddle?

 

Been looking at the nodes*.dat files, which I guess are the compiled path nodes, so the game doesn't have to process them and can load faster. There are 64 files, each one for an area of 750 * 750 units, starting with the SW corner. Also, these node files are streamed by the game, only the active area and those surrounding it are loaded, not more.

 

Now the problem is, these files have a very ugly and hard to decode format. There is a line in the stream.ini, "dontbuildpaths", which might have disabled the internal path compiler, but removing it doesn't seem to have any effect, maybe someone could confirm that. So unless someone is able to re-activate it, we have to try to decode the format and write our own path compiler.

 

And this is where the fun starts. There are tons of parameters and flags in those files, so I assume there is even some advanced stuff stored there, such as traffic lights and such. What would be a nightmare of course.

 

- - - - -

 

Update: file format specification moved to the wiki

 

- - - - -

 

Download empty path nodes (745 Bytes)

 

Use these to get rid of all paths, they contain an empty header and the section 4 data. See here on how to use them.

 

- - - - -

 

And btw, SA only uses the nodes*.dat files inside an img, not those from the data\paths\ directory.

Edited November 14, 2006 by steve-m

[Hammer83]

Can't say I can contribute much to what you already discovered. Basically I can cofirm that what you describe here is correct. I was able to create a parser for node*.dat files according to your structure and was able to plot the paths:

 

http://suchy63.rhlmod.ru/images/nodesdat.gif (WARNING: 1 MB image)

 

So, since the game uses these files for paths, are we to assume it ignores the entries in the ipl files? These paths seem complete to me, even interiors (like rinks) are there, so I guess ipl paths are ignored.

[JernejL]

so the dontbuildpaths maybe says the game not to use ipls but nodes.dat files?

 

[steve-m]

Thanks Hammer. No, ipl paths aren't used, that's for sure. But they were used to create those node files by R*'s compiler, which is either an extra program, or built into the game but not included in the retail version, or just disabled. But as I said, the "dontbuildpaths" doesn't seem to have any effect.

 

The problem atm is that I don't know all sections, so I can't create my own files for testing. I wasn't even able to create empty files, which would be needed, since the original files always try to connect to each other (unless I remove all external links).

 

Edit: But the graphic doesn't include ped paths, does it? Or no car paths? I'm confused...

 

Edit 2: Yes, peds are missing. Compare to this map (1.32 MB).

[steve-m]

Made my own plot map, and it's getting interesting:

 

SA-nodes-plot.png (1.22 MB)

 

I tried to visualize the value I called 'node type'. Red for car nodes (1), Blue for boat nodes (2), and yellow for other nodes from section 1a. All ped nodes from section 1b are green. For green and yellow nodes, the type is printed next do the dot. As you can see, yellow tracks are used either for special vehicles, or mission specific routes (like races), and are identified by an ID > 2. I assume cars can be attached to paths with a certain ID, so in races they don't leave their circuit and continue driving on normal roads. Same for ped paths, they have their own "circuits"; each city, village and separate path have their own IDs, not sure what that's used for.

 

Edit: The value before type seems to be the path width, multiplied by 8 (or something), with 0 being default.

 

Edit 2: The value after type contains 2 things. The lower 4 bits are the number of nodes it is connected to (aka link count), the higher 4 bits are unknown. Quite high for boat paths, probably defines what vehicle types are allowed (as flags).

 

Edit 3: Second value after type - unknown flags. Third value after type - more flags, 15 default, 47 for parking, 1 used quite often north of vegas, 7 used in vegas as well.

Last value is always zero, so I assume the last 4 byte can be seen as DWORD, full of flags.

 

Edit 4: Most flags in section 1 discovered, first post updated.

[steve-m]

Section 2: Still no idea, very confusing. Seems to be extended information for links of vehicle nodes, such as traffic light stops, lanes and directions.

 

Section 5: lower 10 bits - index into section 2 items; upper 6 bits - area ID; only used for vehicle nodes

 

Section 6: Link lengths (distance between two linked nodes) in full units. Example (double size):

 

 

We definitely need a new path editor, this is way too complex for MaxScript. And just converting the old ipl paths won't work either, since they don't have all the information these files have. I already have lots of ideas, but I think I lack the time and motivation to do it. But technically I think it'd be easier than an editor for ipl paths, due to the used data structures.

 

(and yes, I enjoy talking to myself)

[Jevon]

(and yes, I enjoy talking to myself)

And there was us thinking you were going crazy!

[JernejL]

We definitely need a new path editor, this is way too complex for MaxScript. And just converting the old ipl paths won't work either, since they don't have all the information these files have. I already have lots of ideas, but I think I lack the time and motivation to do it. But technically I think it'd be easier than an editor for ipl paths, due to the used data structures (undirected graphs).

i'm too lame for anything map related right now. try and make a editor and a converter ..

 

 

[Gforce]

 

We definitely need a new path editor, this is way too complex for MaxScript.

try and make a editor and a converter ..

i second that request.

 

you're not talking to yourself steve, it's just that some of us are not clever enough to add anything constructive to your finding's, so i guess all i can really say is .... well done so far and keep going

[steve-m]

 

We definitely need a new path editor, this is way too complex for MaxScript.

try and make a editor and a converter ..

i second that request.

But I don't want to...

 

Edit: bumped and moved

[mickarrow]

I don't know if this is helpfull, but the guys that are working on the BLUE platform were making a race editor. Wouldn't they need access to the path nodes to do this? Maybe they can help.

[steve-m]

I thought their in-game editor was only for placing objects and to define race track checkpoints for human competitors, no?

 

I was thinking, maybe it would be easier to explore the node data when displayed in a proper graph. I could write a small tool that prints everything into a text .gdl file, which is rendered by software such as aiSee, for example. Will look into that next week, when I hopefully have some more time.

[aad]

So its true that the game doesnt uses the paths folder in the data map but you cant remove it eather because the game crashes but the weird thing is you can just clean out every file to 0KB in size even the carrec.img dont know where that is for but what i was saying its still searching for the map if it isnt there the game wont load and will just crash so it has to do something. Also in the gta3.img are a lot of ipl files that are called stream1 stream2 etc although those are not the same as normal ipl files that can be edited with notepad or something similair so i hope someone can find out how it all works the flags are known but the making of is not

 

 

[JernejL]

Steve, so is this the only way to edit paths?

 

[steve-m]

@aad: It uses the paths folder, just not the nodes*.dat files there (they are just a copy of those in the gta3.img). The files in carrec.img are special paths that are used within missions for car chases and such; tracks*.dat and train*.dat are used for the trains. Dunno about roadblox.dat, but I think it isn't used since such info is included in the node files as well.

The IPLs in gta3.img are in binary format - that's offtopic and explained everywhere, just search.

 

@Delfi: Yes. Unless R* hid their path compiler somewhere.

[NathanHenry]

So does this mean that the tool at the top can edit the San Andreas paths nodes because it would really help the Liberty City in San Andreas engine that I am on with xrk and AdTec_224 and dastardly.

[grovespaz]

I don"t know if this helps you or not, but:

I tryed to delete the nodes from the img, and run, it didn"t work(duhuh)

Then i removed teh dont build paths line from the streams.ini

didnt work

then i added the line buildpaths, didnt work, then i added the line dobuildpaths, didnt work.

So i decided to try sumthing else that might help you guys in developing a paths editor. I created a new IMG in the paths directory called paths.img(i dont really think the name matters) and added it to the gta.dat. In the img i put the nodes0-63. It worked.

 

This might help you if you are developing a editor so it you dont have to rebuild the gta3.img over and over again.

[steve-m]

Yea, I already checked that stream.ini stuff back then. Didn't try moving to another img though, apparently we can put absolutely everything in external imgs now, and double entries are loaded from the file mentioned first.

 

I didn't work on this anymore, did someone else?

[commando]

So i guess for building new paths in SA the old path editor code for VC wouldnt be much useful and a new program must be builded to create the node paths and max script could be used if coded properly.

 

Am i missing something?

[JGuntherS@NL]

I made empty nodes*.dat and the game doesn't even crash!

I put them in a new img and deleted the old ones in gta3.img and I tested them in two environments, normal SA and GTI, and they both work.

 

Here they are: Download (contains a nodes__.dat and an img)

[steve-m]

I've tried exactly those before, and using yours the game still crashes for me.

[JGuntherS@NL]

 

I've tried exactly those before, and using yours the game still crashes for me.

Strange... I wil try to make a simple path on my SA (peds walking in a square or something).

 

EDIT: Could section 4 mark the end of section 3 and the start of section 5? Sounds a bit strange, but why should that FF FF 00 00 be put in the nodes**.dat and not just in the executable?

 

EDIT2: (Steve, could you give me some more information about the links, it seems to me with your explanation that a link is done between node ID and area ID... A link must be between two nodes, isn't it? Or must it be this way to link node 00 to 01 (in area 36):

24 00 00 00 24 00 01 00) Sorry, I didn't read it well

 

EDIT3: My SA seems to be extremely tolerant. I've corrupted my section 6 (just deleted half of the entries), and the game still works. The only way to crash it is to delete a file or to write nonsense in a file. (That file was corrupted anyway, I made it myself, so I don't know what would happen if I corrupted an original)

 

EDIT4: Disabling "dontbuildpaths" does absoluty nothing. I've disabled it while having empty paths, and nothing happened.

 

EDIT5: (I made a simple nodes36.dat, but it doesn't work, can anyone help?) It was wrong anyway, I'll try to make a new one.

 

EDIT6: I finally got my game to crash... When I approach the path it crashes

 

EDIT7: (What is the link ID used for? If you have a node using link ID 2 and the second link is area ID 36 node ID 3, is the node then linked to node 3 (in area 36)? If it is that way, I understand it) Sorry I didn't read well

 

EDIT8: In section 2, the fifth argument is the area ID. The last two argument are (nearly) always zero. The tenth argument is for a whole bunch of lines the same. I can't find any coordinate-like data in it, so one of the arguments must be a node ID or link ID. The ninth argument is (nearly) always zero) (For information, I'm analyzing nodes36.dat, because it's from (0.0, 0.0) to (750.0, 750.0) and I like positive coordinates)

 

EDIT9: Are you sure the section 6 data is in full units? It seems to me a bit high (for example: 54, 31, 31, the first sec. 6 entries in nodes36). I think it's in 1/8's of the size, I'll check it later.

 

EDIT10: I think the link ID refers to the first link of that node. For example, in nodes36 (again ) node 0 has link ID 0, but no node has link ID 1. These are the first few entries:

 

 

#0000:     44   106#0001:     36     1#0002:     36     0#0003:     36     3#0004:     36    38#0005:     36     3#0006:     36     1

 

 

As you can see, node 0 links to an external node AND to an internal node, namely 1. And node 1 is linking back in line 0002, so that's another proof for Steve-m's double-link-theory.

 

 

#0213:     44   245#0214:     44   109#0215:     36     0#0216:     44    16#0217:     44    78

 

 

In line 0215, the external node (area 44, node 106) links back to area 36, node 0. (I see you already discovered this)

 

EDIT11: My decompiler reads all the stuff in structure-arrays, and it's pretty fast (about 2 sec. for a file). Check this topic on gtaforum.nl: SA Path Editor. Progress going faster than I thought, a compile function is already available (no edit function!).

 

(Maybe this post can go into the Guinness Book of Records for the most edited post )

Edited December 4, 2005 by JGuntherS@NL

[steve-m]

(Maybe this post can go into the Guinness Book of Records for the most edited post  )

Hehe, sorry for ignoring you. Maybe you should've bumped the topic a couple of times to get attention.

 

 

EDIT: Could section 4 mark the end of section 3 and the start of section 5? Sounds a bit strange, but why should that FF FF 00 00 be put in the nodes**.dat and not just in the executable?

Dunno, I think it's just a placeholder for data that's put there at runtime.

 

 

EDIT9: Are you sure the section 6 data is in full units? It seems to me a bit high (for example: 54, 31, 31, the first sec. 6 entries in nodes36). I think it's in 1/8's of the size, I'll check it later.

Yes, I'm quite sure. I checked it on the map I plotted, with 1 pixel = 1 unit, and there it was true. And IIRC I even calculated some examples by hand and compared.

 

 

EDIT10: I think the link ID refers to the first link of that node. For example, in nodes36 (again  ) node 0 has link ID 0, but no node has link ID 1.

Yes, that's what it does. Just the ID of the first one is given, but the number of links tells you how many of them belong to that node.

[NathanHenry]

So is this any closer to having a SA Path Editor Program.

[JGuntherS@NL]

So is this any closer to having a SA Path Editor Program.

I'm writing one right now.Check this topic on GTAForum.nl: SA Path Editor

[NathanHenry]

So there is gonna be a SA Path Editor soon really that is cool.

 

[JGuntherS@NL]

Progress on my editor is going pretty fast. I decompiled and recompiled every file, and the game still works, so that means that my (de)compile-subroutines are good. (Unusable fact: my editor is multithreading )

[JGuntherS@NL]

Woohoo

 

My path compiler can now edit section 1a entries (finally...)

 

Topic on GTAForum.nl: SA Path Editor

[NathanHenry]

So now paths are editable???

 

[BenMillard]

Please run an English thread about your paths editor in this area of GTAF! You could even develop it in this topic.

 

When you are able to reliably construct paths for GTASA using your program, you could contact TonyWob. He is making a GTA3/VC/SA Map Editor (MEd) and a 3D paths builder would be a great addition to it, once the core is made stable.