General OpenIV Discussion and Announcements

[reditec]

The changes to files should be tracked and manual resolution required if there's a conflict when installing or uninstalling.

 

If package management is not a priority for the OpenIV team, I'd like to appeal for them to share the info on how to decrypt the archives, so that it could be implemented by someone else. Currently managing mods is just painful.

Looks like you don't know what you're talking about.

The changes could be tracked.

But it would consume a lot of disk space to store all the backups.

If you read my previous posts, you would understand that they won't share any information about the game data (though decryption is porbably even possible to be reverse engineered by someone else) But that doesn't have an influence on the mod management...

[fastman92]

If you read my previous posts, you would understand that they won't share any information about the game data (though decryption is porbably even possible to be reverse engineered by someone else) But that doesn't have an influence on the mod management...

 

I might have give a try at reverse engineering it.

[nelabs]

The changes could be tracked.

But it would consume a lot of disk space to store all the backups.

 

Using the mods folder means that the original game files are already backed up. The package manager would track the location and checksum of the files it adds or replaces, and also know about the original files, and would stop any operation that would require changing files that it doesn't know about (manually edited, for example) or that belong to a different package, so there's no conflicts or breakage.

 

One additional function a package manager should have is dependency management, so that dependencies would be installed as separate packages instead of being bundled, and packages with the same dependencies wouldn't conflict.

 

Unfortunately, patching the XML files in OpenIV is done with string matching, which is prone to breakage and should really be done with something that uses selectors (like XPath, for example), so this would have to be worked around if basing the packages on the OpenIV format.

 

I might have give a try at reverse engineering it.

 

It would be really fantastic. Here's a thread with some info from the last time I asked about this a month ago.

Edited January 30, 2016 by nelabs

[GooD-NTS]

Is there any timeline for when OpenIV packages will support uninstallation?

Unfortunately, no.

 

Using the mods folder means that the original game files are already backed up.

The "mods" folder is optional thing, so this can't be used for users without "mods" folder.

 

Unfortunately, patching the XML files in OpenIV is done with string matching, which is prone to breakage and should really be done with something that uses selectors (like XPath, for example), so this would have to be worked around if basing the packages on the OpenIV format.

The string matching was made for text files. As for XML I have the similar ideas with XPath selection. I made few test on this week and it's looks promising.

For example, in handling.meta this request can be used to select node for BUFFALO handling:

 

/CHandlingDataMgr/HandlingData/Item[@type=CHandlingData]/handlingName[.=BUFFALO]/..

Still, there is some things to think about. But, I hope to provide command for XML editing in Packages soon. In future it also might be possible to use the same commands to edit some .ymt files.

If you have any ideas or suggestions how this commands for XML might look like, please share it.

 

Also, you need to know we working/thinking on packages and mood management for a long time, so we know it need dependencies and stuff.

[nelabs]

 

The "mods" folder is optional thing, so this can't be used for users without "mods" folder.

 

 

 

Limiting uninstallation to mods in the mods folder should be fine. Changing the original files is just overall bad because it can't be easily reversed.

 

 

It's nice that you're working on packages, but I wish such an important feature like being able to properly install/remove mods would have taken priority. I mean, the whole point of mods is for people to use them. Right now I don't even want to bother with mods because I can't easily try them out and remove them after without doing it one by one and making manual backups.

 

[reditec]

because it can't be easily reversed.

It can not be reversed actually. Imagine two mods installed to the mods folder.

Both overwriting the same file. That can not be prevented and it would destroy the previous mod.

You simply cannot install or uninstall mods. Deal with it.

The only thing is, if you would create custom DLCs in the update.rpf.

Then you could just remove a folder and it would become way more simpler to handle.

Probably this will be the final solution.

And direct edits will always be supported... Think about the space the mods folder consumes when you change large amounts of files in different archives.

Also this feature is not even close as important as the open formats for drawables...

[nelabs]

A proper mod manager helps with mod conflicts by detecting them and preventing the game from getting into a broken state, so resolving conflicts by not installing the conflicting packages is part of the whole reason to have automation. Fixing the mods to not conflict would be way outside the scope of a mod manager.

Uninstallation would only work for users who have the space for backups, and that's fine, same as it's fine that using the "mods" folder also requires more space. In fact, a manager could be made to back up only the changed files, not the whole archives, reducing the space requirements compared to the "mods" folder, although never changing the original files should be preferable.

What are the limitations for creating DLCs? Can it replace existing vehicles, for instance?

Edited January 31, 2016 by nelabs

[GooD-NTS]

Also this feature is not even close as important as the open formats for drawables...

And why exactly you are so sure about this?

 

 

What are the limitations for creating DLCs? Can it replace existing vehicles, for instance?

Yes, Rockstar did it many times. Look at patchday DLCs.

[reditec]

And why exactly you are so sure about this?

Because you guys are now trying to put the "features based on researches"' priority over the priority of researching itself.

You try to build a PC, but you only have a GPU and CPU, but forgot the motherboard...

Mod management is nothing bad, but creating a good management system would require tons of hours spent on this instead of researching the file formats and implementing them.

Also you already created the .oiv format and that format won't work well with the uninstall / install system, because of the structure of the format. Either that, or you'd have to create a new package installer / change the existing one again (it it just got released)...

Edited January 31, 2016 by reditec

[GooD-NTS]

Because you guys are now trying to put the "features based on researches"' priority over the priority of researching itself.

You literally have no idea what we are doing or what priorities we have.

 

 

Also you already created the .oiv format and that format won't work well with the uninstall / install system, because of the structure of the format. Either that, or you'd have to create a new package installer / change the existing one again (it it just got released)...

The OIV format will be changed, because that's how things work. It will evolve for changing requirements. It doesn't matter when it was released.

[nelabs]

It's actually better to make breaking changes early when there's been less adoption, since less things need to be fixed. Adding features like XML patching would be more of an extension of the format anyway.

 

There's a proposed standard for patching XML based on XPath: RFC 5261. Using the standard, the relevant patching operations would be defined something like this:

<xml path="common\data\dlclist.xml">	<add sel="/SMandatoryPacksData/Paths">		<item>dlcpacks:\foo\</item>	</add></xml><xml path="common\data\levels\gta5\vehicles.meta">	<replace sel="/CVehicleModelInfo__InitDataList/InitDatas/PovCameraOffset[@x=0.000000][@y=:0.190000][@ z=0.630000]">		<PovCameraOffset x="0.000000" y="-0.190000" z="0.570000" />	</replace></xml>

Selectors returning any non-element nodes (like text nodes or comments) should result in an error; same with add or replace selectors returning multiple nodes. Removing multiple nodes should be okay.

 

Reversibility for these operations could be implemented by adding attributes with the package's identifier to any changed elements and renaming the removed elements. Using the above example, the changed elements could look like this:

<item oiv:package="{package guid}">dlcpacks:\foo\</item><oiv:removed oiv:package="{package guid}" oiv:elemName="PovCameraOffset" x="0.000000" y="-0.190000" z="0.570000" />

Reversing any changes made by a specific package then becomes trivial by just renaming oiv:removed back and removing any other elements with the matching oiv:package attribute. Same with avoiding conflicts; if a selector returns an element that already has an oiv:package attribute, stop the installation with an error.

Edited January 31, 2016 by nelabs

[nelabs]

Dependencies could be defined like this (using SemVer for version numbers):

<dependencies>
  <dependency guid="{foo}" version="1.*" />
  <dependency guid="{bar}" version="0.1-beta" />
</dependencies>

The most basic functionality would be just to block the installation if the dependencies aren't met. It would make sense to store the list of installed dependencies in the "mods" folder, for example, to make it easy to remove everything. The format could be just some data serialization format like JSON to keep it simple. The same file could also store the list of installed files which would be used to check for conflicts on install and for uninstallation.

A more advanced feature would be to automatically try to fetch unmet dependencies using some centralized source, but that's a lot of effort to make and maintain, so maybe it would be best to leave to a complementary project, like how apt-get is based on dpkg or yum on rpm. I'm considering making a tool for fetching and repackaging mods in the .oiv format based on a repo of 'recipes', so it could be a step towards that.

Edited January 31, 2016 by nelabs

[reditec]

The OIV format will be changed, because that's how things work. It will evolve for changing requirements. It doesn't matter when it was released.

It will evolve. But if you change the entire structure of the format with every update of OpenIV, people could start using the .oac files for distributing their mods again... And that's just what I see happening right now...

 

I recommend you to share more news with us if users reading everything you post here don't have an idea what's going on

OpenIV never was a mod manager, but a mod creator. No doubt it can be both, but if I buy a game (=mod manager) and don't have a computer (=researches to make modding possible), I can't play it. No matter how good the game (= mod manager) is. So I strongly recommend you to put 90% of the available resources into researching until that stuff is done.

 

Why would someone put time into a format that doesn't even come close to the state the author is aiming for and advice people to use it, if it gets changed with every update anyway and makes old versions unusable...?

[GooD-NTS]

There's a proposed standard for patching XML based on XPath: RFC 5261. Using the standard, the relevant patching operations would be defined something like this

Thanks for the link and examples.

 

 

Reversibility for these operations could be implemented by adding attributes with the package's identifier to any changed elements and renaming the removed elements. Using the above example, the changed elements could look like this:

I don't think keep those things in game files is a good idea, we can store changes/diffs/etc in a specific place.

 

 

I recommend you to share more news with us if users reading everything you post here don't have an idea what's going on

I can't share things because they are not in right state. When people see things they start to think it will come really soon. We have a lot of ideas, concepts, prototypes, and some of they really awesome, but they are not ready for public yet. Of course, I can tell about for example "Mod store", "Asset viewer" or "Map tools" for GTA V, but they are not coming anytime soon.

Same with features planned for exact next release, even when we have complete scope as long as feature is not finished I can't share info about it.

 

 

Why would someone put time into a format that doesn't even come close to the state the author is aiming for and advice people to use it, if it gets changed with every update anyway and makes old versions unusable...?

Who said it will be unusable? You still can install OIV packages in 1.0 & 1.1 format which was created a few years ago.

Same here, we will have 2.1 format with XML commands for example, but people will be still able to install 2.0 like they can today.

Edited February 1, 2016 by GooD-NTS

[Mr.Arrow]

How difficult is it to edit the particles effects? (WPFL/YPT)

 

GTA 4 doesn't have openformats for WPFL for like 7 years and still no clue how to edit them.It makes a sh*tty looking like SA look good especially on explosion effects...

 

 

 

I hope someone find a way to edit them ;_;

Edited February 1, 2016 by Cyberzone2

[GooD-NTS]

How difficult is it to edit the particles effects? (WPFL/YPT)

Technically it the same like all other resources like textures or models.

But the structure of object inside WPFL is quite complicated.

 

 

GTA 4 doesn't have openformats for WPFL for like 7 years and still no clue how to edit them.

Actually we never had plan to create openFormats for WPFL (GTA IV).

But, for GTA V (YPT) we are going to create openFormats, and we already have some research of YPT files.

[reditec]

Who said it will be unusable?

Okay, if you upgrade the package version and still support older versions, it's fine. But an eisting file format should not be changed without changing the version of that format

It's fine for me like this.

[UltraGizmo64]

Is it possible to export embedded textures (.dds) into different formats such as PNG?

[GooD-NTS]

@@UltraGizmo64 yes.

[UltraGizmo64]

@GooD-NTS

 

How do you do it?

It seems like the application automatically exports embedded textures as. dds files with no other options.

Edited February 2, 2016 by UltraGizmo64

[nelabs]

There should probably be a more helpful error message for syntax errors in assembly.xml; right now it just says that file is "probably corrupted".

 

Edit: My test package now says: "The installation failed. Something happened." This is just unusable without proper error messages. You should at least have an error log.

 

Also, a GUID with lowercase letters produces this error: "Unable to read package Id". The same GUID with uppercase letters works. The tools linked in the docs all generate the GUIDs in lowercase.

Edited February 3, 2016 by nelabs

[GooD-NTS]

How do you do it?

It seems like the application automatically exports embedded textures as. dds files with no other options.

First of all, you can change the default texture format in OpenIV options:

 

Also, you can open model file in viewer, click on "More Information" link:

Navigate to "Textures" tab, and click on "Export all embedded textures" button:

And select the format you want:

 

 

 

There should probably be a more helpful error message for syntax errors in assembly.xml; right now it just says that file is "probably corrupted".

 

Edit: My test package now says: "The installation failed. Something happened." This is just unusable without proper error messages. You should at least have an error log.

Agreed, but unfortunately I not had time to made it in OpenIV 2.7 release.

I hope I will be able to make more detailed errors in next release.

 

 

Also, a GUID with lowercase letters produces this error: "Unable to read package Id". The same GUID with uppercase letters works. The tools linked in the docs all generate the GUIDs in lowercase.

Okay, this is odd. I will check it, thanks for the report. Edited November 24, 2016 by GooD-NTS

[nelabs]

Agreed, but unfortunately I not had time to made it in OpenIV 2.7 release.

I hope I will be able to make more detailed errors in next release.

Would be nice to get updated builds between the official releases. I've started making tooling to port mods to .oiv and batch install them, but not being able to properly debug problems complicates it too much. An error log should be trivial to implement, so maybe you could make a build with at least that.

[reditec]

Putting the automatation now on the feature list of openiv package creator.

[nkjellman]

When will ymap editing become a reality? There are several Map Editor mods that I am interested in converting over.

[dr0yed]

When will ymap editing become a reality? There are several Map Editor mods that I am interested in converting over.

Road to Heaven V4!!!!!

[mati36]

You will free openformat to core.ypt?

[ThePrince]

When i try to view the Main.sco, OpenIV Crashes

 

Time: "12:22:03"
Type: "EArgumentOutOfRangeException"
Message: "Argument out of range"

Additional information:
[Application Context]
GetThreadLocale=1036 (0x040C)
Game::Path=D:\Grand Theft Auto IV\
Core::OpenRageArchive::LastCall=D:\Grand Theft Auto IV\common\data\cdimages\script.img
GetSystemDefaultLangID=1036
App::Lang=en_GB
App::Path=C:\Users\PC\AppData\Local\New Technology Studio\Apps\OpenIV\
Main::ExploreArchiveNode::WorkItem=D:\Grand Theft Auto IV\common\data\cdimages\script.img|
Main::ExploreArchiveNode::ParentArchive=D:\Grand Theft Auto IV\common\data\cdimages\script.img
App::Temp=C:\Users\PC\AppData\Local\Temp\OpenIV_007EAA26\
ApplicationPath=C:\Users\PC\AppData\Local\New Technology Studio\Apps\OpenIV\
OS=Windows 7 Service Pack 1 (Version 6.1, Build 7601, 64-bit Edition)
GetSystemDefaultLCID=1036
Game::ID=IV (GTA IV)
Game::Platform=pc
SysLocale={ DefaultLCID: "1036", PriLangID: "12", SubLangID: "1", FarEast: "True", MiddleEast: "True" }

[Application Windows]
TRageScriptWindow=SCO Viewer - main.sco
TMainWindow=OpenIV - script.img - [Edit mode]
TActionsModule=ActionsModule
TErrorWindow=OpenIV - Application error

Release: 2.7.0.680 04/01/2016
Address: "0x0084D654"
Procedure: "Rage.Script.RASM.Opcodes.Classes.{System.Generics.Collections}TList<Rage.Script.RASM.Opcodes.Classes.TRageScriptASMBaseOpcode>.GetItem"
Unit: "System.Generics.Defaults.pas", Line: "38"

Stack:
[0084D64F] Rage.Script.RASM.Opcodes.Classes.{System.Generics.Collections}TList<Rage.Script.RASM.Opcodes.Classes.TRageScriptASMBaseOpcode>.GetItem (Line 38, "System.Generics.Defaults.pas" + 0) + $1F
[0084AE5F] Rage.Script.RASM.Opcodes.Classes.TRageScriptASMContent.GetItem (Line 761, "Rage.Script.RASM.Opcodes.Classes.pas" + 0) + $3
[0084AE8A] Rage.Script.RASM.Opcodes.Classes.TRageScriptASMContent.GetOpcode (Line 771, "Rage.Script.RASM.Opcodes.Classes.pas" + 1) + $4
[0085E736] Rage.Script.ControlFlow.Block.TRageScriptAsmFlowBlock.ExtractWhiles (Line 174, "Rage.Script.ControlFlow.Block.pas" + 6) + $8
[00407D5E] [email protected] + $E
[0085EACC] Rage.Script.ControlFlow.Block.TRageScriptAsmFlowBlock.ExtractFlow (Line 268, "Rage.Script.ControlFlow.Block.pas" + 2) + $2
[00860D05] Rage.Script.ControlFlow.TRageScriptControlFlowAnalyzer.GenerateCFAObjects (Line 129, "Rage.Script.ControlFlow.pas" + 13) + $A
[0083FF69] Rage.Script.Threads.TSCOThread.Execute (Line 60, "Rage.Script.Threads.pas" + 6) + $3
[004C4EA2] System.Classes.ThreadProc + $42
[0040970C] System.ThreadWrapper + $28
[75CF3378] BaseThreadInitThunk + $10
[77D29880] Unknown function at RtlInitializeExceptionChain + $61
[77D29850] Unknown function at RtlInitializeExceptionChain + $31

 

[UltraGizmo64]

Will it ever be possible to support the html pages from GTA IV (the .WHM files)?

Edited February 10, 2016 by UltraGizmo64

[GooD-NTS]

Would be nice to get updated builds between the official releases. I've started making tooling to port mods to .oiv and batch install them, but not being able to properly debug problems complicates it too much. An error log should be trivial to implement, so maybe you could make a build with at least that.

I can't promise intermediate builds.

 

 

 

When will ymap editing become a reality? There are several Map Editor mods that I am interested in converting over.

The ymap/ytyp editing coming after ODR/ODD formats.

 

 

 

You will free openformat to core.ypt?

What?

 

 

 

When i try to view the Main.sco, OpenIV Crashes

Do you have original Main.sco or it is custom?

OpenIV is unable to decompile user created sco.

 

 

 

Will it ever be possible to support the html pages from GTA IV (the .WHM files)?

No. It was announced long time ago, no new features for GTA IV is planned.