[REL] Unity 3D + .DFF format

[BlenMiner]

This was designed to work with San Andreas GTA.

 

Title says it all. With this package you can import dff files right into unity with uv coordinates and multi material support (just add more materials and they will behave naturally).

 

How to use u ask? Well just import the package to your project and drag and drop any dff file into assets folder.

 

The source code isn't compiled.. it is a mess since i copied it from my other project which i am importing the map files with the textures..

 

http://adf.ly/1BBlnF- Mediafire link, no password no virus

http://www.mediafire.com/download/213ktw74us5f8rf/DFFImporter.unitypackage

Edited March 22, 2015 by BlenMiner

[TJGM]

Thread moved to the correct section and link removed, we only allow adfly links if there is also a direct link too.

[The_GTA]

I have taken a look at your source code.

 

Looks like a quick loader of DFF data. It expects a fixed version of the DFF RenderWare stream and skips many fields. There is no debug checking

whether the serialization makes any sense, so there are many opportunities where your code may crash.

 

Looks okay for a first version of a DFF loader.

Edited March 22, 2015 by The_GTA

[BlenMiner]

I have taken a look at your source code.

 

Looks like a quick loader of DFF data. It expects a fixed version of the DFF RenderWare stream and skips many fields. There is no debug checking

whether the serialization makes any sense, so there are many opportunities where your code may crash.

 

Looks okay for a first version of a DFF loader.

 

This was my first attempt to work with binary files at all xD

That was my first attempt

[The_GTA]

This was my first attempt to work with binary files at all xD

That was my first attempt

Cool. I recommend that you carefully look at the specification and add in debug output of some sort, so that people can easily tell you which DFFs fail to load and for what reason.

 

I personally do not like adfly links, so I am being skeptical. But I am willing to digress.

[BlenMiner]

 

This was my first attempt to work with binary files at all xD

That was my first attempt

Cool. I recommend that you carefully look at the specification and add in debug output of some sort, so that people can easily tell you which DFFs fail to load and for what reason.

 

I personally do not like adfly links, so I am being skeptical. But I am willing to digress.

 

 

Hey can I ask you something? A problem I am facing while trying to import the map to unity is that the ipl files holdes the low res lod names ... how can i get the high res model? Is the lod trees stored somewhere?

[The_GTA]

Hey can I ask you something? A problem I am facing while trying to import the map to unity is that the ipl files holdes the low res lod names ... how can i get the high res model? Is the lod trees stored somewhere?

Sure.

 

.ipl files can have an inst section. In GTA:SA this section holds CSV entries with 11 items. The 11th item is the LOD instance offset.

 

If lod instance offset is smaller than zero, then the instance has no LOD. Otherwise you get the LOD instance by indexing the instances of the parsed IPL file by the lod instance offset.

To build a LOD list simply add every instance that is elected to be a LOD into a list, for every IPL inst section.

 

You should build the LOD list after you have parsed the inst section, because LOD instances can come before or after the current instance.

 

Every instance that is not inside of our LOD list is a high quality instance.

 

 

 

 

fileRoot.scanDirEx("/maps/", "*.ipl", nil,    function(path)        local curOffset = instId;        local local_instances = {};        local file = resRoot.open(path, "rb");        local instanceMap = {};            parseCSVSections(file.read(file.size()),             {                inst = function(entries)                    local id = tonumber(entries[1]);                    local name = entries[2];                    local model = models[id];                                    if (model) then                        local instance = {                            model = id,                            name = name,                            interior = tonumber(entries[3]),                            posX = tonumber(entries[4]),                            posY = tonumber(entries[5]),                            posZ = tonumber(entries[6]),                            quatX = tonumber(entries[7]),                            quatY = tonumber(entries[8]),                            quatZ = tonumber(entries[9]),                            quatW = tonumber(entries[10]),                                                        instId = instId                        };                                                local lodID = tonumber(entries[11]);                                                if (lodID) and (lodID > -1) then                            instance.lodInstance = lodID + curOffset;                        end                                                instanceMap[instId] = instance;                        table.insert(instances, instance);                        table.insert(local_instances, instance);                    else                        map.printout("inst " .. name .. " at " .. fileRoot.relPath(path) .. " has invalid m_id " .. id);                    end                                        instId = instId + 1;                end            }        );                file.destroy();                -- Link all LOD instances        for m,n in ipairs(local_instances) do            if (n.lodInstance) then                local item = instanceMap[n.lodInstance];                                if (item) and not (item == n) then                    n.lodLink = item;                    models[item.model].isLOD = true;                end            end        end    end);

 

 

EDIT: sorry, always get confused with indices and stuff

Edited March 28, 2015 by The_GTA

[BlenMiner]

 

Hey can I ask you something? A problem I am facing while trying to import the map to unity is that the ipl files holdes the low res lod names ... how can i get the high res model? Is the lod trees stored somewhere?

Sure.

 

.ipl files can have an inst section. In GTA:SA this section holds CSV entries with 12 items. The 12th item is the LOD instance offset.

 

If lod instance offset is smaller than zero, then the instance has no LOD. Otherwise you get the LOD instance by indexing the instances of the parsed IPL file by the lod instance offset.

To build a LOD list simply add every instance that is elected to be a LOD into a list, for every IPL inst section.

 

You should build the LOD list after you have parsed the inst section, because LOD instances can come before or after the current instance.

 

Every instance that is not inside of our LOD list is a high quality instance.

 

 

 

fileRoot.scanDirEx("/maps/", "*.ipl", nil,    function(path)        local curOffset = instId;        local local_instances = {};        local file = resRoot.open(path, "rb");        local instanceMap = {};            parseCSVSections(file.read(file.size()),             {                inst = function(entries)                    local id = tonumber(entries[1]);                    local name = entries[2];                    local model = models[id];                                    if (model) then                        local instance = {                            model = id,                            name = name,                            interior = tonumber(entries[3]),                            posX = tonumber(entries[4]),                            posY = tonumber(entries[5]),                            posZ = tonumber(entries[6]),                            quatX = tonumber(entries[7]),                            quatY = tonumber(entries[8]),                            quatZ = tonumber(entries[9]),                            quatW = tonumber(entries[10]),                                                        instId = instId                        };                                                local lodID = tonumber(entries[11]);                                                if (lodID) and (lodID > -1) then                            instance.lodInstance = lodID + curOffset;                        end                                                instanceMap[instId] = instance;                        table.insert(instances, instance);                        table.insert(local_instances, instance);                    else                        map.printout("inst " .. name .. " at " .. fileRoot.relPath(path) .. " has invalid m_id " .. id);                    end                                        instId = instId + 1;                end            }        );                file.destroy();                -- Link all LOD instances        for m,n in ipairs(local_instances) do            if (n.lodInstance) then                local item = instanceMap[n.lodInstance];                                if (item) and not (item == n) then                    n.lodLink = item;                    models[item.model].isLOD = true;                end            end        end    end);

 

 

 

Hmm alright, but for instance we have this line in countn2.ipl:

 

16687, lod_rnway_bit, 0, 329.8984375, 2504.265625, 15.5703125, 0, 0, 0.002497908659, 0.9999969006, -1

 

Its last value which is the lod as you suggested is < 0 . That should mean it has no lods but theres a cn2_rnway_bit.dff which is the same model but high res xD if that makes any sense.

[The_GTA]

Hmm alright, but for instance we have this line in countn2.ipl:

 

16687, lod_rnway_bit, 0, 329.8984375, 2504.265625, 15.5703125, 0, 0, 0.002497908659, 0.9999969006, -1

 

Its last value which is the lod as you suggested is < 0 . That should mean it has no lods but theres a cn2_rnway_bit.dff which is the same model but high res xD if that makes any sense.

I think we have to define stand alone LOD instances. For this you need to know special behaviour of the GTA:SA engine.

 

The maximum high quality draw distance in GTA:SA is (I think) 300 units. Everything above that would fail to stream because of implementation limitations.

So to get a list of all stand-alone LOD enabled models you compare the LOD distance of each model in the .IDE files and check whether it is higher or equal to 300.

 

Example...

 

16684, cn2_rnway_bit, des_ne, 299, 016685, cn2_rnway_bit2, des_ne, 299, 016686, lod_rnway_bit2, lod_countn2, 1500, 12816687, lod_rnway_bit, lod_countn2, 1500, 128

We know for sure that cn2_rnway_bit and cn2_rnway_bit are high quality models, just by their name.

Their LOD counterparts are obvious too, because of their name and the LOD distances.

 

Since I have not yet reverse engineered this part of the engine I take a wild guess.

Because only the LOD instance is listed in the IPL file the engine creates a high quality instance itself by LOD distance and name deduction.

 

So here is a C++ function to do the check...

 

 

static bool parseModelName( const char *modelName, std::string& preToken, std::string& nameString ){    bool isPreToken = true;        while( true )    {        char c = *modelName++;                if ( c == '\0' )            break;                bool ignoreCharacter = false;                if ( isPreToken )        {            if ( c == '_' )            {                isPreToken = false;                                ignoreCharacter = true;            }        }                if ( ignoreCharacter == false )        {            if ( isPreToken )            {                preToken += c;            }            else            {                nameString += c;            }        }    }        if ( preToken.empty() )        return false;        if ( nameString.empty() )        return false;        return true;}static bool isLODModelOf( const char *baseModelName, const char *lodModelName ){    std::string basePreToken, baseModelDescriptor;        bool parseBaseModel = parseModelName( baseModelName, basePreToken, baseModelDescriptor );        if ( parseBaseModel == false )        return false;        std::string lodPreToken, lodModelDescriptor;        bool parseLODModel = parseModelName( lodModelName, lodPreToken, lodModelDescriptor );        if ( parseLODModel == false )        return false;        // Check that the LOD even is a LOD.    if ( stricmp( lodPreToken.c_str(), "lod" ) != 0 )        return false;        return ( baseModelDescriptor == lodModelDescriptor );}

EDIT: a friend at Skype was kind to remind me of binary IPL files.

 

If you plan to properly parse the entire SA world you will have to load them aswell. You can find them in the IMG archives.

Each binary IPL file stands for a section of the SA world. It can contain LOD models aswell but you will mostly find high definition instances in there.

 

They are pretty easy to parse. They contain an instances and a parking vehicles section, tho I will just cover the instances section.

 

 

struct iplInstance_t{    CVector                 position;           // 0    CQuat                   quatRotation;       // 12    modelId_t               modelIndex;         // 28    union    {        struct        {            unsigned char   areaIndex;          // 32            unsigned char   flags;              // 33            unsigned char   pad[2];             // 34        };        unsigned int        uiFlagNumber;    };    int                     lodIndex;           // 36, index inside of the .ipl file pointing at the LOD instance.};struct binaryIPLHeader_t{    int             numIPLInstances;        // 0    BYTE            pad[20];                // 4    unsigned int    iplOffset;              // 24};

 

 

/*=========================================================    CreateIPLBuilding    Arguments:        instance - information about this building        instanceName - name of the instance (can be NULL), unused    Purpose:        Creates an IPL game instance from the information        as described by the instance parameter.    Binary offsets:        (1.0 US and 1.0 EU): 0x00538090=========================================================*/struct iplInstance_t{    CVector                 position;           // 0    CQuat                   quatRotation;       // 12    modelId_t               modelIndex;         // 28    union    {        struct        {            unsigned char   areaIndex;          // 32            unsigned char   flags;              // 33            unsigned char   pad[2];             // 34        };        unsigned int        uiFlagNumber;    };    int                     lodIndex;           // 36, index inside of the .ipl file pointing at the LOD instance.};CEntitySAInterface* __cdecl CreateIPLBuilding( iplInstance_t *instance, const char *instanceName ){    CBaseModelInfoSAInterface *modelInfo = ppModelInfo[ instance->modelIndex ];    if ( !modelInfo )        return NULL;    CEntitySAInterface *building = NULL;    bool isVisible = false;    if ( modelInfo->usDynamicIndex != -1 )    {        building = new CDummySAInterface;        building->SetModelIndexNoCreate( instance->modelIndex );        isVisible = !( IsEntityVisibleByDefault( building ) && building->GetModelInfo()->atomicType == 5 );    }    else    {        if ( modelInfo->GetModelType() == (eModelType)5 && modelInfo->bCollisionless )        {            building = new CNoCOLBuildingSAInterface;        }        else        {            building = new CBuildingSAInterface;        }        building->SetModelIndexNoCreate( instance->modelIndex );        if ( modelInfo->bDontWriteZBuffer )        {            building->bDontCastShadowsOn = true;        }        isVisible = ( modelInfo->GetLODDistance() >= 2.0f );    }    if ( !isVisible )    {        building->bIsVisible = false;    }    // The_GTA: removed some buggy rotation hacks.    {        // MTA quat functions are superior!        // No invert required!        building->AllocateMatrix();        CTransformSAInterface*& trans = building->Placeable.matrix;        if ( !trans )        {            building->AcquaintMatrix();            building->Placeable.GetMatrixFromHeading( *trans );        }        CQuat::ToMatrix( instance->quatRotation, *trans );    }    building->Placeable.SetPosition( instance->position );    // Set some initialization flags.    if ( IS_ANY_FLAG( instance->flags, 0x4 ) )    {        building->bUnderwater = true;    }    if ( IS_ANY_FLAG( instance->flags, 0x8 ) )    {        building->bTunnel = true;    }    if ( IS_ANY_FLAG( instance->flags, 0x10 ) )    {        building->bTunnelTransition = true;    }    if ( IS_ANY_FLAG( instance->flags, 0x1 ) )    {        building->bUnimportantStream = true;    }    building->m_areaCode = instance->areaIndex;    building->m_iLodIndex = instance->lodIndex;    (... traincross rotation hacks ...)    CColModelSAInterface *colModel = GetOriginalColModel( building->GetModelIndex() );    // This may be the only time this is actually done in the engine.    if ( colModel )    {        if ( !colModel->m_isCollidable )        {            building->bUsesCollision = false;        }        else if ( colModel->m_colPoolIndex != 0 )        {            CBounds2D outBounds;            const CBounds2D& bounds = building->GetBoundingBox( outBounds );            Streaming::GetCOLEnvironment().m_pool->Get( colModel->m_colPoolIndex )->ExtendBounds( bounds );        }        const CVector& buildingPos = building->Placeable.GetPosition();        if ( buildingPos.fZ + colModel->m_bounds.vecBoundMin.fZ < 0.0f )        {            building->bUnderwater = true;        }    }    return building;}(...)    bool __cdecl LoadContainer( unsigned char iplId, const char *buf, int size )    {        CIPLFileSA *iplFile = Streaming::GetIPLEnvironment().m_pool->Get( iplId );        CEntitySAInterface **lodArray = NULL;        if ( iplFile->m_iplId != -1 )        {            lodArray = GetIPLInstanceArray( iplFile->m_iplId );        }        bool successful = false;        // Check whether the file is in binary mode.        if ( strncmp( buf, "bnry", 4 ) == 0 )        {            const binaryIPLHeader_t& header = *(binaryIPLHeader_t*)( buf + 4 );            short numIPLInstances = (short)header.numIPLInstances;            iplInstance_t *instanceArray = (iplInstance_t*)( buf + header.iplOffset );            for ( unsigned int n = 0; n < (unsigned int)numIPLInstances; n++ )            {                iplInstance_t& instance = instanceArray[n];                m_manager.OnInstance( instance );                CEntitySAInterface *entity = CreateIPLBuilding( &instance, NULL );                RegisterInstance( entity, lodArray, iplFile, iplId );            }            m_manager.OnReadAdditionalSections( buf, header );            successful = true;        }        else        {            (... textual loading code ...)        }        return successful;    }(...)

 

 

In the code above you see the loader implementation the SA engine uses. It has been redacted to cover only the important points.

 

Every binary IPL file starts with the FOUR-CC 'bnry'. If it is not there, then the file is treated as a textual binary IPL which is a subset of the regular IPL files found in the maps folder.

The header of the binary file holds pointers to compiled instance lists. To get a ponter to the instance list you take the pointer to the start of the file and add the value of "binaryIPLHeader_t::iplOffset" to it.

 

 

iplInstance_t *instanceArray = (iplInstance_t*)( (char*)buf + header.iplOffset );

You just loop through it by the "binaryIPLHeader_t::numIPLInstances" value. I believe the lodIndex is used similar to how I described it for the IPL files in the maps folder. Edited March 28, 2015 by The_GTA

[BlenMiner]

Thx ! This is the best that i got so far:

 

 

The white models is just my converter failing on me.. :S

One thing i noticed is that on the area 51 there are some things as the fences and the underground things missing.

[The_GTA]

The white models is just my converter failing on me.. :S

Looks like missing textures. The RenderWare implementation of GTA:SA uses an unique TXD archive system. Basically each model fetches its textures from a chain of TXD archives.

 

I would assume, from looking at your picture, that you only fetch from the TXD archives that are directly associated to a model.

The IDE files contain a TXD parent section that is used to share textures across multiple models. It is used to decrease the required system and video memory of the game.

 

 

One thing i noticed is that on the area 51 there are some things as the fences and the underground things missing.

Try loading the binary IPL files.

[nmzik]

Thx ! This is the best that i got so far:

 

 

The white models is just my converter failing on me.. :S

One thing i noticed is that on the area 51 there are some things as the fences and the underground things missing.

Hello! Is it possible to upload ipl loading script? Thanks!

[_CP_]

Map is mirrored, could you fix it?

[BlenMiner]

Map is mirrored, could you fix it?

 

A simple scale would fix that

 

Thx ! This is the best that i got so far:

 

 

The white models is just my converter failing on me.. :S

One thing i noticed is that on the area 51 there are some things as the fences and the underground things missing.

Hello! Is it possible to upload ipl loading script? Thanks!

 

 

Its very slow at the moment and some of the models dont spawn since i am not loading the binary files yet. Havent done anything else on this script.

[TheDeadlyDutchi]

I know this might be a (very) hard gravedig, I'd still like to contribute to this discussion though. Does anyone have a version of the script that loads the textures? I've modified the script a bit, like this: http://pastebin.com/qE4hWQtg it creates the objects and materials (materials do have the 'textures', but look more like the tga's on a ball). Still can't get the textures on the object, though.

Edited June 16, 2015 by TheDeadlyDutchi

[Swoorup]

You might want to consider using this to save time.

 

https://github.com/alemariusnexus/gtatools

 

 

I used his sources while back to make Dff to Ogre3D format exporter.

https://github.com/Swoorup/prj-reboot/tree/master/vice_convert

[DYOM designer]

if you wonna edit any skin just use 3ds max and download kams scripts it's very easy and awesome

[danipren]

i hate to bring back up an old thread but i have been looking for a unity dff importer for ages and this is the closest thing i can find. I cant get it to work so was just wondering if anyone has a working copy maybe or could help shed a little light on my problem

[Kuxii]

Its the DFF importer Working whit Untity 5?

When yes wee can i Import the Map?

[Muk99]

Hey, for the guys interested, I'm developing a plugin which imports the files from the GTAs of 3D era into Unity, I'll put it in GitHub once it's finished, it currently loads Dff, Txd, Ipl, Ide and Dat files, hopefully it will be useful for you guys, send me a message if you want to know more about it

Take a look on it: http://imgur.com/gallery/2XbFr

[goodidea82]

The 360 screens look very promissing. Can you also port the interiors? And areas that are below water level but are dry?

Edited November 19, 2016 by goodidea82

[Muk99]

Yes, it can import the interiors and objects below water, even cutscenes maps are loaded

Liberty City and Vice City are perfectly fine, but there are still some parts that aren't being loaded in San Andreas you can see in the screenshot the areas near the rivers are missing, I'm working to know why this is happening and try to fix it

[MrGTAmodsgerman]

 

Yes, it can import the interiors and objects below water, even cutscenes maps are loaded

Liberty City and Vice City are perfectly fine, but there are still some parts that aren't being loaded in San Andreas you can see in the screenshot the areas near the rivers are missing, I'm working to know why this is happening and try to fix it

 

Nice dude! What about LODs? And animations?

[Muk99]

LODs are working fine on Gta III and Vice City, on San Andreas they are a little different and aren't perfectly loaded yet

Animations aren't being imported, I'll import them later on, but first I need to read the skin section of dff files and convert it to unity skinned mesh

[Erik_Martins KingMods]

Tutorial how to convert the map of gta san andreas to 3d unity using 3ds max and blender

[The Srilankan Myth Hunte]

I tried to use the map, but Unity doesnt work, i use Unity 2019.4.1

[SaraFoxy15]

So uh. Got it loaded just fine into Unity. Having a kind of problem though, what does it mean by .meta files? I try to import the Burrito van dff into Unity and it says:

A meta data file (.meta) exists but its asset 'Assets/burrito.dff' can't be found. When moving or deleting files outside of Unity, please ensure that the corresponding .meta file is moved or deleted along with it.
UnityEditor.AssetDatabase:Refresh ()

It says this for all dff's, this was the last one I tried