Quantcast

Jump to content

» «
Photo

World To Screen Lag

16 replies to this topic
MickB
  • MickB

    Player Hater

  • Members
  • Joined: 03 Oct 2015
  • Australia

#1

Posted 11 February 2016 - 01:01 AM

Using the _WORLD3D_TO_SCREEN2D native I can successfully draw text over entities in the game world but there is an obvious lag as when moving the camera around the text will move a bit before snapping back to the entity. Obviously I'm running it in a loop every tick but I'm guessing that ScriptHookV threads are not actually called as often as every game frame is rendered which is causing this lag?


InfamousSabre
  • InfamousSabre

    Harpocrates

  • Members
  • Joined: 03 Jan 2013

#2

Posted 11 February 2016 - 07:45 AM Edited by InfamousSabre, 13 February 2016 - 01:46 AM.

That's just how it is. Nothing that you can do about it, unfortunately.

 

Read below :D


CamxxCore
  • CamxxCore

    Mark Chump

  • Members
  • Joined: 19 Oct 2013
  • Canada

#3

Posted 12 February 2016 - 05:57 PM

Using the _WORLD3D_TO_SCREEN2D native I can successfully draw text over entities in the game world but there is an obvious lag as when moving the camera around the text will move a bit before snapping back to the entity. Obviously I'm running it in a loop every tick but I'm guessing that ScriptHookV threads are not actually called as often as every game frame is rendered which is causing this lag?


I'm not sure if it will work with custom textures but you can use natives SET_DRAW_ORIGIN and CLEAR_DRAW_ORIGIN to draw without any stuttering/ lag. I ran into similar issues when making some scripts.
  • jedijosh920, InfamousSabre and LeeC2202 like this

InfamousSabre
  • InfamousSabre

    Harpocrates

  • Members
  • Joined: 03 Jan 2013

#4

Posted 13 February 2016 - 12:11 AM Edited by InfamousSabre, 13 February 2016 - 01:45 AM.

 

Using the _WORLD3D_TO_SCREEN2D native I can successfully draw text over entities in the game world but there is an obvious lag as when moving the camera around the text will move a bit before snapping back to the entity. Obviously I'm running it in a loop every tick but I'm guessing that ScriptHookV threads are not actually called as often as every game frame is rendered which is causing this lag?


I'm not sure if it will work with custom textures but you can use natives SET_DRAW_ORIGIN and CLEAR_DRAW_ORIGIN to draw without any stuttering/ lag. I ran into similar issues when making some scripts.

 

Really? Wow, I'll have to try this.

 

Edit: Wow no sh*t. It's totally lag-less. Thanks, CamxxCore!

  • CamxxCore likes this

MickB
  • MickB

    Player Hater

  • Members
  • Joined: 03 Oct 2015
  • Australia

#5

Posted 13 February 2016 - 01:44 AM

 

Using the _WORLD3D_TO_SCREEN2D native I can successfully draw text over entities in the game world but there is an obvious lag as when moving the camera around the text will move a bit before snapping back to the entity. Obviously I'm running it in a loop every tick but I'm guessing that ScriptHookV threads are not actually called as often as every game frame is rendered which is causing this lag?


I'm not sure if it will work with custom textures but you can use natives SET_DRAW_ORIGIN and CLEAR_DRAW_ORIGIN to draw without any stuttering/ lag. I ran into similar issues when making some scripts.

 

 

Thanks looks promising. Pretty sure it won't support custom textures only the native draw functions.


alex8b
  • alex8b

    Square Civilian

  • Members
  • Joined: 12 Jun 2015
  • Sweden

#6

Posted 23 February 2016 - 10:28 AM Edited by alex8b, 23 February 2016 - 05:54 PM.

You can re-implement this function by reading the view matrix from memory. I use that trick in the GTA V Eye Tracking mod.

 

https://github.com/a...eometry.cs#L238

https://github.com/a...ess/CCamera.cpp

  • InfamousSabre likes this

TriRozhka
  • TriRozhka

    Player Hater

  • Members
  • Joined: 18 Mar 2016
  • Russia

#7

Posted 02 April 2016 - 10:30 PM

Im trying to draw helicopterhud hud_corner but it look like square instead off L. Maybe it doesnt like orange color ?


mmuaz
  • mmuaz

    Player Hater

  • Members
  • Joined: 25 Sep 2017
  • Germany

#8

Posted 25 September 2017 - 01:31 PM Edited by mmuaz, 25 September 2017 - 01:32 PM.

 

Using the _WORLD3D_TO_SCREEN2D native I can successfully draw text over entities in the game world but there is an obvious lag as when moving the camera around the text will move a bit before snapping back to the entity. Obviously I'm running it in a loop every tick but I'm guessing that ScriptHookV threads are not actually called as often as every game frame is rendered which is causing this lag?


I'm not sure if it will work with custom textures but you can use natives SET_DRAW_ORIGIN and CLEAR_DRAW_ORIGIN to draw without any stuttering/ lag. I ran into similar issues when making some scripts.

 

I am trying to get just 2d screen coordinates. I don't want to draw them on the screen. Is it possible that if I could only have non-lagged 2d screen points? Because when I try to use _WORLD3D_TO_SCREEN2D, box lags a lot especially when I move the camera really fast.


K^2
  • K^2

    Vidi Vici Veni

  • Moderator
  • Joined: 14 Apr 2004
  • United-States
  • Best Poster [Technology / Programming] 2016
    Best Poster [Programming] 2015
    Most Knowledgeable [Web Development/Programming] 2013
    Most Knowledgeable [GTA Series] 2011
    Best Debater 2010

#9

Posted 25 September 2017 - 02:48 PM

Yeah, that's a standard issue in game-dev. You run the script, get the screen coordinates, then camera updates, and only then do draw operations happen. As the result, screen positions lag a frame behind. Last game engine I was working on had a special post-update pass that ran after the camera update and allowed screen-space FX to get the up-to-frame coordinates.

There's some smoothing you can do by filtering the positions and estimating next frame position. But that won't completely remove the lag. Just make it less jumpy. To fix it completely, you have to either rely on native functionality designed specifically to make this work, or write a work-around. If your script runs sufficiently late, and it's just the matrix that's out of date, alex8b's suggestion of reading matrix directly from memory might work.

CamxxCore
  • CamxxCore

    Mark Chump

  • Members
  • Joined: 19 Oct 2013
  • Canada

#10

Posted 25 September 2017 - 07:12 PM Edited by CamxxCore, 25 September 2017 - 07:14 PM.

Why not look at what I suggested above already? Using those functions seemed to eliminate the lag entirely. Edit: Oops. Sorry I didn't read.

LeeC2202
  • LeeC2202

    Mark Chump

  • Members
  • Joined: 24 Oct 2015
  • United-Kingdom

#11

Posted 25 September 2017 - 10:38 PM Edited by LeeC2202, 26 September 2017 - 12:04 AM.

Yeah, that's a standard issue in game-dev. You run the script, get the screen coordinates, then camera updates, and only then do draw operations happen.

 

I have never actually seen that problem before... that's not me being my usual argumentative self, I have honestly never seen that problem.

 

When I saw it happen in this game, I presumed it was because the script processes are being hooked and might be being queued for the next redraw, rather than being done at the end of the current one... which is where I would expect all overlay processing/coordinate tracking to be done.

 

Has anyone checked to see if SET_DRAW_ORIGIN kicks in an associated priority change that forces elements to be processed during the current update by the game, rather than being queued? I notice the final parameter in that native is always 0, does changing that affect anything? Does that 0 mean zero delay? I can't see a reason for that final parameter to be there, unless there is a default setting that is different. **see Edit**

 

I mean, the fact that changing the origin cures the problem, does seem to indicate that the resolving of worldspace to screenspace is being accurately handled in the current frame. If it was being delayed, that origin would be being set to a location that no longer exists in the next frame, because it would have already moved... so the lag should remain.

 

Just to put this into perspective with something I am working on, I have a UI being drawn made from UIRectangles and UIDrawTextures. If I move a pair of elements (one of each) left or right, the Rectangles lag behind the DrawTextures, so it appears as though the sprites are being pushed ahead of the cursor. When I stop moving, both elements align perfectly, so it's not as if there are some calculation errors in the process. So I wonder if by default, certain elements are processed immediately and some are pushed back a frame and this SET_DRAW_ORIGIN forces them both to be done immediately. Maybe at the default setting, it is intended for elements that have no bearing on what is on-screen, like cash displays, road names etc...

 

I am mostly just thinking out loud here... but I know curiosity did kill the cat. :D

 

Edit: Being unable to resist, I made a quick mod to test a theory. Setting that final parameter to anything other than 0, seems to stop it working altogether. I tried 1, 2 and -1 but only 0 worked. So that's my theory busted...


K^2
  • K^2

    Vidi Vici Veni

  • Moderator
  • Joined: 14 Apr 2004
  • United-States
  • Best Poster [Technology / Programming] 2016
    Best Poster [Programming] 2015
    Most Knowledgeable [Web Development/Programming] 2013
    Most Knowledgeable [GTA Series] 2011
    Best Debater 2010

#12

Posted 26 September 2017 - 05:12 AM

I have never actually seen that problem before...

It's not as likely to manifest if you build the engine right the first time. The experience I was describing was with an engine that was written as a single-threaded task more than a decade ago, and has since become a massively multi-threaded beast of legacy code running on multiple platforms. When Tiamat fight landed on my desk, it was chugging at 3FPS on XBOne. I am not proud of things I had to do to fix it.

What usually happens is that the engine expects things to happen in certain order. Then you realize, you can get more performance by scheduling this on multiple threads if you change the order a little bit. And now, the character movement controller, which is what's going to adjust the camera matrix, might lag behind the FX update, but still lead the rendering thread. So you end up with FX screen positions a frame out of date. Naturally, if you've thought of it from the start, you could have simply deferred necessary operations. But if you didn't, you end up with an a FX late update pass as an ugly hack to try and fix the issue. It's not pretty, but this console port is already a month behind, and you still have a pile of race conditions to fix.

Now, I'm by no means advocating writing code like that, but you put 20 people who are way over their head on tight schedule, and that's the sort of code quality you're going to get. And that happens in game dev left and right, I'm sure you know. So just count yourself lucky you haven't had to deal with that particular mess.
  • LeeC2202 likes this

alex8b
  • alex8b

    Square Civilian

  • Members
  • Joined: 12 Jun 2015
  • Sweden

#13

Posted 26 September 2017 - 01:44 PM Edited by alex8b, 26 September 2017 - 01:48 PM.

Pure C# implementation of World to Screen projection via getting camera matrix

 

https://github.com/a...CameraHelper.cs

 

https://github.com/a...king/SigScan.cs

using System;
using System.Diagnostics;
using System.Runtime.InteropServices;
using SharpDX;

namespace Gta5EyeTracking
{
	[StructLayout(LayoutKind.Sequential, Pack = 1)]
	public struct CViewPortGame
	{
		public unsafe fixed byte _0x0000[588];
		public unsafe fixed float mViewMatrix[16]; //0x024C 
	};//Size=0x028C

	public static class CameraHelper
	{
		private static IntPtr _gPViewPortGame = IntPtr.Zero;
		private static IntPtr GetViewPortGame(IntPtr baseAddress, int length)
		{
			if (_gPViewPortGame == IntPtr.Zero)
			{
				SigScan.Classes.SigScan sigScan = new SigScan.Classes.SigScan(Process.GetCurrentProcess(), baseAddress, length);
				IntPtr matricesManagerInc = sigScan.FindPattern(new byte[] { 0x48, 0x8B, 0x15, 0xFF, 0xFF, 0xFF, 0xFF, 0x48, 0x8D, 0x2D, 0xFF, 0xFF, 0xFF, 0xFF, 0x48, 0x8B, 0xCD }, "xxx????xxx????xxx", 0);
				if (matricesManagerInc != IntPtr.Zero)
				{
					var offset = Marshal.PtrToStructure<int>(new IntPtr(matricesManagerInc.ToInt64() + 3));
					var ptr = new IntPtr(offset + matricesManagerInc.ToInt64() + 7);
					_gPViewPortGame = new IntPtr(Marshal.PtrToStructure<long>(ptr));
				}
			}

			return _gPViewPortGame;
		}

		public static Matrix GetCameraMatrix()
		{
			IntPtr baseAddress = System.Diagnostics.Process.GetCurrentProcess().MainModule.BaseAddress;
			int length = System.Diagnostics.Process.GetCurrentProcess().MainModule.ModuleMemorySize;

			var viewPortGamePtr = GetViewPortGame(baseAddress, length);

			if (viewPortGamePtr != IntPtr.Zero)
			{
				var viewPortGame = Marshal.PtrToStructure<CViewPortGame>(viewPortGamePtr);
				unsafe
				{
					var matrix = viewPortGame.mViewMatrix;
					return new Matrix(matrix[0], matrix[1], matrix[2], matrix[3], matrix[4], matrix[5], matrix[6], matrix[7], matrix[8],
						matrix[9], matrix[10], matrix[11], matrix[12], matrix[13], matrix[14], matrix[15]);
				}
			}

			return Matrix.Identity;
		}
	}
}
        public static bool WorldToScreenRel(Vector3 entityPosition, out Vector2 screenCoords)
        {
            var mView = CameraHelper.GetCameraMatrix();
            mView.Transpose();

            var vForward = mView.Row4;
            var vRight = mView.Row2;
            var vUpward = mView.Row3;

            var result = new SharpDX.Vector3(0,0,0);
            result.Z = (vForward.X * entityPosition.X) + (vForward.Y * entityPosition.Y) + (vForward.Z * entityPosition.Z) + vForward.W;
            result.X = (vRight.X * entityPosition.X) + (vRight.Y * entityPosition.Y) + (vRight.Z * entityPosition.Z) + vRight.W;
            result.Y = (vUpward.X * entityPosition.X) + (vUpward.Y * entityPosition.Y) + (vUpward.Z * entityPosition.Z) + vUpward.W;

            if (result.Z < 0.001f)
            {
                screenCoords = new Vector2(0, 0);
                return false;
            }

            float invw = 1.0f / result.Z;
            result.X *= invw;
            result.Y *= invw;
            screenCoords = new Vector2(result.X, -result.Y);
            return true;
        }
  • K^2, LeeC2202 and mmuaz like this

mmuaz
  • mmuaz

    Player Hater

  • Members
  • Joined: 25 Sep 2017
  • Germany

#14

Posted 26 September 2017 - 02:31 PM Edited by mmuaz, 26 September 2017 - 02:50 PM.

Thank you everyone for your response and healthy discussion. 

@alex8b, I am also trying to get 2d points using matrix as you are doing in code. Is it working fine for you? 

As what I have tried, you can see in this image that 2d projection of 

e.Position // where e = Game.Player.Character.CurrentVehicle;

 (in green) using your code is not accurate as well.

So what exactly I am doing is I take a screenshot of the game at each tick and save it on 852 x 640 resolution. And then draw following point on the screen.

Math.Abs(my2dPoint.X * 852)
Math.Abs(my2dPoint.Y * 640)

Whereas, the 3d bounding box seems to be perfect calculated using this repo

Also, components of 2d projected points are mostly negative. I don't know what wrong I am doing.


alex8b
  • alex8b

    Square Civilian

  • Members
  • Joined: 12 Jun 2015
  • Sweden

#15

Posted 26 September 2017 - 02:36 PM Edited by alex8b, 26 September 2017 - 02:37 PM.

WorldToScreenRel returns screen coordinates normalized from -1 to 1.

 

In order to convert them to UI coordinates, use the following code:

 

var uiCoords = new Vector2(UI.WIDTH * 0.5f + screenCoords.X * UI.WIDTH * 0.5f, UI.HEIGHT * 0.5f + screenCoords.Y * UI.HEIGHT * 0.5f);

  • mmuaz likes this

Cuky
  • Cuky

    Player Hater

  • Members
  • Joined: 23 Sep 2017
  • Germany

#16

Posted 27 September 2017 - 08:51 AM Edited by Cuky, 27 September 2017 - 08:52 AM.

 

Pure C# implementation of World to Screen projection via getting camera matrix

 

https://github.com/a...CameraHelper.cs

 

https://github.com/a...king/SigScan.cs

using System;
using System.Diagnostics;
using System.Runtime.InteropServices;
using SharpDX;

namespace Gta5EyeTracking
{
	[StructLayout(LayoutKind.Sequential, Pack = 1)]
	public struct CViewPortGame
	{
		public unsafe fixed byte _0x0000[588];
		public unsafe fixed float mViewMatrix[16]; //0x024C 
	};//Size=0x028C

	public static class CameraHelper
	{
		private static IntPtr _gPViewPortGame = IntPtr.Zero;
		private static IntPtr GetViewPortGame(IntPtr baseAddress, int length)
		{
			if (_gPViewPortGame == IntPtr.Zero)
			{
				SigScan.Classes.SigScan sigScan = new SigScan.Classes.SigScan(Process.GetCurrentProcess(), baseAddress, length);
				IntPtr matricesManagerInc = sigScan.FindPattern(new byte[] { 0x48, 0x8B, 0x15, 0xFF, 0xFF, 0xFF, 0xFF, 0x48, 0x8D, 0x2D, 0xFF, 0xFF, 0xFF, 0xFF, 0x48, 0x8B, 0xCD }, "xxx????xxx????xxx", 0);
				if (matricesManagerInc != IntPtr.Zero)
				{
					var offset = Marshal.PtrToStructure<int>(new IntPtr(matricesManagerInc.ToInt64() + 3));
					var ptr = new IntPtr(offset + matricesManagerInc.ToInt64() + 7);
					_gPViewPortGame = new IntPtr(Marshal.PtrToStructure<long>(ptr));
				}
			}

			return _gPViewPortGame;
		}

		public static Matrix GetCameraMatrix()
		{
			IntPtr baseAddress = System.Diagnostics.Process.GetCurrentProcess().MainModule.BaseAddress;
			int length = System.Diagnostics.Process.GetCurrentProcess().MainModule.ModuleMemorySize;

			var viewPortGamePtr = GetViewPortGame(baseAddress, length);

			if (viewPortGamePtr != IntPtr.Zero)
			{
				var viewPortGame = Marshal.PtrToStructure<CViewPortGame>(viewPortGamePtr);
				unsafe
				{
					var matrix = viewPortGame.mViewMatrix;
					return new Matrix(matrix[0], matrix[1], matrix[2], matrix[3], matrix[4], matrix[5], matrix[6], matrix[7], matrix[8],
						matrix[9], matrix[10], matrix[11], matrix[12], matrix[13], matrix[14], matrix[15]);
				}
			}

			return Matrix.Identity;
		}
	}
}
        public static bool WorldToScreenRel(Vector3 entityPosition, out Vector2 screenCoords)
        {
            var mView = CameraHelper.GetCameraMatrix();
            mView.Transpose();

            var vForward = mView.Row4;
            var vRight = mView.Row2;
            var vUpward = mView.Row3;

            var result = new SharpDX.Vector3(0,0,0);
            result.Z = (vForward.X * entityPosition.X) + (vForward.Y * entityPosition.Y) + (vForward.Z * entityPosition.Z) + vForward.W;
            result.X = (vRight.X * entityPosition.X) + (vRight.Y * entityPosition.Y) + (vRight.Z * entityPosition.Z) + vRight.W;
            result.Y = (vUpward.X * entityPosition.X) + (vUpward.Y * entityPosition.Y) + (vUpward.Z * entityPosition.Z) + vUpward.W;

            if (result.Z < 0.001f)
            {
                screenCoords = new Vector2(0, 0);
                return false;
            }

            float invw = 1.0f / result.Z;
            result.X *= invw;
            result.Y *= invw;
            screenCoords = new Vector2(result.X, -result.Y);
            return true;
        }

 

I can't run your code. In WorldToScreenRel, Visiual Studio 2017 is saying, that var result = new SharpDX.Vector3(0,0,0) is wrong. But new SharpDX.Mathematics.Interop.RawVector3(0, 0, 0) works. Is this because of different SharpDX versions?

 

And in CameraHelper.cs, there is no variable type Matrix. When I change it to SharpDX.Mathematics.Interop.RawMatrix,

 return new SharpDX.Mathematics.Interop.RawMatrix(matrix[0], matrix[1], matrix[2], matrix[3], matrix[4], matrix[5], matrix[6], matrix[7], matrix[8],
                        matrix[9], matrix[10], matrix[11], matrix[12], matrix[13], matrix[14], matrix[15]);

I get the compiler error, that there is no such constructor! Can you please tell me how to fix this?

 

Thanks.


alex8b
  • alex8b

    Square Civilian

  • Members
  • Joined: 12 Jun 2015
  • Sweden

#17

Posted 27 September 2017 - 11:17 PM

Hi!

 

You need to install both

https://www.nuget.or...ckages/SharpDX/

and

https://www.nuget.or...DX.Mathematics/

  • Cuky likes this




2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users