How To: Get Screen Coordinates From 3D in Delphi

Problem Overview

A common problem using the Delphi FMX 3D forms and layers is that there is a need to obtain 2D screen coordinates in order to position objects in 3D accurately and to scale them for their target platforms. A solution is hard to find, not well documented, and most answers outside of Delphi are using OpenGL, the GML library, or game engines.

Solution

A detailed solution would bring in the matrix transformations for inversing the camera projection in 3D space and calculating the 2D viewport pixel position. However, Delphi already the functionality required for these calculations... they are hard to find and not well documented! There are two main functions that we have available for these calculations that can be found in the TContext3D shown within the following documentation from Embarcadero https://docwiki.embarcadero.com/Libraries/Sydney/en/FMX.Types3D.TContext3D.

From the TContext3D we want to focus on the WorldToScreen and the ScreenToWorld methods, these allow us to transform a 3D point into a screen 2D point, and inversely the 2D screen point into a 3D point. The following code shows how to use the WorldToScreen method for obtaining the 2D screen projection.

Solving the projection is only part of the solution, in order to obtain the correct coordinates we need to apply the Scaling, Rotations, and Translation accordingly in that order prior to computing the Screen projection. The following Helper function uses the TMatrix3D to calculate the correct 2D screen projection for a point.

Device Screen Resolution

When targeting MacOS, iOS, and Android you will notice that the device screen resolution is much greater than in Windows. To account for this, we use the TPlatformService to obtain the correct scaling, and then to fix this issue we device the WorldToScreen projection result by the Scaling as show in the prior piece of code. Doing so will provide with an accurate position when drawing on screen coordinates.

Sample Application

A sample application demonstrating using both the TForm3D and a TLayer3D shows how we can do the transformations in both scenarios and working with the TContext3D of each can be found under my Github repo under the conclusion. Here are screen shots of the application in the IDE, this first one shows a TForm containing a TLayer3D with a TSphere, where we show the position of the mouse and are able to calculate the position of the TSphere and displayed it on the TMemo field for logging purposes.

GOTCHA: When using a TPainBox for drawing into the screen as shown on the TForm example you will need to substract the position (X an Y values) of the TPainBox to the calculated results so that they are accurate, otherwise you will encounter an alignment issue. The source code demonstrates the fix.

Here is the View.Game.pasTForm3D displaying a TLayer3D and a TCube.

The running application is shown below running in Windows.

The same application running in an Android Device (Samsung A54 5G) is below, after a minor UI update removing the TLabel that had "Coords:" as text..

Conclusion

This is a short and simple tutorial show you how to obtain the screen coordinates for a TObject3D in Delphi. The full source code can be found in the link below.

https://github.com/an01f01/delphi_fmx_3d

NOTE: Minor updates on the Github repo have been added to correct the screen resolutions across different platforms. Try this out on your Android device and Windows machine. If you run into any issues do comment and let me know!

References

https://glooscapsoftware.blogspot.com/2021/01/simple-firemonkey-3d.html

https://stackoverflow.com/questions/71423196/firemonkey-viewport-procedure-calculated-differently-world-to-screen-on-mac

https://stackoverflow.com/questions/519106/projecting-a-3d-point-to-a-2d-screen-coordinate

https://thundaxsoftware.blogspot.com/2012/01/projecting-3d-points-to-2d-screen.html

https://www.gamedev.net/forums/topic/708406-opengl-screen-to-world-coordinates-varying-eye-position-and-look_at-position/5432692/

https://blog.grijjy.com/2021/01/14/shader-programming/

https://stackoverflow.com/questions/21101327/delphi-xe5-android-screen-resolution-device-profiles