Just thought about to add X’Y’Z’ of reflecting physics pixel and the distance from a reflecting physics pixel to a correponding image physics pixel into the parameters of each physics pixel, for without them the model can not handle mutliple reflection/bounce scenario.
With identifying reflection physics pixel for each pair of image/reflected physics pixel, the model cant identify an image physics pixel which is also a reflecting physics pixel which is on the surface or an image/second mirror reflected by first mirror.
The distance from an image physics pixel to a reflecting physic pixel is equal to the distance from the corresponding reflected physics pixel to the reflecting physics pixel, which can be calculated from the Z of the image physics pixel minus Z of the reflecting physics pixel, but making it a parameter or explicit will help model understand the reflection especially for more than one bounce/reflection scenerios.
Below are revised summary for physics pixel including reflection.
******
******
I thought that it’s necessary to identify the reflection in mirror better, for it’s essential for not only video generative application but also control scenarios like mirrors set at corner for cars or robot smashing into mirrors.
In fact, all physics pixels visible except luminating ones have a reflection rate above 0 to reflect light to be visible, so as for this aspect, these physics pixels reflecting may all include RGBA from other physics pixel reflected by them. So it’s important to deal with reflection effect.
A virtual image physics pixel in mirror is a reflection image on the extended straight ray view of its corresponding pixel into the mirror surface,
reflected from a reflected physics pixel which is on opposite side of the mirror and is on the reflected ray view of the correponding pixel,
reflected by a reflecting physics pixel which is on the mirror/reflecting surface and is on the ray view of the corresponding pixel.
So the key is how to represent virtual “image physics pixel” in the mirror, “reflected physics pixel” on the opposite of the mirror, and the real “reflecting physics pixel” on the surface of the mirror, and the 3 physics pixels can be regarded as a reflection physics pixel.
Previously I introduced a prefix “TNC” for coordinates XYZ and X’Y’Z’, which can be used to tag phsics pixels included in a reflection physics pixel. So set the highest 3 bits of “C” of “TNC” as identifier as mirror bits, in which C= mirror bits (3 bits) + coordinate system type (5 bits), and mirror bits are:
the highest bit represents whether it’s an image physics pixel or not: 0 is not and 1 is yes,
the second highest bit represents whether it’s an reflected physics pixel or not: 0 is not and 1 is yes,
the third highest bit represents whether it’s an reflecting physics pixel or not: 0 is not and 1 is yes.
As for coordinate system type (5 bits), coordinate type=”00001″ represents sensor’s own perspective coordinate system, and coordinate type=”00010″ represents the unified rectangular coordinate system.
A reflection physics pixel includes parameters for 3 corresponding physics pixels – image & reflected & reflecting physics pixel, but the reflection physics pixel shall still be represented by same parameters of a real physics pixel to keep format identical, which include:
XYZ, for a real physics pixel, or for an image physics pixel;
X’Y’Z’_ed, for a real physics pixel, or for a reflected physics pixel;
RGBA_im, for a real physics pixel, or for an image physics pixel;
RGBA_ed, “null” for a real physics pixel, or for a reflected physics pixel;
X’Y’Z’_ing, “null” for a real physics pixel, or for a reflecting physics pixel;
distance from the reflecting physics pixel to the image physics pixel;
wherein when it’s a reflection physics pixel of one reflection, all other parameters are of the reflected physics pixel;
wherein, X’Y’Z’_ed have its direction vector and X’Y’Z’_ing have no direction vector.
1>> TNC+XYZ of the image physics pixel
XYZ is the perspective coordinates of the virtual position of the image physics pixel in camera’s view, and the image physics pixel doesnt have its own X’Y’Z’ mapped from its XYZ, and XYZ of the image physics pixel corresponds to the X’Y’Z’_ed of the reflected physics pixel,
wherein C=”100-00001″, C=mirror bits+ type, mirror bits=”100″ represents image physics pixel, coordinate type=”1″ represents the camera’s perspective coordinate system,
8bits: T=sensor type, (“1” means camera, “2” means mm wave radar, “3″ means microphone),
8bits: N=sensor ID, (SN of the sensor in its own type),
2>> C+X’Y’Z’_ed of the reflected physics pixel
X’Y’Z’_ed of the reflected physics pixel is corresponding to the XYZ of the image physics pixel, and the reflected physics pixel doesnt have its own XYZ mapping to X’Y’Z’_ed,
In C+X’Y’Z’_ed of the reflected physics pixel, C=”010-00010″,
the direction vector of X’Y’Z’_ed of the reflected physics pixel is from the near/first object to the far/second object along the reflected ray view of the corresponding pixel,
the direction vector of X’Y’Z’ of a real physics pixel is perpendicular to the interface of the physics pixel and at the point of the physics pixel.
3>> C+X’Y’Z’_ing of a reflecting physics pixel in a reflection physics pixel
X’Y’Z’ of a reflecting physics pixel’s is added into the parameters of its corresponding reflection physics pixel’s parameters as C+X’Y’Z’_ing, in which C= ”001-00010″;
X’Y’Z’_ing of the reflecing physics pixel in the reflection physics pixel is an identifier/reference to the datafield of parameters of this reflecting physics pixel which has a whole set of parameters for a physics pixel.
4>> distance from the reflecting physics pixel to the image physics pixel
distance from the reflecting physics pixel to the image physics pixel is equal to the distance from the reflecting physics pixel to the reflected physics pixel.
5>> a reflecting physics pixel is a real physics pixel
a reflecting physics pixel is a real physics pixel with a whole set of parameters for one physics pixel,
5.1> the reflection rate parameter
if the reflection rate parameter of a physics pixel on a surface is above 0, then the RGBA of this physics pixel include RGBA of one or more other reflected physics pixels, through one or more image physics pixels in extended ray view of its pixel deep into the surface of mirror effect;
for a reflecting physics pixel, there must be at least one reflection physics pixel correspondingly;
there could be a physics pixel with reflection rate above 0 which is not classified as an reflecting pixel, which means the physics pixel has reflecting effect but has corresponding no image/reflected physics pixel pair for the image is invisible or dosent matter.
5.2> in TNC+XYZ of the reflecting physics pixel, C=mirror bits+coordinate type=”001-00001″,
5.3> X’Y’Z’ and the direction vector of X’Y’Z’
X’Y’Z’ and the direction vector of X’Y’Z’ can let the model learn the direction of the incident light reflected by the reflecting pixel,
in C+X’Y’Z’ of the reflecting physics pixel, C=”001-00010″,
5.4> RGBA_im of the reflecting physics pixel shall be like a transparent physics pixel through which the camera can see the image physics pixel,
the reflecting physics pixel is a real physics pixel and not an image physics pixel, so RGBA_ed of the reflecting physics pixel are null.
6>> multiple reflection
6.1> In a multiple reflection case, at least the first reflected pixel is also a virtual image physics pixel, so in its C+X’Y’Z’, C is “110-00010”, and X’Y’Z’_ed is the rectangular coordinates of its virtual position.
6.2> except XYZ, X’Y’Z’_ed, X’Y’Z’_ing, RGBA_im and RGBA_ed, all other parameters of a reflection physics pixel including a virtual image&reflected physics pixel shall be null, in another word, in multiple reflection only the last real reflected physics pixel has all other parameters which are exactly for it.
6.3> for one bounce/reflection case, a reflection physics pixel can be represented by parameters of one physics pixel,
for multiple bounce/reflection case, a reflection physics pixel need be represented by parameters of more than one physics pixel.
********
********
The summary of physics pixel
The physics pixel establishes first perspective physics world model for camera (or similar sensor) in the true sense, in which the whole space is filled up by objects with interfaces in between from camera to deep depth, and a pixel’s ray view passes through interfaces with a physics pixel on each intersection, wherein even a point corresponding to bare sky can be regarded as located on an interface at a certain height between an object of the outdoor space and another object of outer space.
Each pixel in a visual frame from a camera is mapped to one or more physics pixels along its ray of view at one or more different depths.
Each physics pixel is located on an interface between (two surfaces of) two objects , wherein the physics pixel is also located on 2 points on the (two surfaces of) two objects, wherein the physics pixel corresponds to:
an interface,
two objects(near/first object nearer to the camera and far/second object farther from the camera),
two points (near point on the near surface of the near object and far point on the far surface of the far object),
two surfaces (near surface of the near object and far surface on the far object).
Therefore, each physics pixel may include parameters:
for the physics pixel (XYZ, X’Y’Z’_ed with direction vector, X’Y’Z’_ing, RGBA_im, RGBA_ed),
for the interface (interface ID, refractive index, overlap number),
for the two objects (object ID, object class, parent object, velocity, rotation, mass, etc ),
for the two points (velocity, rotation, pressure, temperature, material, etc),
for the two surfaces (surface ID, text on surface, reflection rate, material, etc).
In labeled training, transparent physics pixels and invisible physics pixels can also be labeled, to let the model learn the correlation between invisible part and visible part, to establish a spatial understanding of wholen space including invisible part.
This approach is particularly well-suited for artificial intelligence applications, enabling more accurate modeling of real-world physical interactions and spatial relationships at the pixel level.
Below is the classical example: a camera, through a window of a house, see a wall in the house, and this forms 4 interfaces overlapped, in which: first interface is between camera (near object ID:111) and outdoor space (far object ID: 000), second interface is between outdoor space (000) and glass of window (123), third interface is between glass of window (123) and interior space of house (456), forth interface is between interior space of house (456) and wall in house (789).
———-
Below is shared parameters of physics pixels in a visual frame.
———-
[1] ground velocity vector of the origin of the unified rectangular coordinate system;
[2] ground rotation vector of the origin of the unified rectangular coordinate system.
// The unified rectangular coordinate system could be moving ego/local coordinate system for a physics pixel model, in which the origin is the observer/camera/device of the physic pixel model.
// The relative velocity vector of an object/point is relative to the origin of the unified rectangular coordinate system.
// The ground velocity vector of an object/point/origin is relative to the stationary ground.
// The ground rotation vector of an object/point/origin is relative to the stationary ground.
———-
Below is physics pixels along the ray view of a pixel in the visual frame of the camera.
———-
0. parameters of the pixel
[1] RGB of a pixel in a visual frame;
[2] position of the pixel in the visual frame.
———-
1. first physics pixel on 1st interface
1.1) parameters for the physics pixel
[1] TNC+XYZ, XYZ are in perspective coordinate system of the camera, XY correspond to the pixel’s position in the visual frame, Z corresponds to the distance/depth from the camera to the physics pixel, in which XYZ could be in different metric unit like XY could be in number of pixels and Z is in mm or micrometre;
[2] RGBA_im;
[3] C+X’Y’Z’_ed, mapping of XYZ into a unified rectangular coordinate system, all X’Y’Z’ could be in mm or micrometre;
// for a real physics pixel, XYZ and X’Y’Z’_ed above are mapping to each other //
// for a reflection physics pixel, XYZ above are coordinates of virtual position of the image physics pixel in camera’s perspective view, and X’Y’Z’_ed above are real coordinates of the reflected physics pixel in unified rectagular coordinate system //
[4] direction vector of X’Y’Z’ in the unified rectangular coordinate system;
// the direction vector of X’Y’Z’ of the reflected physics pixel is from the near/first object to the far/second object along the reflected ray view of the corresponding pixel //
// the direction vector of X’Y’Z’ of a physics pixel is perpendicular to the interface of the physics pixel at the point of the physics pixel //
[5] RGBA_ed;
// for a real physics pixel, RGBA_ed = null //
// for a reflection physics pixel, RGBA_ed is of the reflected physics pixel //
[6] C+X’Y’Z’_ing;
// for a real physics pixel, C=”000-00000″, X’Y’Z’=null //
// for a reflection physics pixel, C=”001-00010″, X’Y’Z’ is for the reflecting physics pixel of the reflection physics pixel //
[7] distance from the reflecting physics pixel to the image physics pixel;
// for a real physics pixel, this distance = 0//
[8] object depth of the far object along the physics pixel’s ray view;
// object depth = the distance from 1st interface to 2nd interface along the ray view of the pixel corresponding to the physics pixel//
———-
1.2) parameters for the interface
[1] Interface ID;
[2] Overlap number of the interface = 0;
[3] refractive index;
———-
1.3) parameters of near object and far object
1.3.1) parameter of the near object
[1] object id of the near object: 111;
[2] object class: camera;
[3] parent object: observer;
[4] relative velocity vector;
[5] ground velocity vector;
[6] gound rotation vector;
[7] mass of the object;
———-
1.3.2) parameter of the far object
[1] object id of the object: 000;
[2] object class: outdoor space;
[3] parent object: None;
[4] relative velocity vector;
[5] ground velocity vector;
[6] gound rotation vector;
[7] mass of the object;
———-
1.4) parameters of the 2 points which the physics pixel is located on the near object and far object respectively
1.4.1) parameters of the near point on the near object
[1] relative velocity vector;
[2] ground velocity vector;
[3] gound rotation vector;
[4] material of the point: glass;
[5] pressure on the point;
[6] temperature on the point;
[7] reflection rate;
———-
1.4.2) parameters of the far point on the far object
[1] relative velocity vector;
[2] ground velocity vector;
[3] gound rotation vector;
[4] material of the point: air;
[5] pressure on the point;
[6] temperature on the point;
[7] reflection rate;
———-
1.5) parameters for the two surfaces of the two objects forming the interface
1.5.1) parameters for the near surface of the near surface
[1] surface ID;
[2] reflection rate;
[3] material;
[4] text on surface (this parameter can be put into the dedicated text field of omnitoken) .
———-
1.5.2) parameters for the far surface of the far surface
[1] surface ID;
[2] reflection rate;
[3] material;
[4] text on surface (this parameter can be put into the dedicated text field of omnitoken) .
———-
2. 2nd physics pixel on 2nd interface
……
3. 3nd physics pixel on 3rd interface
……
4. 4th physics pixel on 4th interface
……
Be First to Comment