I thought that it’s necessary to identify the reflection in mirror better, for it’s essential for not only video generative application but also control scenarios like mirrors set at corner for cars or robot smashing into mirrors.
A virtual image physics pixel in mirror is a reflection image on the extended straight ray view of its corresponding pixel into the mirror surface,
reflected from a reflected physics pixel which is on opposite side of the mirror and is on the reflected ray view of the correponding pixel,
reflected by a reflecting physics pixel which is on the mirror/reflecting surface and is on the ray view of the corresponding pixel.
So the key is how to represent virtual “image physics pixel” in the mirror, real “reflected physics pixel” on the opposite of the mirror, and the real “reflecting physics pixel” on the surface of the mirror.
Previously I introduced a prefix “TNC” for coordinates XYZ and X’Y’Z’, which can be used to tag phsics pixels related to mirror effect. So set the highest 2 bits of “C” of “TNC” as identifier for mirror effect in which C= mirror bits (2 bits) + coordinate system type (6 bits), and mirror bits are:
“00” represents a physics pixel without mirror effect,
“01” represents a reflecting physics pixel which reflects light,
“10” represents the virtual “image physics pixel” in the mirror,
“11” represents a real reflected physics pixel reflected by the reflecting pixel.
As for coordinate system type (6 bits), coordinate type=”000001″ represents sensor’s own perspective coordinate system, and coordinate type=”000010″ represents the unified rectangular coordinate system
Following are the related parameters of 3 related physics pixels of an “image physics pixel”, from which AI model learn and understand the physics nature of the image physics pixel in labeled training.
1) the parameters of the image physics pixel
1.1) TNC(C=10,00,01)+XYZ, the perspective coordinates of the image physics pixel with prefix.
C=mirror bits+ type, mirror bits=”10″ represents image physics pixel, coordinate type=”1″ represents the camera’s perspective coordinate system,
8bits: T=sensor type, (“1” means camera, “2” means mm wave radar, “3″ means microphone),
8bits: N=sensor ID, (SN of the sensor in its own type),
1.2) X’Y’Z’ of image physics pixel is the X’Y’Z’ of the reflected physics pixel, so the image physics pixel doesnt have its own X’Y’Z’,
2) the parameters of the reflecting physics pixel
2.1) the reflection rate parameter,
if the reflection rate parameter of a physics pixel on a surface is above 0, then the RGBA of this physics pixel includes RGBA of one or more other reflected physics pixels, through one or more image physics pixels in extended ray view of its pixel deep into the surface of mirror effect;
2.2) X’Y’Z’ and the direction vector of X’Y’Z’
X’Y’Z’ and the direction vector of X’Y’Z’ can let the model learn the direction of the incident light reflected by the reflecting pixel,
in C+X’Y’Z’ of the reflecting physics pixel, C=”01,00,10″,
2.3) in TNC+XYZ of the reflecting physics pixel, C=mirror bits+coordinate type=”01,00,01″,
3) the parameters of the reflected physics pixel
C+X’Y’Z’ of the reflected physics pixel is corresponding to the XYZ of the image physics pixel, and the reflected physics pixel doesnt have its own XYZ,
In C+X’Y’Z’ of the reflected physics pixel, C=”11,00,10″.
********
********
Below is a list of parameters of physics pixels along the ray view of a pixel in a visual frame.
———-
0. parameters of the pixel
RGB of a pixel in a visual frame;
position of the pixel in the visual frame.
———-
1. first physics pixel on 1st interface
1.1) parameters for the physics pixel
TNC+coordinates (X, Y, Z) in perspective coordinate system of the camera, XY correspond to the pixel’s position in the visual frame, Z corresponds to the distance/depth from the camera to the physics pixel, in which XYZ could be in different metric unit like XY could be in number of pixels and Z is in mm or micrometre;
C+coordinates (X’, Y’, Z’), mapping of (X, Y, Z) into a unified rectangular coordinate system, all X’Y’Z’ could be in mm or micrometre;
// for a real physics pixel, XYZ and X’Y’Z’ above are mapping to each other
// for an image physics pixel, XYZ above are coordinates of the image physics pixel in camera’s perspective view, and X’Y’Z’ above are coordinates of the reflected physics pixel in unified rectagular coordinate system //
direction vector of (X’, Y’, Z’) from near object to far object in the rectangular coordinate system;
RGBA;
object depth of the far object along the physics pixel’s ray view: the distance from 1st interface to 2nd interface along the ray view of the pixel corresponding to the physics pixel;
———-
1.2) parameters for the interface
Interface ID;
Overlap number of the interface = 0;
refractive index;
———-
1.3) parameters of near object and far object
1.3.1) parameter of the near object
object id of the near object: 111;
object class: camera;
parent object: observer;
velocity vector of the object in unified rectangular coordinate system;
rotation vector of the object in unified rectangular coordinate system;
mass of the object;
———-
1.3.2) parameter of the far object
object id of the object: 000;
object class: outdoor space;
parent object: None;
velocity vector of the object in unified rectangular coordinate system: 0;
rotation vector of the object in unified rectangular coordinate system: 0;
mass of the object;
———-
1.4) parameters of the 2 points which the physics pixel is located on the near object and far object respectively
1.4.1) parameters of the near point on the near object
velocity vector of the point in unified rectangular coordinate system;
rotation vector of the point in unified rectangular coordinate system;
material of the point: glass;
pressure on the point;
temperature on the point;
reflection rate;
———-
1.4.2) parameters of the far point on the far object
velocity vector of the point in unified rectangular coordinate system;
rotation vector of the point in unified rectangular coordinate system;
material of the point: air;
pressure on the point;
temperature on the point;
reflection rate;
———-
1.5) parameters for the two surfaces of the two objects forming the interface
1.5.1) parameters for the near surface of the near surface
surface ID;
reflection rate;
material;
text on surface (this parameter can be put into the dedicated text field of omnitoken) .
———-
1.5.2) parameters for the far surface of the far surface
surface ID;
reflection rate;
material;
text on surface (this parameter can be put into the dedicated text field of omnitoken) .
———-
2. 2nd physics pixel on 2nd interface
……
3. 3nd physics pixel on 3rd interface
……
4. 4th physics pixel on 4th interface
……
Be First to Comment