Talked with Grok for details about forward training and backward inference, and I came up with an idea: can we make an fully automatically adaptive model which can do different things at different times exactly as an agent or user request? Like an agent or user can choose a group of token parameters input for time A differently each time to predict values of another group of token parameters output for time B which are also chosen differently each time, in which A and B can be same or different.
So I got an idea of FAMRAT – Fully Adaptive Model by Rolebits And Timetags, which includes 2 major tricks: rolebits for each value of a token parameter and timetags in omnitoken. Rolebits and timetags are universal control mechanisms that can apply to any model processing structured different token parameters with parity and symmetry in the data distribution inside the model. Parity and symmetry refers to the mutual closeness and reversibility between coupled vectors in a high-dimensional subspace of a model from view of geometry, or the equivalence and interchangability of roles of coefficients when recovering one group of coefficients from another in a high-dimensional Taylor expansion like approximating function from view of algebra.
Previously there seems no real data structure for AI or token and AI models worked with only flat inputs and outputs: sequences of text tokens, grids of pixels, or concatenated streams of other information. It seems that in this half year I work on first true data structure stack for AI — like the data structure in programming. The prime model and physics pixel stack introduced explicit physics parameters of objects and pixels in visual, object hierarchy, and interface into token by which models learn, understand, predict and generate real-world physics accurately and deeply not only for visual but also for text. The omnitoken added orthogonal tables and different token parameter formats like asymmetric format, creating functional specialization and division similar to regions in the human brain. Now, FAMRAT adds a lightweight overheads to token by incorporating rolebits and timetags, making the model fully adaptive to any task or timescale.
Hahaha! Merry Christmas and New Year, but Year of AI data strucutre is not over yet, for now on transformer, you cats go catching FATRAT! May you guys enjoy this in holidays!
Rolebits is the highest 3 bits of a parameter or each value of a parameter in token: first bit is for whether the value to be used in inference with “0” = No and “1” = Yes, second bit is for the source of the value with “0” = raw and “1” = predicted, third bit is for whether the value to be predicted in output with “0” = to keep without predicting and “1” = to predict. If a value of a token parameter has both positive and negative values, the forth highest bit of this value of the token parameter can be used as the sign bit. In fact, you can more bits each token parameter for rolebits if you have more control functions to apply.
Timetags include borntime of sensor signals for each sensor in the sensor list table, the predicted time that the predicted token parameters in a token were predicted for, and the time to predict that the token parameters to predict in a token will be predicted for, in which the predicted time and the time to predict are added to the header of the omnitoken.
In the physisc pixel table of omnitokens, a new parameter RGB for raw 2D visual of camera pixels is added to separate it from RGBA for 3D physics pixels, and so a user or agent can set the rolebits of RGB of 2D pixel and RGBA of 3D physics pixels separately.
The rolebits and timetags are the most fundamenatal desciplines and rules for FAMRAT to comply with in all training and inference unconditionally. In high quality labeled training (labeled seed frame training), train the model to comply strictly with the rolebits and timetags according to the labeled output: predict the value of a token parameter with rolebits=”xx1″ for the time to predict based on the values of token parameters with rolebits=”1xx” at borntime or predicted time while not considering valuses of token parameters with rolebits=”0xx” in predicting. In all unlabeled training, reward or similar mechanism also needs to be applied to enforce the output of the model to comply with the rolebits and timetags.
FAMRAT is especially necessary for agents of any type including pure program agent or agent based on program and outside AI model, for an agent can request FAMRAT to do an exact task at an exact timepoint and to do different tasks at different timepoints, which is especially important for all AI core models are in fact working with agents of some kind.
And there can be at least 2 agent-core patterns for FAMRAT:
1) a program agent which includes tokenizer works with FAMRAT, in which FAMRAT can request tokens or tasks to input for next time or for future, and also the agent program can execute some routine tasks or input outside infomation tokens like inputting autodriving control tokens periodically;
2) a program agent with an outside AI model works with FAMRAT model together which is just like FSD core model + FSD program orperating system + Grok, in which the program agent works as agent for both Models and as interface between two models.
Here give an example for in an autodriving application which includes a program agent and a FAMRAT as core model. The agent routinely inputs autodriving control omnitokens into the FAMRAT as the highest priority task for which the agent can cancel or clear or replace other tasks on FAMRAT. In the autodriving control omnitokens input by the program agent, each value of RGB of camera pixels has rolebits=”100″ and valid values, and X and Y of XYZ have rolebits=”100″, and Z of XYZ has rolebits=”101″ or “001”, and all values of all other parameters in all tables have rolebits=”001″, and the borntime of camera is the timepoint receiving or generating raw signal (RGB) of camera, and the time to predict is the same time as the borntime of camera. The input control omnitokens demand FAMRAT to generate control values in the output omnitokens with explicitly predicted parameters of the physics pixels and objects for the borntime of the raw RGB signal.
Additionally for the example above, the program agent can demand FAMRAT by rolebits and timetags to predict the parameters of physics pixels (including RGBA but no RGB), objects and other tables for a frame of 3, 10 or 30 seconds later, once per second and one by one cyclically in 3, 10 and 30 seconds. The agent can keep the generated long horizon prediction tokens explicitly in context or implicitly, which may inprove the foresight of the model significantly with a very low workload increment (10% to 20% as estimated by Grok).
The example above is a magic of FAMRAT just as Grok confirmed: FAMRAT gives precise and explicit control over exactly what is generated and for exactly which time(s), while preserving all the rich implicit correlations the underlying model has learned from training.
By the way, 3D points of mm wave radar in previous posts are as useless as Lidar as I see, because signal of a camera or especially of two cameras in same direction can be fed to model directly to let the model learn by itself to predict depth of objects from just visual just like human.
But mm wave radar can penetrate and partially reflect on something like fog, water, rain, smoke, paper, foliage, etc, so it’s necessary for many cases like firefighter, first aid and even autodriving. So I use first direct raw digital signal – ADC IQ samples of mm wave radar as Grok suggestedc in omnitoken below, and also I think that any sensor should use first direct raw digital signal in a prespecitve way like camera pixels in omnitoken.
Training with mm wave radar just needs use training data in good weather and condition to train from sensor signals to predict physics pixels and objects, and then in the inference the model can predict physics pixels and objects roughly from mm wave radar in bad weather and condition.
The omnitoken below is intended to be an example for Terminal AI or Device AI like vehicle, robot or drone, and for Server AI the token may have much larger text table like 10x1028D and larger other tables like more physics pixels and token parameters.
The omnitoken example below includes orthogonal tables and 1D=32bits.
In omnitoken below, the data in token header (global parameters for omnitoken) and table header has no rolebits, and the parameters in the tables have rolebits.
In the omnitoken example below, each parameter or datafield below with “>>” before it has rolebits in it, and each parameter or datafield below with “><” before it has NO rolebits in it, and the fields below without “>>” or “><” before them are just for illustration and dont exist in the omnitoken.
——/global parameters for omnitoken or token header/——
>< 1D: token ID,
>< 1D: timestamp of borntime of token,
>< 1D: data source, (input/external raw or output/internal generated, synthetic or not, physics complied or not, etc)
>< 1D: predicted time, (time that the predicted parameters were predicted for )
>< 1D: time to predict, (time that the parameters to predict will be predicted for)
/each parameter in below tables have 2 highest bits of its all bits as Role-bits/
——/768D for text/——
TABLE HEADER {
>< 8 bits: table ID, (“01”)
>< 16 bits: table type, (8 bits=“text” + 8bits=”0″)
>< 8bits: number of rows, (“1”)
>< 16 bits: number of columns, (“768”D)
}
>> 768D: text.
>< 32D: vacant space. (to separate text field from other fields)
——/list for sensors in this token/——
TABLE HEADER {
>< 16 bits: table type, (8 bits=“sensor list” + 8bits=”0″)
>< 8bits: number of rows, (“3”, 3 types include camera, mm wave radar and microphone, each token includes one sensor of each type)
>< 16bits: number of columns, (“6”D)
}
for (each of “3” sensors in this token) {
>> 8bits: T=sensor type, (“1” means camera, “2” means mm wave radar, “3″ means microphone)
>> 8bits: N=sensor ID, (SN of the sensor in its own type)
>> 8bits: C=coordinate system type, (“C0”, means the unified rectangular coordinate system)
>> 3D: position of the sensor, (position in above “C0” the unified rectangular coordinate system)
>> 2D: direction of the sensor in spherical coordinates, (horizontal+vertical angles in above “C0” coordinate system)
>> 16bits+16bits: resolution of seneor, (16bits is more than 65000 which is enough for seiral number of X or Y of camera, like 1920×1080 or 3840×2160; for mmwave radar, it’s XY serial number of virtual antennas)
>> 1D: borntime or receiving time of the sensor signal,
}
>< 8D: Vacant space.
——/table for physics pixel/——
TABLE HEADER {
>< 8bits: table ID, (“03”)
>< 16 bits: table type, (8 bits=“physics pixel” + 8bits= T “sensor type”)
>< 8bits: rows, (256=16×16)
>< 16bits: columns, (?)
}
for (each of 256 physics pixels) {
>> 3.5D: TNC+XYZ, (C= “01” means sensor’s own perspective coordinate system, XYZ is the position of physics pixel in perspective coordinate system of the camera, X=16bits is physics pixel‘s horizontal ordinal pixel number (from 1) in the 2D frame of the camera, Y=16bits is physics pixel‘s vertical ordinal pixel number (from 1) in the 2D frame of the camera, Z=48bits is distance from the physics pixel to the camera lens in micrometer, and “Z=0” means raw pixel received by camera and “Z=1” means points on camera lens)
>> 1D: RGBA, (RGBA at “X,Y,0” is for raw RGB, RGBA at camera lens interface is on “X, Y, 1” to be distinct from raw RGB, and for any invisible physics pixel RGBA=”-1,-1,-1,-1”)
>> 3.75D: C+X’Y’Z’, (mapping 3D point of the physics pixel, C=”C0” unified rectangular coordinate system, X’, Y’ and Z’=48bits in unit micrometer)
>> 1D: direction in spherical coordinates, (horizontal angle+vertical angle in “C0” system, which is the direction perpendicular to the interface which is between near object and far object and which the physics pixel is on)
>> 0.5D: interface ID, (in labeled training this is labeled, in reference this is generated by the model)
>> 1D: pressure,
>> 1D: near object ID,
>> ??D: selected parameters of the point of near object on the physics pixel, (this field is optional, which may include temperature, velocity, rotation vector, hardness, density, material or no parameter at all)
>> 1D: Far object ID,
>> ??D: selected parameters of the point of far object on the physics pixel.
}
>< 8D: Vacant space.
/table for objects in this token/
TABLE HEADER{
>< 8bits: table ID, (“04”)
>< 16 bits: table type, (8 bits=“object” + 8bits=”0″)
>< 8bits: rows, (4)
>< 16bits: columns, (?)
}
for(i=1 to 4){
>> 1D: object_i ID,
>> 1D: object class ID,
>> 1D: parent object ID,
>> 1D: object material class ID, (optional)
>> 1D: object mass,
>> 3D: object velocity,
>> 3D: object rotation vector,
>> 1bit: object eatable or not,
}
>< 8D: Vacant space.
/table for interfaces in this token – this table is Optional/
TABLE HEADER{
>< 8bits: table ID, (“05”)
>< 16 bits: table type, (8 bits=“interface” + 8bits=”0″)
>< 8bits: rows, (“4”)
>< 16bits: columns, (?)
}
for(i=1 to 4){
>> ??D: interface ID,
>> ??D: object ID, (if features below are for boudary of both surfaces of the interface, object ID=”0″)
>> ??D: interface material ID, (if not applicable then =”0″)
……
}
>< 8D: Vacant space.
/table for mm wave radar/
TABLE HEADER{
>< 8bits: table ID, (“06”)
>< 16 bits: table type, (8 bits=“raw signal” + 8bits= T “mm wave radar”)
>< 8bits: rows, (“4”)
>< 16bits: columns, (?)
}
/each table includes 4 ADC samples generated by this mm wave radar at one time/
For (i=1 to 4) {
>> ??D: TNC+X’Y’, (T=”2” mm wave radar, N=SN of radar in its type, C=”C0” unified rectangular coordinate system, X’ and Y’ are both 24 bits)
>> 32bits+32bits: I (In-phase)+Q (Quadrature).
}
>< 8D: Vacant space.
/table for microphone/
TABLE HEADER{
>< 8bits: table ID, (“07”)
>< 16 bits: table type, (8 bits=“raw signal” + 8bits= T “microphone”)
>< 8bits: rows, (?)
>< 16bits: columns, (?)
}
{
>> ??D: TNC+Sound signal of 1/15 sec.
}
>< 16D: Vacant space.
/table for control/
TABLE HEADER{
}
{
>> ??D: user set parameters,
>> ??D: internal sensors,
>> ??D: an actuator – position sensor+operating time+operating value,
……
}
>< 16D: Vacant space.
/table for agent/
TABLE HEADER{
}
{}
>< 1024D: vacant
Be First to Comment