Treating Numbers, UUIDs, Dates and Codes as Entities in Language Models: What Worked and What Failed

Can a language model reproduce an identifier exactly? Can it select the right identifier when several similar values occur in the same context? Can the same representation support both exact copying and operations such as addition or date arithmetic?

These questions arose during Tamil-to-English translation experiments. A small translation model would sometimes change a number, omit it, or place a number in the wrong position. The same class of error appears in tool use with UUIDs, account IDs, dates, amounts and product codes.

Consider this example:

Anu's red code is BRD-ZYF93TJGRVHKZ0MH.
Bala's blue code is BRD-ZYF93TJGRVHKZ0MJ.
Request: return Bala's blue code.

An answer ending in Z0MH rather than Z0MJ is plausible-looking but wrong. There are two possible causes:

1. the model understood that Bala's code was required but corrupted the string while generating it; or
2. the model selected Anu's code.

The first is an exact-copy error: the model chose the right code but changed its text. The second is a binding error: the model confused Bala's code with Anu's. They require different solutions.

Current frontier language models can usually recall values, associate them with the right people or objects, and perform operations on them. Generic Transformers can therefore learn these associations. This project did not begin with a claim that they cannot. Research has found internal mechanisms for keeping track of such associations in sufficiently large Pythia and LLaMA models. Later work reports that retrieval becomes less reliable when there are more entities or when they are arranged in more difficult ways (Feng and Steinhardt, 2023; Gur-Arieh, Geva and Geiger, 2025).

There is no general error rate for this capability. Exact recall, entity selection, arithmetic, tool use and compliance with output formats are usually measured on different datasets. Results also depend on the model, prompt, input length, number of entities and use of external tools. Our experiments did not include a controlled test of frontier models. We asked a narrower question: can a small system guarantee that selected values remain exact, and can we measure the mistakes that remain?

This question is also related to Richard Sutton's Bitter Lesson. Sutton argues that general methods based on learning and computation tend to overtake systems built around human-designed knowledge. An entity register is clearly a piece of human-designed structure. It could make a narrow system more reliable while becoming a limit as language and entity types grow more varied. But the register does not try to encode how language works. It leaves language understanding to the model and uses ordinary software only to preserve or calculate with a value after the model has chosen it. The experiments test whether this is a useful division of work.

I explored this distinction in a series of small experiments. The system stored each exact value outside the language model in a table called an entity register. The model worked with short references to the table entries. Ordinary code then copied or operated on the stored values.

This prevented changes to a value once the model had selected the correct entry. It did not prevent the model from selecting the wrong entry. Choosing the right entity became the main problem as the experiments moved from fixed templates toward less familiar language.

The Proposed Representation

Suppose the input contains:

Anu's yellow number is 89910075.
Ezhil's blue number is 745362696.
Request: add Anu's yellow number and Ezhil's blue number.

The system stores the exact values and replaces them in the text:

Anu's yellow number is <ENTITY_0 type=NUMBER>.
Ezhil's blue number is <ENTITY_1 type=NUMBER>.
Request: add Anu's yellow number and Ezhil's blue number.

The external register contains:

The model can predict:

ADD(ENTITY_0, ENTITY_1)

Ordinary code retrieves the stored values and performs the calculation. If the request asks for a copy, it returns the stored value directly.

The model sees the surrounding language and the short references, while the register retains the original values. The final calculation uses the stored numbers rather than values generated by the model.

This lets us measure three questions separately:

Storing values outside a model and restoring them later is not a new idea. It is related to delexicalization and relexicalization, Pointer Networks, CopyNet, pointer-generator networks, constrained output and model-generated programs. This project did not produce a new kind of copying system. Its useful result is a clear separation between exact copying, choosing the right entity and performing the requested operation.

Current Prototype

The final prototype works as follows:

input text
    ↓
rules find values and store them with their types
    ↓
exact values replaced by short references that include their type
    ↓
pretrained IndicSBERT reads the surrounding language
    ↓
small trained component chooses the requested entities
    ↓
small trained components choose the operation and each value's job
    ↓
ordinary code restores values or performs the operation

The first stage uses ordered pattern-matching rules to find UUIDs, ISO dates, numbers, money, email addresses, URLs and structured codes. The final full experiments use numbers, UUIDs, dates and codes. Names such as Anu and Bala help describe a value, but the system does not store the names as entities.

Each value is replaced by a short reference that also states its type. The language model used to read the text is the 237.6-million-parameter l3cube-pune/indic-sentence-similarity-sbert model. Its weights remain frozen. In other words, this large model is not retrained during our experiments.

For each stored value, the system summarizes four parts of the model's output:

• the replacement reference;
• the words before it in the same record;
• the words after it; and
• the whole record.

Each summary is the average of the model's token representations for that part of the text. The four summaries are combined into a 128-number description of the stored entry. A small trained component then compares the request with every entry and can also choose NONE when a word refers to no stored value.

Another component chooses the task: COPY, ADD, MAX, SUB, YEAR, DAYS_BETWEEN, MIXED_ADD, or one of three reasons to decline the request. A third component assigns each chosen value a job, such as the starting date or the value to subtract. Together, these trained components have about 659,000 parameters. The much larger language model remains fixed.

Fixed rules handle value detection, storage, type checks, exact copying and calculation. Trained model components choose the values, the task and the order in which the values should be used.

Main Findings

The experiments produced eight main findings:

1. A stored value can be reproduced perfectly. Once the model chooses the correct register entry, ordinary code can return its number, UUID, date or code without changing a character.
2. The model can still choose the wrong value. The register prevents damage to the value itself, but it does not guarantee that the model has understood which value the request refers to.
3. A correct final answer does not always mean that the model chose correctly. Swapping two numbers does not change their sum or maximum. We therefore need to measure which values were chosen, what each value was associated with, their order and the final answer separately.
4. Matching a clear description to a record was easy; finding that description in varied language was hard. Near-perfect results were possible when the model was told that it needed, for example, “Anu's yellow number.” Results fell when it had to work this out from the complete request.
5. A pretrained language model helped much more than a small model trained from scratch. Our best system used a large pretrained language model whose weights were kept fixed, together with a much smaller component trained for this task.
6. The system needed to name the job of each value. The first value mentioned is not always the first value used. In “subtract B from A,” for example, the calculation must use A followed by B. Labels such as “starting value” and “value to subtract” reduced these errors.
7. The training data had to include requests with different numbers of values. A model trained mostly on one or two values did not automatically handle longer lists. Replacing some training examples with three-, four- and five-value requests improved this substantially.
8. Each stage can fail independently. The detector can miss a value or add an irrelevant one. The model can choose the wrong value, put correct values in the wrong order or choose the wrong operation. The calculation can then succeed or fail. A single accuracy score hides these differences.

Several seemingly reasonable approaches did not solve the problem. Replacing values with placeholders was not enough by itself. Asking the model to point at the input did not ensure that it pointed at the right part. A learned stopping rule did not cope with requests containing unfamiliar numbers of values. Token labels and stricter decoding rules still failed on unfamiliar wording. A separate component for deciding which possible values to include caused more omissions. Finally, the translation baseline was too weak to support a meaningful comparison of entity representations.

Initial Copy Experiment

The first experiment used a small Transformer trained from scratch on 4,000 synthetic lookup examples. In the raw-value version, the model had to write the answer one token at a time. Its tokenizer could break any unfamiliar value into smaller pieces, including individual bytes, so no value was impossible to represent. But the model still had to produce every piece in the right order.

IID means independent and identically distributed. In these experiments, an IID test set contained new examples drawn from the same templates, entity types and value-length ranges as the training set. The examples and values were not copied from training. The other test sets deliberately changed one or more of these conditions.

The raw model did not reproduce a complete target entity on the fresh IID or long-value sets. This was a strict entity-level score: one incorrect character made a UUID, date, number or code a miss. On the 120 IID examples, none of the 175 requested entities appeared completely in the generated output. The model often produced plausible-looking values instead. For example, it generated 07e7b7d-b7e-b7e7d2f7e7e7d2f when the requested source UUID was f9cff2ae-b167-b01d-0789-068870664e2e.

The raw-value model was probably too small or had too little data for this task. It had 572,032 parameters and trained for 20 passes through the 4,000 examples. Every test value was new. The model was allowed to look back at the input, but nothing forced it to copy from there. Byte fallback made every character available; it did not force the model to choose the source characters. The model learned the shape of an answer but not exact copying of arbitrary values.

The experiment cannot tell us whether more data, a larger model, different training or a better method of choosing output tokens would have helped most. Raw answers were also about twice as long as answers made from references. Giving each system the same examples and the same number of training passes therefore did not give them the same amount of work per answer. The zero is a result for this small model trained from scratch, not a limit of Transformers in general.

References separated by type improved both recall and binding with the same number of training examples. Long values were no harder than ordinary values once the model no longer had to write them itself.

The advantage became smaller when requests contained many records or several values of the same type that differed by only one character. The reference- based models often chose the right values in the wrong order, or chose an irrelevant value. A type label helped distinguish a date from a UUID. It did little when every possible choice was a number.

For this small model and dataset, a tokenizer that could represent every value did not provide exact copying. The experiment also showed that returning a stored value exactly does not mean that the model chose the right value.

Operations Over Entity References

The next experiment added arithmetic. Requests could ask the system to copy a value, add or subtract numbers, or choose the largest number. One model wrote the final answer directly. The other models produced an instruction such as ADD(ENTITY_0, ENTITY_1), which ordinary code then carried out.

Executing an instruction in code worked much better than asking the model to write a new numeric answer. The models almost always chose the right operation, but chose all the right input values only 37–38% of the time.

Even final-answer accuracy could be misleading. Swapping two numbers does not change their sum or maximum. A model could associate the numbers with the wrong parts of the request and still reach the right answer. Later experiments therefore measured the chosen values, their associations, their order, their types and the final answer separately.

Matching Supplied Descriptions to Records

The strongest early result came from a simpler task. The system was given a separate description of each required value. It compared each description with every record and chose the best match. A final rule prevented it from choosing the same record twice.

Across three separately trained models, the score was 1.000 on familiar examples and on tests with difficult irrelevant records, reordered records, similar values, long values and paraphrased requests. With eight records, it was 0.998. By comparison, an ordinary input-to-output model and a model that tried to point from the complete request stayed near 0.35.

The important qualification is that the data generator supplied these descriptions. The model received a phrase such as “the yellow number owned by Anu” and matched it to a record. This phrase was an extra input; the model did not have to extract or infer it from the original request.

The experiment showed that this match was easy to learn in the controlled setting:

description of a requested value ↔ matching record

It did not show that a model could derive the description from the request by itself.

Finding the Requested Values in Complete Requests

I next tested several ways to find descriptions of the required values inside a complete request.

The first model labelled each small piece of text as irrelevant, part of the first requested value, or part of the second. It reached 0.952 on familiar language, but only about 0.38–0.50 on sentence patterns excluded from training. Because it labelled each piece separately, it could also divide one identifier between two roles and produce a phrase that never appeared in the input.

A second model selected a continuous section of the request by marking its start and end. This guaranteed that every selection came directly from the input. It could still select the wrong section, but it could not invent or partly corrupt one. The rule guaranteed faithful copying, not correct choice.

Using the pretrained IndicSBERT language model instead of a small language model trained from scratch produced a large improvement:

Every selection still came directly from the source. The pretrained model was better at choosing the right section, while the start-and-end rule guaranteed that the selected text remained unchanged.

This result also changes what “small model” means. Only a small added component was trained, but it depended on a fixed language model with 237.6 million parameters.

Automatic Entity Registers

The next dataset replaced explicit tables with sentences that looked more like ordinary records. Fixed pattern-matching rules found values in the input and added them to the register.

On this clean dataset, the automatic register matched the known-correct register on all 3,300 test examples. Every value was found with the correct type and stored without changing a character.

This perfect detection result follows from how the dataset was made. It uses standard UUIDs, dates in YYYY-MM-DD form, numbers and codes with regular patterns. It does not test names in general, local date formats, identifiers without a clear format, repeated mentions, pronouns, text-recognition errors or Tamil numerals.

Detection errors had two effects. If the rules missed a value, the rest of the system could never choose it. If the rules added an irrelevant value, the model had one more chance to choose incorrectly. Training with these extra values raised binding from 0.582 to 0.828 on a test containing irrelevant numbers. It could not recover values missing from the register.

Choosing the Number and Order of Values

One model chose values one at a time and then predicted STOP. This was meant to support requests containing any number of values. It fit the training data but failed on unfamiliar list lengths. On a test that changed only the number of required values, it chose the correct count 20.0% of the time and achieved 0.286 binding.

A fixed rule that counted separators in the request raised count accuracy to 0.953 and exact final answers from 0.173 to 0.470. This rule depended on the synthetic wording. It chose how many values to return, but not their correct order.

When the system was again given a separate model representation for every requested value, binding on unfamiliar list lengths reached 0.912 ± 0.010. Clear descriptions worked well, but their boundaries still came from rules that knew the synthetic request grammar.

The next experiments trained on one family of sentence patterns and tested on another. A model given the known-correct text sections reached 0.877 binding on familiar examples. Models that had to find those sections themselves reached only 0.499–0.675 on familiar examples and 0.382–0.511 on unfamiliar list lengths.

A more complicated model considered a text section and a register entry together. Rules prevented overlapping sections, wrong types and the wrong number of answers. It fit the training data but reached only 0.611 on familiar examples and 0.199 on unfamiliar list lengths. All the methods failed on the same change in wording. More output rules did not help the models understand language they had not seen during training.

Giving Each Value a Named Job

On a later synthetic test, the model often found the right set of values but returned them in the order in which they appeared in the request.

For example, “subtract B from A” mentions B first but requires the calculation A - B. Start and end dates create the same problem.

A small trained component gave each value a named job, such as starting number, number to subtract, start date or end date. This raised binding on the separate IID test to 0.724 ± 0.003. On requests that combined copying and addition, binding rose from 0.090 to 0.832.

Choosing the right set of values is therefore not enough. The same values in the wrong order can produce the wrong calculation.

The model still struggled with unfamiliar list lengths. I added another component that judged each possible value independently as “use” or “do not use.” It chose fewer unnecessary values but missed more required ones. Binding fell to 0.540 on familiar examples and 0.260 on unfamiliar list lengths, so I did not keep it.

Training on More List Lengths

The next change was to the training data rather than the model. The total remained 3,883 examples. I replaced 942 familiar paraphrases with requests containing more varied numbers of values.

The gain came from showing the model a wider range of cases, not from giving it more examples. The model had not learned on its own how to extend its behavior from short requests to longer ones.

Independently Authored Requests

The earlier tests still used requests written by our own data generator. For a less direct test, I gave Claude descriptions of the entities and the job of each required value. It did not see the literal values, correct answers, training requests or model predictions. Claude then wrote 70 requests without using our query templates.

The already-trained model produced these results:

Binding varied substantially by operation:

To find the main source of error, I replaced selected model decisions with the known-correct decisions and measured the result:

Most errors came from linking the words in the request to the correct stored values. The register and the code that carried out the instruction were not the main problem.

This was still a small test written by one model family, not by people. The Claude command-line tool recorded the requested sonnet name but not the exact model version that served the requests. The result is evidence of a wording change, not a general test of natural human language.

Using More of Each Record's Context

The earlier system described each possible value using only the language model's output at the replacement reference. The final system also summarized the words before the reference, the words after it and the complete record. This gave it more information about properties such as owner, color and status.

Across three separately trained models:

The change also made the trained part of the system larger: it grew from about 364,000 to 659,000 parameters. To check whether size alone explained the gain, I tested a model of similar size that simply repeated the original reference summary four times. In this paired run, binding on unfamiliar list lengths was 0.433 for the repeated-reference model and 0.643 for the model with real surrounding context.

I then removed one source of context at a time without retraining the model:

The words before the value contributed most. In these synthetic records, that part often contained the owner, status and type.

The improvement was not uniform. On a test defined after the earlier results had been saved:

• addition with unfamiliar list lengths improved by 0.216;
• maximum selection with unfamiliar list lengths improved by 0.222;
• requests containing five required values improved by 0.184;
• date-difference requests fell by 0.167; and
• requests containing two required values fell by 0.068.

After training was complete, I rewrote the records using unfamiliar sentence patterns. Some placed the value before its description. Using more surrounding context raised binding on unfamiliar list lengths from 0.437 to 0.511, but binding on familiar list lengths fell from 0.667 to 0.652. More context helped with longer requests, but did not help every kind of wording change.

I did not test the final model on the earlier Claude-written requests. I had already studied those results and used them when designing the final model, so the requests were no longer an untouched test.

Tamil-to-English Translation Screen

The project did not produce a new translation result.

The translation test used 5,000 Tamil–English sentence pairs from the BPCC Wiki dataset prepared for an earlier project. It used a standard AI4Bharat IndicBERTv2 tokenizer for Tamil and a conventional SentencePiece tokenizer for English. It did not use our experimental Tamil semantic tokenizer.

The baseline was a 3.1-million-parameter Transformer trained from scratch. After 12 passes through the training data, its first 200 development examples produced:

The model often repeated text or failed to stop. I did not train the planned versions with entity references because the baseline had not learned to translate well enough. Measuring value preservation inside a failed translator would not tell us whether the new representation preserved values without damaging translation quality.

Can Generic Transformers Bind Entities?

Generic Transformers can associate entities with the right attributes. Our experiments do not show otherwise.

How do Language Models Bind Entities in Context? reports that sufficiently large pretrained models develop internal representations that help keep each entity connected to its attributes while processing a prompt.

The small models trained from scratch in this project did not learn reliable binding from several thousand synthetic examples. The large pretrained model provided much better language representations. A small added component could then bind values well when the request descriptions and possible records were clearly exposed.

The project did not compare the final register system with a similarly sized pretrained text-generating model or with a frontier model. It therefore does not show that frontier models need typed registers.

Frontier products also place ordinary software around their language models. They use structured tool arguments, stored objects, restricted output formats, search, checks and code execution. An entity register is one way to build such a safety layer. It is not evidence that the Transformer itself cannot associate entities with their attributes.

Practical Use

For an application that must return exact values, a practical design would be:

1. find the original objects and give them short internal references;
2. give the model descriptions and references instead of asking it to write long, arbitrary values;
3. name the job of each selected value instead of relying on its position;
4. check that every selected reference exists and has the right type;
5. perform arithmetic and date operations in code;
6. restore the exact stored text only when producing the final output; and
7. decline the request or ask a question when the correct entity is unclear.

This design can be used with a small local model or a frontier model. It can guarantee that a chosen value is returned unchanged. It cannot guarantee that the model chooses the right value.

Conclusion

The original problem was changed or missing numbers in translation. Storing values outside the language model addressed the copying part directly. Once a value was in the register, its length and character pattern no longer affected exact reproduction.

The remaining errors were missed values, wrong choices, the wrong number of choices and the right choices in the wrong order. Clear descriptions, a pretrained language model, named jobs for each value, more varied list lengths and more record context all helped. None made the system reliable across unrestricted language.

Seen through the Bitter Lesson, the results are mixed but not contradictory. The hand-written detector was perfect on the clean data it was designed for, but that result did not cover unfamiliar formats. Several specialized model components worked on familiar templates and then failed when the wording or number of values changed. The largest improvement in understanding language came from a general pretrained model, and wider training data helped more than another specialized component. These results support Sutton's warning against building too much task knowledge into a learning system.

The register still provided a guarantee that the learned components did not: once the correct entry was chosen, its stored value remained exact. This is not necessarily a rejection of the Bitter Lesson. A database is better than a language model at preserving a UUID, just as a calculator is better at exact arithmetic. The useful boundary may be to let general learning handle the open-ended problem of understanding language, while ordinary software enforces narrow requirements that have exact answers.

The conclusion is limited but useful. When exact text matters, numbers, UUIDs, dates and codes do not need to be generated one token at a time. They can be stored and restored by ordinary code. Choosing the correct stored value remains a language-understanding problem.